Benchmarking is a critical and unmet need for the bioinformatics and genomics communities. Users have no impartial guidance to indicate what tool will be best suited for their analyses, while algorithm developers lack a baseline to compare their approaches to others.

Mission Statement

  1. Design and execution of four global challenges to benchmark algorithms for:
    • RNA-Seq data-analysis
    • Detection of intra-tumoural heterogeneity
    • Compression and anonymization of genome-sequencing data
    • Somatic Mutation Calling in cancer
  2. Generation of gold-standard datasets for algorithm evaluation
  3. Development of new techniques for analyzing and integrating multiple algorithms
  • Management Team

  • Collaborators

    • Gustavo Stolovitzky, Collaborator


By indicating which specific algorithms and classes thereof have performed best, this work will give guidance as to the most promising directions for future researchers. While journals are supportive of large-scale benchmarking activities, there remains a paucity of published results in this area. This core will benefit the entire genomics community with objective and timely assessments of the best algorithms in four central problem areas. It will create a template upon which similar cloud based algorithmic benchmarking challenges can be designed.

Latest Publications & Presentations


Benchmarking & Biomarkers: Barriers to Bringing Molecular Discoveries to Clinic

Presenter: Paul Boutros

Date: September 2015

University of Waterloo; Waterloo, Canada


Cancer Sequencing Quality & The ICGC-TCGA DREAM Somatic Mutation Calling Challenge

Presenter: Paul Boutros

Date: July 2015

The 23rd International Conference on Intelligent Systems for Molecular Biology (ISMB); Dublin, Ireland


Ewing AD, Houlahan KE, Hu Y, Ellrott K, Caloian C, Yamaguchi TN, Bare JC, P'ng C, Waggott D, Sabelnykova VY, ICGC-TCGA DREAM Somatic Mutation Calling Challenge participants., Kellen MR, Norman TC, Haussler D, Friend SH, Stolovitzky G, Margolin AA, Stuart JM, Boutros PC, (2015)

Combining tumor genome simulation with crowdsourcing to benchmark somatic single-nucleotide-variant detection.

Nature methods, 2015;12(7):623-30