Numbat 3: Peer-to-peer systematic review manager


The Dat link to the most up-to-date Numbat 3 is here. Open it with Beaker Browser and fork to use.

Introducing Numbat 3

Figure 1. A systematic review. Pai et al.
Figure 1. A systematic review. Pai et al.

Systematic reviews and meta-analyses provide an important part of the evidentiary basis for clinical care. They also aid in evaluating risk and benefit when launching clinical trials, inform clinical practice guidelines, and can be used as a window into medical science generally.1 However, systematic reviews are also complicated, difficult to do properly, and prone to the common errors associated with repetitive manual tasks. See Figure 1.2

Traditionally, systematic reviews were completed by filling in, by hand, photocopied forms printed on sheets of paper. Paper is notoriously easy to lose, handwriting can be difficult to interpret, and manually copying information from paper to a computer for analysis is time-consuming and represents an unnecessary step where errors may be introduced.

Spreadsheets are also not well-suited to the task of coordinating systematic reviews. Large spreadsheets are difficult to read and manage at best, and become unwieldy when entering data for large sets of references. When tasks are distributed among many workers, spreadsheets can be modified either by user accident, or by changes introduced by different versions of spreadsheet software, either of which may introduce errors. Microsoft Excel, for example, is a particularly bad choice of software for managing systematic reviews or large scientific data sets, and has many well-documented cases where its use has inadvertently introduced errors.3,4

To avoid these kinds of frustrations and errors, the STREAM research group at McGill University developed and used previous versions of Numbat5 since 2013 to coordinate systematic reviews for various publications.6,7 Numbat was designed to manage multi-user extraction of data points from large sets of references and to coordinate the merging of double-coded data into a single master copy for analysis.

Numbat was built for use on a LAMP stack,8 and maintaining a web server for the purposes of coordinating extractions presented a barrier for other research groups’ use of the software.

Dat and Beaker Browser

Dat is a peer-to-peer protocol that was originally designed for sharing large and complicated scientific data sets that change over time.9 Beaker Browser adds an application programming interface on top of the Dat protocol to allow hosting and sharing data, websites and embedded programmes directly from a browser. Together, these make an ideal platform on which to build a new version of Numbat that does not require a research group to maintain a web server.

Numbat 3 is free and open source software licensed under a Creative Commons Attribution 4.0 International License.10 Anyone running Beaker Browser can fork a blank Numbat archive, and immediately use it to manage a systematic review project.

How to use Numbat 3

  1. Install Beaker Browser from
  2. In consultation with a statistician, prepare a protocol for your systematic review and define the set of questions necessary to answer your research question
  3. Define your extraction form using the editor in Numbat
  4. Install Zotero from
  5. Consult a librarian to construct a list of references in Zotero for extraction using databases such as Medline, Embase, etc.
  6. Screen references at the title-and-abstract level for inclusion using Zotero
  7. Install Beaker Browser, and fork the Numbat Source Code archive
  8. Enter search details into Numbat under “Meta” for future reference
  9. Export included references from Zotero as a CSV, import into Numbat
  10. Assign extractions to multiple users
  11. Each user must fork the Numbat archive at this point in order to allow for double-coding of the same reference
  12. Complete assigned extractions
  13. Using the built-in tool, copy completed extractions together into a single archive
  14. Reconcile completed extractions by copying the correct response into the master copy
  15. Export data set as a CSV for analysis


Consult this page for the most up-to-date version of Numbat and for links to previous versions.


Date Event Timestamp
2018-09-15 Numbat public beta, v. 3.0.19 published


  • Landing page has project stats
  • Reconcile page has a button to copy extraction notes
  • Removed “master” language in favour of “final”
2018-07-23 Numbat public beta, v. 3.0.17 published


  • Better assignments page
  • Export is better for large data sets
2018-07-07 Numbat public beta, v. 3.0.11 published


  • Safer update function (downloads first, THEN deletes previous software)
  • Better bug-catching on software updating
2018-07-06 Numbat public beta, v. 3.0.10 published


  • New feature: notes for reconciliation page, similar to notes for extractions
  • Minor redesign of greeter page to better reflect ownership, project details
  • Minor bug fixes
2018-07-05 Numbat public beta, v. 3.0.9 published


  • New feature: automatically randomize order of references
  • Now supports Firefox web browser with dat-fox extension
2018-07-04 Numbat public beta, v. 3.0.8 published


  • Fixes bug in rendering section headings in reconciliation page
  • New feature: auto-reconcile merges answers to questions where all extractors agree
2018-07-03 Numbat public beta, v. 3.0.7 published


  • Includes self-updater button on About page, that will update itself to the newest version of Numbat fetched from
2018-06-28 Numbat public beta, v. 3.0.6 published


  • Minor UX changes to prevent user surprises
  • Minor bug fixes
2018-06-11 Numbat public beta, v. 3.0.5 published


  • Fixed broken back-button
  • Minor formatting errors
2018-03-27 Numbat public beta, v. 3.0.4 published


  • Extraction form includes show/hide buttons
  • Non-reconciled extractions can be exported
2017-12-12 Numbat public beta, v. 3.0.3 published


  • Minor bug fixes in display name and codebook
2017-11-22 Numbat public beta, v. 3.0.1 published


  • Reference sets need not be formatted by Zotero. Numbat now allows the user to specify which columns correspond to a reference’s title, author, journal, year and unique identifier.
  • Added “reference data” form element to display data from the references CSV inline during extractions
2017-11-19 Numbat public beta, v. 3.0.0 published Link



Numbat was built using Dat,9 Beaker Browser, Bootstrap v 3.3.7, jQuery v 3.2.1, jQuery UI v 1.12.1 and jQuery CSV v 0.8.3.

Numbat was originally designed for use by the STREAM research group at McGill University in Montreal Canada to work on the Signals, Safety and Success CIHR grant.


1. Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G. & Group, T. P. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLOS Med. 6, e1000097 (2009).

2. Pai, M. et al. Systematic reviews and meta-analyses: an illustrated, step-by-step guide. Natl. Med. J. India 17, 86–95 (2004).

3. Zeeberg, B. R. et al. Mistaken Identifiers: Gene name errors can be introduced inadvertently when using Excel in bioinformatics. BMC Bioinformatics 5, 80 (2004).

4. Ziemann, M., Eren, Y. & El-Osta, A. Gene name errors are widespread in the scientific literature. Genome Biol. 17, 177 (2016).

5. Carlisle, B. G. Numbat Meta-Analysis Extraction Manager. (Studies of Ethics, Translation and Medicine (STREAM), Biomedical Ethics Unit, McGill University, 2015).

6. Carlisle, B. et al. Benefit, Risk, and Outcomes in Drug Development: A Systematic Review of Sunitinib. JNCI J. Natl. Cancer Inst. 108, djv292-djv292 (2015).

7. Mattina, J., Carlisle, B., Hachem, Y., Fergusson, D. & Kimmelman, J. Inefficiencies and Patient Burdens in the Development of the Targeted Cancer Drug Sorafenib: A Systematic Review. PLOS Biol. 15, e2000487 (2017).

8. LAMP (software bundle). Wikipedia (2017).

9. Maxwell Ogden, Karissa McKelvey & Mathias Buus Madsen. Dat – Distributed Dataset Synchronization And Versioning. Code Sci. (2017).

10. Creative Commons — Attribution 4.0 International — CC BY 4.0. Available at: (Accessed: 18th November 2017)