If you’ve done a systematic review using Numbat, you may want to estimate inter-rater reliability for one or more of the data points extracted.
First, make sure that all the extractors have completed all the extractions for all the references. If there is one missing, you will get an error.
When the extractions are complete, log in to your Numbat installation, and choose Export data from the main menu. Export the extractions, not the final version.
This will give you a tab-delimited file that contains a row for every extraction done for every user, which is not the format that the Fleiss’ kappa function as implemented by the
irr package in R requires, unfortunately. (Hence the R script below.)
Next, choose which of the data points you wish to assess for inter-rater reliability. Let’s imagine that you were extracting whether a clinical trial is aimed at treatment or prevention, and this column is called
tx_prev in the exported extractions file.
You could delete all the columns from the extractions file except the
userid columns, and the data point of interest, in this case
tx_prev. The following CSV is an example that you can use. A typical Numbat export will contain many more columns than this. These are just the relevant ones.
referenceid,userid,tx_prev 1,1,treatment 1,2,treatment 1,3,treatment 2,1,treatment 2,2,prevention 2,3,prevention 3,1,treatment 3,2,treatment 3,3,treatment 4,1,prevention 4,2,prevention 4,3,prevention 5,1,treatment 5,2,treatment 5,3,treatment 6,1,treatment 6,2,treatment 6,3,treatment 7,1,treatment 7,2,treatment 7,3,treatment 8,1,treatment 8,2,treatment 8,3,treatment 9,1,treatment 9,2,treatment 9,3,treatment 11,1,treatment 11,2,treatment 11,3,prevention
If you saved this CSV to your Downloads folder as
numbat-export.csv, you could use the following function to convert this CSV into a data frame that is compatible with
library(tidyverse) library(irr) read_csv("numbat-export.csv") %>% spread(userid, tx_prev) %>% select(! referenceid) %>% kappam.fleiss()
This should give you a console printout that looks like this:
Fleiss' Kappa for m Raters Subjects = 10 Raters = 3 Kappa = 0.583
z = 3.2p-value = 0.0014
Congrats, you just calculated Fleiss’ kappa from your Numbat extractions!