How to calculate Fleiss’ kappa from a Numbat extractions export

If you’ve done a systematic review using Numbat, you may want to estimate inter-rater reliability for one or more of the data points extracted.

First, make sure that all the extractors have completed all the extractions for all the references. If there is one missing, you will get an error.

When the extractions are complete, log in to your Numbat installation, and choose Export data from the main menu. Export the extractions, not the final version.

This will give you a tab-delimited file that contains a row for every extraction done for every user, which is not the format that the Fleiss’ kappa function as implemented by the irr package in R requires, unfortunately. (Hence the R script below.)

Next, choose which of the data points you wish to assess for inter-rater reliability. Let’s imagine that you were extracting whether a clinical trial is aimed at treatment or prevention, and this column is called tx_prev in the exported extractions file.

You could delete all the columns from the extractions file except the referenceid and userid columns, and the data point of interest, in this case tx_prev. The following CSV is an example that you can use. A typical Numbat export will contain many more columns than this. These are just the relevant ones.

referenceid,userid,tx_prev
1,1,treatment
1,2,treatment
1,3,treatment
2,1,treatment
2,2,prevention
2,3,prevention
3,1,treatment
3,2,treatment
3,3,treatment
4,1,prevention
4,2,prevention
4,3,prevention
5,1,treatment
5,2,treatment
5,3,treatment
6,1,treatment
6,2,treatment
6,3,treatment
7,1,treatment
7,2,treatment
7,3,treatment
8,1,treatment
8,2,treatment
8,3,treatment
9,1,treatment
9,2,treatment
9,3,treatment
11,1,treatment
11,2,treatment
11,3,prevention

If you saved this TSV to your Downloads folder as numbat-export.tsv, you could use the following function to convert this CSV into a data frame that is compatible with kappam.fleiss() from irr.

library(tidyverse)
library(irr)

extractions <- read_csv("numbat-export.csv")

get_numbat_extraction_data <- function (data, refid, uid, row) {
  return( as.character( unlist( data[data$referenceid == refid & data$userid == uid, row] ) ) )
}

numbat_transform_row <- function (data, row) {

  new_data <- data %>%
    select(referenceid) %>%
    unique()

  users <- data %>%
    select(userid) %>%
    unique() %>%
    unlist() %>%
    as.numeric()

  rids <- new_data %>%
    unlist() %>%
    as.numeric()

  for (user in users) {
    colname <- paste0("user", user)
    new_data[[colname]] <- ""
  }

  for (user in users) {
    for (rid in rids) {
      colname <- paste0("user", user)
      new_data[new_data$referenceid == rid, colname] <- get_numbat_extraction_data (
        data,
        rid,
        user,
        row
      )
    }
  }

  new_data$referenceid <- NULL

  return(new_data)

}

numbat_transform_row (extractions, "tx_prev") %>%
  kappam.fleiss()

This should give you a console printout that looks like this:

Fleiss' Kappa for m Raters
Subjects = 10
Raters = 3
Kappa = 0.583
z = 3.2
p-value = 0.0014

Congrats, you just calculated Fleiss’ kappa from your Numbat extractions!

Published by

The Grey Literature

This is the personal blog of Benjamin Gregory Carlisle PhD. Queer; Academic; Queer academic. "I'm the research fairy, here to make your academic problems disappear!"

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.