statcheck-ml
Machine-learned extraction of reported statistics, in Python, R and the browser
statcheck-ml replaces the extraction step of the R package statcheck. statcheck finds reported test results with regular expressions and then recomputes each p-value. The arithmetic is sound, but the regular expressions fail when PDF conversion damages the text, and in a real corpus about half of the results are damaged that way. statcheck-ml uses a small character-level model to find those results. The p-value check itself stays closed-form arithmetic: the model finds results, it never judges them.
On a holdout of unseen documents, running statcheck first and letting the model add what it missed raises F1 from 0.636 to 0.908.
The project ships in three forms, all built from one shared specification and checked against the same parity cases:
- Web app: try it in the browser. Drop in up to ten PDFs; everything runs in the tab and nothing is uploaded. Source
- Python package and command line tool:
statcheck-ml check paper.pdf. Source, models, dataset and full report - R package:
statcheckml, withsc_check("paper.pdf"). Source
The training labels are bronze: three language-model raters annotated each passage and a fourth adjudicated disagreements, and no human gold set exists yet. The report gives the agreement study and every number with its interval.
statcheck-ml builds on statcheck by Michèle B. Nuijten, Sacha Epskamp and colleagues.