Reverse Dependency Checks for Empirical Outputs
When you change an R package, CRAN expects you to run reverse dependency checks: build every package that depends on yours and see what breaks. When you fix a bug in a statistical package, it might be worth running the equivalent check against papers.
To run the check, you need to know which papers depend on what. softverse builds that graph: it parses replication archives from 34 social science journals deposited on Dataverse and records which libraries each paper actually called. lfe's felm is the most-called function in the corpus, at 245 scripts, and it has had exactly the kind of bug the check is for. Before 2.5, felm with two clustering dimensions could understate the standard error by nearly 6x. fixest serves as the second example: a change to how it handles singleton fixed effects changed the observation count and the standard errors.
The cheap version of the check is a grep over replication scripts for the conditions that trigger the bug. Does this script call felm with two clustering dimensions? Does it fit fixest with fixed effects that have singleton levels? But a condition is only half a check. You also need the version. Absent a renv lockfile or a sessionInfo() dump in the archive, a publication date only brackets it. Even a full match, condition and version both, says only that the paper ran the buggy code, not that the results changed substantively. With such weak evidence, the response should be cheap to make and easy to ignore. One option is a pull request: flag the line that trips the condition, link the bug report, leave the rest to the author. That needs an archive that can take a pull request, and a Dataverse deposit accepts neither comments nor patches. The other is for package maintainers to ship an advisory with the fix: which conditions trigger the bug, which specifications are exposed, which version range is affected.
The expensive option is to refit the paper's specification across versions, or with the fix switched off where the package allows it, and see which run reproduces the published numbers. The caveat is that more than one thing changes between versions, so a match is a likelihood, not a verdict.