AFib: Apple Watch Did Not Increase Atrial Fibrillation Diagnoses

A new paper purportedly shows that the release of the Apple Watch 2018, which supported the ECG app, did not cause an increase in AFib diagnoses (mean = −0.008).

They make the claim based on 60M visits from 1270 practices across 2 years.

Here are some things to think about:

1. Expected effect size. Say the base AF rate is .41%. Let’s say 10% have the ECG app + Apple Watch. (You have to make some assumptions about how quickly people downloaded the app. I am making a generous assumption that 10% do it the day of release.) For the 10%, say it is .51%. Additional diagnoses expected = .01*30M ~ 3k.
2. Time trend. 2018-19 line is significantly higher (given the baseline) than 2016-2017. It is unlikely to be explained by the aging of the population. Is there a time trend? What explains it? More acutely, diff. in diff. doesn’t account for that.
3. Choice of the time period. When you have observations over multiple time periods, pre-treatment and post-treatment, the inference depends on which time period you use. For instance,  if I do an “ocular distortion test”, the diff. in diff. with observations from Aug./Sep. would suggest a large positive impact. For a more transparent account of assumptions, see diff.healthpolicydatascience.org (h/t Kyle Foreman).
4. Clustering of s.e. Some correlation in diagnosis because of the facility (doctor), which is unaccounted for.