Sign in Subscribe

Topic

ML/Statistics

A collection of 130 issues

Superhuman: Can ML Beat Human-Level Performance in Supervised Models?

A supervised model cannot do better than its labels. (I revisit this point later.) So the trick is to make labels as good as you can. The errors in labels stem from four sources: 1. Lack of Effort: The more effort people spend labeling something, presumably, the more accurate it

Too Much Churn: Estimating Customer Churn

A new paper uses financial transaction data to estimate customer churn in consumer-facing companies. The paper defines churn as follows: There are three concerns with the definition: 1. The definition doesn’t make clear what the normalizing constant is for calculating the share. Given that the value “can vary between

94.5% Certain That Covid Vaccine Will Be Less Than 94.5% Effective

“On Sunday, an independent monitoring board broke the code to examine 95 infections that were recorded starting two weeks after volunteers’ second dose — and discovered all but five illnesses occurred in participants who got the placebo.”Moderna Says Its COVID-19 Vaccine Is 94.5% Effective In Early Tests The data

Fat Or Not: Toward 'Proper Training of DL Models'

A new paper introduces a DL model to enable ‘computer aided diagnosis of obesity.’ Some concerns: 1. Better baselines: BMI is easy to calculate and it would be useful to compare the results to BMI. 2. Incorrect statement: The authors write: “the data partition in all the image sets are

Not So Robust: The Limitations of "Doubly Robust" ATE Estimators

Doubly Robust (DR) estimators of ATE are all the rage. One popular DR estimator is Robins’ Augmented IPW (AIPW). The reason why Robins’ AIPW estimator is called doubly robust is that if either your IPW model or your y ~ x model is correctly specified, you get ATE. Great! Calling something

Instrumental Music: When It Rains, It Pours

In a new paper, Jon Mellon reviews 185 papers that use weather as an instrument and finds that researchers have linked 137 variables to weather. You can read it as each paper needing to contend with 136 violations of the exclusion restriction, but the situation is likely less dire. For

Unmatched: The Problem With Comparing Matching Methods

In many matching papers, the key claim proceeds as follows: our matching method is better than others because on this set of contrived data, treatment effect estimates are closest to those from the ‘gold standard’ (experimental evidence). Let’s side-step concerns related to an important point: evidence that a method

Optimal Recruitment For Experiments: Using Pair-Wise Matching Distance to Guide Recruitment

Pairwise matching before randomization reduces s.e. (see below, for instance). Generally, the strategy is used to create balanced control and treatment groups from available observations. But we can use the insight for optimal sample recruitment, especially in cases where we have a large panel of respondents with baseline data,

AFib: Apple Watch Did Not Increase Atrial Fibrillation Diagnoses

A new paper purportedly shows that the release of the Apple Watch 2018, which supported the ECG app, did not cause an increase in AFib diagnoses (mean = −0.008). They make the claim based on 60M visits from 1270 practices across 2 years. Here are some things to think about: