Sign in Subscribe

Topic

ML/Statistics

A collection of 147 issues

(No) Value Added Models

This note is in response to some of the points raised in the Agnoff Lecture by Ed Haertel. The lecture makes two big points: 1) Teacher effectiveness ratings based on current Value Added Models are ‘unreliable.’ They are actually much worse than just unreliable; see below. 2) Simulated counterfactuals of

Impact of selection bias in experiments where people treat each other

Selection biases in the participant pool generally have limited impact on inference. One way to estimate population treatment effect from effects estimated using biased samples is to check if treatment effect varies by ‘kinds of people’, and then weight the treatment effect to population marginals. So far so good. When

Sampling on MTurk

In many of the studies that use MTurk, there appears to be little strategy to sampling. A study is posted (and reposted) on MTurk till a particular number of respondents take the study. If the pool of respondents reflects true population proportions, if people arrive in no particular order, and

Why Were the Polls so Accurate?

The Quant. Interwebs have overflowed with joy since the election. Poll aggregation works. And so indeed does polling, though you won’t hear as much about it on the news, which is likely biased towards celebrity intellects rather than the hardworking many. But why were the polls so accurate? One

Sample This

What do single shot evaluations of MT (replace it with anything else) samples (vis-a-vis census figures) tell us? I am afraid very little. Inference rests upon knowledge of the data (here – respondent) generating process. Without a model of the data generating process, all such research reverts to modest tautology – sample

Representativeness Heuristic, Base Rates, and Bayes

From the Introduction of their edited volume: Tversky and Kahneman used the following experiment for testing the ‘representativeness heuristic’ – Subjects are shown a brief personality description of several individuals, sampled at random from 100 professionals – engineers and lawyers. Subjects are asked to assess whether the description is of an engineer

Comparing Datasets and Reporting Only Non-Duplicated Rows

The following is in response to a question on the R-Help list. Consider two datasets: reported <- structure(list(Product = structure(c(1L, 1L, 1L, 1L, 2L, 2L, 3L, 4L, 5L, 5L), .Label = c("Cocoa", "Coffee C", "GC", "Sugar No 11"

Correcting for Differential Measurement Error in Experiments

Differential measurement error across control and treatment groups or in a within-subjects experiment, pre- and post-treatment measurement waves, can vitiate estimates of treatment effect. One reason for differential measurement error in surveys is differential motivation. For instance, if participants in the control group (pre-treatment survey) are less motivated to respond