Sign in Subscribe

Topic

ML/Statistics

A collection of 151 issues

Size Matters, Significantly

Achieving statistical significance is entirely a matter of sample size. In the frequentist world, we can always distinguish between two samples if we have enough data (except of course if the samples are exactly the same). On the other hand, we may fail to reject even large differences when sample

Adjusting for Covariate Imbalance in Experiments with SUTVA Violations

Consider the following scenario: control group is 50% female while the participant sample is 60% female. Also, assume that this discrepancy is solely a matter of chance and that the effect of the experiment varies by gender. To estimate the effect of the experiment, one needs to adjust for the

Poor Browsers and Internet Surveys

Given, 1. Older browsers are more likely to display the survey incorrectly. 2. The type of browser can be a proxy for the respondent’s proficiency in using computers, and the speed of the Internet connection. People using older browsers may abandon surveys at higher rates than those using more

Reducing Errors in Survey Analysis

Analysis of survey data is hard to automate because of the immense variability across survey instruments—different variables, differently coded, and named in ways that often defy even the most fecund imagination. What often replaces complete automation is ad-hoc automation—quickly coded functions, e.g., recoding a variable to lie

Cricket: An Unfairly Random Game?

In many cricket matches, it is claimed that there is a clear advantage to bowling (batting) first. The advantage is pointed to by commentators, and by captains of the competing teams in the pre-toss interview. And sometimes in the post-match interview. The opportunity to bowl or bat first is decided

help(R): matrix indexing

In R, some want to treat matrix like a data frame. It is a bad idea. And no – one cannot use the dollar sign to pick out a column of a matrix. The underlying data structure for a matrix in R is a vector, whereas data.frame object in R

GSS and ANES: Alike Yet Different

The General Social Survey (GSS), run out of National Opinion Research Center at University of Chicago, and American National Election Studies (ANES), which until recently ran out of University of Michigan’s Institute for Social Research, are two preeminent surveys tracking over-time trends in social and political attitudes, beliefs and

The Perils of Balancing Scales

Randomization of scale order (balancing) across respondents is common practice. It is done to ‘cancel’ errors generated by ‘satisficers’ who presumably pick the first option on a scale without regard to content. The practice is assumed to have no impact on the propensity of satisficers to pick the first option,

Join Or Die

How to join datasets correctly is well understood. But we continue to do it badly. Here are notes on how to do fuzzy joins and long to wide transformations starting with some general preprocessing advice. Preprocessing Keys and Tables 1. Remove stuff irrelevant to matching, e.g., font style, spaces,

Q-Tips For R

Distance Between Sounds library(stringdist) > stringdist("ham", "jam", method ="soundex") [1] 1 > stringdist("ham", "banana", method ="soundex") [1] 1 This is pretty bad. Character-wise cosine distance or some other distance based on Soundex codes may be