Sign in Subscribe

Topic

Statistics

A collection of 123 issues

Talking On a Tangent

What is the trend over the last X months? One estimate of the ‘trend’ over the last k time periods is what I call the ‘hold up the ends’ method. Look at t_k and t_0, get the difference between the two, and divide by the number of time

Targeting 101

Targeting Economics Say that there is a company that makes more than one product. And users of any one of its products don’t use all of its products. In effect, the company has a \textit{captive} audience. The company can run an ad in any of its products about

Estimating Bias and Error in Perceptions of Group Composition

People’s reports of perceptions of the share of various groups in the population are typically biased. The bias is generally greater for smaller groups. The bias also appears to vary by how people feel about the group—they are likelier to think that the groups they don’t like

Sample This: Sampling Randomly from the Streets

Say you want to learn about the average number of potholes per unit paved street in a city. To estimate that quantity, the following sampling plan can be employed: 1. Get all the streets in a city from Google Maps or OSM 2. Starting from one end of the street,

Clustering and then Classifying: Improving Prediction for Crudely-labeled and Mislabeled Data

Mislabeled and crudely labeled data are common problems in data science. Supervised prediction of such data is expectedly to yield poor results—coefficients are biased, and accuracy with regard to the true label is poor. One solution to the problem is to hand-code labels of an `adequate’ sample and infer

The Missing Plot

Datasets often contain missing values. And often enough—at least in social science data—values are missing systematically. So how do we visualize missing values? After all, they are missing. Some analysts simply list-wise delete points with missing values. Others impute, replacing missing values with mean or median. Yet others

About 85% Problematic: The Trouble With Human-In-The-Loop ML Systems

MIT researchers recently unveiled a system that combines machine learning with input from users to ‘predict 85% of the attacks.’ Each day, the system winnows down millions of rows to a few hundred atypical data points and passes these points on to ‘human experts’ who then label the few hundred

Interpeting Clusters and 'Outliers' from Clustering Algorithms

Assume that the data from the dominant data generating process are structured so that they occupy a few small portions of a high-dimensional space. Say we use a hard partition clustering algorithm to learn the structure of the data. And say that it does—learn the structure. Anything that lies

Sampling With Coprimes

Say you want to sample from a sequence of length n. Multiples of a number that is relatively prime to the length of the sequence (n) cover the entire sequence and have the property that the entire sequence is covered before any number is repeated. This is a known result