Sign in Subscribe

Topic

ML/Statistics

A collection of 158 issues

Cell Mates: Extracting Useful Information From Tables for LLMs

We seem to have cracked the art of distilling information in words and images using large machine learning models. However, our ability to exploit useful information in tabular data using large models is mostly missing. The upshot is that LLMs don't largely encode the knowledge from these tabular

Resolution Lost: The Deadweight Costs of Strict Isotonicity

Calibration aligns model scores with event frequencies. For a binary outcome $Y\in{0,1}$ and a score $S$, the calibration function is $g(s)=\mathbb{E}[Y\mid S=s]$. Post hoc calibration estimates $g$ on a holdout set and applies the estimate to future scores. A common difficulty

The GitHub Sampler

GitHub is the world's largest developer platform, making it a compelling place to build developer tools and study software development. However, with over half a billion repositories and an API that limits responses to a thousand results, it feels nigh impossible to answer even simple questions like, "

Coding At a Distance

Senior professors have long lived the future of coding. Professors ask. Underlings code. Professors critique the output, e.g., "I prefer statistical significance stars for the betas but not the constant term." Underlings revise the code. The cycle continues. Something similar can play out at companies. Now, thanks

Using Reward Models to Select Pre-Training Data

Large code models currently treat all code as equal during pre-training, then use RLHF to fix the resulting problems. This is backwards. We should allocate pre-training compute toward good code from the start. The approach is straightforward: score code quality using verifiable signals, then weight training examples proportionally to their

Diabetes as a Dummy Variable

“Adults with diabetes are nearly twice as likely to have heart disease or stroke as adults without diabetes.” NIH These alarming numbers are used to justify more pre-emptive testing, e.g., Calcium score for the heart, more aggressive use of statin therapy, etc., among diabetics. The analyses that underlie the

Not so Prompt: Prompt Optimization as Model Selection

Prompt optimization is a budgeted black-box optimization problem. It is made worse by the fact that: 1) the mapping from prompt to performance is opaque, and 2) measurements are noisy because of sampling (which rows you draw), decoding randomness (temperature/top-p), retrieval/context variability, serving load, and judge variability. Under

The Welfare Costs of Low-Friction Idea Production

Humans are naturally curious, and curiosity often takes us into domains where our understanding is shallow. Large language models (LLMs) have radically lowered the marginal cost of experimentation in such domains. Minutes of prompting can yield a plausible business strategy, technical architecture, or research framework—outputs that look like the

When OLS and SUR Disagree

Under Zellner’s classical SUR assumptions—the same regressors in each equation and a common sample of observations—OLS and SUR produce the same coefficient estimates, but SUR yields smaller standard errors. Differences in coefficients appear in practice for good and bad reasons: Efficient, “good” differences arise from intentional GLS

Take 80 mg Atorvastatin for Myalgia

Here’s a drug pamphlet with a table about side effects The same table can be found here: https://www.rxlist.com/lipitor-drug.htm#side_effects As you can see, for a range of conditions, the rate at which patients experience side effects is greater in the Placebo arm than