Not so Prompt: Prompt Optimization as Model Selection
Prompt optimization is a budgeted black-box optimization problem. It is made worse by the fact that: 1) the mapping from prompt to performance is opaque, and 2) measurements are noisy because of sampling (which rows you draw), decoding randomness (temperature/top-p), retrieval/context variability, serving load, and judge variability. Under