We don’t always know what problems are hard for LLMs. So devs evaluate on tasks HUMANS find hard or on broad benchmarks. What if we could instead anticipate which scenarios a model will fail on—all without evaluating specific input examples?
🧵NEW PAPER by @jenniferlumeng.bsky.social
really excited to head home for icml:) and attending the co-located FAR.ai alignment workshop (for the first time)! would love to meet others interested in training & interpretability
work w/ Emmy Liu, Cathy Jiao @brihi.bsky.social, Dani Yogatama, Fazl Barez, @saxon.me
since i'm headed home for icml, presented by amazing @brihi.bsky.social!
this was my first time writing a position paper, which turned into a grant, which i'm turning into multiple projects 🙂 stay tuned
6/6
also, blog: iglee.me/papers/inte...
7/6
3. Predicting failures. A distinction: scientific prediction (not the ML kind) is how scientists validate our understanding. A hypothesis proves its strength w/ predictive power. Used as eval, interp can predict failures from internals. Meaning, we generate eval from interp.
5/6
2. Reproducibility detects faulty mechanisms. If we were to actually act on it, we want our claim to identify mechanisms robustly against variations wrt input, method, etc. Our claim needs to be reproducible under specified conditions.
4/6
1. Falsifiability enables debugging. First argued by Leavitt & Morcos, it has to produce hypotheses that can be proven wrong. And if it can, we can then act on it, to trace back to the source of error and attempt a fix.
3/6
Our ICML 2025 workshop on Actionable Interpretability drew massive interest. But the same questions kept coming up: What does "actionable" mean? Is it achievable? How?
We're ready to answer.
🧵
What does it mean for interp to meet scientific standards? We argue that it has to meet 3 criteria: falsifiability, reproducibility and predictability.
2/6