We don’t always know what problems are hard for LLMs. So devs evaluate on tasks HUMANS find hard or on broad benchmarks. What if we could instead anticipate which scenarios a model will fail on—all without evaluating specific input examples?
🧵NEW PAPER by @jenniferlumeng.bsky.social
The pedestrianization of Broadway through the heart of Manhattan just expanded, and it's glorious:
Anthropoic founder: “How will we ensure the gains of AI are shared globally? We do not have a mechanism for this.”
@bcmerchant.bsky.social: “They're called taxes. And just because AI companies don't really like that idea, they don't like the concept of being taxed, doesn't mean they don't exist.”
He sure as hell does!
@tyrellturing.bsky.social you know about brain stuff, why does this happen?
Why is it that some days, you have zero useful ideas, and other days you get multiple really solid research ideas? What the hell, brain, pace yourself.
big fish supper club, bena, minnesota, 1980
Wooooooooooooooooooooooooo!
Me getting an email from TMLR
In film, "we'll fix it in post" is what you say when something went wrong on set and you don't want to redo it. AI research has made it our entire methodology: train the model, then patch whatever comes out. Our new ICML oral argues this can't be the basis of a science of AI. 🧵
Video
Alt: Al Pacino as Michael Corleone from The Godfather Part III, saying "Just When I Thought I Was Out They Pull Me Back In"
