When it comes to text prediction, where does one LM outperform another? If you've ever worked on LM evals, you know this question is a lot more complex than it seems. In our new #acl2025 paper, we developed a method to find fine-grained differences between LMs:
🧵1/9
The sycophantic tone of ChatGPT always sounded familiar, and then I recognized where I'd heard it before: author response letters to reviewer comments.
"You're exactly right, that's a great point!"
"Thank you so much for this insight!"
Also how it always agrees even when it contradicts itself.
bigram of the year: "surprisal brainrot"
Saturday, I'll have another poster on work with Philip Resnik & @colinphillips.bsky.social exploring how we can enrich a toy cognitive model of reaction times in a speeded cloze task with data from language models.
hsp2026.org/abstracts/su...
✨ Updated preprint ✨
Iris van Rooij & Olivia Guest (2026). Combining Psychology with Artificial Intelligence: What Could Possibly Go Wrong? PsyArXiv osf.io/preprints/psyarxiv/aue4m_v2 @olivia.science
Our aim is to make these ideas accessible for a.o. psych students. Hope we succeeded 🙂
NEW paper! 💭🖥️
“Combining Psychology with Artificial Intelligence: What could possibly go wrong?”
— Brief review paper by @olivia.science & myself, highlighting traps to avoid when combining Psych with AI, and why this is so important. Check out our proposed way forward! 🌟💡
osf.io/preprints/ps...
If any friends are at Cog Sci, I’ll be in SF tomorrow! Let me know if you’d like to meet!
I'll be presenting some work comparing how humans and LMs make predictions at #HSP2026 this week, please reach out if you'd like to meet!
Tomorrow, I'll have a poster on work with @byungdoh.bsky.social comparing cloze & LM surprisals as predictors of reading time hsp2026.org/abstracts/su...
Sathvik
What (if anything) can LLMs tell us about human language processing?
I discuss this question and how psycholinguistics is moving forward with @colinphillips.bsky.social , to appear in BBS as a commentary on Futrell & Mahowald's article about LLMs & linguistics.
arxiv.org/abs/2604.09466
