PhD student at University of Montreal // Mila ··· mechanistic understanding of LLMs + Human-AI collaboration for science ··· http://mirandrom.github.io
Andrei Mircea
Loading...
Thanks to my collaborators and mentors @katelobacheva.bsky.social, Irina Rish, Supriyo Chakraborty, and Nima Chitsazan.
Also Ashwinee Panda for coining "zero-sum learning", which is honestly a pretty great name.
TL;DR We find two new phenomena (loss deceleration + zero-sum learning) and show quantifiably how scaling improves LLMs by mitigating these.
What’s cool is that these could potentially be mitigated independent of scaling (Step 2).
Exactly how to do this remains an open question.
Mechanistic understanding of systematic failures in language models is something more research should strive for IMO. This is really interesting work in that vein by @ziling-cheng.bsky.social, highly recommend you check it out.
Do LLMs hallucinate randomly? Not quite.
Our #ACL2025 (Main) paper shows that hallucinations under irrelevant contexts follow a systematic failure mode — revealing how LLMs generalize using abstract classes + context cues, albeit unreliably.
📎 Paper: arxiv.org/abs/2505.22630 1/n
All of our code and artefacts are also open, which hopefully will help.
Code: github.com/mirandrom/zsl
Checkpoints: huggingface.co/mirandrom/zs...
Wandb logs: wandb.ai/amr-amr/zsl/...
Step 1: Understand how scaling improves LLMs.
Step 2: Directly target underlying mechanism.
Step 3: Improve LLMs independent of scale. Profit.
In our ACL 2025 paper we look at Step 1 in terms of training dynamics.
Project: mirandrom.github.io/zsl
Paper: arxiv.org/pdf/2506.05447
