This is an incredible paper that I've longed to do for a long time. However the engineering challenges were far too daunting, so my collaborators and I settled for indirect evidence for this hypothesis instead (or did other things).
Stella Biderman
How do LLMs learn to reason from data? Are they ~retrieving the answers from parametric knowledge🦜? In our new preprint, we look at the pretraining data and find evidence against this:
Procedural knowledge in pretraining drives LLM reasoning ⚙️🔢
🧵⬇️
