OLMo 2 tech report is out!
We get in the weeds with this one, with 50+ pages on 4 crucial components of LLM development pipeline:
New paper. We show that the representations of LLMs, up to 3B params(!), can be engineered to encode biophysical factors that are meaningful to experts.
We don't have to hope Adam magically finds models that learn useful features; we can optimize for models that encode for interpretable features!
Julius Adebayo
Luca Soldaini 🎀
Is the final output actually “causally” dependent on the long COT generated? How key are these traces to the search/planning clearly happening here? Some many questions but so little answers.
Pinging into the void.
Great to see clarification comments. o3 is impressive nonetheless.
Played around with o1 and the ‘thinking’ Gemini model. The cot output (for Gemini) can confusing and convoluted, but it got 3/5 problems right. Stopped on the remaining 2.
These models are an impressive interpretability test bed.
Looks like Tesla’s models sometimes confuse train tracks with road lanes.
