Two simple ideas for building improved brain encoding models: 1. learn to use representations from all model layers via a gating mechanism + 2. start from natively multimodal features for multimodal predictions. State of the art performance; see mirage-brain.epfl.ch for details #NeuroAI 🧠🤖🧪
🧠When you watch a movie, your brain blends sight, sound, and speech into a single experience.
Should models of the brain blend them too, or keep the senses separate until the very end?
We built MIRAGE to find out. It sets a new SOTA for predicting whole-brain fMRI from movies. 🧵
