Code, models and paper is available on our project site!
🌐 aaltoml.github.io/BayesVLM/
1k thanks to @antonbaumann.bsky.social, @ruili-pml.bsky.social, @smentu.bsky.social, @shyamgopal.bsky.social, @zeynepakata.bsky.social, @arnosolin.bsky.social, @trappmartin.bsky.social for this collaboration!🫰
BayesVLM improves calibration in zero-shot classification without sacrificing accuracy. The uncertainties are also useful for data selection in active fine-tuning, which actually was our target in the beginning, i.e. fetch new samples that will reduce the model's uncertainty on current observations.
The paper includes quite some math about estimating Hessians from CLIP efficiently which turned out harder as we initially thought due to the InfoNCE loss being cross-modal and contrastive. Anton, Rui, and Martin did a heckuva job to get this done in a rigorous way!
BayesVLM requires estimating Hessians over the image and text proj. layers, where access to the pretraining data (or proxy of this) is needed, but only 10 batches is sufficient. Estimating pseudo-data count and prior precision params is also needed, which is similar to what temp. scaling needs.
We put the Laplace approximation on top of CLIP to enable estimating uncertainties without inference overhead. AFAIK this is the first method that can produce zero-shot uncertainties in VLMs without architecture changes or retraining from scratch.
👋🇧🇷 If you are at #ICLR2026 today, you should talk to @antonbaumann.bsky.social who is presenting our paper about turning pre-trained VLMs into probabilistic models without retraining or fine-tuning.
Poster Session 3
⌚: 10:30am - 1:00pm (local time)
📍: Pavilion 3 P3 - #313
@iclr-conf.bsky.social
Want to work on Trustworthy AI? 🚀
I'm seeking exceptional candidates to apply for the Digital Futures Postdoctoral Fellowship to work with me on Uncertainty Quantification, Bayesian Deep Learning, and Reliability of ML Systems.
The position will be co-advised by Hossein Azizpour or Henrik Boström.
