Straightening also makes the loss landscape closer to convex and better conditioned, improving gradient-based planning. We test on four goal-reaching tasks and observe a significant boost in open-loop and MPC success rate using gradient descent.
What is a good latent space for world modeling and planning? 🤔
Inspired by the perceptual straightening hypothesis in human vision, we introduce temporal straightening to improve representation learning for latent planning.
📝: agenticlearning.ai/temporal-str...
The resulting embedding space has many good properties! We find that (i) implicit straightening can happen when training the encoder using the predictor loss alone; (ii) adding straightening regularization further decreases curvature of the resulting embeddings;
What's better, (iii) straightening encourages the latent Euclidean distance to better align with the geodesic distance; (iv) near-perfect reconstruction can be attained with a very low feature dimensionality (we can reduce embedding dimension from 384-->8!)
Inspired by the perceptual straightening hypothesis ( human visual systems transform natural videos into straighter internal representations), we introduce a simple fix: jointly learn an encoder & a predictor (JEPA-style) with regularization on curvatures of latent trajectories.
Large-scale visual pretraining is useful but NOT enough! It's not tailored to the dynamics of the environment and retains many planning-irrelevant low-level details. e.g. In DINOv2 feature space, the latent trajectories are curved & L2 distances don't reflect geodesic distances.
More details can be found in our paper arxiv.org/abs/2603.12231. Many hanks to my collaborators @oumayma @gaoyuezhou.bsky.social @randall @timrudner.bsky.social and amazing advisors @yann-lecun.bsky.social @mengyer.bsky.social for their guidance and support 💜💜💜
yingwww.bsky.social
Learning good representations is essential for latent planning with world models. While pretrained visual encoders produce strong semantic visual features, they are not tailored to planning and contai...
