Check this recent work by my PhD student Moayed. He has been doing amazing work on Generative AI for images, video and audio. We introduce AV-Link âžď¸, an unified approach for audio-video generation. Our generated audio is the best in terms of synchronization with video actions. Check thread below.
Besides Video to Audio (đ˝ď¸ âĄď¸đ), we also support Audio to Video (đâĄď¸đ˝ď¸) generation under the same unified framework.
Compared to Meta Movie Gen Video to Audio, we achieve significantly better temporal synchronization with a 90% smaller scale model.
Vicente Ordonez
Can pretrained diffusion models be connected for cross-modal generation?
đ˘ Introducing AV-Link âžď¸
Bridging unimodal diffusion models in one self-contained framework to enable:
đ˝ď¸ âĄď¸ đ Video-to-Audio generation.
đ âĄď¸ đ˝ď¸ Audio-to-Video generation.
đ: snap-research.github.io/AVLink/
â¤ľď¸ Results
ICLR rejections go brrrr
After x (aka good old twitter) kept shadow-banning me for no apparent reason, I decided to give Blue Sky a try. Posting this tweet to test my reach
While current approaches uses external pretrained features (e.g. Meta CLIP, BEATs), we found that diffusion activations hold rich, semantically and temporally aware features, making them perfect for cross-modal generation in a self-contained framework.
đâĄď¸đ˝ď¸ Example:
A great collaboration with
W. Menapace, A. Siarohin, I. Skorokhodov, A. Canberk, K.S Lee, V. Ordonez, and S. Tulyakov.
Please repost to support our work and check out our
Arxiv preprint: arxiv.org/abs/2412.15191
Webpage: snap-research.github.io/AVLink/
Moayed Haji ALi
Moayed Haji ALi
Moayed Haji ALi
Moayed Haji ALi
Moayed Haji ALi
Moayed Haji ALi
recise temporal synchronization remains a significant challenge for current video-to-audio models. AV-Link addresses this by leveraging diffusion features to accurately capture both local and global temporal events, such as hand slides on a guitar and fretboard pitch changes.
Moayed Haji ALi
Moayed Haji ALi
Can pretrained diffusion models be connected for cross-modal generation?
đ˘ Introducing AV-Link âžď¸
Bridging unimodal diffusion models in one self-contained framework to enable:
đ˝ď¸ âĄď¸ đ Video-to-Audio generation.
đ âĄď¸ đ˝ď¸ Audio-to-Video generation.
đ: snap-research.github.io/AVLink/
â¤ľď¸ Results
