SyncFusion: Multimodal Onset-Synchronized
Video-to-Audio Foley Synthesis

Marco Comunità, Riccardo F. Gramaccioni, Emilian Postolache,
Emanuele Rodolà, Danilo Comminiello, Joshua D. Reiss

Paper Dataset Code

Abstract


Sound design involves creatively selecting, recording, and editing sound effects for various media like cinema, video games, and virtual/augmented reality. One of the most time-consuming steps when designing sound is synchronizing audio with video. In some cases, environmental recordings from video shoots are available, which can aid in the process. However, in video games and animations, no reference audio exists, requiring manual annotation of event timings from the video. We propose a system to extract repetitive actions onsets from a video, which are then used - in conjunction with audio or textual embeddings - to condition a diffusion model trained to generate a new synchronized sound effects audio track. In this way, we leave complete creative control to the sound designer while removing the burden of synchronization with video. Furthermore, editing the onset track or changing the conditioning embedding requires much less effort than editing the audio track itself, simplifying the sonification process. We provide sound examples, source code, and pretrained models to faciliate reproducibility


Samples


Examples of audio generated by SyncFusion when conditioned on ground truth onset tracks and audio or text embeddings (CLAP).

Source Video Conditioning Audio Generated Audio Conditioning Text Generated Audio
metal, scratch, static
wood, hit, static
paper, scratch, rigid-motion
cloth, hit, deform
paper, hit, static
plastic, hit, rigid-motion
metal, hit, static
metal, scratch, static
cloth, scratch, deform
metal, hit, static
leaf, hit, scatter
rock, hit, static
tile, hit, static
paper hit deform
leaf, hit, scatter
dirt, hit, static
leaf, hit, rigid-motion
leaf, hit, deform
leaf, hit, deform
rock, hit, static
gravel, hit, rigid-motion
wood, hit, static
wood, hit, scatter
rock, hit, static
leaf, scratch, rigid-motion
dirt, scratch, scatter
leaf, scratch, scatter
dirt, hit, scatter
wood, hit, static
rock, scratch, static
wood, hit, static
water, scratch, splash
water, hit, splash
gravel, scratch, scatter
water, scratch, splash
leaf, hit, rigid-motion
water, hit, splash