Sound Effects / Foley¶
Audio
Open Source¶
-
ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling Discovered by AI
2026
A unified and controllable multimodal video-to-audio (V2A) generation framework that enables precise control over generated sound effects using video, text, and reference audio, with robust handling of cross-modal conflicts.
foleymultimodalsfxvideo-to-audio -
Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text Discovered by AI
2026
A unified text-to-audio generation model that can jointly synthesize intelligible speech, music, sound effects, and environmental acoustics from text descriptions. It uses structured multi-view captions and a high-dimensional semantic-acoustic representation to generate coherent mixed audio scenes.
foleysfxsound-generationtext-to-audio -
Foley-Omni Discovered by AI
2026
A unified multimodal audio generation model that extends isolated task-level synthesis to complete video soundtrack generation. It jointly models speech, sound effects, and music within a shared latent generation process, conditioned on text and video inputs.
foleymultimodalsfxvideo-to-audio -
FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scripts Discovered by AI
2026
A video-to-audio generation framework that introduces Structured Temporal Scripts (STS) to enable fine-grained temporal control over sound effects. It allows users to specify precise time intervals for generating specific sounds (e.g., car horns or footsteps) while keeping other segments silent.
foleysfxtemporal-controlvideo-to-audio -
Woosh Discovered by AI
2026
A sound-effect foundation model developed by Sony AI that supports high-quality text-to-audio (T2A) and video-to-audio (V2A) synthesis, designed specifically for professional sound design and foley generation.
foleysfxtext-to-audiovideo-to-audio -
WorldSonus Discovered by AI
2026
An interactive video-to-audio framework designed for world models and game environments, supporting real-time sound generation, mid-stream instruction control, and spatially aligned stereo audio.
foleygame-audiosfxworld-models -
Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis Discovered by AI
2025
An extension of the Stable Audio Open architecture that introduces four time-varying acoustic control signals (loudness, pitch, spectral centroid, and timbre) for fine-grained, artist-centric Foley synthesis.
controllablediffusionfoleysfx -
CAFA Discovered by AI
2025
A Controllable Automatic Foley Artist (CAFA) that generates synchronized sound effects for silent videos with enhanced user control via text prompts and reference audio.
controlfoleysfxvideo-to-audio -
DreamFoley Discovered by AI
2025
A scalable video-to-audio generation framework leveraging vision-language models (VLMs) to produce high-fidelity, synchronized sound effects and background audio. It addresses the limitations of existing diffusion transformer architectures by improving cross-modal alignment and audio quality.
foleysfxvideo-to-audiovlm -
Foley Control Discovered by AI
2025
A lightweight approach to video-guided Foley generation that keeps pretrained single-modality models frozen and learns only a small cross-attention bridge between them. It connects video embeddings to a frozen text-to-audio model to refine timing and local dynamics.
foleysfxstable-audiovideo-to-audio -
FoleyGRAM: Video-to-Audio Generation with GRAM-Aligned Multimodal Encoders Discovered by AI
2025
A multimodal generative audio model that synthesizes synchronized Foley effects using video, text, and audio inputs with unified Gramian Representation Alignment Measure (GRAM) alignment.
diffusionfoleymultimodalsound-effects -
HunyuanVideo-Foley Discovered by AI
2025
An end-to-end video-to-audio generation framework developed by Tencent Hunyuan that synthesizes high-fidelity sound effects precisely aligned with visual dynamics. It utilizes multimodal diffusion with representation alignment to achieve realistic semantic and temporal synchronization.
diffusionfoleysound-generationvideo-to-audio -
Kling-Foley Discovered by AI
2025
A large-scale multimodal video-to-audio generation model that utilizes diffusion transformers to synthesize high-quality audio synchronized with video content. It incorporates a visual semantic representation module and an audio-visual synchronization module to align video conditions with latent audio elements.
diffusion-transformerfoleysfxvideo-to-audio -
StereoSync: Spatially-Aware Stereo Audio Generation from Video Discovered by AI
2025
A diffusion-based framework that generates video-aligned stereo audio by incorporating spatial cues through cross-attention, producing soundscapes that dynamically adapt to the visual scene.
diffusionspatial-audiostereo-audiovideo-to-audio -
ThinkSound Discovered by AI
2025
A unified Any2Audio generation framework that utilizes Chain-of-Thought (CoT) reasoning to produce high-fidelity, video-synchronized sound effects and Foley. It decomposes the generation process into foundational foley, acoustic environment modeling, and temporal alignment.
chain-of-thoughtfoleysfxvideo-to-audio -
AudioLCM Discovered by AI
2024
A text-to-audio generation framework based on Latent Consistency Models (LCMs) that enables high-quality and fast sound effects synthesis with few-step sampling.
consistency-modelfast-inferencesound-effectstext-to-audio -
EzAudio Discovered by AI
2024
An efficient text-to-audio generation framework based on a latent diffusion transformer operating on raw waveforms, designed to produce high-quality, natural-sounding sound effects with low computational demands.
diffusionsound-effectstext-to-audioGitHub · arXiv · Hugging Face
-
FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignment Discovered by AI
2024
A two-stage generative framework that decouples temporal structure extraction and semantically guided generation to synthesize synchronized Foley sound effects using video, text, or audio prompts.
controlnetfoleylatent-diffusionsound-effects -
FoleyCrafter Discovered by AI
2024
A video-to-audio generation framework that produces realistic, semantically relevant, and temporally synchronized sound effects from silent videos using a pre-trained text-to-audio model.
diffusionfoleysound-effectsvideo-to-audio -
Frieren Discovered by AI
2024
An efficient video-to-audio (V2A) generation model based on rectified flow matching. It regresses the conditional transport vector field from noise to spectrogram latents, enabling highly synchronized and realistic sound effects for silent videos.
flow-matchingfoleysound-generationv2a -
Make-An-Audio 3 Discovered by AI
2024
An open-source text-to-audio and video-to-audio generation model based on flow-based large diffusion transformers (Lumina-T2X). It generates high-quality, realistic sound effects and music from text prompts or video inputs.
diffusionfoleysfxtext-to-audio -
MMAudio Discovered by AI
2024
An open-source multimodal joint training framework that synthesizes high-quality, synchronized sound effects and foley from video and optional text prompts.
foleymultimodalsfxvideo-to-audio -
Stable Audio Open Discovered by AI
2024
An open-weights latent diffusion model trained on Creative Commons data, optimized for generating short sound effects, foley, and instrument samples from text prompts.
diffusionfoleysfxtext-to-audioGitHub · arXiv · Hugging Face
-
T-Foley Discovered by AI
2024
A temporal-event-guided waveform-domain diffusion model designed for controllable Foley sound synthesis. It incorporates precise timing control to align generated sound effects with specific visual events.
diffusionfoleysfxwaveform -
Tango 2 Discovered by AI
2024
Tango 2 is an aligned diffusion-based text-to-audio generation model fine-tuned using Direct Preference Optimization (DPO) to produce superior sound effects and environmental audio.
diffusiondposfxtext-to-audio -
TangoFlux Discovered by AI
2024
An efficient and fast text-to-audio generative model with 515M parameters capable of generating up to 30 seconds of 44.1kHz high-quality sound effects and ambient audio. It utilizes flow matching and is aligned via preference optimization for superior text-audio alignment.
flow-matchingsfxsound-generationtext-to-audio -
Video-Foley Discovered by AI
2024
A video-to-sound system that uses Root Mean Square (RMS) as a temporal event condition alongside semantic timbre prompts to ensure high controllability and synchronization. It automates the foley synthesis process by aligning generated audio precisely with visual actions without requiring manual timestamp annotations.
foleysfxtemporal-alignmentvideo-to-audio -
Video-Guided Foley Sound Generation with Multimodal Controls Discovered by AI
2024
A multimodal Foley sound generation model developed by Adobe that supports conditioning on text, audio, and video inputs. It generates high-quality, synchronized 48 kHz sound effects for silent videos, allowing flexible control such as using reference audio to guide the timbre of the generated sound.
foleymultimodalsfxvideo-to-audio -
Audiobox Discovered by AI
2023
A unified generative model by Meta based on flow-matching that is capable of generating high-quality speech and sound effects. It supports both description-based (natural language) and example-based (audio reference) prompting for precise control over vocal and acoustic styles.
flow-matchingfoleysfxtext-to-audio -
AudioLDM: Text-to-Audio Generation with Latent Diffusion Models Discovered by AI
2023
AudioLDM is a text-to-audio generation system built on a latent diffusion model that learns continuous audio representations from Contrastive Language-Audio Pretraining (CLAP) latents. It can generate high-quality sound effects, foley, and music from natural language descriptions.
audio-generationfoleylatent-diffusionsfx -
Diff-Foley Discovered by AI
2023
A synchronized video-to-audio synthesis framework based on a Latent Diffusion Model. It utilizes a contrastive audio-visual pre-training model to extract aligned features, generating high-quality and temporally synchronized sound effects for silent videos.
diffusionfoleysound-generationv2a -
FoleyGen Discovered by AI
2023
An autoregressive transformer-based video-to-audio generation system that synthesizes realistic Foley sound effects synchronized with visual events. It utilizes a neural audio codec to convert waveforms to discrete tokens and incorporates visual features via cross-attention.
foleysfxtransformervideo-to-audio -
Make-An-Audio Discovered by AI
2023
A text-to-audio generation framework utilizing a prompt-enhanced diffusion model with a distill-then-reprogram approach to synthesize diverse sound effects and environmental audio.
diffusionsfxtext-to-audio -
Make-An-Audio 2 Discovered by AI
2023
Make-An-Audio 2 is a latent diffusion-based text-to-audio model that enhances temporal consistency and semantic alignment for high-fidelity sound effects generation.
diffusionsfxtext-to-audio -
Tango Discovered by AI
2023
A latent diffusion model for text-to-audio generation that utilizes an instruction-tuned LLM (Flan-T5) as its text encoder to synthesize realistic sound effects and ambient audio from textual descriptions.
diffusionllm-guidedsound-effectstext-to-audio -
AudioCraft Discovered by AI
2022
Meta's open-source framework for generative audio, featuring AudioGen for text-to-sound generation and MusicGen for music. It is widely used for generating realistic sound effects, environmental audio, and foley from text descriptions.
audiogensfxsound-generationtext-to-audio
Proprietary / Industry Tools¶
-
ElevenLabs Sound Effects Discovered by AI
2024
A commercial text-to-audio generation tool designed to produce high-quality, royalty-free sound effects, foley, and ambient audio from natural language prompts. It supports generating multiple variations, seamless looping, and professional-grade 48 kHz audio output.
commercialfoleysfxtext-to-audio -
OptimizerAI Discovered by AI
2024
A proprietary AI-powered sound effects generator designed for creators, game developers, and video producers. It enables fast creation of high-quality, royalty-free audio assets from text prompts or video uploads, supporting stereo output at 44.1 kHz.
foleyproprietarysfxsound-generation
