DeeperML

Meta is developing MoCha (Movie Character Animator), an AI system designed to generate complete character animations. MoCha takes natural language prompts describing the character, scene, and actions, along with a speech audio clip, and outputs a cinematic-quality video. This end-to-end model synchronizes speech with facial movements, generates full-body gestures, and maintains character consistency, even managing turn-based dialogue between multiple speakers. The system introduces a "Speech-Video Window Attention" mechanism to solve challenges in AI video generation, improving lip sync accuracy by limiting each frame's access to a specific window of audio data and adding tokens to create smoother transitions.

MoCha runs on a diffusion transformer model with 30 billion parameters and produces HD video clips around five seconds long at 24 frames per second. For scenes with multiple characters, the team developed a streamlined prompt system, allowing users to define characters once and reference them with simple tags throughout different scenes. Meta’s AI research head, Joelle Pineau, announced her resignation which is effective at the end of May, vacating a high-profile position amid intense competition in AI development.

ImgSrc: the-decoder.com

References :

• Analytics India Magazine: Meta’s AI Research Head Joelle Pineau Announces Resignation
• THE DECODER: Meta develops AI system MoCha that turns text into animated characters

Classification:

• HashTags: #MetaAI #CharacterAnimation #AIResearch
• Company: Meta
• Target: Researchers
• Product: MoCha
• Feature: MoCha
• Type: AI
• Severity: Informative