OUR MISSION IS TO
BUILD AND DEMOCRATIZE
ARTIFICIAL GENERAL INTELLIGENCE
THROUGH OPEN SCIENCE
AI RESEARCH LAB BASED IN PARIS
OUR MISSION IS TO
BUILD AND DEMOCRATIZE
ARTIFICIAL GENERAL INTELLIGENCE
THROUGH OPEN SCIENCE
AI RESEARCH LAB BASED IN PARIS
Open Science
One of our core values is open science: We aim to build efficient models that integrate multiple input modalities, and to release the tools to use and understand them. To that end, we make our research accessible to as many people as possible through scientific publications and tutorials, demos, and open-sourcing.
We recently open-sourced Pocket TTS, our 100M-parameter multilingual TTS which runs on CPU faster than real-time, bringing voice ability to low resource devices with a single line of code.
In collaboration with Les Invincibles
, we release Invincible Voice, an open-source tool leveraging our speech-to-text and text-to-speech models to give a voice to people suffering from Charcot's disease.
// OUR LATEST OPEN SCIENCE PROJECTS
Audio
Moshi is the first speech-native dialogue system, unveiled during our keynote. Moshi processes speech directly rather than converting to text and back, which means it has minimal latency, and can understand emotions, as well as other non-verbal aspects of communication.
Moshi's multi-stream paradigm also enabled us to create Hibiki-Zero, a speech-to-speech translation model that runs in real time, and Unmute, which allows any LLM to listen and speak by equipping it with our low-latency streaming text-to-speech and speech-to-text models. Going beyond speech, we also release Muscriptor, a lightweight open-source model transcribing any music into multi-instrument MIDI.
Our first spin-off, Gradium, is committed to further turn this open speech research into production-ready systems to support all voice applications and enable real-time voice interactions at scale.
Vision
We explore vision both as an input and output modality, with a strong focus on video data and model efficiency.
Through MoshiVis and CASA, we investigate how to leverage cross-attention to efficiently fuse visual inputs with text or speech for real-time interaction with long-horizon visual streams such as videos.
On the side of generative vision, we release MIRA as a joint project with General Intuition, in collaboration with Epic Games. MIRA is a real-time multiplayer world model trained on 10k hours of Rocket League games, which can simulate up to four players games at 20fps on a single B200 GPU, enabling interactive gameplay.
// OUR LATEST VISION PROJECTS
Language
Our first milestone in language understanding is Helium 1, our modular and multilingual 2B-parameter LLM. In the spirit of open science, we are also releasing the dactorycodebase and tools required to reproduce our training dataset. We further release several research publications on better understanding LLM pretraining and modularity in LLMs.
We also investigate how to leverage pre- and post-training techniques of LLMs in multimodal settings involving speech or vision. For instance, we introduce ARC-Encoder, an efficient method to compress large contexts for LLMs, and use it as a core component to build MoshiRAG, enabling asynchronous knowledge retrieval for full-duplex speech models.