BidirLM-Omni: One Model That Understands Text, Images, and Audio

We built a single 2.5B-parameter open-source model that understands customer messages across text, images, and audio — and beats the previous best omnimodal model and specialists built for just one modality. It was trained entirely on public, open-source datasets — no client data was used at any stage.

For Diabolocom clients, it’s a small look at the research powering Diabolocom.

Why This Matters for Customer Interactions

Behind every great customer interaction sits a layer of AI doing quiet, unglamorous work: finding the right knowledge base article, routing a request to the right team, detecting intent, measuring conversation quality. These tasks rely on a family of models called encoders — and while the industry has spent billions on chatbots and generative AI, encoders have been left behind.

Generative LLMs read left-to-right, one word at a time. That’s great for writing answers, but it’s not how you should understand a customer message. Classifying intent or matching a question to a knowledge base article works best when the model sees the whole message at once. That’s what an encoder does — and almost every strong open-source model released in the last two years is generative, not an encoder.

BidirLM closes that gap. It’s a recipe for converting the industry’s massive investment in generative models into the encoders that production systems actually need.

The Results

Omnimodal Performance

BidirLM-Omni-2.5B outperforms Nemotron-Omni-3B, the previous best omnimodal model, on every modality — +17 points on text, +5 on images — at nearly half the size. And it beats specialist models built to do just one thing:

– Images: ranks first, ahead of SigLIP, CLIP variants, and E5-V.
– Audio: ranks third overall, beating models nearly twice its size.
– Text: matches our best text-only encoder, despite handling two extra modalities.

A single 2.5B model holding its own against specialists designed for one task. For a production system where every millisecond and every euro of compute counts, that’s a meaningful shift.

A New Way to Build Multimodal AI

The traditional way to build a multimodal encoder is to train everything from scratch, for every combination of modalities you care about. That’s expensive, inflexible, and wasteful.

BidirLM-Omni shows a different path: start with a strong text encoder, then merge in specialized open-source models — vision, audio, safety, whatever the community releases next — and run a short alignment phase. New audio model released? Merge it. Better vision backbone? Merge it. Domain-specific variant for legal or medical text? Merge it.

For our clients, this means a single model that can match a spoken complaint to the right knowledge base article, understand a screenshot shared in chat, and detect intent across many languages — all from one compact, open model we fully control.

How We Built It

Three ideas, each validated through controlled experiments on Gemma3 and Qwen3.

1. A two-step adaptation recipe. You can’t just flip a switch to make a generative model bidirectional. It needs an adaptation phase to learn how to use context from both directions, then a second phase that teaches it to produce high-quality embeddings. This pipeline is the backbone of every BidirLM model, and it puts our text encoders on the open-source frontier — BidirLM-270M matches mmBERT-base with 10% fewer parameters; BidirLM-0.6B beats EuroBERT-610m by over a point.

2. Weight merging to prevent forgetting. Most adapted encoders quietly reuse their lab’s original pre-training data. For everyone else, scaling on new data causes the model to forget what it knew — languages, code, math. Our fix: average the adapted model’s weights with the original, and mix 20% multilingual, code, and math examples into training. Result: +2 points on multilingual benchmarks and up to +11 on code — using only publicly available open-source datasets, no proprietary or client data.

3. Merging as a way to absorb whole capabilities. This is the bold part. If averaging weights recovers lost knowledge, can it also absorb capabilities from completely different specialized models? Yes — and dramatically. We merged our text encoder with a safety model and reached 93% of peak performance after just 80 training examples. We pushed the same idea to vision and audio: +30 points on visual-textual understanding, +19 on audio. Merging worked even when the audio specialist had no text understanding to begin with.

BidirLM-Omni is the payoff. We averaged the text backbones of a vision-language model, a speech model, and our text encoder — all built on the same Qwen3 base — plugged in the existing vision and audio components, and ran a lightweight alignment phase on a balanced multimodal dataset. Total cost: 250 GPU hours, versus the thousands spent training each specialist from scratch.

Get Started

Everything is open:

– Models: BidirLM-270M, 0.6B, 1B, 1.7B, and BidirLM-Omni-2.5B
– Data: Our full contrastive training corpus and the Omni-Contrastive multimodal dataset
– Checkpoints: All intermediate experimental variants

This work was led by Nicolas Boizard (CentraleSupélec × Diabolocom) and Théo Deschamps-Berger (Diabolocom), with Hippolyte Gisserot-Boukhlef, Pierre Colombo, and Céline Hudelot. Thanks to the French supercomputers Adastra (CINES), Jean Zay (IDRIS), and ROMEO (Université de Reims).