Hmm. For now, there are a few technical approaches that are close to what you’re describing:
I think the interesting part of your idea is that there are actually several different ways to “marry” an existing language model to new senses, and they preserve different things.
Very roughly:
1. Keep DeepSeek as the central language model,
and let it consult separate vision/audio specialists.
2. Keep the language model frozen,
but train a bridge that feeds visual/audio representations into it.
3. Retrain the language model as part of a new multimodal model.
Your wording — especially “let you learn to use and consult each other” — sounds closest to #1.
And if the main concern is preserving the existing DeepSeek rather than replacing it with a newly trained multimodal successor, I would probably start there. It is also the cheapest version of the idea to test: you do not actually need to train a new vision-language model at all to find out whether the basic idea is useful.
Something as simple as this is already a legitimate architecture:
┌─ vision specialist
├─ OCR/document specialist
User ──► DeepSeek ───────┼─ audio specialist
▲ └─ other specialists
│
└──── structured observations/results
DeepSeek remains the main reasoner/conversational model. When it needs information that it cannot directly perceive, it calls the appropriate specialist, gets the result back, and continues reasoning.
There is quite a bit of research precedent for exactly this general idea. Socratic Models, for example, explored composing independently pretrained vision, language, and audio models through language without requiring joint fine-tuning. MM-ReAct put ChatGPT at the center of a pool of visual experts. LENS similarly lets an ordinary off-the-shelf LLM reason over information supplied by independent vision modules, without multimodal training.
So the basic idea is not at all outlandish. There is a fairly well-developed family of architectures behind it.
The main distinction I would make is this:
Having another AI describe what it sees to DeepSeek is not quite the same thing as making DeepSeek itself visually multimodal.
Both are useful, but they solve slightly different problems.
A useful way to choose between them is to ask what exactly you want to preserve:
If the priority is...
Keep the central DeepSeek model itself unchanged
→ external vision/audio specialists
Keep the same language-model weights,
but give it more direct access to sensory representations
→ frozen-LLM + modality adapters/bridges
Allow DeepSeek itself to change during training,
while trying to preserve its language abilities
→ integrated VLM / multimodal training
Preserve the same "mental set" or perspective
→ that is a separate behavioral question;
none of the architectures above guarantees it by itself
1. The version closest to your 'consult each other' idea
There are several surprisingly direct precedents.
Socratic Models
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language is probably the closest conceptual match I know.
The idea is to leave independently pretrained models specialized in different domains and let them exchange information through language.
That can look like:
vision model ─┐
audio model ─┼──► language interface ──► language model
other model ─┘
The interesting point is that the models do not need to have been jointly trained as one giant multimodal network.
That is very close to your idea of creating a visually oriented AI and a sound-oriented AI and allowing the language model to consult them.
MM-ReAct
MM-ReAct makes this even more concrete.
It uses ChatGPT as the main reasoning component and gives it access to a pool of vision experts. Depending on the problem, the system can obtain things such as recognition results, OCR, spatial information, or other visual analysis and then continue reasoning with those results.
That matters because the interface does not have to be only:
vision AI:
"There is a dog in the image."
It could instead return something much richer:
objects:
- dog: bounding box [...]
- bicycle: bounding box [...]
text:
- "PARK ENTRANCE"
scene:
- outdoor
- daytime
uncertainty:
- possible second animal in background
DeepSeek could then reason over that structured evidence.
LENS
LENS takes a similar modular approach: independent vision modules extract detailed information from an image, while an off-the-shelf language model performs the reasoning.
An important property of that approach is that the language model itself does not require multimodal training.
ViperGPT
ViperGPT illustrates an even more active version of the idea.
Rather than receiving one visual description, the language model can compose calls to visual modules programmatically:
question
↓
LLM decides what visual operation is needed
↓
object detector / OCR / VQA / other function
↓
result
↓
possibly another operation
↓
answer
So “consulting” a visual AI need not mean asking one vision model one question. The language model can potentially orchestrate several specialized perceptual capabilities.
This is one reason I would distinguish multimodal agents from a single end-to-end multimodal model.
2. There is also a middle ground: keep the 'brain' frozen, but add sensory pathways
There is another family that falls between “DeepSeek talks to a separate AI” and “retrain DeepSeek into a new multimodal model.”
The central language model can remain frozen while a smaller trainable component learns how to translate another modality into representations the language model can use.
A simplified picture is:
image
↓
vision encoder
↓
trainable bridge
↓
frozen language model
This gets closer to the intuition of giving the existing model a sensory pathway, rather than giving it a separate witness that reports in text.
BLIP-2
BLIP-2 is a classic example.
It connects a frozen image encoder to a frozen large language model with a relatively lightweight learned component called a Q-Former.
So the big language model itself does not need to be retrained just to establish the vision-language connection.
X-InstructBLIP
X-InstructBLIP is particularly relevant to your idea because it goes beyond images.
It aligns:
- images
- audio
- video
- 3D inputs
to a frozen LLM, while also studying reasoning across modalities.
That is remarkably close to the sequence you suggested:
language model
+
visual system
+
audio system
+
...
except that the communication is through learned representations rather than two conversational agents speaking natural language to one another.
Freeze-Omni
The audio side has a similarly relevant example in Freeze-Omni.
Its central design goal is to connect speech input and speech output components to a textual LLM while keeping that LLM’s parameters frozen during training.
So there is already real work on something very close to:
“Keep this language model, but add ears and a voice without rewriting the language model itself.”
That does not guarantee identical behavior — the model is now receiving information it never received before — but it does give a much stronger technical meaning to “preserve the original model.”
3. DeepSeek itself has already explored the other route
There is also an especially relevant DeepSeek-specific precedent: DeepSeek-VL, with the official repository and Hugging Face model.
The older DeepSeek-VL-7B was constructed from the earlier DeepSeek-LLM-7B family and added visual processing through a hybrid vision encoder and a vision-language adaptor.
So DeepSeek has already explored the broader question:
How do you take an existing language-model foundation and turn it into a vision-language system without destroying its language abilities?
But there is an important difference from your proposal.
DeepSeek-VL did not simply leave the language model permanently untouched and bolt an independent vision AI onto it.
Its training pipeline was approximately:
Stage 1:
vision encoder frozen
language model frozen
VL adaptor trained
Stage 2:
vision encoder frozen
language model trained
VL adaptor trained
Stage 3:
part of vision stack trained
language model trained
VL adaptor trained
So after the initial adaptor stage, the language model itself becomes part of the multimodal training process.
That makes DeepSeek-VL an example of the integrated-model route, rather than the “same DeepSeek consulting another AI” route.
The really interesting part of the DeepSeek-VL paper for your concern is that the authors explicitly encountered a trade-off between multimodal learning and preservation of language capability.
When they trained the language model on multimodal data alone during the joint pretraining stage, multimodal performance improved but language metrics deteriorated sharply. They describe this in terms of catastrophic forgetting and mitigate it by retaining substantial language data during multimodal training.
That does not prove anything about preserving a model’s personality or perspective, but it does show that the more concrete version of your concern —
“If we retrain a good language model to acquire another modality, can we accidentally damage things it already knows how to do?”
— is a real engineering/research problem, including in DeepSeek’s own VLM work.
So I would not say “DeepSeek-VL already did exactly what you proposed.”
I would say:
DeepSeek-VL explored one nearby solution, while your “separate specialist that DeepSeek consults” idea points toward a more modular solution.
4. 'Consult each other' and 'learn together' are actually different operations
I would separate these two phrases, because they lead to very different engineering projects.
Consult
No training may be necessary at all.
DeepSeek:
"I need visual information."
↓
vision model:
"Here are the objects, text, positions, and scene description."
↓
DeepSeek:
"Now I can reason about the user's question."
That can be prototyped with existing models.
Learn to consult
Now the system learns things such as:
- when a specialist is necessary,
- which specialist to call,
- what question to send it,
- whether another call is needed,
- how to combine conflicting results.
That is more like tool-use or routing learning.
LLaVA-Plus, for example, explicitly studies a multimodal assistant that learns to activate relevant pretrained vision/vision-language tools from a skill repository.
Learn from previous consultations
That could instead mean giving the system memory:
task
→ specialist calls
→ successful solution
→ save useful procedure/example
→ reuse it later
The underlying model weights need not necessarily change.
Learn the new sense
Now you are training an adaptor, projector, encoder, or other modality-specific component.
Learn together
If both the central LLM and the specialist are updated through training, that becomes a much deeper form of joint multimodal training — and then preservation of the original model becomes substantially harder to define and measure.
So merely connecting two AIs will not make them automatically learn from each other.
But allowing one model to learn how to use the other is itself a very real and useful design direction.
5. If you keep adding senses one after another
Your proposed sequence —
DeepSeek
↓
add vision
↓
add sound
↓
perhaps add more later
— leads to another research problem: modality-incremental learning.
The difficulty is not only ordinary catastrophic forgetting.
For example, MERA: Merge then Realign studies adding modalities incrementally and reports another failure mode: components trained for a particular modality can become misaligned with the common LLM backbone as the overall system evolves.
So if modalities are being trained into one model sequentially, there are at least two questions:
Did adding audio make the model forget earlier capabilities?
and
Did adding audio alter the shared representation enough
that the old vision connector no longer lines up as well?
That problem is much less severe in the simple external-agent version, because the visual specialist can remain a separate component with a defined input/output contract.
That modularity is one of the strongest practical arguments for trying the loose-coupling version first.
6. What does it mean for it to remain 'the same DeepSeek'?
This is probably the hardest part of the idea, because several different things can be meant by “the same.”
They are worth separating:
| What is preserved? |
Can we check it fairly directly? |
| Same language-model weights |
Yes |
| Same text-only capabilities |
Mostly — with regression tests |
| Same conversational behavior |
Partly — with fixed prompt suites |
| Same reasoning tendencies |
Harder |
| Same “mental set” or perspective |
Much harder to define objectively |
For example, freezing the original LLM gives you a very strong guarantee about the first row:
those language-model weights did not change.
But it does not imply that the entire system will behave identically.
If the old model receives:
"The man is standing beside a car."
and the new system receives:
Vision system:
person = ...
vehicle = ...
distance = ...
facial expression = ...
visible text = ...
scene = ...
the same unchanged language model may reasonably respond differently because its context is different.
Likewise, keeping a language benchmark score unchanged does not establish that some more subtle quality you value as a “perspective” has remained unchanged.
If someone actually wanted to test this experimentally, I would separate the evaluation into four layers:
A. Text-only regression
Run the original model and expanded system on the same fixed text-only prompts.
This asks:
Did the extension disturb what the language model already did?
B. Perception
Test the new sensory component separately.
This asks:
Did the vision/audio system perceive the relevant information correctly?
C. Cross-modal use
Give tasks whose answer genuinely depends on the image or audio.
This asks:
Did the central model actually use the new information correctly?
D. Behavioral continuity
Keep a fixed set of conversations covering whatever qualities you mean by “mental set” or “perspective” and compare the original and expanded systems.
That would not turn “same personality” into a mathematically exact property, but it would at least make the question testable rather than leaving it entirely intuitive.
7. A very small proof of concept could come before any training
If the goal were simply to find out whether your idea has merit, I would not start by training a new DeepSeek-VL.
I would start with something closer to:
┌─────────────────┐
image ──────────►│ vision model │
└────────┬────────┘
│
structured evidence
│
▼
text ───────────► DeepSeek
│
▼
response
For the first version, the visual model could return:
scene:
indoor kitchen
objects:
- saucepan
- cutting board
- knife
ocr:
- "LOW HEAT"
spatial:
- saucepan is on the front-left burner
- knife is right of cutting board
uncertainty:
- small object near sink not confidently identified
rather than one prose caption.
That would make failures easier to diagnose:
Wrong answer
│
├─ Was the scene perceived incorrectly?
│
├─ Did the sensory interface omit the needed information?
│
├─ Did DeepSeek misunderstand correct evidence?
│
└─ Did the orchestration call the wrong specialist?
This is one practical advantage of the modular version: the failure boundaries are visible.
If someone wanted to prototype the orchestration on the Hugging Face side, the current smolagents documentation is one possible starting point. It supports model-agnostic agents, multimodal inputs, tools, Hub/Space integrations, and multi-agent orchestration.
That is only one implementation option — the architectural idea does not depend on that particular library.
The important point is that the experiment could be done before committing to any expensive multimodal training.
So, if I translate your proposal into the terminology used in the literature, I would not reduce it simply to “make DeepSeek multimodal.”
I would describe the interesting choice more like this:
Option A — sensory colleagues
DeepSeek consults separate visual/audio specialists.
Lowest coupling; easiest way to preserve the original model.
Option B — sensory pathways
Keep the language model frozen and train modality bridges.
Deeper integration, while still preserving the LLM weights.
Option C — multimodal successor
Train the language model itself together with the new modalities.
Potentially the deepest integration, but now preserving the old
language behavior becomes an explicit training/evaluation problem.
And there are real examples of all three families.
Given the concern you started with — not wanting a future multimodal system simply to replace the DeepSeek you already value with something different — A seems like the cleanest first experiment, with B as the particularly interesting middle ground.
It lets you test the central idea first:
Can the existing language model become substantially more capable by learning when and how to consult sensory specialists, without having to rewrite the language model itself?
If that works well, deeper integration is still available later. But you would learn quite a lot before crossing that boundary.