A text-only agent calls text tools. A multimodal agent has a richer tool set and a harder routing problem: when the input is a photo, a PDF, and a voicemail all at once, it must decide which tool handles which input, how to normalize the outputs into a shared context, and when to reason across them jointly. This module covers how that routing works, how state is managed across modalities, and which architectural pattern to choose.
By the end of this module you will be able to
Explain how a multimodal agent routes inputs to the correct modality tool
Compare the modality-first and native multimodal architectural patterns and their tradeoffs
Describe two strategies for managing conversation history when inputs include images, audio, and text
Design a multimodal agent for an enterprise claims or contract review use case
When a Claims Agent Gets a Phone Call, a Photo, and a PDF
Imagine the inbox of an insurance claims team: an email arrives with three attachments. A photo of a damaged car taken by the claimant. A PDF repair estimate from an auto shop. And a voicemail file from the claimant explaining what happened. A human adjuster opens all three, cross-references the VIN visible in the photo against the policy, checks whether the estimate items are covered, and drafts a settlement letter.
A multimodal agent does the same sequence, but the mechanics differ from what most people picture when they hear "AI agent." The agent does not simply paste all the content into one prompt and ask for an answer. It routes each input to a specialized tool, receives structured outputs from each, and then reasons across those outputs to produce a decision. Understanding that routing architecture is the core topic of this module.
The key shift from text-only agents
A text-only agent's tool set is text in, text out. A multimodal agent's tool set includes a vision tool (image in, structured description out), a document tool (PDF in, extracted fields out), and an audio tool (audio in, transcript out). The orchestrator decides which tool to invoke based on input type. What the reasoning model sees is always text, but the inputs that produced that text were not.
Tool Routing Across Modalities
The orchestrator is the component that receives the raw input bundle and decides what to call. In practice, routing is implemented as a set of tool definitions, each with an input_type field that the orchestrator uses to match inputs to tools. When the orchestrator receives a file, it inspects the MIME type or a user-provided label and invokes the corresponding tool.
This pattern keeps the routing logic explicit and auditable. A compliance auditor can read the tool definitions and understand exactly which model handled which input. Contrast this with passing a raw image to a single model and relying on that model to internally decide how to process it, which produces the same answer but with no inspectable routing record.
Python pseudocode: tool definitions with input_type routing
tools = [
{
"name": "vision_tool",
"description": "Describe or analyze an image. Extract text, objects, or structured fields.",
"input_type": "image/*",
"model": "vision-model-endpoint"
},
{
"name": "audio_tool",
"description": "Transcribe an audio file. Return timestamped transcript.",
"input_type": "audio/*",
"model": "asr-model-endpoint"
},
{
"name": "document_tool",
"description": "Extract structured fields from a PDF or document.",
"input_type": "application/pdf",
"model": "document-ai-endpoint"
}
]
def route_input(file_obj, tools):
for tool in tools:
if file_obj.mime_type.startswith(tool["input_type"].rstrip("*")):
return tool
return None # falls through to text reasoning
def process_claim(inputs):
normalized = []
for inp in inputs:
tool = route_input(inp, tools)
if tool:
result = call_tool(tool, inp)
normalized.append({"role": "tool", "name": tool["name"], "content": result})
else:
normalized.append({"role": "user", "content": inp.text})
return reasoning_llm(normalized)
State Management with Mixed-Modality Inputs
Conversation history in a multimodal agent is trickier than in a text-only system because some turns are images, some are audio transcripts, and some are text. There are two main strategies for handling this.
Strategy 1
Normalize to text first
Run every non-text input through its specialist tool before it enters the context window. Images become descriptions. Audio becomes transcripts. PDFs become extracted fields. The reasoning model sees only text.
Works with any text-only reasoning model
History is compact and auditable
Easier to replay and debug
Loss: fine visual detail that the vision model did not describe
Strategy 2
Native multimodal context
Use a model that accepts images, audio, and text natively in the same context window. Each turn carries its original modality. The model reasons across all of them simultaneously.
No information lost in normalization
Simpler pipeline: fewer specialist hops
Requires a capable native multimodal model
Loss: harder to inspect what drove the decision
Most enterprise deployments start with the normalize-to-text strategy because it is more controllable and works with a wider range of reasoning models. As native multimodal models mature and audit tooling catches up, native context becomes more viable.
Two Architectural Patterns
At the system level, these strategies correspond to two distinct architectural patterns that differ in how many models are involved and where the reasoning sits.
Fig 1 · Multimodal agent pipeline: input to structured response
Pattern 1
Modality-first
Each input type is handled by a specialist model. A vision model describes images, an ASR model transcribes audio, a document AI model extracts PDFs. All outputs normalize to text before reaching the reasoning LLM.
More controllable: each specialist is independently tunable
Auditable: routing is explicit, not implicit
Works with best-in-class specialists per modality
More infrastructure: multiple model endpoints
Pattern 2
Native multimodal
A single model accepts image, audio, and text inputs natively and reasons across them within one context window. No specialist routing layer is needed.
Simpler pipeline: one model, one endpoint
No normalization loss
Can reason across modalities jointly
Less inspectable: cross-modal reasoning is opaque
Enterprise Example: The Contract Review Agent
A legal operations team receives a contract as a PDF, a recording of the negotiation call as an audio file, and a company policy document as a text file. A multimodal contract review agent routes each input to the appropriate tool: the document tool extracts clause-by-clause text from the PDF, the audio tool transcribes and timestamps the negotiation call, and the text policy is passed directly to the reasoning model. The reasoning model is then asked to flag any discrepancy between what was discussed on the call, what appears in the contract, and what the policy requires.
This kind of cross-modal cross-referencing is the category of task where multimodal agents produce the most enterprise value. It is not about understanding images in isolation: it is about connecting a visual or audio artifact to structured policy and surfacing gaps that a human reviewer might miss when working through three document types separately.
Fig 2 · Agent decision tree: active routing path by input type
Think about it first: why does modality-first routing make compliance audits easier than native multimodal?+
In a modality-first architecture, every routing decision is a logged function call: the orchestrator called vision_tool with file X at time T and received output Y. An auditor can reconstruct exactly what each model saw and what it produced. In a native multimodal model, the same reasoning happens internally and the model does not expose which portion of the input drove which part of the output. For regulated industries, the inspectability of modality-first routing is often worth the added infrastructure cost.
Knowledge Check
In the normalize-to-text state management strategy, what does the reasoning model receive?
The primary advantage of the modality-first pattern over native multimodal for regulated enterprise use cases is:
In the claims agent scenario, what connects the photo of the damaged car to the coverage rules in the policy?
Before you go
A multimodal agent routes each input type to a specialist tool (vision, audio, document) before the reasoning model sees anything. Routing is based on MIME type or an explicit input_type field in the tool definition.
Modality-first architecture produces explicit, auditable routing records and works with best-in-class specialist models. Native multimodal architecture is simpler but less inspectable. For regulated industries, modality-first is usually the right starting point.
State management across modalities comes down to a choice: normalize everything to text before it enters the context window (more controllable), or use a native multimodal context window (more capable but harder to audit).
Reflection: think of a workflow in your organization where inputs routinely arrive in more than one modality. What would the routing architecture look like for that use case, and which pattern would compliance require?
Share this insight: "A multimodal agent does not just understand images. It routes inputs to specialist tools, normalizes outputs to text, and then reasons across all of them. The routing architecture is what makes it auditable. Free course: arjunjaggi.com/course/multimodal-ai/"