Multimodal LLMs: Vision, Audio, and Cross-Modal Reasoning

From Text Tokens to Image Patches

Standard LLMs operate on sequences of token embeddings: each word or subword is mapped to a vector, and the transformer processes that sequence. Extending this to images requires a way to map pixel data into a similar sequence of vectors.

The Vision Transformer (ViT, Dosovitskiy et al., arXiv:2010.11929) introduced the dominant approach. An input image is divided into a regular grid of non-overlapping patches (typically 16x16 or 14x14 pixels each). Each patch is flattened into a vector and linearly projected to the embedding dimension. A special classification token is prepended, and position embeddings are added. The result is a sequence of patch token embeddings that the transformer processes identically to text token embeddings.

For a 224x224 image divided into 16x16 patches, this yields a grid of 14 by 14 patches, or 196 visual tokens. These tokens attend to each other through the standard multi-head self-attention mechanism, with no architectural modifications. The same architecture that learns that words near each other often have related meanings also learns that nearby image regions often have related visual content.

CLIP and Contrastive Pretraining

Training a vision encoder to produce representations that are useful for language tasks requires aligning visual and textual representations in a shared embedding space. Radford et al.'s CLIP (Contrastive Language-Image Pretraining, arXiv:2103.00020) is the foundational work here.

CLIP trains two encoders jointly: a text encoder (a transformer) and an image encoder (a ViT or ResNet). The training objective is contrastive: given a batch of paired image-text examples, the model is trained to maximize the similarity of the paired image and text embeddings while minimizing the similarity of unpaired combinations within the batch. This is computed as a softmax over all possible pairings in the batch, which creates a dense gradient signal involving all examples simultaneously.

The result is an embedding space where the vector representation of a photo of a dog is close to the vector representation of the text "a dog" and far from the vector representation of "a cat" or "a building." This alignment enables zero-shot image classification: you encode a set of candidate class descriptions as text, encode an input image, and select the class whose text embedding is most similar to the image embedding.

CLIP's joint embedding space is not a perfect alignment. It captures statistical co-occurrence between image regions and text descriptions, not causal understanding of what is depicted.

Connecting Vision to Language Generation

CLIP-style pretraining produces a vision encoder that understands images in the context of language, but it does not directly enable language generation from images. Combining a vision encoder with an LLM for generation requires a bridging mechanism.

LLaVA (Liu et al., arXiv:2304.08485) uses a simple approach: a linear projection layer maps vision encoder outputs to the text embedding space, and the resulting visual tokens are prepended to the text token sequence before being fed to the LLM. During pretraining, the projection layer is trained while the vision encoder and LLM are kept frozen. During instruction tuning, the full model is trained on visual question answering and instruction-following data. This two-stage approach produces a capable visual language model with modest compute requirements compared to training from scratch.

More sophisticated approaches use cross-attention layers or Q-Former modules (BLIP-2, Li et al., arXiv:2301.12597) that learn to selectively extract task-relevant visual information rather than passing all visual tokens to the LLM. The tradeoff is between the flexibility of passing all visual tokens (which can be expensive for high-resolution images producing many patches) and the efficiency of learning a fixed-size query that extracts a compressed visual representation.

Audio and Video Modalities

The same general approach extends to audio and video. Audio is typically converted to a log-mel spectrogram (a 2D time-frequency representation) and processed as a sequence of spectrogram patches using the same ViT-style architecture. The Whisper ASR model (Radford et al., arXiv:2212.04356) uses this approach and demonstrates that a single encoder-decoder transformer trained on sufficient data can achieve robust speech recognition across many languages and acoustic conditions.

Video introduces temporal dimension alongside spatial. A video clip can be treated as a sequence of frames, with each frame processed by a vision encoder, and the sequence of per-frame representations fed to the temporal model. Alternatively, 3D patch volumes spanning both spatial and temporal dimensions can be treated as tokens (Video Transformers). The key challenge is computational: a 10-second video at 30 frames per second contains 300 frames, and if each frame generates 196 visual tokens, the full sequence is nearly 60,000 tokens before any text is appended.

Where Multimodal Reasoning Fails

Despite substantial progress, multimodal LLMs have consistent failure modes that matter for enterprise deployment.

Hallucination in visual description is the most well-documented failure. Models trained to generate fluent text from images will sometimes describe objects, text, or relationships not present in the image, especially when the image is ambiguous, low-resolution, or depicts unusual scenes. The fluency of the hallucinated text can make these errors difficult to detect without independent verification.

Spatial reasoning remains challenging. Questions about relative positions ("is the cup to the left or right of the plate?"), counting ("how many people are in the image?"), and geometric relationships ("is the shadow consistent with the stated light source?") are systematically harder than semantic description tasks. These failures reflect the patch-based representation, which does not explicitly encode spatial relationships and must infer them from attention patterns trained on descriptions rather than geometric supervision.

Document understanding presents a specific challenge for enterprise use cases. Forms, tables, and scanned documents require both OCR capability (reading the text in the image) and layout understanding (interpreting how the positions of text elements relate to each other). Models that perform well on natural images often struggle with document-specific layouts, particularly when the document format is unusual or the image quality is low.

Implications for Enterprise Multimodal Deployment

For organizations evaluating multimodal models for enterprise use, the key questions are not about benchmark performance on natural image captioning but about the specific failure modes relevant to the deployment task. A model evaluating whether assembly line components meet visual quality standards has different failure mode risks than a model extracting structured data from invoices or answering questions about architectural drawings.

Evaluating multimodal models requires task-specific test sets that include adversarial examples representing the specific failure modes above, not generic visual question answering benchmarks. Systematic evaluation across resolution, lighting conditions, document formats, and edge cases specific to the deployment domain should precede any commitment to a multimodal system for high-stakes decisions.