A user pastes a screenshot of a Python traceback into a chat interface and, within seconds, receives a precise explanation and a corrected code snippet. No text described the error. The model read the image. This module unpacks exactly how that happens: from raw pixels to patch vectors, through a shared embedding space, into cross-attention with language, and why the same architecture that enables this also produces a distinct class of errors that text-only models do not have.
The Screenshot That Started the Question
Before 2021, a user who encountered a confusing error message would copy the text from the terminal and paste it into a search engine or a help forum. The image on their screen was not part of the query. Models could not read it.
The architectural change that made image-reading possible is not magic and it is not a separate modality bolted onto a language model. It is a principled extension of the same embedding-space logic that makes language models work. Once you understand it at the mechanism level, you can reason clearly about what these systems can and cannot do, why they fail in predictable ways, and what questions to ask before deploying one in an enterprise workflow that touches images.
What this module is not
This module does not cover every architecture variant or training recipe. It gives you the core mechanism shared across the major vision-language model families: patch tokenization, a shared embedding space learned through contrastive training, and cross-attention fusion. That is enough to reason about behavior and failures in practice.
Step 1: Images Become Tokens Through Patch Splitting
A language model processes sequences of tokens. To feed an image into one, the image must become a sequence of something the model can treat like tokens. The approach used across most modern architectures is called patch splitting.
The image is divided into a grid of fixed-size square patches. A common choice is 14x14 pixels per patch or 16x16 pixels per patch, depending on the model family. Each patch is flattened into a vector of numbers, then projected through a learned linear layer into the same embedding dimension the model uses for text tokens.
Consider a 224x224 pixel image divided into 16x16 pixel patches. That gives 14 patches per side, or 196 patches total. Each patch becomes one visual token. The sequence of 196 visual tokens then enters the model alongside the text tokens from the user's prompt.
Position encodings are added to each patch vector before processing. Without them, the model would have no way to distinguish a patch in the upper-left corner from an identical-looking patch in the lower-right. Position encodings encode location in the image grid so the model can reason about spatial relationships: this text is above that button, this chart label is to the left of that bar.
Why patch size matters in practice
A 56x56 pixel patch covers a large area. Fine text, small icons, and dense tables may fall within a single patch, collapsing into a single vector before the encoder ever sees them in detail. Smaller patches preserve more spatial resolution at the cost of more tokens and higher inference compute. When a VLM fails to read small text in a screenshot, patch resolution is often the first thing to examine.
Step 2: Contrastive Pretraining Builds the Shared Space
Splitting an image into patches is necessary but not sufficient. The resulting patch vectors are in a raw projection space that the language model cannot yet interpret as meaningful visual content. The key step that makes it all work is a pretraining method called contrastive learning.
The setup: take millions of image-caption pairs gathered from the internet. Run each image through a vision encoder to get an image embedding. Run each caption through a text encoder to get a text embedding. Then train both encoders together with one objective: make the embedding of a matching image-caption pair as close as possible in the shared space, and make the embedding of every non-matching pair as distant as possible.
The result is a vision encoder that has learned to compress images into vectors that carry semantic meaning recognizable to the language model. An image of a running dog maps to a vector near the text vector for "a dog running across a field." A screenshot of a Python traceback maps near vectors for terms like "error," "exception," and "stack trace." This shared space is what lets the language decoder treat image patches and text tokens as inputs of the same kind.
The most widely known implementation of this approach was published as CLIP (Contrastive Language-Image Pretraining) in 2021, and variants of this contrastive alignment step now appear across most major VLM architectures, though the details of encoder architecture, training scale, and alignment procedure vary.
The practical implication
Contrastive pretraining means the vision encoder was trained on internet image-caption pairs. The shared space reflects the distribution of that training data. Concepts and image types well-represented in internet data (photographs of common objects, screenshots of popular software, printed text in Latin scripts) will have richer, more accurate representations than concepts underrepresented in that data.
Step 3: Cross-Attention Fuses Vision and Language
Once the image is encoded as a sequence of patch vectors and the user's prompt is encoded as text tokens, the language decoder must combine both when generating a response. The mechanism is cross-attention.
At each decoding step, the model computes attention scores between the current state of the generation and every element in the combined context: image patch vectors and text token vectors. These scores determine how much each context element influences the next token that gets generated.
This is why a question like "What is the error on line 7 of this code screenshot?" can be answered correctly. The model attends heavily to the patch regions where line 7 appears while simultaneously processing the semantic meaning of "error" from the text tokens. The output token sequence is shaped by both information sources at every step.
Not all architectures implement this identically. Some models concatenate image patch tokens with text tokens and feed the combined sequence through a unified transformer (no separate cross-attention layer, just a single self-attention mechanism over everything). Others use dedicated cross-attention layers. The user-visible behavior is similar either way: the model can reference image content while generating text.
Why VLMs Hallucinate About Images
VLMs hallucinate about image content for reasons that are partially distinct from why language models hallucinate about facts. Understanding the specific mechanisms matters because the failure mode shapes what mitigations actually work.
The root cause is the same as in text-only models: the model is trained to generate the most plausible next token given its context. When the visual evidence is ambiguous, low-resolution, or falls in an area the model's attention did not weight heavily, the model fills in with the most plausible textual description of what could be there, not a faithful report of what is.
The practical test
Before deploying a VLM on image content that matters, test it explicitly on low-resolution versions of your images, on images with fine print, and on images where the key information is in an unusual location. Hallucinations in these conditions are predictable and worth quantifying before a system touches decisions that depend on accurate image reading.
Python · Calling a vision-language API with a base64-encoded image
import base64
import httpx # or use openai SDK directly
def read_image_base64(image_path: str) -> str:
"""Encode a local image file as a base64 string."""
with open(image_path, "rb") as f:
return base64.standard_b64encode(f.read()).decode("utf-8")
def describe_image(image_path: str, prompt: str, api_key: str) -> str:
"""
Send an image and a text prompt to an OpenAI-compatible vision API.
Returns the model's text response.
The model receives both the image (as base64) and the prompt together.
"""
image_data = read_image_base64(image_path)
# Determine MIME type from extension (extend as needed)
ext = image_path.rsplit(".", 1)[-1].lower()
mime = {"jpg": "image/jpeg", "jpeg": "image/jpeg",
"png": "image/png", "webp": "image/webp"}.get(ext, "image/png")
payload = {
"model": "gpt-4o", # any OpenAI-compatible VLM endpoint
"messages": [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": f"data:{mime};base64,{image_data}",
"detail": "high" # "low" for faster, cheaper processing
}
},
{
"type": "text",
"text": prompt
}
]
}
],
"max_tokens": 1024
}
resp = httpx.post(
"https://api.openai.com/v1/chat/completions",
headers={"Authorization": f"Bearer {api_key}"},
json=payload,
timeout=60
)
resp.raise_for_status()
return resp.json()["choices"][0]["message"]["content"]
# Example usage
if __name__ == "__main__":
import os
api_key = os.environ["OPENAI_API_KEY"]
answer = describe_image(
image_path="error_screenshot.png",
prompt="What Python error does this screenshot show, and what is the most likely fix?",
api_key=api_key
)
print(answer)
Think about it first: if a VLM gives a confident wrong answer about image content, what three things would you check?
+
1. Patch resolution: is the relevant detail (small text, a number, a label) small enough that it may have been collapsed into a single patch? Try a higher-detail mode or a higher-resolution input image.
2. Attention coverage: is the model being prompted to focus on the right region? A more specific prompt ("read the value in the bottom-left cell of the table") often reduces attention mismatch errors compared to a general prompt.
3. Training distribution: does your image type appear frequently in internet image-caption data? Medical imaging, engineering schematics, handwritten forms, and domain-specific charts are often underrepresented. Evaluate on held-out examples from your actual domain before assuming the model generalizes.