Seeing What Text Can't Say: How Vision-Language Models Are Unlocking the Other Half of Enterprise Data

IA FORUM MEMBER INSIGHTS: ARTICLE

By Ravi Dalal, Director of GenAI, Agents & Innovation, Walmart Global Tech, WALMART

A practitioner's deep-dive on VLMs in production - architecture, integration patterns, and the failure modes that matter at enterprise scale.

The Strategic Shift Most Leaders Are Still Underestimating

Most enterprise AI strategies in 2026 are, when you really look at them, text strategies. The last two years have been spent perfecting prompt engineering, retrieval-augmented generation, evaluation harnesses, and guardrails - all built around large language models that read text and emit text.

That work matters. But it operates on a surprisingly narrow slice of what enterprises actually know.

Walk into any business and the most valuable knowledge is rarely sitting in a clean text database. It lives inside scanned contracts and historical PDFs, engineering schematics, factory inspection photos, medical imagery, dashboards captured as screenshots, slide decks that no one ever turned into structured data, product photography, retail shelf images, claims documents, identity scans, and the visual layout of forms that humans read effortlessly and LLMs simply cannot.

This is the half of enterprise data that text-only systems do not see. And it is the half that Vision-Language Models – VLMs - are beginning to open up.

I want to use this piece to go deeper than the hype cycle: what these models actually are at - on an architectural level, where they integrate with the RAG and agentic systems most enterprises are already building, and the failure modes I see consistently in production.

What a VLM Actually Is

A VLM is a model that can take an image (or several) as input alongside text and reason about them jointly in language. Architecturally, modern VLMs converge on a fairly common pattern.

Figure 1. VLM architecture - image patches become tokens that share the LLM's context window with text tokens.

Three (3) things matter about that picture from a system-design perspective.

First, the image becomes tokens. A page, a chart, or a photo is split into patches by the vision encoder - typically a Vision Transformer or a contrastive image-text encoder such as CLIP or SigLIP - projected into the LLM's embedding space - and then sits alongside text tokens in the same context window. This is why context length matters disproportionately for VLMs - a single high-resolution document page can consume several thousand tokens before the model has read a word of the prompt.

Second, the projector is the integration point. It is the layer that makes the visual representation legible to the language model. Designs range from a simple MLP, as in LLaVA-style architectures, to query-based transformers like the Q-Former in BLIP-2, to gated cross-attention layers as in Flamingo. The quality of this mapping is why some open-source VLMs feel sharp on documents while others feel hallucinatory on the exact same input. Two VLMs with the same base LLM can behave very differently because their projectors and visual pre-training differ.

Third, the LLM behind the VLM still hallucinates. Vision does not anchor it the way some demos imply. The model is doing pattern completion over a joint token sequence, and the visual tokens are only as reliable as the encoder, the input resolution, and the training data made them.

This is why "we'll just turn on vision" is a misleading mental model for executives. You are not adding a feature. You are introducing a new modality with its own evaluation, cost, and failure surface.

Why Text-Only RAG Silently Degrades on Visual Data

The single most common architecture I see in enterprise AI today is retrieval-augmented generation: ingest documents, chunk them, embed the chunks, store them in a vector database, retrieve the top-k chunks for a user query, and pass them to a LLM.

Here is what that pipeline looks like for a document-heavy enterprise.

Figure 2. Text-only RAG - OCR linearization quietly drops most of what a human sees on a page.

The orange box is where most enterprise RAG systems quietly bleed accuracy. OCR linearizes a document. A table becomes a stream of cells with no row-column structure. A figure becomes nothing. A chart becomes axis labels with no curve. Stamps, signatures, redactions, and handwritten notes are usually dropped. Cross-references that depended on visual layout collapse.

By the time the chunk reaches the LLM, the LLM is reasoning over a transcription of the document, not the document itself. Users do not see the broken pipe. They see "I couldn't find that information" or, worse, a confident answer drawn from partial context. Trust quietly erodes.

A vision-aware retrieval pipeline raises the floor.

Figure 3. Vision-aware RAG - the page image itself becomes the unit of retrieval and reasoning.

The difference is not cosmetic. The retrieval target is now the page itself, with its layout, figures, and structure intact. The VLM sees what a person would see. Recent late-interaction retrievers such as ColPali and ColQwen demonstrate this directly: on the ViDoRe document-retrieval benchmark, page-image embeddings have produced nDCG@5 gains in the range of fifteen to thirty points over conventional OCR-plus-text-embedding pipelines on visually-rich documents, including financial reports, regulatory filings, and technical manuals.

In my own production work on enterprise document corpora, the direction of the lift has been broadly consistent with that pattern. On a corpus of approximately 50,000 pages of technical manuals, moving from a tuned OCR-plus-text-embedding baseline to a page-image retrieval approach raised top-3 retrieval precision by roughly 20% and reduced unanswered queries by roughly 25%, with the largest gains concentrated on pages containing tables, exploded-view diagrams, schematics, and stamped or signed sections - the parts of a manual where text alone struggles most.

It is not free, which we will come back to. But the ceiling moves, and it moves most for exactly the documents that text-first systems consistently struggle with.

Three Places VLMs Are Already Paying Off in Production

I want to ground the rest of this piece in the patterns I see actually working at enterprise scale, rather than the demos that look impressive on a stage.

Pattern 1: Document Intelligence Beyond OCR

OCR has been "solved" for a decade in the sense that most enterprises already have it deployed somewhere. What was never solved is the semantics of a document. A VLM can look at a one-page invoice and tell you not only what each line says, but which line is the total, which is the vendor, which charges fall under a disputed purchase order, and how this invoice compares structurally to last quarter's batch.

For executive teams the unlock is moving from extraction to understanding. Finance, procurement, claims, legal, KYC, and back-office functions are the early winners. The technical pattern is consistent: a VLM either replaces or sits alongside the OCR layer, and downstream systems consume structured output - JSON, function calls - rather than raw text.

The validator is non-negotiable. VLMs will happily return well-formed JSON that is still subtly wrong - a confident date in the wrong format, a transposed amount, a vendor name pulled from the letterhead instead of the signing block. Schema validation, confidence thresholds, and human-in-the-loop for low-confidence outputs are what turn a clever demo into a production system.

Pattern 2: Operational Visual Inspection

In manufacturing, energy, logistics, claims, telecom, and field services, photos are the operational lingua franca. Crews send images of equipment, damage, infrastructure, packages, and worksites. Historically these images have been a dead end for automation - useful for human triage, useless downstream.

VLMs are turning those images into structured signals: damage type, severity, recommended follow-up, safety risk, even a draft work order. The unlock here is not just speed; it is consistency across thousands of inspectors who otherwise interpret things differently.

The architecture is similar to document intelligence, but the evaluation problem is harder. Domain-specific evaluation sets need to be assembled deliberately, and active learning becomes essential. The first version of one of these systems is rarely good enough. The second or third - trained against the model's own mistakes captured from production - typically is.

Pattern 3: Agents That Can See

This is where VLMs and agentic AI converge, and it is the pattern I am personally most excited about.

A traditional agent operating on tools and APIs - even one built on well-established reasoning-and-acting patterns such as ReAct - lives in a small, well-structured world. An agent that can also read a screen - recognize layouts, parse dashboards, follow UI flows - can finally operate the long tail of internal applications that enterprises will never get around to rewriting with clean APIs. Visual GUI agents such as CogAgent, and the broader class of computer-using agents emerging from frontier labs, point clearly in this direction.

The agent loop itself changes shape when vision is in the picture.

Figure 4. Agentic loop with vision - the screen becomes a first-class observation, not a black box.

Browser-using and computer-using agents are still fragile and still expensive per step. But the trajectory is clear. The agents that survive in production over the next eighteen to twenty-four months will be the ones that can see - not because vision is a nice feature, but because the enterprise software estate is fundamentally visual.

A Reference Architecture: VLM + RAG + Agent

In real deployments these patterns rarely live in isolation. The most capable enterprise AI stacks I see today weave them together.

Figure 5. A reference stack for enterprise VLM-powered AI: vision-aware retrieval, tool-use, GUI agents, guardrails, and online evaluation in one loop.

Three architectural points are worth calling out explicitly.

The retriever is multimodal. It can pull pages, screenshots, charts, and text snippets into the same context window. The orchestration layer is where reasoning happens, but the retriever is what determines whether that reasoning is grounded.

Guardrails apply to both modalities. Output validation, PII detection, prompt-injection defenses - these need to be repeated for vision, not inherited from a text-only stack. Indirect and visual prompt injection, in which malicious instructions are embedded inside an image or a retrieved document asking the model to ignore its rules, is real, well documented, and consistently underestimated.

Evaluation is online, not just offline. Static benchmarks tell you the model can pass a test. Production telemetry tells you whether the system is still working three months in, after the document mix has drifted, the UIs have changed, and the user queries have moved on.

Where VLMs Are Still Fragile

VLMs are powerful, but they are not the finished product some demos suggest. The failure modes are predictable enough that I would name them as the three (3) to design for from day one.

Calibration: VLMs hallucinate visually the way LLMs hallucinate textually. They will confidently misread a poorly scanned figure, miscount items in a photo, assert structure in a table that isn't there, or invent text that the image does not contain. The mitigation is not a single trick - it is structured outputs with strict schemas, confidence scoring, citations back to image regions, and human-in-the-loop on low-confidence outputs. Anyone shipping a VLM system without those scaffolds is shipping a liability.

Cost & Latency: Image tokens are expensive, often an order of magnitude more than text tokens for the same page. A naive design that pipes every page of every document through a frontier VLM will produce excellent accuracy and a budget conversation no one wants. The teams getting this right tier their inference: cheaper classifiers triage which pages need a frontier model, smaller VLMs handle high-volume routine work, and frontier models are reserved for the minority of inputs that genuinely warrant them. Tiering is the answer, not heroics.

Governance: Visual data carries higher sensitivity than most text. Faces, signatures, badge photos, medical imaging, anything regulated. Every guardrail conversation that LLM teams have already had needs to be repeated, deliberately, for vision. Visual prompt injection, biometric data handling, jurisdictional rules on facial data, retention policies for raw images - none of this inherits cleanly from the text-AI playbook.

Evaluating a VLM System Honestly

Most VLM deployments I see fail not because the model is bad, but because the evaluation is bad. A robust evaluation framework for a VLM-driven system has at least four layers, and each layer needs its own evidence.

Figure 6. Four honest layers of evaluation. Collapsing them into a single score is the most common reason VLM systems fail in production.

The mistake to avoid is collapsing these into a single number. A system can have excellent OCR accuracy and still fail business-level evaluation because retrieval is bad. It can have great end-to-end task success on a static benchmark and quietly leak PII in production. Each layer is its own discipline, and the four together are the closest thing we have to a real readiness check.

Leadership Questions Worth Asking

For senior technology and transformation leaders, I would suggest the right question is no longer "Should we adopt VLMs?" That question will answer itself over the next twelve months as the technology becomes embedded in every major AI platform. The more useful questions are sharper.

Where in our enterprise is visual data quietly bottlenecking value, and what would change if a system could read it as fluently as a person? Which of our current RAG and agent investments are silently hitting a ceiling because they were designed text-first? What is our evaluation discipline for visual outputs, and is it actually as rigorous as the one we built for text? Where do governance, privacy, and bias considerations shift when models start seeing rather than reading? And who, organizationally, owns the visual data layer - given that it tends to sit awkwardly between data engineering, application teams, and the business functions that generate the images in the first place?

These are not technical questions. They are leadership questions about where the next wave of value will land - and which teams will be ready to capture it.

A Closing Thought

For most of the last two years, "enterprise AI" has effectively meant "enterprise text AI". That era is ending faster than the conversation around it suggests. The organizations that recognize the visual half of their data as a strategic asset - and design their agents, their retrieval systems, and their governance models accordingly - are going to look meaningfully different from the organizations that don't.

The interesting work, in other words, is no longer just teaching models to read. It is teaching the enterprise to be seen.

Author Disclaimer: The views and opinions expressed herein are those of the Author alone and are shared in a personal capacity, in accordance with the Chatham House Rule. They do not reflect the official views or positions of the Author’s employer, organization, or any affiliated entity.

References

• Dosovitskiy, A., et al, "An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale", Proc. ICLR (2021), arXiv:2010.11929

• Radford, A., et al, "Learning Transferable Visual Models from Natural Language Supervision", Proc. ICML (2021), arXiv:2103.00020

• Zhai, X., et al, "Sigmoid Loss for Language Image Pre-Training", Proc. ICCV (2023), arXiv:2303.15343

• Liu, H., et al, "Visual Instruction Tuning", Proc. NeurIPS (2023), arXiv:2304.08485

• Li, J., et al, "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models", Proc. ICML (2023), arXiv:2301.12597

• Alayrac, J.-B., et al, "Flamingo: a Visual Language Model for Few-Shot Learning", Proc. NeurIPS (2022), arXiv:2204.14198

• Lewis, P., et al, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", Proc. NeurIPS (2020), arXiv:2005.11401

• Faysse, M., et al, "ColPali: Efficient Document Retrieval with Vision Language Models", Proc. ICLR (2025), arXiv:2407.01449

• Yao, S., et al, "ReAct: Synergizing Reasoning and Acting in Language Models", Proc. ICLR (2023), arXiv:2210.03629

• Hong, W., et al, "CogAgent: A Visual Language Model for GUI Agents", Proc. CVPR, (2024), arXiv:2312.08914

• Greshake, K., et al, "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection", Proc. AISec @ CCS (2023), arXiv:2302.12173