The Second Kind of AI Value

IA FORUM MEMBER INSIGHTS: ARTICLE

By Dr. Pavan Kumar Nagula, Director, Data Science, VIALTO PARTNERS

A New Lens On AI Value

Four AI applications in tax and immigration services reveal a consistent pattern: the automation objective is valuable. What the automation reveals is more so.

• AI value should be measured in two layers: automation value and diagnostic value. Most organizations are capturing the first. The second is more consequential.

•  The diagnostic layer reveals hidden business constraints, weak performance metrics, untapped data signals, and process ceilings that have been accepted as fixed.

• The firms that benefit most from AI will not simply deploy more models. They will learn more from each deployment.

• Ask of every AI project: what did this automate - and what did it reveal?

A Typology of Discoveries

The four discoveries that emerge from this body of work are distinct in character. Some concern measurement - the way performance is defined and reported. Some concern method - whether the analytical tools being used are suited to the problem domain. Some concern structure - the invisible constraints embedded in processes that have never been examined from the outside. And some concern the opportunity to make existing capability more legible and compelling.

Together they form a map of the places where professional services organizations are routinely operating with less clarity than their data would allow. The diagnostic value of AI is, in each case, the act of making that map legible.

DISCOVERY 01: THE WRONG METRIC HIDES THE RIGHT ANSWER

Volume Forecasting: When the Right Answer Hides Behind the Wrong Metric

The forecasting scenario that opens this article is not an edge case. It reflects a structural problem in how AI performance is evaluated across the profession: aggregate metrics that look strong at the headline level while concealing material variance at the operational level.

In tax questionnaire volume forecasting, the insight that unlocks genuine predictive power is upstream. Briefings delivered to employees - a distinct service activity that precedes the questionnaire cycle - emerge as one of the most powerful leading indicators of future questionnaire volumes. The relationship is logical in retrospect: a briefing is delivered when an employee is approaching a situation that will generate a tax obligation. The briefing anticipates the questionnaire well in advance of when that demand materializes.

Months Ahead: Upstream Signal - Tax briefing activity is a meaningful upstream signal for questionnaire demand, embedded in an earlier stage of the same service cycle and invisible to models trained only on current-period data.

Identifying this signal is only half the problem. Exploiting it correctly is where the methodological precision becomes critical - and where a naive implementation actively misleads.

The lesson here is not simply that the model needed more data. It needed the right business sequence. Briefings must be aligned to the downstream questionnaire demand they create - because the forecasting model does not automatically know that one precedes the other, or by how long, or in which months the downstream effect concentrates by market. Without that alignment, the model learns whatever accidental relationship exists between the two unaligned data series and encodes it as a predictive signal. Leadership receives a forecast that looks technically sound but is built on the wrong relationship.

The practical implication is that the forecasting model is not simply told that briefings predict questionnaire volumes. It is given a precisely constructed picture of briefings activity - one that accounts for the full annual cycle of engagement and reflects the specific months in which each country and segment historically concentrates its questionnaire demand. The model therefore sees the leading signal in the form most relevant to the operational question being asked: not how many briefings happened, but when their downstream effect will be felt, and at what scale, in each market.

This is a class of methodological trap that is invisible without domain understanding. A data scientist unfamiliar with the operational relationship between briefings and questionnaires would have no basis for knowing what lag to engineer. A tax operations expert unfamiliar with SARIMAX's limitations around exogenous variable lag estimation would not know that engineering was required at all. The insight sits precisely at the intersection of the two - which is where the most consequential AI methodology decisions in professional services tend to live.

The discovery here operates on two levels. The first is measurement: headline accuracy metrics can conceal operational exposure, and the wrong metric optimizes the wrong outcome. The second is methodological: a powerful predictive signal exists in the data, but extracting it correctly requires knowing both that the lag structure matters and that the model class cannot estimate it automatically. Miss either level and the forecasting capability either looks weaker than it is, or worse, encodes counterintuitive patterns that undermine rather than support operational decisions.  

Leadership Implication: Before accepting any AI forecast as operationally sound, ask two questions: does the accuracy metric reflect operational impact, or merely aggregate volume? And do the signals feeding the model reflect the actual business sequence, or just the most readily available data? The answers to both determine whether the forecast can be trusted to drive resource and planning decisions.

DISCOVERY 02: YOU CANNOT SEE THE DIFFERENCE BETWEEN HAVING CAPABILITY AND PROVING IT

RFP Intelligence: Why Win/Loss Is the Wrong Instrument for Content Quality

Win/loss data is the natural starting point for RFP performance analysis - CRM systems capture it, leadership asks for it, and it reflects the outcome that matters commercially. The insight that drives this work is a recognition that win/loss data, while essential for commercial tracking, conflates content quality with many other factors: relationship history, pricing, incumbency advantage, procurement timing. Separating the content quality signal from this composite outcome opens a more precise and more actionable form of analysis - one that tells content teams exactly what is within their control to strengthen.

The pipeline built to address this starts upstream, at the RFP itself. Requirements and Value Propositions - the specific claims and promises the bid must address - are extracted from incoming client RFPs and grouped into themes, each with a narrative that articulates what the theme is arguing. RFP response content is indexed in a vector store per client. For each Value Proposition identified, retrieval-augmented generation pulls the three most relevant response chunks, and alignment between the retrieved content and the VP is assessed with a rationale explaining the score. CRM data is then merged to tag each record as won or lost.            

What the merged dataset reveals is the finding that reframes the entire purpose of the analysis. Alignment score and win/loss outcome are independent variables - winning depends on many factors beyond content quality alone, which is precisely what makes content quality worth measuring on its own terms. A well-written, tightly aligned response loses bids that incumbency or pricing decide. A loosely aligned response wins bids that relationship or timing decide. Using win/loss as the sole lens for content improvement therefore limits the precision of the guidance it can offer.            

The response to this insight is to decouple content quality measurement from outcome entirely. A Claude skill is built on historical VP-Response-Rationale records where alignment scored at or above 50% - a threshold derived empirically through histogram analysis of alignment score distributions across industry-theme pairs, rather than set arbitrarily. The analysis revealed a natural distributional structure in the scores, with 50% representing the point at which responses transition from partial to substantive alignment with the Value Proposition. This threshold is explicitly designed to rise as more RFP response data accumulates - making the baseline progressively more precise as the library grows. The result is a curated baseline of what strong content alignment looks like for a given Value Proposition type, independent of whether the bid was won or lost. When a new RFP response is reviewed, the skill compares it against this baseline, segmented by theme, and produces a report that reflects how well the current response aligns with historically strong responses for that VP type. The output is an estimated alignment probability: a pure measure of content quality, kept separate from outcome noise.

Partial Alignment: Content Quality Zone - The zone where most RFP response content clusters - capable and relevant, but not yet structured to answer the evaluator’s implicit question: not “can you do this?” but “prove it.”

The more granular insight emerges from the language of the model's rationale text, examined across hundreds of records. The differentiating factor between high-scoring and mid-scoring content is almost never capability itself - the response consistently demonstrates the firm can address the Value Proposition. What varies is the presence of structured proof: specific metrics, named deliverables, quantified SLAs, concrete client-facing examples. The capability exists. The evidence that makes it legible to a procurement evaluator often does not.

Analyzed across the full library, theme-level patterns emerge that no individual bid review could surface. Certain VP themes show consistently lower alignment scores against the high-scoring baseline - a content investment signal that is precise, evidence-based, and entirely independent of win/loss noise. It points to exactly where response templates and content libraries can be further strengthened - and does so from a baseline of what strong alignment actually looks like for that VP type, not from assumption.

The discovery this pipeline surfaces is precise. The insight this pipeline surfaces is that content quality and bid outcome, while related, are distinct variables that reward distinct measurement instruments. Win/loss analysis tracks commercial performance. A VP-Response-Rationale baseline calibrated to alignment ≥50%, independent of outcome, tracks content performance. Both are valuable. Together, they give content teams a complete picture: what won, and whether the writing was as strong as it could have been regardless of outcome.            

Leadership Implication: Alignment score alone was not predictive of whether a bid was won or lost. Win/loss data reflects many variables beyond the writing team’s control. Requiring content teams to improve on the basis of win rate alone conflates commercial outcome with content quality. Ask instead: how does this response compare against the best responses ever written for this type of client requirement?

DISCOVERY 03: YOU CANNOT SEE WHEN YOUR TOOLS DO NOT FIT YOUR PROBLEM

PERM Clustering: When Standard Algorithms Meet Specialist Legal Text

The Department of Labor publishes certified PERM applications at scale, embedding within that data a detailed map of the skills, experience, and educational requirements associated with successfully certified immigration pathways. Clustering this data into coherent archetypes would enable practitioners to benchmark new job descriptions against established precedent - a meaningful enhancement to both legal strategy and client counsel.

The business value at stake is significant: reliable job description archetypes enable immigration counsel to benchmark new roles against certified precedent with greater consistency and speed, reducing both advisory risk and the time required to assess a new position’s immigration viability. Achieving that reliability required confronting a methodological reality.

The analytical challenge is that legal text occupies a high-dimensional semantic space that is structurally incompatible with the assumptions underlying most standard clustering algorithms. K-means, hierarchical clustering, DBSCAN, the default instruments of data science practice, all were tested and found to be better suited to lower-dimensional data. The clusters they produce on high-dimensional legal text do not achieve the coherence required for practical legal application. This is a question of fit between tool geometry and problem geometry.

Accuracy Standard: Stable Role Archetypes - Modularity score achieved through graph partitioning on PERM application data - a result no conventional clustering algorithm could approach, because the geometry of high-dimensional legal text requires a fundamentally different analytical approach.

Graph partitioning approaches the problem differently: treating job descriptions as nodes in a semantic network, with edges weighted by similarity, and identifying natural community structure through network analysis. The result indicates genuine, stable cluster structure. Similar roles grouped together consistently, while materially different role types remained separated - reflecting real distinctions in how the labour market categorizes professional roles, not artifacts of algorithmic assumptions.

What this experience surfaces is that general-purpose data science tools carry assumptions that do not transfer cleanly to specialist legal and regulatory domains. The structural specificities of immigration law - the vocabulary, the dimensional complexity, the nature of the distinctions that matter - require methodological choices that a generalist data science practice would not make by default. Identifying this early - and choosing the method that fits the geometry of the problem - is what makes the difference between a clustering result that is academically interesting and one that is practically useful for legal counsel.            

Leadership Implication: The most advanced method is not always the right method. In specialist domains, the distinctions that matter professionally must be respected analytically. Some more semantically aggressive methods risk grouping conceptually related skills together, even where the domain treats those distinctions as materially different - placing Python and Java in the same cluster, for example, because they are conceptually related, while immigration practice treats them as distinct. Knowing which distinctions matter in your domain, and choosing the method that honours them, is a leadership question as much as a technical one. Ask your AI teams not just what method they used, but why that method fits the specific distinctions your domain requires.

The implication extends beyond clustering: any AI application in a specialist professional domain should treat methodological fit as an explicit design question, not an assumption.

DISCOVERY 04: YOU CANNOT SEE THE CEILING UNTIL YOU HIT IT

Certificate Review: The Capacity Constraint Hidden in Plain Sight

The review of MSW, CoC, and A1 certificates is meticulous, consequential work. Cross-referencing submitted documents against received certificates, verifying field consistency, and flagging discrepancies for professional attention requires - when performed with appropriate rigour - significant professional time per application.

That processing time is not simply an efficiency metric. It is a capacity ceiling. Regardless of team size, the maximum throughput of a manual certificate review process is fixed by the time each review requires. In a high-volume global mobility environment, this ceiling constrains everything downstream: filing volumes, client commitments, the ability to respond to demand growth. Adding resources moves the ceiling upward incrementally. It does not remove it.

Significant Time: Capacity Redefined - Per-application time previously absorbed by manual extraction and cross-referencing - restructured through document intelligence and generative AI prompting so that professional attention is reserved for exception-based review, not mechanical verification.

The AI-enabled approach - document intelligence for page-by-page content extraction, combined with generative AI prompt-based field extraction - does not remove professional judgement from the process. It restructures where that judgement is applied. The model performs the extraction and comparison. The professional reviews the exceptions the model surfaces. The four-eye principle is preserved; the two sets of eyes are redirected to the work that genuinely requires them.

What this restructuring materially raises is the effective capacity ceiling. The constraint on filing volumes and client commitments is no longer a function of how long each review takes, but of how many exceptions require professional attention - a materially different and more manageable constraint.

What this experience reveals is that process time, long accepted as an inherent property of the work, is actually a design variable - one whose implications for commercial capacity became visible only when an alternative architecture made it possible to separate the mechanical from the expert elements of the task.            

Leadership Implication: The redesign revealed that the practice now has the commercial freedom to grow filing volumes and deepen client commitments in ways that were previously unavailable - not because demand was absent, but because capacity was structurally constrained by process architecture. What AI unlocked was not just efficiency. It was a new range of what the business can credibly promise and deliver. That is a growth conversation, not a cost conversation.

The Strategic Implication

The firms that will compound their advantage in this space are not necessarily those deploying the most AI. They are the ones with the intellectual discipline to read the full answer - to treat the diagnostic output of every model as a strategic input, and to build the organizational habit of asking not just “What did the model produce?” - but - “What did the model show us?”

The following leadership imperatives emerge from this body of work:

• Do not approve AI business cases based only on time saved or headline accuracy metrics. Require every deployment to define what operational success looks like before the model runs - and challenge the metric if the operation still struggles when the number looks strong.

• Require AI teams to report what the model revealed, not just what it produced. Every deployment generates diagnostic intelligence alongside its automation output. Build the organizational habit of asking both questions: what did this automate, and what did it show us about how the business actually works?

• Ask whether current performance metrics reflect operational impact or merely activity. Aggregate accuracy, win rate, and throughput are useful measures. They are not sufficient measures. For each AI deployment, identify the metric that would change a leadership decision - and ensure that is the metric being optimized.

• Build cross-functional teams where domain experts and data scientists jointly define the problem. The most consequential methodological decisions in professional services AI are invisible to either discipline working alone. The insight emerges at the intersection of domain knowledge and analytical method.      

• Treat AI outputs as management intelligence, not only workflow automation. The forecasting model reveals how accuracy should be measured. The RFP skill reveals where content investment should be directed. The PERM model reveals how to benchmark legal positions against certified precedent. Each of these began as a process improvement. Each became a strategic input to leadership decisions.

Every AI deployment is already generating diagnostic value alongside its automation value. The question is whether that value is being extracted - or whether the organization brings the curiosity to ask what the model reveals, not just whether it worked.

Conclusion: The Second Kind of Value

The next advantage in AI will not belong simply to the firms that automate the most work. It will belong to the firms that learn the most from every deployment - that treat what the model reveals as a strategic input, not a byproduct.

These are insights that change how a practice is designed, measured, and managed. They are available to any firm willing to look carefully at what its models show, not just what they produce.

The industry has been asking AI the right questions. The invitation now is to read the full answer.

Author Disclaimer: The views and opinions expressed herein are those of the Author alone and are shared in a personal capacity, in accordance with the Chatham House Rule. They do not reflect the official views or positions of the Author’s employer, organization, or any affiliated entity.