Separating genuine AI use cases from vendor theater

# Separating genuine AI use cases from vendor theater

A fund COO sits through a vendor demo for an "AI-powered distribution engine." The slides show a neural network diagram, the word "proprietary" appears eleven times, and the annual license is $400,000. When she asks what the model predicts, the answer is: "which advisors are most likely to allocate to your funds next quarter." When she asks how they measure that, the room goes quiet.

That silence is the whole lesson. Most "AI" pitched to asset managers is either (a) a linear ranking that a spreadsheet could do, or (b) a genuine machine learning problem dressed up so heavily you cannot tell if it works. Your job is to tell them apart before you sign.

First question: does this problem even need machine learning?

Machine learning (ML) earns its keep when three conditions hold at once:

1. The pattern is real but too complex for hand-written rules. Think unstructured text, images, or thousands of interacting variables.

2. You have enough labeled examples to learn from. Labels are the "right answers" the model trains on.

3. The prediction changes a decision. If the output does not alter what anyone does, it is a dashboard, not a use case.

Run any vendor claim through this. A tool that scores 12,000 financial advisors for allocation propensity based on past purchases, fund flows, and web engagement? Plausibly ML-worthy: many variables, real historical labels (they either bought or did not), and it reroutes your sales team's calendar.

A tool that flags "advisors who redeemed more than 20% last quarter"? That is a rule. Calling it AI is theater.

The checklist

Print this and hold it up in the next demo.

• What is the exact prediction or output? (Not "insights." A number, a class, a ranking.)
• What decision does it change, and who acts on it?
• What is the label? How did the model learn what "good" looks like?
• What is the baseline? What would a simple rule or a human analyst achieve? A model that beats nothing is worth nothing.
• How is accuracy measured, and on data the model never saw? (Out-of-sample testing.)
• What happens when it is wrong? Cost of a false positive vs a false negative.
• Can the vendor show performance on YOUR data, not a case study?

If a vendor cannot answer the first four in plain language, stop.

Two live examples from the asset management value chain

Example 1: the NLP earnings-call tool

Natural language processing (NLP) is the branch of AI that reads text. A common pitch: a tool that transcribes earnings calls, then extracts "sentiment" and "management tone" to feed your equity analysts.

Does this need ML? Reading thousands of transcripts for subtle tone shifts is genuinely hard for rules. So yes, the *category* is legitimate. Large language models (LLMs, the technology behind tools like ChatGPT) are now good at summarizing and extracting from transcripts.

But stress-test it:

• Label problem. "Positive sentiment" is not a fact, it is a judgment. What did the vendor train on? If they scraped generic financial news sentiment, it may not mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.Voir la définition complète → to whether a stock outperforms.
• The alpha question. If the signal were reliable and cheap, it would already be arbitraged away. Ask: is this a research-productivity tool (saving analyst hours) or an alpha-generation tool (predicting returns)? These are wildly different value claims. The first is defensible. The second demands out-of-sample proof across market regimes.

A useful, honest version of this tool says: "We cut transcript review from 40 minutes to 5, and we surface every mention of guidance changes." That is real ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.Voir la définition complète → you can measure in analyst hours. The dishonest version says "our sentiment score predicts alpha" with a backtest on 18 months of a bull market.

For grounding on how LLMs actually process text, the Google Machine Learning crash course is free and refreshingly jargon-light.

Example 2: the AI distribution engine

Back to our COO. The distribution engine ranks advisors and financial intermediaries by likelihood to allocate. Let us actually test it with a worked example.

Suppose the vendor claims their model identifies the "top 20%" of advisors who will drive 80% of next-quarter inflows. You need to compare this to a baseline: just ranking advisors by their historical purchase volume.

Here is the kind of check your data team should run.

# Compare AI model vs simple baseline on last quarter's ACTUAL outcomes
# "converted" = advisor allocated to our funds this quarter (the label)

from sklearn.metrics import precision_score

# Top 20% flagged by the vendor model
ai_precision = precision_score(actual_converted, ai_top20_flag)

# Top 20% flagged by simply sorting on past 12-month purchase volume
baseline_precision = precision_score(actual_converted, volume_top20_flag)

print(f"AI model precision:  {ai_precision:.2f}")
print(f"Baseline precision:  {baseline_precision:.2f}")

Precision here means: of the advisors the model flagged, what fraction actually allocated. If the AI scores 0.34 and the dumb volume-sort scores 0.31, you are paying $400,000 for a three-point lift that may be noise. If the AI scores 0.52 vs 0.31, now you have a conversation.

The point is not the code. It is that you must *demand this comparison* before purchase, on your own historical data, with a holdout period the vendor never touched.

ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.Voir la définition complète →: translate the claim into hours or basis points

Vendors love vague value. Force a translation into one of two currencies.

Time saved. The earnings-call tool saves 35 minutes per transcript. If your team covers 500 calls a quarter, that is roughly 290 analyst hours a quarter. Fully loaded analyst cost is an input your firm knows. This ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.Voir la définition complète → is boring, measurable, and usually the honest core of a "productivity AI" pitch.

Flows or cost influenced. The distribution engine claims it reprioritizes sales effort. The ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.Voir la définition complète → test is not "the model is accurate." It is: run a controlled pilot. Give half your sales team the model's rankings, give the other half business as usual, and compare inflows over two quarters. This is an A/B testA/B testA/B testing is a controlled experiment that compares two versions of something (A and B) by splitting traffic randomly to learn which performs better on a chosen metric.Voir la définition complète →, and vendors who resist it are usually hiding something.

Red flags that signal theater

• "Proprietary algorithm" with no description of inputs or outputs.
• Backtests only, never a live or forward test.
• No mention of a baseline.
• Accuracy quoted without a time period or a definition.
• The demo uses the vendor's cherry-picked data, and they resist a pilot on yours.
• "AI" applied to a problem that is deterministic (fund fact-sheet generation is templating, not ML).

Where AI genuinely fits in asset management

To keep perspective, here is the honest mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.Voir la définition complète → as of 2026. These are areas with real, repeated deployment, not hype.

• Research productivity. LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.Voir la définition complète → summarization of filings, transcripts, and broker notes. High adoption, clear time-saved ROI. This is where most genuine value currently sits.