Building a defensible ROI model for AI pilots

Finance committees, internal audit, and model risk teams (governed in the US under the Federal Reserve's SR 11-7 guidance on model risk management) are trained to distrust soft claims. If your ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → model blends the two without separating them, the whole case becomes suspect, even the parts that are true.

The fix is structural: build two separate ledgers, hard and soft, and never let them touch until the end.

The hard savings ledger: trade surveillance example

Trade surveillance systems monitor trading activity for market abuse: insider dealing, spoofing (placing orders with no intent to execute), layering, and wash trading. In the EU this obligation sits under the Market Abuse Regulation (MAR); in the US it falls under FINRA and SEC rules, notably FINRA Rule 3110.

Hard, defensible savings typically fall into three buckets:

1. Headcount and workload reduction

If your legacy rules-based system generates 10,000 alerts a month and analysts spend an average of 20 minutes triaging each false positive, that's a calculable labor cost. An AI system that cuts false positives by a verified percentage (from a controlled pilot, not a vendor brochure) frees analyst hours that can be reallocated or reduced.

2. Penalties and fines avoided

This is harder to claim credibly (you can't prove a fine that didn't happen), but you can anchor it to industry benchmarks. Deutsche Bank, Goldman Sachs, and other major banks have paid nine-figure surveillance-related settlements over the past decade for gaps in detecting manipulative trading. Citing these as *sector risk exposure*, not a guaranteed avoided cost, is defensible; claiming "this will prevent a $100 million fine" is not.

3. Investigation and remediation cost

Every alert that escalates to a full investigation costs money: compliance officer time, legal review, sometimes external counsel. Reducing false-positive escalation rates reduces this directly.

A worked example (illustrative, not vendor-sourced)

Assume (all figures illustrative estimates for teaching purposes, not sourced from any real vendor or bank):

• 12 surveillance analysts, fully loaded cost $130,000/year each (US estimate) = $1.56 million/year
• Current false-positive rate on alerts: 95% (a commonly cited pain point in surveillance industry commentary, treat as illustrative)
• AI pilot reduces false positives by 30 percentage points in a controlled A/B testA/B testA/B testing is a controlled experiment that compares two versions of something (A and B) by splitting traffic randomly to learn which performs better on a chosen metric.View full definition →
• Analyst time freed: roughly 25% of total triage hours

Rough calculation:

Note what happened: only half the freed capacity became a "hard" saving. The other half is redeployment, which is a soft benefit (better coverage, not lower cost) unless you actually cut a role. This distinction is what survives committee questioning.

The soft claims ledger: what to do with "efficiency"

Soft benefits are real but must be labeled as such, quantified separately, and never added directly to the hard savings total. Typical soft claims in trade surveillance AI:

• Faster analyst onboarding (less institutional knowledge required)
• Better alert prioritization ("analysts see the right cases first")
• Improved audit trail quality for regulators
• Analyst morale and retention (alert fatigue is a genuine driver of compliance staff turnover)

Treat these as a secondary narrative, presented after the hard numbers, framed as "additional expected value not included in the ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → calculation." This protects your core numbers from being dismissed as inflated.

Building the model: a simple structure

A defensible ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → model for an AI pilot has four components, always kept visible and separate:

1. Cost of the pilot: licensing, integration, data engineering, model validation, ongoing monitoring (this last one is often omitted and shouldn't be)

2. Hard savings: labor, penalty exposure reduction (framed conservatively), investigation cost reduction

3. Soft benefits: listed, described, explicitly not counted in the ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → ratio

4. Risk adjustments: model risk, vendor lock-in, regulatory approval delay

A basic ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → ratio:

If your hard savings are $195,000/year and your pilot costs $150,000/year in software, model validation, and integration, ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → is roughly 30%. That's a modest, credible number, far more persuasive than an unsubstantiated "5x productivity gain."

Model risk and validation costs: the line item everyone forgets

AI surveillance tools, especially those using machine learning to score trades, fall under model risk management frameworks. In the US, SR 11-7 requires independent validation of models used in decision-making, including ongoing performance monitoring. In the EU, the EU AI Act (entered into force 2024, phased obligations through 2026-2027) classifies certain financial-sector AI uses, and supervisory expectations from the European Central Bank (ECB) and European Banking Authority (EBA) reinforce similar validation duties.

This means: budget for a validation team, documentation, and periodic re-testing. These are recurring costs, not one-time pilot costs, and they must appear in your model or your ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → is overstated.

Presenting to a finance committee: format matters

Committees scrutinizing AI spend want three things on one page:

1. A hard-savings number they can trace to a headcount or cost line

2. A clearly labeled "not included" section for soft benefits

3. A sensitivity range, not a single number ("if false-positive reduction is only 15%, not 30%, ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → drops to X%")

Sensitivity analysis is the single most credibility-building element you can add. It signals you're not cherry-picking the best-case scenario.

🎬 [VIDEO: "How Banks Use AI for Fraud and Trade Surveillance" - youtube.com - search for recent explainer content from Bank for International Settlements (BIS) or major consultancy channels covering surveillance AI use cases, useful for a visual walkthrough of alert triage workflows]

Common pitfalls to flag explicitly

• Vendor-supplied benchmarks as your only evidence. Vendors report best-case pilot results. Insist on a controlled internal test before finalizing numbers.
• Ignoring integration cost. Connecting an AI surveillance tool to existing order management systems and market data feeds is often the largest cost, not the software license.
• Counting redeployed time as pure savings. As shown above, this needs to be split.
• No decommissioning cost for the legacy system. Old rules-based systems often run in parallel during validation, doubling cost temporarily; build this into year-one numbers.

For a broader grounding in how AI is applied and governed across banking more generally, the Bank for International Settlements' work on AI in finance is a solid, free, non-vendor resource.

Key Takeaways

• Separate hard savings (headcount, penalty exposure, investigation cost) from soft claims (efficiency, morale, prioritization) into two distinct ledgers, never blended.
• Build a simple worked ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → calculation with explicit assumptions, and always include a sensitivity range, not a single confident number.
• Budget recurring model validation and monitoring costs (per SR 11-7 in the US, EU AI Act and ECB/EBA expectations in Europe) as ongoing costs, not one-time pilot expenses.
• Redeployed analyst time is a soft benefit unless it results in an actual headcount reduction; don't count it twice.
• Committees trust models that show what's excluded as much as what's included.