Evaluating vendors and build-versus-buy tradeoffs

# Evaluating vendors and build-versus-buy tradeoffs

Three vendors walk into a procurement meeting. All three claim their AI can detect subrogation opportunities, cases where your insurer paid a claim but another party (a negligent driver, a contractor, a product manufacturer) should ultimately foot the bill. All three show a demo with impressive recall numbers. Only one has ever connected to a claims system as messy as yours. This lesson gives you the checklist to tell them apart.

Why subrogation detection is a good test case

Subrogation is the process where an insurer that has paid a claim seeks reimbursement from the party actually responsible. Missed subrogation is pure margin loss: the claim is paid, the file is closed, and nobody ever goes back to collect. Industry estimates (as of 2024, sources vary) suggest US property and casualty insurers recover well under half of theoretically recoverable subrogation dollars, often cited in the 30 to 50% range depending on line of business.

AI models can flag subrogation candidates by reading adjuster notes, police reports, and claim metadata (claim type, injury codes, liability indicators) to surface patterns a busy adjuster might miss. This makes it a realistic, bounded use case for comparing vendors, not a moonshot.

The three-vendor scoring exercise

Imagine you are scoring Vendor A (a specialized insurtech startup), Vendor B (a module bolted onto a large claims-platform incumbent), and Vendor C (a general-purpose AI/analytics consultancy offering a custom build). Score each 1 to 5 on the criteria below.

Criterion 1: Data access and integration effort

Ask concretely: does the vendor read structured claims data only, or also unstructured text (adjuster notes, PDFs, call transcripts)? Subrogation signals often hide in free text, so a model limited to structured fields will underperform.

• Vendor A (startup): built APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → connectors for major claims systems (e.g. Guidewire, Duck Creek) but has never touched your legacy mainframe extract. Integration effort: medium-high, timeline risk.
• Vendor B (incumbent module): already sits inside your existing platform, near-zero integration effort, but the underlying model is a generic add-on not tuned to subrogation specifically.
• Vendor C (consultancy): will build a custom pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → against your exact data, but that means months of data engineering before any model output.

Criterion 2: Explainability requirements

Explainability means the ability to show, in human terms, why a model produced a given output. In insurance this is not optional. Many US states require that adverse decisions affecting policyholders be explainable, and regulators like the NAIC (National Association of Insurance Commissioners) have issued model governance guidance on AI use in insurance. Subrogation flags feed decisions that touch claims handling, so if the flag influences reserve-setting or litigation strategy, your legal and compliance teams will ask "why did the model say this."

• Vendor A: provides feature-level explanations (which words or fields drove the flag), decent audit trail.
• Vendor B: black-box score with a generic confidence percentage, weak explainability.
• Vendor C: fully custom, so explainability is whatever you specify in the contract, but that specification work is on you.

Criterion 3: Track record and validation

Ask for a live reference client in your line of business (auto, workers' comp, property), not just a logo on a slide. Ask how the vendor measured accuracy: against a held-out historical dataset, or only against their own training data (a red flag for overstated performance).

A simple scoring table

| Criterion (weight) | Vendor A | Vendor B | Vendor C |

|---|---|---|---|

| Data access / integration (30%) | 3 | 5 | 2 |

| Explainability (30%) | 4 | 2 | 3 (if specified) |

| Track record (20%) | 3 | 4 | 2 |

| Cost / time to value (20%) | 3 | 4 | 1 |

Worked calculation (weighted score, out of 5):

• Vendor A: (3×0.3) + (4×0.3) + (3×0.2) + (3×0.2) = 0.9 + 1.2 + 0.6 + 0.6 = 3.3
• Vendor B: (5×0.3) + (2×0.3) + (4×0.2) + (4×0.2) = 1.5 + 0.6 + 0.8 + 0.8 = 3.7
• Vendor C: (2×0.3) + (3×0.3) + (2×0.2) + (1×0.2) = 0.6 + 0.9 + 0.4 + 0.2 = 2.1

Vendor B wins on this weighting, but only because integration and cost were weighted at 50% combined. If your legal team insists explainability must be weighted at 40% minimum (common in claims-adjacent use cases), Vendor A overtakes B. The point of the exercise is not the arithmetic, it is forcing your team to state weights explicitly *before* seeing vendor demos, so the sales pitch doesn't set your priorities for you.

Build versus buy: the real tradeoffs

"Build" rarely means building a large language modellarge language modelA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → from scratch, almost no insurer does that. It means building a custom pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → (data extraction, feature engineering, a fine-tuned or prompted model, a review workflow) using existing AI infrastructure (cloud provider APIs, open-source models).

Buy makes sense when:

• The use case is common across the industry (subrogation detection, first-notice-of-loss triage) so vendors have already refined it on many clients' data.
• Your team lacks in-house ML engineering capacity to maintain a model over time.
• Time to value matters more than perfect fit; a mediocre tool live in Q2 beats a perfect tool live in Q4 next year.

Build makes sense when:

• Your data or process is genuinely unusual (a niche specialty line, a proprietary claims taxonomy).
• The use case is core to competitive advantagecompetitive advantageA lasting edge over competitors: a resource, capability or position they cannot easily replicate, letting a firm earn above-average returns over time.View full definition →, not a commodity function.
• You already have an MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.View full definition → (machine learning operationsmachine learning operations, the practices for deploying and monitoring models in production) capability from other projects.

A frequent middle path: buy a vendor's base model, but negotiate access to fine-tune it on your own historical claims data. This gets you speed plus some customization, at added contractual complexity.

A minimal technical sanity check

Even non-technical evaluators should ask vendors to show a confusion matrix on a held-out test set, not just an overall accuracy number. Example of what to request:

                Predicted: Subrogation   Predicted: No Subrogation
Actual: Subrogation         420 (TP)             180 (FN)
Actual: No Subrogation      90 (FP)             9,310 (TN)

From this: precision = TP/(TP+FP) = 420/510 ≈ 82%. Recall = TP/(TP+FN) = 420/600 = 70%. A vendor who only quotes "95% accuracy" on a dataset where subrogation cases are rare (a classic class-imbalance problem) is likely hiding a weak recall number behind a misleading headline figure. Always ask for precision and recall separately.

Governance and procurement checklist

Before signing, confirm the vendor can answer these, in writing:

1. Where is training and inference data stored, and does it cross borders? Relevant for GDPR (General Data Protection Regulation, EU) if you operate in Europe, and for US state data privacy laws.

2. Who owns model outputs and any model improvements derived from your data?

3. What is the process to contest or override a model's subrogation flag?

4. Can the vendor demonstrate bias testing across claimant demographics? Regulators including several US state insurance departments have begun requiring bias audits for claims-related AI.

For a broader grounding in AI risk management frameworks applicable to vendor evaluation, the NIST AI Risk Management Framework is a solid, free reference used well beyond insurance.

🎬 [VIDEO: "How Insurers Are Using AI for Claims and Fraud Detection" - youtube.com - search for recent industry panel discussions covering claims AI deployment, useful for seeing vendor demos discussed critically rather than in a sales context]

Key Takeaways

• Score vendors against explicit, weighted criteria (data access, integration effort, explainability, track record) decided *before* demos, not after.
• Explainability is not a nice-to-have in claims-related AI; regulators and internal legal teams will demand it.
• Always request precision and recall separately, not a single accuracy figure, especially for rare-event detection like subrogation.
• Buy for commodity, well-trodden use cases; build (or customize) when your data or process is genuinely differentiated.
• A hybrid path, buying a base model and on your , is often the realistic middle ground in 2026.