Evaluating vendors and build-versus-buy tradeoffs

Imagine you are scoring Vendor A (a specialized insurtech startup), Vendor B (a module bolted onto a large claims-platform incumbent), and Vendor C (a general-purpose AI/analytics consultancy offering a custom build). Score each 1 to 5 on the criteria below.

Criterion 1: Data access and integration effort

Ask concretely: does the vendor read structured claims data only, or also unstructured text (adjuster notes, PDFs, call transcripts)? Subrogation signals often hide in free text, so a model limited to structured fields will underperform.

• Vendor A (startup): built APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.Voir la définition complète → connectors for major claims systems (e.g. Guidewire, Duck Creek) but has never touched your legacy mainframe extract. Integration effort: medium-high, timeline risk.
• Vendor B (incumbent module): already sits inside your existing platform, near-zero integration effort, but the underlying model is a generic add-on not tuned to subrogation specifically.
• Vendor C (consultancy): will build a custom pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → against your exact data, but that means months of data engineering before any model output.

Criterion 2: Explainability requirements

Explainability means the ability to show, in human terms, why a model produced a given output. In insurance this is not optional. Many US states require that adverse decisions affecting policyholders be explainable, and regulators like the NAIC (National Association of Insurance Commissioners) have issued model governance guidance on AI use in insurance. Subrogation flags feed decisions that touch claims handling, so if the flag influences reserve-setting or litigation strategy, your legal and compliance teams will ask "why did the model say this."

• Vendor A: provides feature-level explanations (which words or fields drove the flag), decent audit trail.
• Vendor B: black-box score with a generic confidence percentage, weak explainability.
• Vendor C: fully custom, so explainability is whatever you specify in the contract, but that specification work is on you.

Criterion 3: Track record and validation

Ask for a live reference client in your line of business (auto, workers' comp, property), not just a logo on a slide. Ask how the vendor measured accuracy: against a held-out historical dataset, or only against their own training data (a red flag for overstated performance).

A simple scoring table

| Criterion (weight) | Vendor A | Vendor B | Vendor C |

|---|---|---|---|

| Data access / integration (30%) | 3 | 5 | 2 |

| Explainability (30%) | 4 | 2 | 3 (if specified) |

| Track record (20%) | 3 | 4 | 2 |

| Cost / time to value (20%) | 3 | 4 | 1 |

Worked calculation (weighted score, out of 5):

• Vendor A: (3×0.3) + (4×0.3) + (3×0.2) + (3×0.2) = 0.9 + 1.2 + 0.6 + 0.6 = 3.3
• Vendor B: (5×0.3) + (2×0.3) + (4×0.2) + (4×0.2) = 1.5 + 0.6 + 0.8 + 0.8 = 3.7
• Vendor C: (2×0.3) + (3×0.3) + (2×0.2) + (1×0.2) = 0.6 + 0.9 + 0.4 + 0.2 = 2.1

Vendor B wins on this weighting, but only because integration and cost were weighted at 50% combined. If your legal team insists explainability must be weighted at 40% minimum (common in claims-adjacent use cases), Vendor A overtakes B. The point of the exercise is not the arithmetic, it is forcing your team to state weights explicitly *before* seeing vendor demos, so the sales pitch doesn't set your priorities for you.

Build versus buy: the real tradeoffs

"Build" rarely means building a large language modellarge language modelA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.Voir la définition complète → from scratch, almost no insurer does that. It means building a custom pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → (data extraction, feature engineering, a fine-tuned or prompted model, a review workflow) using existing AI infrastructure (cloud provider APIs, open-source models).

Buy makes sense when:

• The use case is common across the industry (subrogation detection, first-notice-of-loss triage) so vendors have already refined it on many clients' data.
• Your team lacks in-house ML engineering capacity to maintain a model over time.
• Time to value matters more than perfect fit; a mediocre tool live in Q2 beats a perfect tool live in Q4 next year.

Build makes sense when:

• Your data or process is genuinely unusual (a niche specialty line, a proprietary claims taxonomy).
• The use case is core to competitive advantagecompetitive advantageA lasting edge over competitors: a resource, capability or position they cannot easily replicate, letting a firm earn above-average returns over time.Voir la définition complète →, not a commodity function.
• You already have an MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.Voir la définition complète → (machine learning operationsmachine learning operationsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.Voir la définition complète →, the practices for deploying and monitoring models in production) capability from other projects.

A frequent middle path: buy a vendor's base model, but negotiate access to fine-tune it on your own historical claims data. This gets you speed plus some customization, at added contractual complexity.

A minimal technical sanity check

Even non-technical evaluators should ask vendors to show a confusion matrix on a held-out test set, not just an overall accuracy number. Example of what to request:

From this: precision = TP/(TP+FP) = 420/510 ≈ 82%. Recall = TP/(TP+FN) = 420/600 = 70%. A vendor who only quotes "95% accuracy" on a dataset where subrogation cases are rare (a classic class-imbalance problem) is likely hiding a weak recall number behind a misleading headline figure. Always ask for precision and recall separately.

Governance and procurement checklist

Before signing, confirm the vendor can answer these, in writing:

1. Where is training and inference data stored, and does it cross borders? Relevant for GDPR (General Data Protection Regulation, EU) if you operate in Europe, and for US state data privacy laws.

2. Who owns model outputs and any model improvements derived from your data?

3. What is the process to contest or override a model's subrogation flag?

4. Can the vendor demonstrate bias testing across claimant demographics? Regulators including several US state insurance departments have begun requiring bias audits for claims-related AI.

For a broader grounding in AI risk management frameworks applicable to vendor evaluation, the NIST AI Risk Management Framework is a solid, free reference used well beyond insurance.

🎬 [VIDEO: "How Insurers Are Using AI for Claims and Fraud Detection" - youtube.com - search for recent industry panel discussions covering claims AI deployment, useful for seeing vendor demos discussed critically rather than in a sales context]

Key Takeaways

• Score vendors against explicit, weighted criteria (data access, integration effort, explainability, track record) decided *before* demos, not after.
• Explainability is not a nice-to-have in claims-related AI; regulators and internal legal teams will demand it.
• Always request precision and recall separately, not a single accuracy figure, especially for rare-event detection like subrogation.
• Buy for commodity, well-trodden use cases; build (or customize) when your data or process is genuinely differentiated.
• A hybrid path, buying a base model and fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → on your own dataown dataData collected directly from your own customers and prospects through your own channels: your most reliable and privacy-compliant source.Voir la définition complète →, is often the realistic middle ground in 2026.