Value metrics and adoption

# Value metrics and adoption

In 2019, a global consumer-goods company we'll call NorthStar stood up a demand-forecasting platform that cost $14 million to build. The data science was excellent. The model beat the incumbent statistical baseline by 22% on backtested accuracy. Eighteen months later, the platform was quietly decommissioned. Not because it was wrong, because the regional demand planners never used it. They kept their spreadsheets. The forecast was more accurate and less trusted, so it sat on the shelf while the business ran on gut feel and Excel.

This is the central tension of your job at scale: a data product's technical quality and its realized value are two different variables, and they correlate far less than most CDOs assume. You have already learned to build governed, well-architected data products. This lesson is about the harder question, proving they create value, and reading adoption as the early-warning signal that tells you whether that value will ever materialize.

Why adoption is the leading indicator

Most value frameworks fail because they measure the wrong thing at the wrong time. Financial impact, revenue lift, cost avoided, margin protected, is a lagging indicator. It arrives 6 to 18 months after you ship, filtered through a dozen confounding variables, and by the time it's measurable the money is spent and the political capital is gone.

Adoption is the leading indicator. It tells you in weeks, not quarters, whether a data productdata productA data asset managed like a product, with an owner, defined users, guaranteed quality, and measurable business value.View full definition → is on the path to value or on the path to the NorthStar graveyard. The logic is a simple causal chain:

> Usage → Behavior change → Decision change → Financial outcome

Value cannot appear at the end of that chain if the first link is broken. A dashboard nobody opens changes no decisions. A model nobody trusts protects no margin. So while your CFO will eventually ask about dollars, your operating dashboard on Monday morning should be watching adoption, because that's the variable you can still influence.

The mistake is treating adoption as a vanity metric ("we have 4,000 monthly active users!"). Raw logins are noise. The signal is depth and durability of use by the people whose decisions the product was built to change.

The adoption pyramid

Segment usage into a pyramid, because each layer means something different:

• ReachReachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition →, Who has access and has logged in at least once. This is your denominator, not your success.
• Active use, Who returns on the product's natural decision cadence (daily for an ops dashboard, monthly for a planning model). This is the first real signal.
• Embedded use, Who has changed a workflow around the product. The demand planner who deleted their spreadsheet. The pricing analyst who now starts their day in your tool. This is where value begins.
• Advocacy, Who pulls colleagues in without your promptingpromptingPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.View full definition →. This is self-sustaining and the only kind of adoption that survives your attention moving elsewhere.

At NorthStar, the platform had strong ReachReachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition → (mandated access), weak Active use, and zero Embedded use. Anyone reading the pyramid would have killed or fixed it at month three instead of month eighteen.

Measuring value: a framework that survives CFO scrutiny

You need a value model that a finance partner will actually sign. That means separating three tiers of rigor and being honest about which one you're claiming.

Tier 1: attributed value (defensible)

Value you can tie to the product through a control or a counterfactual. This is the gold standard and you should build the mechanism to capture it *before* you ship, not after.

The cleanest tool is a staggered rollout as a natural experiment. Don't deploy to all 40 regions at once. Deploy to 10, hold 10 as a comparison, and measure the delta on the target metric. If the demand platform reduces stockouts, the treated regions should diverge from the control regions on stockout rate. That divergence is attributable value, and no one can wave it away as "the market moved."

Tier 2: modeled value (credible)

When a true control isn't possible, model the value chain with explicit, agreed assumptions. The discipline is to pre-register the assumptions with the business owner and finance. A pricing-optimization tool's value model might read:

Value = (addressable_transactions)
      × (share_using_recommendation)      ← this is an ADOPTION variable
      × (avg_margin_uplift_per_txn)
      × (persistence_factor)              ← decays if adoption isn't durable

Notice that two of the four terms are adoption terms. Your financial value is mechanically capped by your adoption rate. A 3% margin uplift applied to 15% of transactions is a fifth of the same uplift at 75% adoption. This is why the two halves of this lesson are the same conversation.

Tier 3: Anecdotal Value (directional only)

Testimonials, "this saved my team two days a week," qualitative wins. Useful for narrative and momentum, dangerous as your primary evidence. Use it to bring a Tier 1 or Tier 2 number to life in a board deck, never as the number itself.

The governance move here: maintain a value register. Every data productdata productA data asset managed like a product, with an owner, defined users, guaranteed quality, and measurable business value.View full definition → carries a stated value hypothesis, its rigor tier, its owner, and its measurement mechanism. When someone proposes a new product, the first question isn't "can we build it" but "which tier of value can we credibly claim, and how will we measure it?" This kills vanity projects at intake.

Instrumenting adoption: the practical how-to

You cannot manage what you don't instrument. Most data teams instrument their pipelines obsessively and their *usage* not at all. Fix that.

Capture depth, not just presence

Log events that reveal the behavioral chain, not just page loads. For a decision-support product, the events worth capturing:

{
  "event": "recommendation_actioned",
  "user_role": "regional_demand_planner",
  "product_id": "demand_fcst_v3",
  "recommendation_id": "r-88213",
  "action": "accepted",        // accepted | overridden | ignored
  "override_reason": null,
  "session_decision_latency_s": 42,
  "ts": "2024-06-11T09:14:22Z"
}

The single most valuable field is action. The ratio of accepted to overridden to ignored recommendations tells you whether the product is changing decisions or being politely bypassed. A high override rate isn't necessarily failure, but a high *ignore* rate is the NorthStar signal. And override_reason is a gift: it's your product backlog, written by your users.

Watch the leading-indicator dashboard

Your team's operating view should track, per product and per user segment:

• Week-4 retention, Of users active in week 1, how many are still active in week 4? Below ~40% and you have a shelf-ware trajectory regardless of technical quality.
• Recommendation action rate, accepted + informed-override, as a share of total. This is your closest proxy for decision influence.
• Time-to-first-value, Days from access granted to first meaningful action. Long tails here reveal onboarding friction that quietly caps adoption.
• Depth distribution, Where your users sit on the pyramid. Watch the *movement* between layers, not the static counts.

Instrument the counterfactual

The hardest and most valuable habit: capture what users did *instead*. When a planner overrides the forecast, log the override. When they don't open the tool during a decision window, that absence is data. If you can join usage logs to outcomes (the stockout actually happened, the override was wrong 70% of the time), you close the loop from adoption all the way to value, and you can show a skeptical planner their own track record against the model. Nothing drives adoption like showing someone their overrides cost them money.

Diagnosing and fixing an adoption gap

When a product has good technical quality but poor adoption, resist the reflex to add features. The gap is almost always one of four failures, and each has a different fix. Diagnose before you spend.

1. Trust failure

Users don't believe the output. This was NorthStar's core problem, the planners had no window into *why* the model said what it said, so they defaulted to the thing they understood. Fix: explainability and track-record transparency, not accuracy. Show the model's historical hit rate versus the human's. Expose the top drivers of each recommendation. Trust is earned through visible reliability over time, and it collapses instantly after one unexplained bad call, so build the explanation layer *into* the product, not as a follow-on.

2. Workflow failure

The product is good but lives outside the user's flow of work. If a planner has to leave SAP, log into a separate portal, and re-key context, the friction tax exceeds the perceived benefit. Fix: embed the data product where the decision is already being made, inside the ERP screen, the CRMCRMCustomer Relationship Management: software and strategy to manage and analyse customer interactions throughout their lifecycle.View full definition →, the ticketing tool. The highest-adoption data products are often invisible; they surface a recommendation inside a workflow the user never leaves. ReachReachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition → declines matter less than embeddingembeddingAn embedding is a numerical vector that represents data (text, images, or items) in a way that captures meaning, so similar items sit close together in space.View full definition →.

3. Incentive failure

Using the product is not in the user's interest. If a sales rep's comp rewards volume and your tool recommends walking away from low-margin deals, they will never adopt it, and no amount of UX will fix that. Fix: this is not a data problem, escalate it. Adoption at scale requires the CDO to negotiate incentive alignment with the business owner. You change what the product optimizes for, or you change what the human is measured on. This is where your leadership mandate meets your value mandate.

4. Capability failure

Users don't know how to interpret or act on the output. Fix: targeted enablement, not mass training. Identify the advocates at the top of your pyramid and invest in them as champions who spread capability peer-to-peer. Central training decks are the weakest lever; a respected planner showing three colleagues over coffee is the strongest.

The intervention sequence

When adoption stalls, run the diagnosis in this order because the fixes get progressively more expensive and political: workflow (cheapest) → capability → trust → incentives (hardest). A CDO who reaches for "let's retrain the model" when the real problem is that the tool lives in the wrong window has misdiagnosed the patient and will burn a quarter fixing the wrong thing.

The scaling insight: adoption doesn't scale linearly with quality. It scales with the *removal of friction and the alignment of incentives*. Once you've proven Tier 1 value in your control group and diagnosed the adoption levers, scaling is a distribution and change-management exercise, which is precisely why the best CDOs treat their most-adopted product as a template, not just a win. You've learned what makes a decision-maker actually change behavior; that learning is more portable than any single model.

Key Takeaways

1. Treat adoption as your leading indicator and financial value as the lagging one. Watch week-4 retention and recommendation action rate weekly; you can still influence value when it's an adoption number, not when it's a P&L number six months later.

2. Instrument the decision, not the login. Capture accepted-vs-overridden-vs-ignored actions and override reasons. The ignore rate is your shelf-ware alarm; the override reasons are your product backlog.

3. Pre-register value before you ship, and default to staggered rollout. A hold-out group turns "we think it helped" into attributable value your CFO will sign, and it costs you nothing but the discipline to not deploy everywhere at once.

4. Diagnose adoption gaps in order, workflow, capability, trust, incentives, before adding features. Most stalled data products fail on friction or incentives, not accuracy. Adding features to a trust or incentive problem wastes a quarter.

5. Maintain a value register with a rigor tier for every product. Force the "which tier of value can we credibly claim?" question at intake, and you kill vanity projects before they become expensive monuments to good data science nobody used.