Bias, explainability & model cards

# Bias, explainability, and model cards

In 2019, Apple and Goldman Sachs launched a co-branded credit card. Within weeks, a software developer named David Heinemeier Hansson tweeted that his credit limit was 20 times higher than his wife's, despite her superior credit score and their jointly filed taxes. Steve Wozniak chimed in with the same experience. The New York Department of Financial Services opened an investigation. Goldman's defense was revealing: "There is no gender input into the algorithm." That statement, technically true, was also the entire problem. The bank had no way to demonstrate the *absence* of proxy discrimination, no documentation of how the model behaved across protected groups, and no pre-deployment review that would have caught disparate outcomes. The model wasn't the failure. The governance around it was.

This is the CDO's real exposure in AI. You will rarely be the one who wrote the biased code. You will always be the one asked to explain, under oath or under a headline, why nobody caught it. This lesson is about building the machinery that catches harm before it ships.

Fairness is a choice, not a metric

The first thing to internalize, and to teach your teams, is that there is no single, universal definition of "fair." This is not a philosophical dodge; it is a mathematical fact with operational consequences.

Consider a lending model. You can optimize for demographic parity (approval rates equal across groups), equalized odds (equal true-positive and false-positive rates across groups), or calibration (a predicted 80% repayment probability means the same thing for every group). A landmark result, surfaced during the debate over the COMPAS recidivism algorithm, proved that when base rates differ between groups, you *cannot* satisfy all three simultaneously. Optimizing one degrades another.

This means fairness is a business and ethics decision that a CDO must force into the open, not delegate to a data scientist's default settings. The wrong move is to let the modeling team quietly pick a metric because it's the one their library implements most easily. The right move is a documented, cross-functional decision:

• Demographic parity when you have an affirmative obligation to equalize access (some hiring or outreach contexts).
• Equalized odds when the cost of errors is what matters and must be borne equally (medical triage, fraud flags).
• Calibration when the score itself is consumed downstream as a probability (risk-based pricing).

The CDO's job is to make legal, the business owner, and the model team sit in one room and choose, then record *why*. When the regulator calls, "we chose equalized odds because false positives carry legal jeopardy for the applicant, and here is the signed decision memo" is a defensible position. "The library defaulted to it" is not.

Hunting for proxies

Removing the protected attribute, "there is no gender input", is worse than useless, because it creates false confidence while proxies do the discriminating. Zip code encodes race. Shopping categories encode gender. The alma mater and first name encode both. A model denied the sensitive feature will happily reconstruct it from correlated inputs.

The practical technique is to run proxy detection: train a secondary model to predict the protected attribute from your feature set. If it succeeds with high accuracy, your features leak the attribute, and your "blind" model is not blind. You then either drop the offending features, use adversarial de-biasing, or, critically, *keep* the protected attribute in your possession specifically so you can measure disparate impact. You cannot audit what you refuse to collect. Many organizations get this backwards, deleting demographic data for "privacy" and thereby making bias measurement impossible.

Explainability: match the explanation to the stakeholder

Explainability fails in most organizations because teams treat it as a single deliverable. It is not. There are at least three distinct audiences, each needing a different artifact.

The regulator or auditor needs *global* explainability: how does the model behave across the entire population? Which features drive decisions in aggregate? Is the logic stable and non-arbitrary?

The affected individual needs *local*, actionable explainability: "You were declined because your debt-to-income ratio was 47% and our threshold is 40%; reducing it below 40% would change the outcome." This is a legal requirement under the U.S. Equal Credit Opportunity Act (adverse action notices) and the EU's GDPR "right to explanation." Note the word *actionable*, a SHAP value chart is not a reason a human can act on.

The internal model owner needs *debugging* explainability: where is the model brittle, where does it rely on spurious correlations, where will it fail on distribution shift?

The dominant tools are SHAP (game-theoretic feature attributionattributionA framework for assigning credit to the touchpoints that contributed to a conversion, so you can measure which channels and interactions actually drive results.View full definition →, strong for both global and local, computationally heavy) and LIME (fast local approximations, less consistent). The CDO doesn't need to choose the algorithm. The CDO needs to insist that each audience gets a fit-for-purpose artifact, and that the "explanation" delivered to a declined applicant is a human-readable reason, not a data-science visualization.

A caution worth putting in front of your teams: post-hoc explanations can be *misleading*. SHAP on a black box tells you what the model correlated with, not why, and adversaries can even game explanation methods. For genuinely high-stakes decisions, an inherently interpretable model (a monotonic gradient-boosted model, a scorecard) often beats a black box plus an explanation you can't fully trust. Interpretability by design is a legitimate architectural choice, and sometimes the responsible one.

Model cards: the documentation that makes review possible

A model card is a short, standardized document, proposed by Google researchers led by Margaret Mitchell, that travels with every model into production. Think of it as the nutrition label: it forces the team to state, before deployment, what the model is for, how it performs, and where it should *not* be used.

The reason a CDO should mandate model cards is not documentation for its own sake. It is that the *act of filling out the card* surfaces the failures. A team that must write "intended use" is forced to name the boundaries. A team that must report performance *disaggregated by subgroup* cannot hide a model that works well overall but fails for one population. The card is a forcing function, not a formality.

A minimal, enforceable model card contains:

model_card:
  model_name: credit_line_assignment_v3.2
  owner: risk_ml_team
  date: 2024-11-01
  intended_use: "Assign initial credit limits for approved applicants."
  out_of_scope_use: "NOT for approve/deny decisions. NOT for existing-customer limit changes."
  training_data: "18 months applicant data; excludes pre-2022 (policy change)."
  fairness_metric_chosen: equalized_odds
  performance:
    overall_auc: 0.84
    disaggregated:
      - group: female,   approval_rate: 0.61, fpr: 0.09
      - group: male,     approval_rate: 0.63, fpr: 0.10
  known_limitations: "Underperforms for thin-file applicants (<2yr history)."
  human_review_trigger: "Any limit >5x median for applicant income band."
  approved_by: [cdo_office, legal, model_risk]

Notice what this structure enforces. The out_of_scope_use field would have flagged the Apple Card problem: a model built for one purpose being trusted for another. The disaggregated performance would have exposed a gender gap before launch. The human_review_trigger builds a circuit breaker into the deployment itself.

Make the card a merge requirement. A model without a completed, signed card does not deploy, enforced in the CI/CD pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition →, not in a policy PDF nobody reads. When documentation is a gate, it gets written. When it's a "best practice," it doesn't.

Building the review process that catches harm

Frameworks are inert without a process that applies them. Here is the operating model a CDO installs, and the honest failure modes to design against.

Tier your review by risk, not by team

Do not review every model equally; you will drown, and your best people will start rubber-stamping. Adopt a risk tiering aligned to the EU AI Act's logic, which is becoming the global default:

• Prohibited / unacceptable: don't build (social scoring, manipulative systems).
• High-risk: models affecting credit, employment, healthcare, housing, essential services. These get the full treatment: mandatory model card, fairness audit across chosen metric, human-in-the-loop, independent sign-off.
• Limited-risk: internal efficiency tools, content ranking. Lightweight card, self-certification, spot audits.
• Minimal-risk: spam filters, internal search. Register and move on.

The CDO defines the tiering rubric so a team cannot self-classify a high-risk model as limited to escape scrutiny. Classification is reviewed by your office, not the builder.

Separate the builders from the reviewers

This is the single most important structural principle, borrowed directly from model risk management in banking (SR 11-7). The team that builds the model cannot be the team that approves it. You need an independent validation function, reporting to you, not to the business line whose revenue depends on the model shipping. In a smaller organization this might be one person plus an external auditor on retainer; the principle holds regardless of scale. Independence is what turns review from theater into control.

Make review a gate, then a loop

Pre-deployment review is necessary but insufficient, because models degrade. A model fair at launch drifts as the population shifts. Your process needs two clocks:

1. The gate: no high-risk model deploys without a completed card, a fairness audit against the chosen metric, and independent sign-off. This is binary and enforced in the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition →.

2. The loop: production monitoring for *both* performance drift and fairness drift, with disaggregated metrics recomputed on live data monthly. A human_review_trigger and an automatic rollback threshold are configured before launch, not after an incident.

The failure mode to design against is the launch-and-forget pattern, where all the governance energy goes into the gate and none into the loop. Most public AI harms, including biased outcomes that emerge months post-launch, are drift failures, not launch failures.

The escalation path must have teeth

A review board that can only *recommend* is a review board that gets overruled by a VPVPA clear statement of the benefits your product delivers, the problems it solves and why customers should choose you over alternatives.View full definition → with a quarterly target. The CDO must secure, in writing, from the CEO or board, the authority to block deployment of a high-risk model. Without that, everything above is documentation of your own powerlessness. The Apple Card episode cost Goldman regulatory scrutiny and reputation precisely because the incentive to ship overrode the discipline to review. Your job is to make "we blocked it" a survivable career move for the people who raise the flag.

Key Takeaways

• Force the fairness decision into the open. You cannot satisfy demographic parity, equalized odds, and calibration at once. Convene business, legal, and modeling to choose one, document why, and keep the signed memo, that memo is your regulatory defense.

• Never delete the protected attribute; use it to audit. Removing sensitive features creates false confidence while proxies discriminate. Run proxy-detection models, and retain demographic data specifically to measure disparate impact.

• Deliver explanations fit for the audience. Regulators need global logic, individuals need actionable adverse-action reasons, model owners need debugging views. For the highest-stakes decisions, consider an inherently interpretable model over a black box you can only approximate.

• Make the model card a pipeline gate, not a policy. Enforce a signed card with disaggregated performance, out-of-scope-use limits, and human-review triggers as a merge requirement. A model without a card doesn't deploy.

• Separate builders from reviewers, and gate plus loop. Install independent validation reporting to you, tier review by risk, monitor fairness drift in production, and secure written authority to block deployment. Review that can only recommend is theater.