# Building an AI governance structure that scales with your roadmap
A mid-sized SaaS company ships three AI features in one sprint: a smarter search bar, an auto-summarize button, and a "predict churn" dashboard widget. Nobody outside the engineering team knows all three exist. Legal finds out when a customer asks why their data trained a model. This is not a hypothetical. It is the default state of most product-led SaaS companies in 2026, and it is exactly the failure mode this lesson fixes.
Traditional governance (annual model reviews, quarterly risk committees) was built for banks deploying a handful of models a year. SaaS companies ship AI features weekly. If your governance cadence is slower than your release cadence, governance will always be reacting to shipped features instead of shaping them.
Two forces pull against each other:
The fix is not "review everything equally." It's tiering: light-touch checks for low-risk features, heavier review for high-risk ones. This is the same logic regulators use.
A model inventory is a live registry of every AI/ML model in production, including third-party models called via APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → (OpenAI, Anthropic, Cohere, etc.) and embedded models inside vendor tools (e.g., a support ticket triage model inside Zendesk).
Minimum fields per entry:
| Field | Example |
|---|---|
| Model name/version | GPT-4o-mini via APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition →, v2026-01 |
| Owner (team + person) | Growth team, J. Alvarez |
| Purpose | Auto-summarize support tickets |
| Data inputs | Ticket text, customer name |
| Risk tier | Low / Medium / High |
| Last review date | 2026-02-10 |
| Human oversight point | Agent approves summary before send |
Why this matters for regulation: the EU AI Act (in force since August 2024, with obligations phasing in through 2026-2027) requires providers and "deployers" of certain AI systems to maintain documentation and traceability. You cannot comply with a law about "high-risk AI systems" if you don't know which systems you have. A model inventory is the prerequisite, not an optional nicety. The NIST AI Risk Management Framework (US, voluntary but widely adopted) makes the same point: "MapMapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition →" is step one, before "Measure" and "Manage."
Practical tip: put the inventory where engineers already work (a repo, a Notion database linked to your CI/CD pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition →), not in a compliance tool nobody opens.
Not every AI feature deserves a board review. Tier by two questions:
1. Does it make or influence a decision about a person? (pricing, hiring signals, account suspension, credit-like scoring)
2. How much autonomy does it have? (suggests vs. auto-executes)
A simple tiering model:
This mirrors the EU AI Act's own structure (unacceptable, high-risk, limited-risk, minimal-risk categories), which is worth knowing even if you're US-based, because any SaaS company with EU customers falls under its extraterritorial reachreachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition →.
Skip the 12-person committee. For a company shipping AI weekly, the board should be 3 to 5 people who can convene asynchronously:
Cadence: Tier 2 and 3 features get a standing async review (Slack thread + a one-page risk memo), not a scheduled meeting. Reserve live meetings for genuine Tier 3 disputes. Target: decisions in days, not sprints.
Keep it under 300 words. Sections: purpose, data used, risk tier, known failure modes, human oversight point, rollback plan. If a team can't fill this in an hour, the feature isn't ready to ship.
Ambiguous ownership is the single most common governance failure. Write it down:
A useful pattern: a RACI matrix (Responsible, Accountable, Consulted, Informed) per risk tier, published where every PM can find it before kickoff, not after launch.
Regardless of tier, four checks scale down cheaply:
1. Data provenance check: What data trained or fine-tuned this model? Is customer data being used to train a third-party model's future versions? (Check vendor terms; OpenAI's APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → data, for instance, is not used for training by default as of their current enterprise terms, but this must be verified per contract, not assumed.)
2. Bias/output spot-check: Run the model against a small adversarial test set (edge cases, protected-class proxies) before launch.
3. Security review: Prompt injection testing if the model takes user input; access scoping so the model can't read more data than the feature needs.
4. Human-in-the-loop point: Define explicitly where a human can intervene before harm occurs, and log when they do.
# Minimal pre-deploy check script, run in CI
def preflight(feature):
checks = {
"data_provenance_documented": feature.data_source is not None,
"risk_tier_assigned": feature.risk_tier in ["1", "2", "3"],
"human_oversight_defined": feature.oversight_point is not None,
"rollback_plan_exists": feature.rollback is not None,
}
failed = [k for k, v in checks.items() if not v]
if failed:
raise Exception(f"Cannot ship: missing {failed}")
return "Preflight passed"This isn't compliance theater. It's a gate that catches the "nobody wrote down what happens if it's wrong" problem before it reaches production.
Knowledge check
1. Why does traditional governance modeled on annual reviews and quarterly risk committees fail for product-led SaaS companies?
2. What is the core logic behind a 'tiered' governance approach as described in the lesson?
3. A support ticket triage model embedded inside a third-party vendor tool (not built by the company's engineers) is being used in production. According to the model inventory concept, how should this be treated?
4. Select ALL correct answers about the underlying tension that AI governance in SaaS companies must manage.
Select all the correct answers.
5. Select ALL correct answers about the purpose and scope of a model inventory as described in the lesson.
Select all the correct answers.
The trap most companies fall into: building governance that works for month one, then collapsing under month twelve's feature volume. Three habits keep it scalable:
🎬 [VIDEO: "The EU AI Act Explained" - youtube.com/@EUAIact - a concise walkthrough of the EU AI Act's risk tiers and what they mean for companies building AI products, useful context for the tiering logic in this lesson]