AML pipelines: three design calls examiners test

September 22, 2026 · 5 min · Free, with full transcript

Why do fintech AML pipelines fail regulatory examination when the models themselves are sound? The position here is that examiners do not score area under the curve, they ask reproducibility questions, and most data architectures cannot rebuild a March alert with March data. Three design decisions decide the outcome: immutable timestamped feature storage, documentation that records who approved a feature and which regulation it maps to, and explanation captured at inference time rather than manufactured later. You walk away knowing why dbt column level lineage maps transformations but not business intent, why SHAP values may be plausible rather than faithful, and how to run a one hour reproduction test on your last high value alert.

Listen to the podcast

5 min

Chapters

Key takeaways

  • Freeze every model input as it existed on the decision date so an alert can be rerun with that day's data.
  • Keep design records for each feature covering who built it, when, why, and which regulation it maps to.
  • Log contributing features live at inference time instead of reconstructing a reason with post hoc explainers.
  • Treat vendor lineage statistics, such as dbt Labs' 80% column level lineage claim, as marketing until checked against your own graph.
  • Take your last high value AML alert and try to reproduce inputs, output and reason in under an hour.
Read the full transcript

Host:Leaders' Insights. Five minutes on three pipeline design decisions that determine whether your AML model survives its first regulatory examination.

Expert:Picture a conference room in a mid-sized lending startup, 8 a.m., three examiners from the state regulator on one side, and a very quiet data science lead on the other. They've asked one question. Show us why this transaction was flagged in March. And nobody in the room can answer. That's the moment everything gets decided. And by then, it's already too late. The model was fine. The pipeline couldn't reconstruct its own past.

Host:You've said most anti-money laundering pipelines fail examination for reasons that have nothing to do with the model. That sounds like an excuse data teams tell themselves.

Expert:It would be, except I've watched it happen at four institutions. AML, anti-money laundering, the machinery that spots suspicious cash flows, lives or dies on whether you can defend the data, not the math. A gradient-boosted model with a great score and no lineage is a liability with good manners.

Host:Let me put three beliefs to you. Widely held, possibly wrong. First one, if our model is accurate and well-tested, we'll pass the exam.

Expert:Wrong. Flatly. Examiners don't score your area under the curve. The statistical measure of how well the model separates good from bad. They ask reproducibility questions. Rerun this March alert with March's data. If your pipeline overwrites features in place, you can't. O'Reilly's engineering surveys keep flagging the same gap. Teams instrument accuracy obsessively and version their data almost never.

Host:So accuracy is table stakes and everyone's optimizing the wrong table.

Expert:They're polishing the trophy nobody asked to see. The first design decision is immutable. Time-stamped feature storage. Every input frozen as it existed the day the decision was made. Do that and half the exam evaporates.

Host:Second belief, modern tooling handles lineage for us. DBT, orchestration layers, it's solved.

Expert:Half true and the half that's false will cost you. Lineage, the documented trail of where each number came from and what transformed it, is genuinely better in 2026 than it was three years ago. DBT Labs claims something like 80% of their enterprise users now run column level lineage automatically though they sell that tooling. So treat the figure as a brochure until you've checked it against your own graph. In the false half? Tooling maps transformations. It does not explain business intent. A regulator asks, why does this feature exist and who approved it? Your lineage graph shows the join. It doesn't show the human judgment. MIT Sloan's work on model governance is blunt about this. The documentation gap is organizational, not technical. The tool draws the pipe. It can't tell the examiner why you built the plumbing. So the second design decision is documentation that captures decisions, not just data flow. Design records tied to each feature. Who, when, why, what regulation it maps to. Boring. Survives examinations.

Host:Third belief, and this is the one people defend hardest. Explainability tools mean we can always justify a flag after the fact.

Expert:Wrong, and dangerously so. Post hoc explainers. Tools that guess why a model decided something after it already decided. Give you a plausible story, not the actual reason. Sage app values. The popular method for attributing a prediction to its inputs are an approximation. An examiner who knows their statistics will ask whether your explanation is faithful to the model or just comfortable. If those diverge, you've now lied to a regulator with a chart.

Host:That's a genuinely uncomfortable thought.

Expert:The third decision is designing for explanation at inference time. Capturing the real reason as the decision happens, not manufacturing one during the exam. You log the contributing features live, alongside the frozen inputs. When they ask why March, you don't reconstruct. You retrieve.

Host:You're describing a lot of work for a threat that shows up once a year, if that.

Expert:The exam shows up once a year. The lawsuit shows up whenever it wants. And the cost of retrofitting lineage after you're already flagged is roughly 10 times building it in. That's from O'Reilly's practitioner data, independent of any vendor.

Host:Give me the one thing a data lead does Monday morning.

Expert:Pick your last high value AML alert and try to reproduce it exactly. Why unceased? Same inputs, same output, same reason. If you can't do it in under an hour, you don't have a model problem. You have an exam you're going to fail.

Host:What we read for this one, the new stack towards data science, dbt labs, vendor, data tooling, O'Reilly radar, MIT Sloan management review. We'll leave it there. The CDO self-assessment at MBA-training.com will tell you where you really stand.

Read and go deeper