Promotion gates · drift and anomaly monitoring · segment stability
A production-shaped pipeline over 30,000 real card accounts. The modelling is deliberately ordinary. Everything around it — what is allowed to deploy, what fires when serving goes wrong, and who reads it at 3am — is the work.
The discrimination floor is measured, not chosen: the best single-feature
rule (months_delinquent, AUC 0.6311) plus 0.02.
| Model | AUC | ECE | Segment gap | Outcome | |
|---|---|---|---|---|---|
| v1 | logistic_baseline | 0.6853 | 0.0134 | 0.1072 | promoted |
| v2 | hist_gradient_boosting | 0.7135 | 0.0115 | 0.0830 | promoted |
| v3 | naive_bayes | 0.6760 | 0.1249 | 0.1001 | blocked: calibration |
Naive Bayes ranks well enough to clear the floor. Its probabilities are wrong by 0.125. Limits and provisions are set off those probabilities, so AUC alone would have shipped it.
| Injected fault | Output rules | + input PSI | Fired |
|---|---|---|---|
schema_drop | caught | caught | input_drift, null_rate |
stuck_feature | missed | caught | input_drift |
score_shift | caught | caught | input_drift, score_shift |
latency_spike | caught | caught | latency |
The stuck upstream feature moved the mean score by 0.0006 — 0.2 standard deviations of normal batch variation. The model leaned on correlated features and kept producing plausible numbers. Only the input side could see it. Detection rates are on deliberately injected faults, not production incidents.
Score-level PSI 0.0136. Two of 14 features cross the 0.10 investigate threshold, both on the repayment side; the balance and limit features did not move. Same accounts in both windows, so this is how the features moved, not how a fresh cohort behaved.
| Age | n | Declined | False declines |
|---|---|---|---|
| 21-30 | 2,752 | 20.5% | 14.6% |
| 31-40 | 2,655 | 16.7% | 11.1% |
| 41-50 | 1,519 | 21.6% | 15.3% |
| 51-60 | 502 | 27.7% | 17.8% |
| 61+ | 72 | 36.1% | 23.9% |
| Education | n | Declined | False declines |
|---|---|---|---|
| graduate | 2,649 | 15.2% | 10.3% |
| high school | 1,196 | 26.8% | 18.6% |
| other | 115 | 9.6% | 8.7% |
| university | 3,540 | 21.6% | 15.2% |
Widest gap across reliable segments: 8.3 points. Greyed rows hold fewer than 200 accounts and are reported but excluded from the gate — without that rule a 72-account band set the gap at 12.8 points. The check reports; what gap is acceptable is a policy decision, not a modelling one.
UCI Default of Credit Card Clients (Yeh & Lien, 2009) — 30,000 Taiwanese card accounts, April–September 2005, 22.1% default rate. Public research dataset, no personal identifiers. · Source and runbook on GitHub