Same IBM holdout as telco-churn-nba-engine

Read the score before anyone acts.

This is a reading of the gradient-boosting churn model pinned from telco-churn-nba-engine. The pack does not train a replacement. Nothing below was measured again for this page.

Open the interactive demo Keep reading the write-up

Independent portfolio project built on public data. The holdout is telco_nba.pipeline.split_customers, seed 42, test size 0.250000, stratified on Churn: 5282 training rows and 1761 test rows. The table is the public IBM Telco Customer Churn sample, 7043 rows, churn rate 0.265370. It is not an operator extract and it is not Kenyan network performance.

A group gap on this page is a measurement, not a certificate. SHAP describes the model. It is not a cause of churn.

Poster: a churn score flags senior customers more often. The pack explains scores, measures group rates, checks drift, and gates the held-out metrics. — A churn score flags senior customers more often. This pack explains scores, measures group rates, checks drift, and gates the held-out metrics.assets/hero.png

Findings at a glance

Two published gaps, one split, one cutoff.

gender0.030582Demographic parity difference. The groups are close.SeniorCitizen0.223259Demographic parity difference. Flagged for review. Not a CI floor.

The README flags these two differences for review. They are not a CI floor, and they are not a fairness certificate.

Source: reports/fairness.json

Model card

What the pinned model is for.

Intended use

Rank customers in the IBM telco sample by the model's estimate of P(Churn=Yes).
Show the TreeSHAP contributions behind one score so a person can read the model.
The intended user of this pack is someone reviewing that public model.

Out of scope

Any live operator's customers, including customers in Kenya.
Deciding credit, disconnection, eligibility, or a penalty.
Sending an offer or placing a call with no person in the loop.
Treating a SHAP value as a cause of churn.
Treating logistic regression or the add-on models as this scoring model.
Claiming a fairness pass.

Source: MODEL_CARD.md

Held-out metrics

Gradient boosting stays the scoring file.

Recomputed with the saved pipelines on the 1761 held-out rows. Top-decile k is 176. The test base rate is 0.265190. matches_upstream_metrics_at_six_decimals: true.

Logistic regression is higher on ROC-AUC. Gradient boosting is higher on PR-AUC and on top-decile lift. The scoring artifact remains gradient boosting.
Model	Role	ROC-AUC	PR-AUC	Top-decile lift
Dummy prior	Comparison	0.500000	0.265190	1.000000
Logistic regression	Comparison	0.846490	0.638090	2.785308
Gradient boosting	Scoring model	0.846001	0.656070	2.806733

CI floors in gates.yaml are ROC-AUC 0.82, PR-AUC 0.63, and top-decile lift 2.60. The last gate check passed: true.

Held-out ROC-AUC, PR-AUC, and top-decile lift for the dummy prior, logistic regression, and gradient boosting. Gold lines mark the gates.yaml floors. — Held-out ROC-AUC, PR-AUC, and top-decile lift for the dummy prior, logistic regression, and gradient boosting. The gold lines are the floors in gates.yaml. Deep green is the scoring model.assets/model_metrics.png · reports/metrics_recomputed.json

Drift baseline

PSI on one split is not a later month.

The two slices are a stratified random split of one public table, not two time periods. A small PSI here checks the function. A production monitor would keep this training split as the reference and pass a later batch as the current sample.

Shift a bin in the demo. The chart below stays on the published counts.

Reference: training rows from telco_nba.pipeline.split_customers. Current: held-out rows from the same split. n_train 5282, n_test 1761. Review trigger 0.250000. n_above_trigger 0. The largest value is MonthlyCharges at 0.012474. Numeric columns ask for 10 quantile bins. Shares below the clip floor 0.000001 are raised to that floor.

MonthlyCharges is numeric. PSI 0.012474. above_trigger: false. The red mark at the right of each track is the review trigger 0.250000.

PSI by feature between the training split and the held-out split. A gold line marks the review trigger. — Each bar is a feature PSI from reports/drift_psi.json. The gold line is the review trigger.assets/drift_psi.png

Limitations

What this holdout cannot say.

The table is IBM's public US sample, 7043 rows. These metrics are not local performance for any network.
gender and SeniorCitizen are inputs. The file has no race, ethnicity, region, language, or disability column.
There is one stratified holdout. The model card does not report a confidence interval.
Hard-label rates use P(Churn=Yes) > 0.5. They are not the upstream policy cutoffs.
Low PSI here compares two halves of one random split.

In this demo

Continue the reading.

Provenance

Upstream commit 21f6115931f4358ebc7cc87d9ba1f4d87fd015aa

Scoring artifact SHA-256 fe1cd3c3ea69565c133f9964744c9d4f5d55908ef883908cfb380f1fc37d3509

CSV SHA-256 16320c9c1ec72448db59aa0a26a0b95401046bef5d02fd3aeb906448e3055e91

Example customer 5343-SGUBI. Scoring model gradient_boosting. Positive label Yes. gender is an input: true. SeniorCitizen is an input: true.