Data Leakage Detective
Find the hidden leaks that make models look brilliant in testing and fail in production
A taste of a lesson
Predicting hospital readmission, my top feature is 'number of follow up appointments booked'. AUC 0.97. Suspicious?
Very. Let us question that feature. When are follow up appointments booked: at discharge, or in the weeks after, possibly because the patient got worse? If your table counts appointments over the whole 30 day window, it partly records the outcome you are predicting. Ask the data owner for the booking timestamps and rebuild the feature using only appointments booked before discharge. Then compare AUC. What do you predict will happen to that feature's importance after the fix?
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Name and recognise the main types of data leakage
- Trace every feature to the moment it becomes known
- Detect entity and temporal leakage in split designs
- Compute target encodings and rolling features without leaks
- Run a leakage audit checklist on a real project
Lesson plan
- 1 Case one: the feature from the future Spot target leakage from features created after the outcome. Start
- 2 Case two: the familiar stranger Detect entity overlap between training and test data. Start
- 3 Case three: the peeking pipeline Find contamination from preprocessing fitted on all data. Start
- 4 Case four: tomorrow's average Catch temporal leakage in splits and rolling features. Start
- 5 Case five: the encoding that knew Use out of fold target encoding and avoid proxy identifiers. Start
- 6 Your leakage audit Assemble a checklist and apply it to your own project. Start
Try asking
About this tutor
An advanced tutor built around case files: models with suspiciously good scores that you investigate like a detective. You will learn the main kinds of leakage, from features recorded after the outcome to entity overlap, temporal leaks, target encoding without folds and preprocessing on the full dataset. Each lesson gives you a realistic scenario, a feature list and a score, and you find the leak before the tutor reveals it. You finish with a leakage audit checklist you can run on any project, and the habit of asking 'when was this known?' about every feature.
Reviews
4.7
3 ratingsSample
- Lucia R.Sample
The case format is excellent. I found a status field leak in our churn model the week after the first lesson.
- Ahmed H.Sample
Rigorous and fun. The backfilled records case was something I had never considered.
- Elif D.Sample
Great content. Sometimes the tutor held back the answer longer than I wanted, but that was the point.
About the teacher
Model evaluation you can trust: splits, metrics, leakage and tuning
9 tutors 439 lessons taught Sample
Most of the machine learning failures I have seen were not about the algorithm. They came from a test set that was used too often, a feature that leaked the answer, or a metric that hid the real cost of mistakes. I teach the unglamorous discipline that makes model results believable: how to split data, how to validate, which metric...
See Lukas's profile and tutorsMore like this
Other tutors on the same or nearby topics.