Overfitting, regularisation and dropout
Recognise overfitting from your curves and pick the right fix, from more data to dropout
A taste of a lesson
I added dropout 0.5 everywhere and now my model underfits. Did I do it wrong?
Not wrong, just too strong. A rate of 0.5 on every layer removes a lot of signal, especially in small networks or convolutional layers, so the model struggles to fit even the training data. Try lower rates, such as 0.1 to 0.3, and apply them mainly before the larger dense layers. Also check the alternatives: augmentation and weight decay often help with fewer side effects. Then compare training and validation curves again. Quick check: if both training and validation accuracy are now low and close together, which problem are you looking at?
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Tell overfitting from underfitting by reading training and validation curves
- Set up training, validation and test splits that avoid leakage
- Apply augmentation, weight decay, dropout and early stopping appropriately
- Choose a next experiment for an overfitting model and justify it
Lesson plan
- 1 Reading learning curves Recognise overfitting, underfitting and healthy training from loss and accuracy curves. Start
- 2 Honest data splits Build training, validation and test splits that measure real generalisation. Start
- 3 Data first: more data and augmentation Use better data and augmentation as the first line of defence. Start
- 4 Weight decay and dropout Understand how penalties and random dropping of units reduce overfitting. Start
- 5 Early stopping and choosing the next experiment Stop at the right time and plan a sensible sequence of fixes. Start
Try asking
About this tutor
For beginners who have trained a model that looked brilliant on training data and disappointing on anything new. You will learn to read training and validation curves, tell overfitting from underfitting, and set up data splits that give honest answers. Then you work through the main remedies in order of usefulness: more or better data, augmentation, a smaller model, weight decay, dropout and early stopping. Each lesson uses a small realistic scenario, such as an image classifier at 99 percent training accuracy and 70 percent validation accuracy, and asks you to choose and justify the next step. Light maths only.
Reviews
4.7
3 ratingsSample
- Olumide A.Sample
Clear and not preachy. I wanted a bit more on augmentation for text rather than images, but the principles transferred.
- Isabel F.Sample
The splits lesson exposed my problem: the same customers were in train and validation. My 'great' model was just remembering them.
- Grace L.Sample
I liked that it never jumped to a fix before asking what my curves looked like. That habit is the main thing I kept.
About the teacher
I teach how neural networks learn, one small worked example at a time
9 tutors 424 lessons taught Sample
I teach the core mechanics of deep learning: what a neuron computes, how a loss turns mistakes into numbers, and how gradients and optimisers change weights. My background is in building and training models for applied research teams, which mostly meant staring at loss curves that refused to go down. That shaped how I teach. I start every idea with...
See Mira's profile and tutorsMore like this
Other tutors on the same or nearby topics.