Self supervised and contrastive learning
Learn how models build useful representations from unlabelled data, and how to test them
A taste of a lesson
In contrastive learning, why does a bigger batch usually help?
In the common setup, every other example in the batch acts as a negative for each anchor. The loss is a softmax over the anchor's similarity to its positive and to all those negatives. With more negatives, the model sees more examples it must push away, including harder ones that look similar, so the task becomes more informative and the representation sharper. That is why memory queues and large batches were popular. The benefit has limits, and some non contrastive methods avoid the need for negatives entirely. Quick check: with a batch of 256 images and two views each, how many negatives does each anchor have?
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Explain how pretext tasks and masked modelling create learning signals without labels
- Compute a small contrastive loss and explain the role of temperature and negatives
- Choose augmentations deliberately, knowing they define what the model ignores
- Explain representational collapse and how non contrastive methods prevent it
- Evaluate representations with linear probes, nearest neighbours and retrieval
Lesson plan
- 1 Learning without labels Understand how tasks built from the data itself teach useful features. Start
- 2 Masked modelling Compare masked prediction for text and images and why mask ratios differ. Start
- 3 Contrastive learning and InfoNCE Compute a contrastive loss from a small similarity matrix. Start
- 4 Augmentations define invariance Choose augmentations based on what the downstream task needs to keep. Start
- 5 Collapse and non contrastive methods Explain why trivial solutions appear and how designs avoid them without negatives. Start
- 6 Image text pretraining and evaluation Use paired image and text embeddings and evaluate representations fairly. Start
Try asking
About this tutor
For practitioners who know deep learning and want to understand how models learn from data without labels, the idea behind most modern pretrained encoders. You will compare pretext tasks, masked modelling for text and images, and contrastive learning, where a model pulls together two views of the same example and pushes apart different examples. You will compute a small contrastive loss, see how temperature and augmentations shape what is learned, and understand representational collapse and how non contrastive methods avoid it. The final lessons cover image and text contrastive pretraining, which powers zero shot classification and search, and how to evaluate representations with linear probes and retrieval.
Reviews
Students can review a tutor after a paid lesson. Nobody has yet.
About the teacher
Generative models and representation learning, explained with intuition first and maths second
9 tutors 335 lessons taught Sample
I teach how models learn useful representations and how they generate new data: autoencoders, GANs, diffusion models, self supervised learning and language model pretraining. I came to this through research engineering work where we had to decide which kind of model was worth the compute, so I teach with trade offs in mind. Each topic starts with a picture or...
See Mateo's profile and tutorsMore like this
Other tutors on the same or nearby topics.