Eigenvalues, Eigenvectors and PCA
Derive principal component analysis from eigenvectors and use it with full understanding
A taste of a lesson
My first principal component explains 92 percent of variance. But one feature is salary in dollars and the rest are 0 to 1 scores. Fine?
Almost certainly not fine. PCA finds directions of maximum variance, and salary measured in dollars has a variance millions of times larger than features on a 0 to 1 scale. So the first component is essentially 'salary', and the 92 percent reflects units, not structure. Standardise each feature to mean 0 and standard deviation 1 first, so PCA works on correlations. Then compare the explained variance again. Before running it, what do you expect to happen to that 92 percent?
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Compute eigenvalues and eigenvectors of small matrices by hand
- Explain why covariance matrices have orthogonal eigenvectors
- Derive PCA as maximising projected variance
- Relate PCA to the singular value decomposition and explained variance
- Apply PCA with correct centring, scaling and leakage precautions
Lesson plan
- 1 Vectors that keep their direction Understand eigenvectors as directions a matrix only stretches. Start
- 2 Symmetric matrices See why symmetric matrices have real eigenvalues and orthogonal eigenvectors. Start
- 3 Variance along a direction Express projected variance with the covariance matrix. Start
- 4 PCA step by step Run PCA by hand on a small dataset. Start
- 5 PCA through SVD Connect PCA to the singular value decomposition. Start
- 6 Limits and good practice Use PCA responsibly in machine learning pipelines. Start
Try asking
About this tutor
An advanced tutor for learners comfortable with vectors and matrix multiplication who want the real mathematics behind PCA. You will find eigenvalues and eigenvectors of small matrices by hand, understand why covariance matrices have orthogonal eigenvectors, and derive PCA as finding directions of maximum variance. Lessons connect PCA to the singular value decomposition, explained variance, standardisation choices and reconstruction error, and finish with the practical limits of PCA: linearity, interpretability, sign ambiguity and leakage. Expect worked calculations and precise notation, always introduced with a geometric picture first.
Reviews
4.7
3 ratingsSample
- Rodrigo P.Sample
Exactly the depth I wanted before reading papers that use spectral methods.
- Mateusz Z.Sample
Rigorous and clear. Deriving PCA from maximising u transpose C u was the explanation I had been missing for years.
- Aiko F.Sample
The SVD connection lesson was dense but excellent. I would have liked one more worked example on reconstruction error.
About the teacher
Linear algebra for AI, with geometry first and notation second
9 tutors 338 lessons taught Sample
I teach the linear algebra behind modern AI: vectors, matrices, similarity, eigenvectors and the methods built on them, such as PCA, clustering and recommender systems. I trained in applied mathematics and later worked on search and recommendation features, so I like to connect each idea to something a real system does. My lessons begin with pictures and small numbers you...
See Katarzyna's profile and tutorsMore like this
Other tutors on the same or nearby topics.