Auditing a model for bias: a hands-on method
Run a structured fairness audit with clear metrics, subgroup tests and an honest written report
A taste of a lesson
Our model approves 58% of men and 52% of women. Is that a fairness problem?
Not automatically, but it is worth investigating. The ratio is 52 / 58, about 0.90, above the common 0.8 rule of thumb, which is only a screening heuristic anyway. Next steps: how many people are in each group, and what is the uncertainty on that gap? Then look at error rates: among people who would repay, are approval rates similar? Also ask whether repayment labels are biased by past approvals. Exercise: if 400 women applied, compute a rough 95% interval for the 52% rate.
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Scope a fairness audit by defining the decision, affected groups and harms
- Compute subgroup selection rates, error rates and disparate impact ratios
- Account for uncertainty, small groups, intersections and selective labels
- Design counterfactual tests for generative models
- Write an audit report with honest limits and recommendations
Lesson plan
- 1 Scoping: decisions, people and harms Define what is being audited and why before choosing any metric. Start
- 2 Data for auditing Assemble evaluation data with outcomes and group membership, and know its flaws. Start
- 3 Computing subgroup metrics Calculate selection, error and calibration measures per group. Start
- 4 Uncertainty and intersections Avoid false conclusions from noise and hidden subgroup disparities. Start
- 5 Auditing generative models Measure bias in text and image generators with counterfactual tests. Start
- 6 Root causes and the audit report Trace disparities to causes and report findings honestly. Start
Try asking
About this tutor
For analysts, data scientists, product people and auditors who need to actually check a model, not just discuss bias. You follow a structured method: define the decision and the people affected, choose fairness criteria with stakeholders, gather or construct evaluation data with group information, compute subgroup metrics including confusion matrix rates and calibration, test intersections and small groups with uncertainty in mind, probe generative models with counterfactual prompts, and write a report that states limits honestly. You work through a realistic loan approval case with small tables of numbers. Code is optional and kept generic.
Reviews
4.5
2 ratingsSample
- Ingrid T.Sample
Finally a structured method instead of philosophy. The selective labels point changed our whole audit plan for a credit model.
- Omar Q.Sample
Rigorous and demanding, as advertised. The generative model lesson needed more examples, but the pre registration advice was gold.
About the teacher
I teach AI ethics as practical judgment: privacy, fairness and accountability you can act on
9 tutors 327 lessons taught Sample
I teach the ethics of AI as something you do, not something you recite. My lessons cover privacy and personal data, bias and fairness, explainability and the effects of AI on work and high stakes decisions. I use real cases and simple methods so learners can reason through a new situation on their own. My background combines data analysis with...
See Aisha's profile and tutorsMore like this
Other tutors on the same or nearby topics.