Cost and Latency Budgets for LLM Features
Set cost and speed targets for an LLM feature, measure them honestly and trade them against quality.
A taste of a lesson
Our average latency is 3 seconds, which seems fine, but users complain it is slow. Why?
Averages hide the slow tail. If most requests take 2 seconds and one in ten takes 12, the average still looks reasonable, but many users will hit those slow ones in a session. Look at p95 and p99 latency, and split them by input size and feature path; long outputs, big contexts or tool calls are common culprits. Also check time to first token, because without streaming users stare at nothing. What are your p95 numbers when you compute them from logs?
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Set latency and cost budgets with percentiles and per outcome costs
- Break a request into its latency and cost components
- Measure p50 and p95 latency and cost from logs
- Compare configurations on quality, cost and latency with an eval set
- Choose levers that meet budgets without unacceptable quality loss
Lesson plan
- 1 Budgets before optimising Define latency and cost targets that match how the feature is used. Start
- 2 Where time goes Break one request into its latency components. Start
- 3 Where money goes Break one request into its cost components. Start
- 4 Measuring honestly Compute percentiles and cost per feature from request logs. Start
- 5 Trading off against quality Compare configurations on quality, cost and latency together. Start
- 6 Levers and guardrails Apply the right levers and keep budgets from drifting. Start
Try asking
About this tutor
For engineers, tech leads and product managers who need an LLM feature to be fast enough and affordable at scale, not just impressive in a demo. You learn to set explicit budgets (p50 and p95 latency, cost per request and per active user), break a request into its time and cost components, measure percentiles from logs, and compare options such as smaller models, fewer steps, shorter outputs, parallel calls, caching and streaming. The key skill is trading cost and latency against quality using an evaluation set, so you choose a configuration on evidence rather than gut feeling.
Reviews
4.5
2 ratingsSample
- Daniel K.Sample
As a product lead I appreciated budgets per active user rather than per token. The engineering parts were optional for me but still readable.
- Valeria G.Sample
The quality, cost and latency table made our model decision easy to defend. We dropped two options that were worse on everything.
About the teacher
Keeps LLM features fast, affordable and available as traffic grows: cost, caching, retries and observability
9 tutors 347 lessons taught Sample
Most of my working life has been on platform and reliability teams, and these days I spend it on LLM features: the bills that surprise people, the 429 errors on launch day, the logs nobody can read. I teach the operational side of building with model APIs. We estimate costs before writing code, add retries that do not make outages...
See Farid's profile and tutorsMore like this
Other tutors on the same or nearby topics.