Inference
Running a trained model to produce predictions — the production-time work, as opposed to training.
What is Inference?
Inference is what happens every time a user sends a prompt. The model takes the input, runs forward through its layers, and produces an output. Training is rare (occasionally); inference is constant (every request).
Inference cost dominates Gen AI economics. Training GPT-4 cost ~$100M one time; serving GPT-4 to ChatGPT users costs that much every few weeks. Optimising inference (smaller models, quantization, batching, caching, speculative decoding) is where engineering effort pays off.
Key metrics: **latency** (time to first token, total time), **throughput** (tokens per second), **cost per million tokens**. Indian product teams optimise different metrics depending on use case — chat needs low latency, batch summarisation needs high throughput.
Inference economics is the constraint that shapes every production Gen AI decision. Engineers who understand it ship cheaper, faster systems.
A Mumbai chatbot was running every query through Claude Opus at ₹4 per conversation. The team routed simple queries to Claude Haiku (₹0.20) and reserved Opus for complex ones — costs dropped 70% without quality loss.
Want to master this?
Learn Inference in a structured cohort
3-month live program with mentors, real projects, and 50+ partner placement support.