Token
The basic unit of text an LLM processes — typically a word fragment, not a full word.
What is Token?
LLMs do not see words; they see tokens. Tokens are pieces of text that the model's tokeniser has chunked the input into. "Cat" might be one token; "predominantly" might be three.
The rough rule: 1 token ≈ 4 characters of English text ≈ 0.75 words. A 1000-word document is about 1300–1400 tokens. Hindi, Tamil, and other Indian languages typically have higher tokens-per-character because they are less common in training data.
LLM pricing is per token (both input and output). Context windows are measured in tokens. Cost engineering = token engineering — shorter prompts are cheaper and lower latency.
Everything in production Gen AI economics is measured in tokens. Cost optimisation, latency, context limits all depend on token counts.
A Chennai chatbot was costing ₹3 per conversation until the team rewrote prompts to be 60% shorter. Same quality, ₹1.20 per conversation — pure margin.
Related terms
Want to master this?
Learn Token in a structured cohort
3-month live program with mentors, real projects, and 50+ partner placement support.