Technology & AI
What is Tokens (LLM Text Units)?
Tokens are the units in which LLMs process text — short words become 1 token, long words split into many. Billing and context limits are in tokens.
Rule of thumb in English: ~4 characters = 1 token, or ~0.75 words = 1 token. In Portuguese, words split into more tokens (accents, suffixes). 1,000 PT words ≈ 1,500 tokens.
They matter for 2 things: cost (charged per million input and output tokens) and context (model truncates past the limit). Prompt optimization = removing redundancy = paying less.
Frequently asked questions
How to count tokens before sending?
OpenAI has tiktoken (JS/Python lib). Anthropic and Google have their own counters. 1 character ≠ 1 token.
Why are input + output billed separately?
Output requires more processing (generation) than input (reading). Output usually costs 3-5× more per token.