Quick answer: A token is the basic unit of text an LLM reads and generates. It is usually a chunk of a word, not a whole word: common words are one token, rarer words split into several. Everything about an LLM, from cost to context limits, is measured in tokens, so understanding them is the difference between guessing at your bill and controlling it.
If you have ever wondered why your API cost does not match your word count, or why a document that "looks short" blows past a context limit, the answer is tokens. Let me explain what they are and why they matter more than you might expect.
Why Tokens, Not Words
You might assume a model reads words. It does not. It reads tokens, which are pieces of text produced by a tokenizer before the model ever sees the input.
The reason is practical. If a model used whole words as its unit, its vocabulary would need to include every word in every language, plus every typo, name, and made-up term. That is unmanageable. Instead, tokenizers break text into subword pieces. Common words get their own token. Rare or complex words get split into smaller, reusable chunks.
This subword approach means the model can represent any string, including words it has never seen, by combining pieces it knows. The tradeoff is that "one word" and "one token" are not the same thing, and the gap trips people up constantly.
A Tokenization Example
Take the sentence:
Tokenization affects your API costs.
A typical tokenizer might split this into something like:
["Token", "ization", " affects", " your", " API", " costs", "."]
Seven tokens for five words. Notice a few things:
- "Tokenization" splits into two tokens because it is a longer, less common word.
- The leading space is part of the token (" affects" includes the space).
- Punctuation is often its own token.
As a rough rule of thumb for English, one token is about four characters, or roughly three-quarters of a word. So 1,000 tokens is around 750 words. But that ratio shifts hard depending on content. Code, JSON, URLs, and non-English text tokenize far less efficiently, often using many more tokens per visible character.
Why Tokens Control Your Cost
LLM APIs bill per token, split into input tokens (what you send) and output tokens (what the model generates). Output tokens usually cost more than input tokens.
This has direct consequences:
- A long system prompt you send on every request is paid for every request.
- Pasting a full web page into a prompt can cost more than the actual question.
- Verbose model output costs more than concise output, so asking for brevity saves money.
Because both directions are metered, trimming what you send and what you ask for is the most reliable lever on your bill. We go deep on this in token optimization: feeding web data to LLMs, which is required reading if you are piping web content into a model.
Why Tokens Control Context Limits
Every model has a context window, the maximum number of tokens it can consider in a single request. That budget covers everything: the system prompt, the conversation history, any retrieved documents, and the space reserved for the response.
When people say a document "won't fit," they mean its token count plus everything else exceeds the window. A page that looks modest on screen can be surprisingly heavy once tokenized, especially if it is code-dense or full of tables.
If you want the full picture on how that budget works and how to manage it, see what is a context window.
Here is how different content types compare in token efficiency:
| Content type | Token efficiency | Notes |
|---|---|---|
| Plain English prose | High | Close to the 0.75 words-per-token rule |
| Clean markdown | High | Structure with minimal overhead |
| Raw HTML | Low | Tags, classes, and scripts waste tokens |
| Minified JSON | Medium | Keys repeat; punctuation adds up |
| Source code | Low to medium | Symbols and indentation fragment badly |
| Non-English text | Variable | Often 2x to 3x more tokens per character |
The pattern is clear: the cleaner and more prose-like your input, the more information you fit per token.
The Hidden Cost of Feeding Web Pages to an LLM
This is where tokens become a real engineering concern rather than trivia.
Say you want a model to answer questions about a web page. The naive approach is to download the HTML and paste it in. That HTML is full of navigation menus, cookie banners, inline CSS, tracking scripts, and markup, and every character of it becomes tokens you pay for and tokens that eat your context window.
I have seen pages where the actual article is 800 words but the raw HTML tokenizes to tens of thousands of tokens. You are paying to feed the model a pile of <div class="..."> noise, and worse, that noise dilutes the signal and can confuse the answer.
The fix is to strip the page down to its real content before it reaches the model. Converting HTML to clean markdown removes the boilerplate and keeps the meaning. A single call to link.sc does exactly that:
import requests
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_your_key_here"},
json={"url": "https://example.com/article", "format": "markdown"},
)
content = resp.json()["content"]
# `content` is clean markdown, far fewer tokens than the raw HTML
The markdown version of a page is routinely a fraction of the token count of its HTML, which means lower cost, more room in the context window, and cleaner input for the model to reason over. You can read more about the fetch and search endpoints in the link.sc docs.
Practical Rules to Remember
A few habits that pay off once you internalize tokens:
- Estimate in tokens, not words. When budgeting a prompt, count tokens with a tokenizer rather than eyeballing word count.
- Clean your inputs. Never feed raw HTML to a model when clean markdown will do. It is cheaper and more accurate.
- Trim system prompts. Anything you send on every request should earn its place.
- Ask for concise output when you can. Output tokens are the expensive direction.
- Watch code and non-English content. They tokenize badly and will surprise your budget.
The Bottom Line
A token is a small piece of text, the atomic unit an LLM actually operates on. Words are how humans think about text; tokens are how models bill it, limit it, and process it. Once you start thinking in tokens, a lot of otherwise confusing behavior (surprising costs, documents that will not fit, jumbled answers from messy input) becomes obvious.
Get your inputs clean and count in tokens, and you will spend less and get better results at the same time.
Feeding web pages to an LLM without burning tokens on HTML junk? link.sc returns clean markdown from any URL in one call. Try it free.