LLM Token Counter: Count Tokens for GPT and Claude Before You Send

You paste an article into your prompt, hit send, and get back a "context length exceeded" error. Or worse, the request succeeds and you quietly pay five times what you expected. Both problems have the same fix: count your tokens before the request leaves your machine.

Here's how token counting actually works for GPT and Claude, how to do it in a few lines of code, and why the counts you get from web content are usually way higher than they need to be.

What a Token Counter Actually Measures

A token is the unit LLMs read and bill in. It's not a word and not a character. It's a chunk of text produced by the model's tokenizer, usually 3 to 4 characters of English on average.

The catch is that every model family tokenizes differently. The same paragraph produces different counts in GPT-4o and Claude because they use different vocabularies. A token counter that doesn't tell you which tokenizer it uses is giving you a guess, not a count.

Rough rules of thumb for plain English text:

• 1 token is about 4 characters
• 1 token is about 0.75 words
• 100 words is about 130 to 140 tokens

Those break down fast for code, non-English languages, and anything with lots of punctuation or markup. JSON and HTML are especially token-dense because every brace, quote, and angle bracket costs you.

Counting Tokens for GPT Models

OpenAI publishes its tokenizers in the tiktoken library. GPT-4o and its variants use the o200k_base encoding:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

text = "Count me before you send me."
tokens = enc.encode(text)
print(len(tokens))  # 8

That count is exact, it runs locally, and it costs nothing. If you're targeting older models like GPT-4 Turbo, swap in cl100k_base instead. The difference between encodings on the same text is usually 5 to 15 percent, which matters at scale.

One caveat: chat requests add a few tokens of overhead per message for role formatting. For budgeting purposes, add roughly 4 tokens per message plus 3 for the reply priming. For a single long document in one message, the overhead is noise.

Counting Tokens for Claude

Anthropic doesn't ship a public offline tokenizer for current Claude models. Instead, the API has a free token counting endpoint:

import anthropic

client = anthropic.Anthropic()

count = client.messages.count_tokens(
    model="claude-sonnet-4-20250514",
    messages=[{"role": "user", "content": long_document}],
)
print(count.input_tokens)

This is exact because it runs the real tokenizer server-side, and the endpoint itself is free to call. The tradeoff is that it needs a network round trip and an API key.

If you need a fast local estimate for Claude, the character-based heuristic works fine: divide character count by 3.5 and treat the result as an upper-ish bound. In my experience Claude's counts on English prose land within about 10 percent of GPT's, so a tiktoken count is a reasonable proxy when you just need a sanity check before sending.

Estimating Cost, Not Just Count

A count is only useful once you multiply it by a price. At the time of writing, representative input prices per million tokens look like this (always check the providers' pricing pages, these move):

Model	Input price / 1M tokens	Cost of a 10,000-token page
GPT-4o mini	$0.15	$0.0015
GPT-4o	$2.50	$0.025
Claude Haiku	$0.80	$0.008
Claude Sonnet	$3.00	$0.03

A 10,000-token page sounds cheap until you multiply. Feed 50,000 pages a month through GPT-4o and that's $1,250 in input tokens alone, before the model writes a single word of output. This is why counting before sending isn't pedantry. It's the difference between a bill you predicted and one you explain to your manager.

A minimal cost estimator is about ten lines:

PRICES = {  # $ per 1M input tokens
    "gpt-4o": 2.50,
    "gpt-4o-mini": 0.15,
    "claude-sonnet": 3.00,
    "claude-haiku": 0.80,
}

def estimate_cost(text: str) -> dict:
    n = len(tiktoken.get_encoding("o200k_base").encode(text))
    return {m: round(n * p / 1_000_000, 4) for m, p in PRICES.items()}

Paste text in, get per-model counts and dollar figures out. That's the whole "free token counter tool" concept, and you can own it in your own codebase instead of pasting confidential documents into a random website. That last part deserves emphasis: a lot of online token counters are just textareas that ship your text somewhere. For anything sensitive, count locally or use the official endpoints.

The URL Case: Where Counts Go Wrong

Counting a pasted paragraph is easy. The interesting case is counting a URL, because what you fetch determines what you count.

Fetch a typical article as raw HTML and you'll count 15,000 to 50,000 tokens. Fetch the same article as clean Markdown and you'll count 500 to 2,000. The article didn't change. The wrapper did. Navigation menus, cookie banners, footers, inline scripts, and tracking markup are all tokens, and they're all useless to the model.

I covered the extraction side of this in HTML to Markdown for LLMs, but the token counter angle is simple: if your counter says a blog post costs 30,000 tokens, the correct response is not "pay it," it's "clean the input."

Here's the full loop with the link.sc fetch API, which returns pages as LLM-ready Markdown:

import requests, tiktoken

resp = requests.post(
    "https://api.link.sc/fetch",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={"url": "https://example.com/some-article", "format": "markdown"},
)
content = resp.json()["content"]

enc = tiktoken.get_encoding("o200k_base")
print(f"{len(enc.encode(content))} tokens as Markdown")

Run the same page through both formats once and the ratio will convince you faster than any benchmark table. On typical article pages I see 90 percent or more of the raw HTML token count disappear, with nothing a model cares about lost.

A Simple Pre-Send Checklist

Before web content goes into a prompt, this is the whole discipline:

1. Fetch as Markdown, never raw HTML.
2. Count with the target model's real tokenizer (tiktoken for GPT, the count endpoint for Claude).
3. Multiply by the model's input price and check the number against your budget.
4. If the count is still high, trim sections you don't need instead of paying for them. The chunking strategies in Token Optimization: Feeding Web Data to LLMs cover this in depth.

Counting after the fact, from the usage field in the API response, tells you what you already spent. Counting before tells you what you're about to spend, whether the content fits the context window, and whether your extraction step is doing its job. Only one of those is actionable.

The token counter itself is the easy part. Ten lines of Python gets you exact GPT counts and solid Claude estimates. The payoff is in what you feed it: clean content in, small counts out, and a bill that matches the number you predicted.

---

Want smaller token counts before you even start counting? Get a free link.sc API key and fetch any page as clean, LLM-ready Markdown.