Token Optimization: How to Feed Web Data to LLMs Efficiently

Every token counts when you're feeding web data to LLMs. Raw HTML from a typical web page can consume 10,000-50,000 tokens, while the actual useful content might only need 500-2,000. Here's how to optimize.

The Token Tax of Raw HTML

Consider a typical news article page. The raw HTML includes:

• Navigation menus and header
• Sidebar widgets and related articles
• Footer with hundreds of links
• Ad scripts and tracking code
• Cookie consent banners
• Social sharing buttons
• CSS and inline styles
• Schema.org markup and meta tags

All of this is noise for an LLM. It wastes tokens, increases cost, and — critically — degrades the quality of LLM responses by diluting relevant context.

Token Usage Comparison

Content Format	Avg. Tokens	Useful Content
Raw HTML	15,000-50,000	5-15%
Cleaned HTML	3,000-10,000	30-50%
Markdown (link.sc)	500-2,000	85-95%
Extracted JSON	200-800	95-100%

Using link.sc's Markdown output instead of raw HTML typically reduces token usage by 90-95% while preserving all semantically meaningful content.

Strategy 1: Use Markdown Format

The single biggest optimization is converting web content to clean Markdown:

# Instead of this (raw HTML, ~20,000 tokens)
result = client.fetch(url=url, format="html")

# Do this (clean Markdown, ~1,500 tokens)
result = client.fetch(url=url, format="markdown")

link.sc's Markdown engine:

• Strips all navigation, ads, footers, and boilerplate
• Preserves headings, lists, tables, and code blocks
• Maintains link text (removes URLs unless essential)
• Converts images to alt-text descriptions
• Removes duplicate whitespace

Strategy 2: Intelligent Chunking

Don't feed entire pages when you only need part of the content:

def chunk_by_headings(markdown: str, max_tokens: int = 500) -> list:
    """Split Markdown into semantic chunks at heading boundaries."""
    sections = re.split(r'\n(?=#{1,3} )', markdown)
    chunks = []
    current_chunk = ""

    for section in sections:
        if estimate_tokens(current_chunk + section) > max_tokens:
            if current_chunk:
                chunks.append(current_chunk.strip())
            current_chunk = section
        else:
            current_chunk += "\n" + section

    if current_chunk:
        chunks.append(current_chunk.strip())

    return chunks

Strategy 3: Query-Relevant Extraction

Only include content relevant to the user's query:

def get_relevant_context(query: str, pages: list, max_tokens: int = 4000):
    """Select the most relevant chunks from fetched pages."""
    all_chunks = []
    for page in pages:
        chunks = chunk_by_headings(page.content)
        for chunk in chunks:
            score = compute_relevance(query, chunk)
            all_chunks.append((score, chunk))

    # Sort by relevance and take top chunks within token budget
    all_chunks.sort(reverse=True, key=lambda x: x[0])

    selected = []
    total_tokens = 0
    for score, chunk in all_chunks:
        tokens = estimate_tokens(chunk)
        if total_tokens + tokens <= max_tokens:
            selected.append(chunk)
            total_tokens += tokens

    return "\n\n---\n\n".join(selected)

Strategy 4: Summarize Before Embedding

For RAG pipelines, consider summarizing long content before embedding:

# Fetch the page
page = client.fetch(url=url, format="markdown")

# If content is very long, summarize first
if estimate_tokens(page.content) > 3000:
    summary = llm.generate(
        f"Summarize the key points:\n\n{page.content}"
    )
    embed_and_store(summary, metadata={"source": url})
else:
    embed_and_store(page.content, metadata={"source": url})

Strategy 5: Use Structured Extraction

When you only need specific data points, use JSON extraction:

result = client.fetch(
    url="https://example.com/product",
    format="json",
    schema={
        "name": "string",
        "price": "number",
        "description": "string",
        "features": ["string"]
    }
)
# Returns only the fields you asked for — minimal tokens

Measuring Token Efficiency

Track these metrics for your pipeline:

• Token ratio: useful tokens / total tokens consumed
• Cost per answer: total API cost (fetch + LLM) per user query
• Context utilization: % of provided context actually used by the LLM
• Answer quality: accuracy/relevance scores with different token budgets

Real-World Impact

A customer migrating from raw HTML to link.sc Markdown output saw:

• 92% reduction in token usage per query
• $3,200/month savings on LLM API costs
• 15% improvement in answer relevance (less noise in context)
• 2x faster LLM response times (smaller context = faster inference)

---

Optimize your token usage with link.sc. Get started free — clean Markdown output from any URL.