How to Build a RAG Pipeline That Uses Live Web Data

Most RAG tutorials assume your knowledge lives in a folder of PDFs that never changes. Real questions don't work that way. Users ask about pricing that changed last week, a docs page that got rewritten, news from this morning. If your index was built once and never updated, your grounded answers are grounded in stale facts — which is arguably worse than no grounding at all, because it's confidently wrong.

This post is about building a RAG pipeline that pulls in live web data: the loop, why content cleaning is the step that quietly determines your retrieval quality, a concrete end-to-end example, and the freshness/cost tradeoffs you'll have to make.

The RAG Loop, With the Web in It

The core loop doesn't change when you add web data — you're just adding a live source at the front:

fetch → clean → chunk → embed → retrieve → generate

1. Fetch the source — a specific URL, or a search query that returns several URLs.
2. Clean the HTML down to the actual content (this is where most pipelines silently lose quality).
3. Chunk the clean text into passages sized for your embedding model.
4. Embed each chunk into a vector and store it with metadata.
5. Retrieve the top-k chunks for a user query by vector similarity.
6. Generate an answer with the LLM, passing the retrieved chunks as context.

There are two ways live web data enters. You can pre-index a set of known sources and refresh them on a schedule, or you can retrieve on-demand — run a web search at query time, fetch the top results, and build a tiny ephemeral index just for that question. Most production systems do both: a warm index for known-important sources, plus on-demand search for the long tail.

Why Clean Markdown Decides Your Retrieval Quality

The step people underinvest in is cleaning, and it's the one that matters most for retrieval.

Here's the mechanism. When you embed a chunk, the vector represents everything in that chunk. If your chunk is half navigation links and cookie-banner text — because you chunked raw HTML or badly-extracted text — the embedding is a blend of "the actual content" and "generic website boilerplate." Two problems follow:

• Recall drops. The signal you care about is diluted, so the right chunk doesn't rank as highly for the right query.
• False similarity. The same nav and footer appear on every page, so chunks from unrelated pages look similar to each other. Your top-k fills up with near-duplicate boilerplate.

Clean markdown avoids both. It's almost all content, and its heading structure gives you natural, meaningful chunk boundaries — so a chunk maps to a coherent idea instead of an arbitrary character window. Clean input is not a nice-to-have here; it's the difference between retrieval that works and retrieval that mostly returns noise.

A Concrete Pipeline

Here's an on-demand pipeline end to end. Start by getting live content — either a direct fetch or a search.

import requests

HEADERS = {"x-api-key": "lsc_your_key_here"}

# Option A: you know the URL
def fetch_page(url):
    r = requests.post(
        "https://api.link.sc/v1/fetch",
        headers=HEADERS,
        json={"url": url, "format": "markdown"},
    )
    return r.json()["content"]

# Option B: you have a question, not a URL
def search_web(query):
    r = requests.post(
        "https://api.link.sc/v1/search",
        headers=HEADERS,
        json={"query": query},
    )
    # search returns results with full-page markdown already attached
    return r.json()["results"]

Because link.sc returns clean markdown, chunking is straightforward — split on heading boundaries and pack up to a token budget:

import re

def chunk_markdown(md, max_chars=1500):
    sections = re.split(r"\n(?=#{1,3} )", md)
    chunks, current = [], ""
    for s in sections:
        if len(current) + len(s) > max_chars and current:
            chunks.append(current.strip())
            current = s
        else:
            current += "\n" + s
    if current.strip():
        chunks.append(current.strip())
    return chunks

Embed the chunks and store them with source metadata so you can cite and expire them later:

def index_url(url, store, embed):
    md = fetch_page(url)
    for chunk in chunk_markdown(md):
        vector = embed(chunk)            # any embedding model
        store.upsert(
            vector=vector,
            text=chunk,
            metadata={"source": url, "fetched_at": now()},
        )

Then retrieve and generate:

def answer(question, store, embed, llm):
    q_vec = embed(question)
    hits = store.query(q_vec, top_k=5)
    context = "\n\n---\n\n".join(h.text for h in hits)
    sources = {h.metadata["source"] for h in hits}
    prompt = (
        f"Answer using only the context below. Cite sources.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    return llm(prompt), sources

That's the whole loop. /search is the shortcut when you don't have URLs yet — it hands you results with full-page markdown, so you skip a separate fetch round-trip.

Keeping the Index Fresh (and the Bill Sane)

Freshness is a policy decision, not a feature you turn on. A few patterns that work:

• Tier your sources by volatility. A pricing page or a news feed might need re-fetching hourly; a stable reference doc, monthly. Store fetched_at in metadata and re-index on a TTL that matches how fast each source actually changes.
• Prefer on-demand for the long tail. You can't pre-index the whole web. For rare or unpredictable questions, search + fetch at query time and don't persist the result.
• Cache aggressively for the head. For your common, high-value sources, a warm pre-built index is faster and cheaper than fetching live on every request.
• Watch two costs, not one. Every query can incur fetch/search cost and embedding cost and LLM generation cost. On-demand fetching is fresh but adds latency and per-query cost; pre-indexing front-loads the cost but risks staleness. Pick per source based on how much freshness that source actually buys you.

The honest summary: freshness and cost trade directly against each other, and there's no universal right answer — only the right answer for a given source's volatility.

You can assemble all of this yourself with a headless browser, an extraction library, and a scraping proxy. link.sc just collapses the fetch-and-clean front of the pipeline into one call — clean markdown from a URL via /fetch, or from a query via /search — and it's open core, so you can self-host the same thing if you'd rather own it.

---

Ready to give your RAG pipeline live, clean web data? link.sc includes 500 free credits every month — get an API key or dig into the docs.