How to Combine Search and Fetch in One Pipeline

Quick answer: Run search first to get candidate URLs, then fetch the most promising ones as clean markdown so your LLM reads full page content instead of thin snippets. Between the two steps, rank and dedupe the candidates so you only pay to fetch what matters. The result is a retrieval pipeline that hands the model grounded, citable text in one pass.

Search alone gives you snippets. Fetch alone assumes you already know the URL. Almost every real retrieval task needs both, in that order, wired into a single pipeline. Here is the pattern I reach for and the code to run it.

Why Search-Then-Fetch Beats Either One Alone

A search API returns titles, URLs, and short excerpts. That is enough to decide what is relevant, but not enough to answer a question well. Snippets drop context, truncate the important paragraph, and often miss the part of the page that actually matters.

Fetching the full page fixes that, but fetching everything a search returns is wasteful. A ten-result search where you fetch all ten spends most of its budget on pages the model will never cite.

The pipeline threads the needle: search to find candidates cheaply, then fetch only the ones that survive a ranking pass.

query -> search -> rank + dedupe -> fetch top N -> assemble -> LLM (with citations)

Step One: Search for Candidates

Start with the query and get a batch of candidates. link.sc search returns titles, URLs, and descriptions; full page content comes from a follow-up fetch per URL, which is exactly what this pipeline does selectively.

curl https://api.link.sc/v1/search \
  -H "x-api-key: lsc_..." \
  -H "Content-Type: application/json" \
  -d '{
    "q": "electric vehicle tax credit changes 2026",
    "engine": "google"
  }'

You get back serpData.results, a ranked list: each item has a targetUrl, a title, and a description. Slice it to the first ten client-side. That is your candidate pool.

Step Two: Rank and Dedupe Before You Fetch

This middle step is the one people skip, and it is the one that saves the most money and the most tokens. Two jobs here.

Dedupe. Search often returns three URLs from the same domain, or the same article syndicated across sites. Collapse near-duplicates so you do not fetch the same content twice.

Rank. Score each candidate against the query using the snippet you already have, and keep only the top few to fetch.

Step	Input	Output	Cost
Search	Query	10 candidates + snippets	1 search call
Rank + dedupe	Candidates	Top 3 unique URLs	Local, near free
Fetch	Top URLs	Full markdown	3 fetch calls
Assemble	Markdown	Prompt with citations	Local

from urllib.parse import urlparse

def rank_and_dedupe(results, query_terms, keep=3):
    seen_domains = set()
    scored = []
    for r in results:
        domain = urlparse(r["targetUrl"]).netloc
        if domain in seen_domains:
            continue                     # one page per domain
        seen_domains.add(domain)
        text = (r["title"] + " " + r.get("description", "")).lower()
        score = sum(t in text for t in query_terms)
        scored.append((score, r))
    scored.sort(key=lambda x: x[0], reverse=True)
    return [r for _, r in scored[:keep]]

Keeping one result per domain is a blunt but effective dedupe. If you need multiple pages from the same site (documentation, for example), swap it for a content-hash check after fetching instead.

Step Three: Fetch the Survivors as Markdown

Now fetch the pages that made the cut. Markdown is the right format here because it keeps structure (headings, lists, tables) that the model uses to understand the page, without the HTML noise.

curl https://api.link.sc/v1/fetch \
  -H "x-api-key: lsc_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.gov/ev-credit-2026",
    "format": "markdown"
  }'

Fetch the top three in parallel to keep latency low. Each fetch handles its own rendering and proxying, so a heavy JavaScript page and a plain article come back in the same clean shape.

Step Four: Assemble the Prompt With Citations

The last step turns fetched pages into a grounded prompt. The key move is tagging each chunk with its source URL so the model can cite it and you can verify it.

def assemble(query, pages):
    blocks = []
    for i, p in enumerate(pages, 1):
        body = p["content"][:6000]       # trim to a token budget
        blocks.append(f"[{i}] Source: {p['url']}\n\n{body}")
    context = "\n\n---\n\n".join(blocks)
    return (
        f"Answer the question using ONLY the sources below. "
        f"Cite sources by their [number].\n\n"
        f"Question: {query}\n\n"
        f"Sources:\n{context}"
    )

Because every block carries its URL, the model's answer comes back with [1], [2] markers you can resolve straight to the original pages. That is what makes the output trustworthy instead of a confident guess.

The Whole Pipeline

def pipeline(query):
    terms = query.lower().split()
    results = search(query)["serpData"]["results"][:10]  # /v1/search
    top = rank_and_dedupe(results, terms, keep=3)
    pages = [{"url": r["targetUrl"],
              "content": fetch_markdown(r["targetUrl"])["content"]}
             for r in top]                               # /v1/fetch, parallelize
    prompt = assemble(query, pages)
    return call_llm(prompt)                              # grounded, cited answer

A handful of lines of orchestration. Each stage does one job, and you can tune the candidate slice and keep independently to trade cost against recall.

When to Skip the Rank Step

There is a shortcut. If your query is narrow and you trust the search ranking, skip the ranking pass and fetch the top two or three results as-is. That is the simplest possible version of this pipeline, and it is covered in depth in search results with full page content.

Use the ranked version (search, then fetch selectively) when:

• You need tight control over how many pages you fetch.
• You want to dedupe or filter candidates before spending fetch credits.
• Some candidates should be dropped for reasons the search API cannot know (domain allowlists, freshness, language).

Use the shortcut (fetch the top results directly) when the query is simple and you want the least code.

Keep It Honest

A pipeline that cites sources is only as good as the sources. A few habits keep it trustworthy: prefer primary sources over aggregators, keep the fetched text long enough that you are not citing a stub, and respect the fetched sites (rate limits, robots.txt, public pages only). If two sources disagree, surface both rather than letting the model pick one silently.

For a tour of wiring this into an agent that runs the loop itself, give your AI agent internet access walks through the connection layer. Full parameters for both endpoints are in the link.sc docs, and pricing for higher volumes is at link.sc/pricing.

Search finds it. Fetch reads it. Ranking in between keeps you from paying for pages nobody cites. That is the entire pattern, and it scales from a weekend script to a production retrieval layer without changing shape.

---

Want search and fetch behind one clean API? Start free at link.sc.