Quick answer: You have two options for getting search results with full page content: the two-step pattern (call a search API for URLs, then scrape each URL yourself) or a single-call API that returns full content with the results. The two-step pattern gives you control but you own the scraping headaches. With link.sc you pair POST /v1/search with POST /v1/fetch and wrap both in one tool that returns each result as clean markdown, which is what most RAG pipelines and agents actually want.
I keep seeing the same story: someone wires a search API into their agent, ships it, and the agent starts confidently answering from 160-character snippets. Let me walk through why that fails and what to do instead.
Why Snippets Fail for RAG and Agents
A search snippet is a preview engineered to make a human click. It is not a summary, and it is definitely not the document.
Three concrete failure modes:
- Truncated claims. Snippets cut mid-sentence. "The study found that the treatment was not..." Not what? Your model will guess, and guessing with confidence is the worst failure mode an AI product can have.
- Missing the actual answer. The snippet shows the paragraph that matched keywords, which is often the introduction. The answer is in section four, which the snippet API never fetched.
- No context for citations. If your agent cites a source it never read, that's not a citation, it's decoration.
For tool-calling agents the damage is subtler but real: the agent burns a turn on a search, gets snippets, decides it needs more, and now you're paying for a multi-turn loop that one content-rich call would have avoided. That loop pattern is most of what separates good agentic search from bad, which I covered in what is agentic search.
Option 1: The Two-Step Search-Then-Fetch Pattern
The classic approach: search API for URLs, then fetch and extract each page.
import requests
HEADERS = {"x-api-key": "lsc_your_key"}
# step 1: search (any search API works here)
serp = requests.post("https://api.link.sc/v1/search", headers=HEADERS, json={
"q": "postgres connection pooling best practices",
"engine": "google",
}).json()
results = serp["serpData"]["results"][:5] # no limit param; slice client-side
# step 2: fetch each result as markdown
docs = []
for r in results:
page = requests.post("https://api.link.sc/v1/fetch", headers=HEADERS, json={
"url": r["targetUrl"],
"format": "markdown",
}).json()
docs.append({"url": r["targetUrl"], "content": page["content"]})
This works, and it's the right shape when your search provider and your fetch provider are different (say, Brave for search, a fetch API for content). The downsides are operational: N+1 requests per query, per-URL failures you have to handle, and latency that stacks up because the slowest page gates the batch unless you parallelize.
If you do the fetching with your own scraper instead of an API, add anti-bot walls, JS rendering, and paywalled duds to that list. This is exactly the mess that pushed us to build a fetch endpoint in the first place.
Option 2: Single-Call Search With Content Included
The pattern that's replacing two-step for most agent work: expose one tool that returns content-rich results. Some providers bundle content into the search response itself. link.sc keeps search and fetch as separate endpoints, so you get the same ergonomics by wrapping them in a single function and parallelizing the fetches:
import requests
from concurrent.futures import ThreadPoolExecutor
HEADERS = {"x-api-key": "lsc_your_key"}
API = "https://api.link.sc/v1"
def search_with_content(q, n=5):
serp = requests.post(f"{API}/search", headers=HEADERS,
json={"q": q, "engine": "google"}).json()
results = serp["serpData"]["results"][:n]
def fetch(r):
page = requests.post(f"{API}/fetch", headers=HEADERS,
json={"url": r["targetUrl"], "format": "markdown"}).json()
return {"url": r["targetUrl"], "title": r["title"], "content": page["content"]}
with ThreadPoolExecutor(max_workers=n) as pool:
return list(pool.map(fetch, results))
One tool call from the agent's side, and each result carries its full page body as markdown. The fetch endpoint handles rendering, extraction, and the anti-bot failure cases behind the scenes. For an agent tool definition this is a huge simplification: one tool, one call, readable output. The quickstart has the response shapes.
The trade-off with APIs that bundle content into the search response is control. You get the provider's extraction, on the provider's timeline. The wrapper above keeps the standalone fetch call in the loop, so custom headers, JS rendering, or specific per-URL behavior are still one JSON field away.
The Comparison, Honestly
| Approach | Requests per query | You handle scraping | Latency | Best for |
|---|---|---|---|---|
| Snippets only | 1 | No content at all | Fastest | Link discovery, rank tracking |
| Two-step (search + fetch API) | 1 + N | No, API does it | Medium, parallelize | Mixed providers, per-URL control |
| Two-step (search + own scraper) | 1 + N | Yes, all of it | Slow and spiky | Rarely worth it now |
| Single-call with content | 1 | No | Medium | Agents, RAG, answer engines |
The Token-Cost Question
Full content is more tokens, and tokens are money. So does full content blow up your LLM bill?
Only if you're careless. The pattern that works:
- Retrieve full pages (cheap relative to LLM inference).
- Chunk and rank locally (nearly free).
- Send only the top-ranked chunks to the model.
Full retrieval with selective inference beats snippet retrieval every time, because you're choosing what the model reads from the whole document instead of letting a search engine's snippet algorithm choose for you. A typical full article is 1500-4000 tokens as clean markdown. The same page as raw HTML is often 5-10x that, which is why extraction format matters as much as the retrieval pattern.
And budget-wise: you cut LLM tokens by ranking chunks, not by retrieving less. Skimping on retrieval to save money is saving at the wrong layer.
When Snippets Are Actually Fine
Fairness demands a section here. Snippets are the right tool when:
- You only need to know that something exists, not what it says (monitoring for new mentions).
- You're tracking rankings, where position is the data.
- A human will click through anyway, so the snippet is genuinely just a preview.
If your consumer is a model rather than a person, though, snippets are almost never enough. Models don't click.
Wiring It Into a Real Pipeline
For RAG specifically, single-call search with content collapses your ingestion path: query goes in, chunk-ready markdown comes out, and your pipeline starts at the chunking step. I walked through the full setup in building RAG pipelines with real-time web data if you want the end-to-end version.
Whichever pattern you pick, the principle is the same: never let your model answer from a preview. Feed it the page.
Stop building on snippets: get a free link.sc API key and pull full-page search results with one tool call.