Quick answer: A Perplexity-style answer engine is a four-stage pipeline: take the user's question, run a web search, fetch the full content of the top results, then have an LLM synthesize an answer with numbered citations back to the sources. You can build a working version in about a hundred lines of Python with a search API, a fetch API, and one LLM call. The hard parts are citation accuracy, freshness, and keeping the context window under budget.
I've built this pipeline several times now, and the surprising thing is how little of the magic lives in the model. Most of the quality comes from what you feed it.
The Architecture at a Glance
user question
-> query rewriting (optional but useful)
-> web search (get 5-10 candidate URLs)
-> fetch full pages as markdown
-> chunk, dedupe, rank against the question
-> LLM synthesis with numbered sources
-> answer with [1][2] citations
That's it. No vector database required for the basic version, because you're retrieving fresh content per question, not searching a pre-built corpus. If you want persistent knowledge on top, that becomes a RAG system, and I covered that split in RAG pipelines with live web data.
Stage 1: Search (and Rewrite the Query First)
Users type questions. Search engines want queries. "why did my sourdough starter die after I moved" searches worse than "sourdough starter inactive after moving causes".
A cheap LLM call to rewrite the question into 1-3 search queries improves everything downstream. Then run the search:
curl https://api.link.sc/v1/search \
-H "x-api-key: lsc_your_key" \
-H "Content-Type: application/json" \
-d '{
"q": "sourdough starter inactive after moving causes",
"engine": "google"
}'
The response nests the ranked results under serpData.results, and each result carries a targetUrl plus a title and description. There is no result-count parameter, so slice the list client-side and keep the top 5-6. Search gives you candidates, not content: getting the full page is a follow-up fetch call per targetUrl, which is stage 2. link.sc keeps both endpoints behind the same key, so the whole loop stays simple.
Stage 2: Fetch the Full Pages
Loop over serpData.results and fetch each targetUrl as markdown. The same endpoint also handles URLs from elsewhere (a user pastes a link, or you follow a citation):
curl https://api.link.sc/v1/fetch \
-H "x-api-key: lsc_your_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/long-article",
"format": "markdown"
}'
Markdown matters here. Raw HTML burns 5-10x the tokens for the same information, and models cite cleaner text more accurately.
Stage 3: Chunk, Dedupe, Rank
Full pages are long, and you can't ship six full articles into every answer. So:
- Chunk each page into 500-1000 token passages, keeping headings attached.
- Dedupe near-identical passages (syndicated articles are everywhere).
- Rank chunks against the question, with embeddings or a small reranker, and keep the top 10-15.
Here's the whole pipeline as a compact Python sketch:
import requests
LSC = "https://api.link.sc/v1"
HEADERS = {"x-api-key": "lsc_your_key"}
def answer(question: str) -> str:
# 1. search, keep the top 6 results client-side
r = requests.post(f"{LSC}/search", headers=HEADERS, json={
"q": question,
"engine": "google",
})
results = r.json()["serpData"]["results"][:6]
# 2. fetch each page as markdown, build numbered sources
sources = []
for i, res in enumerate(results, start=1):
page = requests.post(f"{LSC}/fetch", headers=HEADERS, json={
"url": res["targetUrl"],
"format": "markdown",
})
body = page.json()["content"][:6000] # crude budget per source
sources.append(f"[{i}] {res['targetUrl']}\n{body}")
context = "\n\n---\n\n".join(sources)
# 3. synthesize with citations
prompt = (
"Answer the question using ONLY the numbered sources below. "
"Cite claims inline like [1] or [2][3]. If the sources do not "
f"contain the answer, say so.\n\nQuestion: {question}\n\n{context}"
)
return call_your_llm(prompt) # any chat completion API
Replace the crude [:6000] truncation with real chunk ranking when you outgrow it. The prompt's "ONLY the numbered sources" clause is doing serious work: without it, models blend in training-data knowledge and your citations stop meaning anything.
Stage 4: Synthesis That Cites Honestly
Three prompt rules that consistently improve citation quality:
- Number sources and require inline
[n]markers. Free-form "according to TechCrunch" citations drift. - Instruct the model to say "the sources don't cover this" rather than fill gaps. You want visible holes, not confident padding.
- Ask for the citation immediately after the claim it supports, not clustered at paragraph ends.
If you're curious how the commercial engines handle this, I broke down what happens when ChatGPT searches the web, and the pattern is recognizably the same pipeline.
The Pitfalls That Will Actually Bite You
Citation drift
The model cites [2] for a claim that lives in source [4], or worse, for a claim in neither. Mitigations: fewer, cleaner sources beat many noisy ones; markdown beats HTML; and a cheap verification pass ("does source [2] support this sentence?") catches most of the rest. Budget for this. It's the number one complaint users have about answer engines.
Freshness
Search indexes lag. For "what happened today" questions, bias toward news-heavy queries and check publish dates from the fetched content, then tell the model each source's date. An answer engine that confidently summarizes last year's pricing page feels broken even when the pipeline worked perfectly.
Token budgets
Six full pages can easily exceed 50k tokens. Decide your budget per answer up front (I aim for 8-15k tokens of context), and spend it on ranked chunks rather than page prefixes. The first 6000 characters of a page are often navigation, cookie banners, and author bios, which is another reason clean markdown extraction matters.
Query-shaped failures
Some questions need two searches (comparisons, "X vs Y"), and some need zero (math, definitions). Route them. A single search call per question is a decent default and a poor ceiling.
What This Costs to Run
Per answer, the basic version is: one search call, a fetch call per source you keep, one small rewrite call, one synthesis call. The web-data side of that is a handful of credits on link.sc's pricing, and the free tier's 500 monthly credits are plenty to build and demo the whole thing before paying anyone.
The LLM synthesis call usually dominates cost, which is one more argument for aggressive chunk ranking: every junk token you cut is money.
Where to Take It Next
Once the basic loop works, the upgrades that matter most, in order: a reranker for chunk selection, streaming the answer token-by-token, a follow-up question loop that reuses fetched sources, and caching fetched pages for an hour so repeat questions are near-free.
None of these change the architecture. Search, fetch, rank, synthesize. Get that loop solid and the rest is polish.
Build your own answer engine this weekend: sign up for link.sc and get 500 free credits a month for search and fetch.