Quick answer: Reranking is a second scoring pass in a RAG pipeline. Your retriever pulls back a broad set of candidate chunks quickly, then a reranker reads each candidate alongside the query and reorders them by actual relevance. You keep the top few and drop the rest. The retriever optimizes for recall (do not miss anything); the reranker optimizes for precision (put the best answer first).
If your RAG system retrieves the right documents but the LLM still gives mediocre answers, reranking is usually the cheapest fix available. It does not require re-embedding your corpus, changing your vector database, or retraining anything.
Why retrieval alone is not enough
Most RAG systems retrieve with a bi-encoder: the query and every document are embedded into vectors separately, and you find the nearest neighbors by cosine similarity. This is fast because you embed all your documents once, ahead of time, and only embed the query at request time. (If vectors are new to you, see what are vector embeddings.)
The problem is that a bi-encoder never sees the query and the document together. It compresses each into a fixed vector and hopes the geometry lines up. That works well enough to get relevant documents into the top 50 or top 100. It works less well at getting the single best document into position one.
Consider a query like "how do I cancel a subscription after the trial ends." A bi-encoder might rank a general "subscription management" page above the specific "canceling during or after your free trial" page, because the general page mentions "subscription" more often. The specific page is the better answer, but the vector similarity does not know that.
What a reranker actually does
A reranker takes the query and one candidate document as a single input and produces one relevance score. It does this for every candidate, then you sort by that score.
Because the reranker reads the query and document jointly, it can catch relationships a bi-encoder misses: negation, specificity, whether the document actually answers the question versus merely mentioning the topic.
Here is the two-stage pattern:
Query
│
▼
[Retriever / bi-encoder] → top 50 candidates (fast, high recall)
│
▼
[Reranker / cross-encoder] → rescore all 50 (slower, high precision)
│
▼
Top 5 → LLM context window
The retriever casts a wide net. The reranker is the judge that reads each catch carefully. You only pay the expensive judging cost on 50 documents, not your whole corpus.
Cross-encoders vs bi-encoders
The distinction between the two model types is the core idea behind reranking.
| Bi-encoder (retriever) | Cross-encoder (reranker) | |
|---|---|---|
| Input | Query and document encoded separately | Query and document encoded together |
| Output | A vector per item | A single relevance score per pair |
| Precomputation | Document vectors cached ahead of time | Nothing can be cached; scored at query time |
| Speed | Very fast over millions of docs | Slow; practical only over tens of candidates |
| Relevance quality | Good | Better |
You cannot use a cross-encoder as your primary retriever, because scoring the query against every document in a large corpus at request time would be far too slow. And you would not want a bi-encoder as your final ranker, because it leaves precision on the table. The two-stage design uses each where it is strongest.
When reranking is worth it
Reranking adds latency and a second model call, so it is not free. It earns its place when:
- Your top-k is noisy. You retrieve 20 chunks, and the right answer is often somewhere in there but not at the top.
- You feed few chunks to the LLM. If you only pass 3 to 5 chunks into the prompt to save tokens, getting the best 3 matters enormously. Reranking pairs well with token optimization.
- Your queries are specific. Narrow, intent-heavy questions are exactly where bi-encoder ranking degrades and reranking helps most.
Reranking helps less when your retriever already nails position one, or when you dump 30 chunks into a long context window and let the LLM sort it out. In that case you are paying for a reorder the model does not need.
A code sketch
Here is the shape of a retrieve-then-rerank flow. The retrieval step returns candidates; the rerank step scores and sorts them.
# Stage 1: fast retrieval from your vector store
candidates = vector_store.search(query, top_k=50)
# Stage 2: rerank the candidates with a cross-encoder
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [(query, c.text) for c in candidates]
scores = reranker.predict(pairs)
ranked = sorted(
zip(candidates, scores),
key=lambda pair: pair[1],
reverse=True,
)
top_chunks = [c.text for c, score in ranked[:5]]
context = "\n\n".join(top_chunks)
# Pass `context` into your LLM prompt
The retriever hands over 50 candidates. The cross-encoder rescores all 50. You keep the best 5 for the prompt. Everything else is discarded before it ever costs you a token.
Where clean input data fits in
Reranking can only reorder what retrieval found, and retrieval can only find what you indexed. If your source documents were scraped as messy HTML full of navigation menus, cookie banners, and boilerplate, both stages suffer: your embeddings are polluted and your reranker wastes its scoring budget on junk.
That is why clean extraction is upstream of everything. When you build a corpus from web pages, fetching them as clean markdown gives your chunker, embedder, and reranker better material to work with.
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/docs/billing", "format": "markdown"}'
The response is the article content as markdown, without the surrounding page furniture. Feed that into your chunker, embed the chunks, retrieve, rerank, and the whole pipeline improves because the raw material is clean. You can read more about how link.sc handles rendering and parsing in the docs.
The bottom line
Reranking is the precision layer of a RAG system. Retrieval gives you recall over a large corpus; reranking gives you the ordering that actually helps the LLM answer. If your retrieval is broadly right but the final answers feel off, add a reranker before you touch anything else in the stack. For the wider picture of how these pieces fit together, see what is RAG.
Building a RAG pipeline that needs clean web data as its source? Start free with link.sc and fetch any URL as structured markdown.