Quick answer: Give your agent a loop that fetches a page, extracts its links, scores each one for relevance to the goal, and follows only the top candidates within a fixed budget. Add depth limits, a visited set, and clear stop conditions so it never wanders forever or falls into a crawler trap. The whole thing runs on two calls: fetch a URL to markdown, and search when you need fresh candidates.
Letting an agent decide what to read next is where most people either get magic or get a runaway process that burns credits on a calendar widget's "next month" links. The difference is almost entirely in the control structure around the fetch, not the model.
Below is how I build that loop.
The Shape of an Autonomous Link-Following Loop
Every version of this I have shipped comes down to the same five parts:
- A frontier: the queue of URLs the agent might visit, each with a priority score.
- A budget: hard caps on total fetches, depth, and wall-clock time.
- A visited set: URLs and content hashes already seen, so you never fetch the same thing twice.
- A scorer: the logic that decides which links are worth following.
- A stop condition: what "done" means, so the loop exits on purpose instead of on exhaustion.
Miss any one of these and the agent misbehaves. Skip the budget and it never stops. Skip the visited set and it loops. Skip the scorer and it reads everything, which is slow and expensive.
Step One: Fetch a Page to Clean Markdown
You cannot reason about links buried in ad scripts and nav soup. Get the page as clean markdown first, with the links preserved as real markdown links.
curl -X POST https://api.link.sc/v1/fetch \
-H "x-api-key: lsc_..." \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/guide",
"format": "markdown"
}'
The response gives you the body as markdown in a content field, with outbound links preserved as standard markdown links. A short regex pulls them out, and that list is the raw material for the frontier. Because link.sc handles rendering and proxying, links that only appear after JavaScript runs are already in there, which matters for single-page apps.
Step Two: Score the Links Before You Follow Them
This is the part that separates a research agent from a dumb crawler. You do not follow links in document order. You follow the ones most likely to advance the goal.
A cheap, effective scorer combines a few signals:
| Signal | Why it helps | Cheap heuristic |
|---|---|---|
| Anchor text relevance | The link text usually describes the target | Keyword overlap with the goal, or an embedding similarity |
| URL path shape | Deep content paths beat utility pages | Downrank /login, /cart, /tag/, /?page= |
| Same vs. off domain | On-domain links stay on topic; off-domain widens scope | Small bonus for same domain early, allow off-domain later |
| Novelty | Avoid near-duplicates of pages you have | Skip if the URL or a normalized title is already visited |
For anchor-text relevance you can ask the model directly, but that costs a call per batch. I usually run a fast local pass (keyword and path heuristics) to cut the list to a shortlist, then let the model rank the shortlist. That keeps the expensive judgment for the decisions that matter.
import re
LINK_RE = re.compile(r"\[([^\]]*)\]\((https?://[^)\s]+)\)")
def extract_links(markdown):
return [{"text": t, "url": u} for t, u in LINK_RE.findall(markdown)]
TRAP_PATTERNS = re.compile(
r"/(login|signup|cart|checkout|tag|search|print)|[?&](page|sort|filter)=",
re.I,
)
def prescore(link, goal_terms):
url, text = link["url"], (link.get("text") or "")
if TRAP_PATTERNS.search(url):
return -1 # drop utility and pagination traps
overlap = sum(t in text.lower() for t in goal_terms)
depth_penalty = url.count("/") * 0.1
return overlap - depth_penalty
Anything scoring below zero never enters the frontier.
Step Three: Budget, Depth, and Stop Conditions
The budget is your safety rail. I set three limits and honor whichever trips first:
- Max fetches: total pages the run may read (for example 25).
- Max depth: how many link-hops from the seed (for example 3).
- Goal satisfied: the model reports it has enough to answer, so the loop exits early.
The last one is the good exit. After each fetch, ask the model a yes/no question: "Given what you have read, can you fully answer the goal, or is a specific piece still missing?" If it can answer, stop. If it names a missing piece, that becomes the next search query.
Step Four: When to Fetch a Link vs. Run a Fresh Search
Following links is depth-first exploration of what one page points to. But sometimes the page you are on does not link to what you need, and you should widen out with a search instead.
curl -X POST https://api.link.sc/v1/search \
-H "x-api-key: lsc_..." \
-H "Content-Type: application/json" \
-d '{
"q": "post-2023 pricing changes for the vendor",
"engine": "google"
}'
The rule I follow: use link-following when the current document is clearly on-topic and its links look promising, and fall back to search when the scorer's best candidate is weak or when the model names a gap the current page cannot fill. Search returns fresh candidates under serpData.results, each with a targetUrl; feed the good ones back into the same frontier.
Putting the Loop Together
def run_agent(seed_url, goal, goal_terms, max_fetches=25, max_depth=3):
frontier = [(seed_url, 0, 1.0)] # (url, depth, score)
visited = set()
context = []
while frontier and len(visited) < max_fetches:
frontier.sort(key=lambda x: x[2], reverse=True)
url, depth, _ = frontier.pop(0)
norm = url.split("#")[0].rstrip("/")
if norm in visited or depth > max_depth:
continue
visited.add(norm)
page = fetch_markdown(url) # POST link.sc /v1/fetch
context.append({"url": url, "text": page["content"]})
if model_can_answer(goal, context): # early stop
break
for link in extract_links(page["content"]):
s = prescore(link, goal_terms)
child = link["url"].split("#")[0].rstrip("/")
if s > 0 and child not in visited:
frontier.append((link["url"], depth + 1, s))
return synthesize(goal, context) # answer with citations
Notice what keeps it safe: the visited set uses a normalized URL so page#section and page/ collapse to one entry, depth is checked before every fetch, and the outer while cannot exceed max_fetches no matter how large the frontier grows.
Avoiding Loops and Traps
A few failure modes show up again and again:
- Infinite pagination:
?page=2,?page=3forever. Cap pagination depth or drop paginated URLs from scoring, as the trap regex above does. - Calendar and faceted-search traps: date pickers and filter combinations generate near-infinite unique URLs. Downrank query-string-heavy URLs hard.
- Redirect cycles: A links to B links back to A. The normalized visited set handles this, since you record the URL before following its links.
- Content duplication: many URLs, same body. Hash the fetched markdown and skip bodies you have already stored.
A Note on Being a Good Citizen
Autonomous following can hit a lot of pages quickly. Respect robots.txt, keep concurrency modest, add a small delay between fetches to the same host, and only follow public links. The goal is legitimate access to public content, not evasion of access controls or authentication. If a site asks crawlers to slow down, slow down.
If you want the broader picture of wiring an agent to live web data, see give your AI agent internet access. And once your agent is deciding what to read, the natural next step is combining that with search, covered in search results with full page content.
The takeaway: the model does not need to be smart about crawling. Your loop does. Give it a scored frontier, hard budgets, a visited set, and an honest stop condition, and a modest model will follow links like it knows exactly where it is going. Full request details are in the link.sc docs.
Ready to give your agent a fetch-and-follow loop that stays on the rails? Start free at link.sc.