Most "web scraping AI agent" tutorials stop at the demo moment: the model calls a fetch tool, gets some markdown back, and everyone claps. Then you try to use it for real work, like building a dataset of 200 SaaS pricing pages, and it falls apart. The agent fetches three pages, hallucinates a fourth, returns half the fields empty, and declares victory.
The difference between a demo and a working scraping agent is not the model. It is the loop around the model. In this post I will build that loop end to end: plan, search, fetch, extract, validate, and only then write to the dataset.
What a scraping agent actually is
A traditional scraper is a script: hardcoded URLs, hardcoded CSS selectors, brittle by design. A scraping agent replaces both hardcoded parts with decisions.
- It finds its own targets by searching, instead of working from a fixed URL list.
- It extracts by reading the page content, instead of matching selectors that break on the next redesign.
That flexibility is the whole point, and also the whole risk. A script fails loudly when a selector breaks. An agent fails quietly by making things up. So the architecture has one job: let the model do the flexible parts (planning, extraction) while deterministic code does the strict parts (fetching, validation, storage).
Here is the loop I use:
- Plan: turn the goal into concrete search queries.
- Search: find candidate URLs.
- Fetch: pull each page as clean markdown.
- Extract: have the model fill a fixed JSON schema from that markdown only.
- Validate: check every record in code, and reject or retry what fails.
Steps 1, 2, and 4 are LLM calls. Steps 3 and 5 are plain code. Keep that boundary and most agent horror stories disappear.
Step 1 and 2: plan and search
Say the goal is "collect pricing for popular uptime monitoring tools." Do not let the agent freestyle. Ask for a plan as structured output:
import anthropic, json, requests
client = anthropic.Anthropic()
LSC = {"x-api-key": "lsc_..."}
plan = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=500,
messages=[{"role": "user", "content":
"Goal: collect pricing data for popular uptime monitoring tools. "
"Return JSON only: {\"queries\": [..up to 5 search queries..]}"}],
)
queries = json.loads(plan.content[0].text)["queries"]
Then run each query through a search API and collect candidate URLs:
candidates = set()
for q in queries:
r = requests.get("https://api.link.sc/v1/search",
params={"q": q}, headers=LSC).json()
for item in r["results"]:
candidates.add(item["url"])
I use the link.sc search endpoint here because it returns results already shaped for LLM consumption, but the pattern works with any search API. The important part is that the code owns the candidate list. The model proposed queries; it does not get to invent URLs.
Step 3: fetch pages as markdown, not HTML
Raw HTML is the wrong input for extraction. A typical pricing page is 300 to 600 KB of markup, and maybe 3 KB of it is actual pricing content. Feeding HTML to the model burns tokens, and worse, it buries the signal. In my experience extraction accuracy drops noticeably when the content is wrapped in navigation, cookie banners, and script tags.
So fetch clean markdown:
def fetch_markdown(url):
r = requests.get("https://api.link.sc/v1/fetch",
params={"url": url}, headers=LSC)
r.raise_for_status()
return r.json()["content"]
This also handles the two problems that kill DIY fetchers: JavaScript rendering (many pricing pages are empty shells without it) and bot walls on datacenter IPs. If you want to solve those yourself, that is a project of its own; I wrote about the tradeoffs in how to give an AI agent internet access.
Step 4: extract into a fixed schema
This is where most agents go wrong. If you ask "summarize the pricing on this page," you get prose. Prose is not a dataset. Define the schema once, in code, and force every extraction through it:
SCHEMA_PROMPT = """Extract from the page content below. Return JSON only:
{
"product": string,
"plans": [{"name": string, "monthly_usd": number or null,
"annual_usd": number or null}],
"free_tier": boolean,
"source_confidence": "high" | "low"
}
Rules: use ONLY the page content. If a field is not on the page,
use null. If the page is not a pricing page, set plans to []."""
def extract(markdown):
msg = client.messages.create(
model="claude-sonnet-4-5", max_tokens=1000,
messages=[{"role": "user",
"content": SCHEMA_PROMPT + "\n\n" + markdown[:30000]}],
)
return json.loads(msg.content[0].text)
Two details matter more than they look. First, "use ONLY the page content" plus an explicit null policy is your main defense against the model backfilling from training data. Second, source_confidence gives the model a sanctioned way to say "I am not sure," which beats forcing a confident guess.
Step 5: validate like the model is lying to you
Because sometimes it is. Every record goes through deterministic checks before it touches the dataset:
def valid(rec, markdown):
if not rec.get("plans"):
return False
for p in rec["plans"]:
m = p.get("monthly_usd")
if m is not None:
if not (0 <= m <= 10000):
return False
# the number must literally appear in the source
if str(int(m)) not in markdown and str(m) not in markdown:
return False
return rec.get("source_confidence") == "high"
That grounding check, verifying the extracted price actually appears in the fetched text, is the single highest-value line in the whole agent. It catches hallucinated numbers, currency confusion, and stale training-data prices in one move.
Records that fail get one retry with the failure reason appended to the prompt. Records that fail twice get logged and skipped. Do not let the agent negotiate its way past validation.
Wiring the loop together
dataset = []
for url in candidates:
try:
md = fetch_markdown(url)
rec = extract(md)
if valid(rec, md):
rec["source_url"] = url
dataset.append(rec)
except Exception as e:
print(f"skip {url}: {e}")
with open("pricing.json", "w") as f:
json.dump(dataset, f, indent=2)
Notice what the model never controls: which URLs get fetched, what counts as valid, and what gets written to disk. The agent is autonomous about judgment (queries, extraction) and constrained about actions. That split is the framework I would tattoo on every agent tutorial.
Honest limitations
A few things this design will not save you from. Sites you are not permitted to scrape are still off limits; check terms and robots directives, and read up on whether web scraping is legal for your use case. Extraction on genuinely ambiguous pages (usage-based pricing, "contact us" tiers) will produce nulls, which is correct behavior, not a bug. And at hundreds of pages you will want concurrency and caching, which I left out to keep the loop readable.
But the core holds at any scale: let the model plan and read, let the code fetch and verify. Build it that way and your scraping agent stops being a demo and starts producing datasets you can actually trust.
Get search and fetch APIs built for agents, with 500 free requests a month, at link.sc/register.