Quick answer: Collecting LLM training data from the web is a five-stage pipeline: define the distribution of examples you want, discover source URLs (search queries plus sitemaps), fetch every page to clean markdown, deduplicate and quality-filter the corpus, and format it as JSONL for your fine-tuning framework. The fetching is the easy part. The parts that determine whether your fine-tune works are the distribution definition and the filtering.
Before the pipeline, though, one honest question you should answer first.
First: are you sure you need fine-tuning?
Most teams reaching for fine-tuning actually want retrieval. If your goal is "the model should know our domain's facts," RAG gets you there faster, cheaper, and with updatable knowledge: fetch relevant pages at query time and put them in the context window. Facts baked in by fine-tuning go stale the day you train.
Fine-tuning earns its cost when you want to change behavior, not knowledge: a consistent output format, a specific tone, a specialized task the base model does clumsily, or shorter prompts by baking in instructions.
If you are unsure, build the RAG version first. It doubles as a test: if good context in the window does not fix your problem, fine-tuning on similar data probably will not either. Everything below assumes you have passed this checkpoint.
Define the target distribution before you collect anything
A fine-tune learns the distribution of its training set, including the parts you did not intend. Decide up front:
- Task shape. Instruction-response pairs? Documents for continued pre-training? Conversations?
- Domain mix. If you want 70 percent medical device docs and 30 percent regulatory text, write that down and measure against it. Whatever ratio you collect is the ratio the model learns.
- Quality bar. One thousand excellent examples beat fifty thousand mediocre ones for most behavioral fine-tunes. Decide what "excellent" means concretely: length range, structure, language, no boilerplate.
Write this as a one-page spec. Every later stage is a filter that either serves this spec or does not.
Source discovery: search plus sitemaps
You need URLs before you can fetch. Two complementary approaches:
Search for coverage. Enumerate the subtopics in your spec and turn each into queries. A search API that returns full page content lets you evaluate a source's quality in the same call that discovers it:
import requests
def discover(query: str) -> list[dict]:
r = requests.post(
"https://link.sc/v1/search",
headers={"Authorization": "Bearer lsc_your_api_key"},
json={"q": query},
timeout=60,
)
return r.json()["results"] # title, url, full markdown content
urls = {res["url"] for q in my_subtopic_queries for res in discover(q)}
Sitemaps for depth. Once search surfaces a high-quality domain, do not sample it one result at a time. Grab https://thesite.com/sitemap.xml, parse the URL list, and filter by path patterns (/docs/, /guides/) to get the whole section. A handful of great domains crawled deep usually beats hundreds of domains sampled shallow.
Keep the URL list as a file with a source column. When a domain turns out to be junk later, you want to delete its contribution in one line.
Fetch everything to markdown
Raw HTML is a terrible training format: nav bars, cookie banners, and footers become patterns your model learns to emit. Fetch straight to markdown so boilerplate is stripped before it enters the corpus:
def fetch_md(url: str) -> str | None:
r = requests.post(
"https://link.sc/v1/fetch",
headers={"Authorization": "Bearer lsc_your_api_key"},
json={"url": url, "format": "markdown"},
timeout=60,
)
if r.status_code != 200:
return None # log and move on; never train on error pages
return r.json()["content"]
Using a fetch API here rather than raw requests matters more than it does for casual scraping, because training corpora need volume across many domains, and that is exactly where JavaScript rendering and bot walls eat DIY pipelines. link.sc handles both behind the endpoint; the quickstart covers batching patterns. The conversion quality question (what good markdown extraction preserves and drops) is its own topic: see HTML to markdown for LLMs.
Store the markdown with its URL and fetch date. Provenance is not optional; compliance (below) depends on it.
Dedupe, then filter for quality
Web data is massively redundant: syndicated articles, scraped copies, templated pages. Duplicates make the model overweight whatever got copied most.
Exact dedup is a hash set over normalized text. Do it first; it is free.
Near-dedup is where MinHash earns its reputation. The idea: shingle each document into overlapping n-grams, hash the shingles, and keep a small signature whose collision probability approximates Jaccard similarity. Documents whose signatures collide in enough bands are near-duplicates; keep one per cluster. The datasketch library implements this (MinHash plus MinHashLSH) and handles corpora of hundreds of thousands of documents on a laptop.
Then filter what remains. Cheap heuristics catch most junk:
- Length bounds (drop stubs and 200k-character dumps)
- Language ID matches your target language
- Symbol-to-word and repetition ratios (catches menus, logs, spam)
- Boilerplate phrases ("accept cookies", "404", "please enable JavaScript")
For the final cut, an LLM-as-judge pass scoring each document against your quality spec is affordable at fine-tuning scale and catches what heuristics cannot, like factually thin listicles that are statistically normal.
Licensing, copyright, and PII
This section is not optional reading.
Terms of service. Sites state scraping and reuse terms. Training a model is a heavier use than reading, and some ToS address ML training explicitly now. The general legal landscape for scraping is genuinely unsettled (I walk through the case law in is web scraping legal), and training use adds copyright questions on top: several lawsuits over exactly this are still working through courts as of mid-2026. Prefer permissively licensed sources (government publications, Creative Commons, open documentation) where you can, and get a lawyer's opinion for anything commercial.
Respect robots.txt and rate limits. Beyond ethics, a corpus built by hammering sites is a corpus someone may make you delete.
PII scrubbing. Names, emails, phone numbers, and addresses in training data can be memorized and regurgitated by the model. Run a PII detection pass (Microsoft's Presidio is a solid open-source option) and drop or mask hits before formatting. Under GDPR, personal data in a training set is still personal data.
Keep the provenance log from the fetch stage. If a source objects later, you can identify and remove their data and retrain, which is a much better conversation than "we cannot tell what is in the corpus."
Format as JSONL and version it
Every mainstream fine-tuning stack eats JSONL, one example per line. For instruction tuning in chat format:
{"messages": [{"role": "system", "content": "You are a medical device regulatory assistant."}, {"role": "user", "content": "What does IEC 62304 class B require for software verification?"}, {"role": "assistant", "content": "IEC 62304 class B requires..."}]}
Getting from cleaned documents to pairs usually means generating questions per document with an LLM and grounding the answers in the document text, then spot-checking a sample by hand. Version the resulting file like code: hash it, tag it, and record which model checkpoint trained on which corpus version. When quality regresses (it will), the diff between corpus versions is where you look first.
Then run the smallest fine-tune your framework supports, evaluate against a held-out set, and only scale the corpus when the small run shows the behavior moving in the right direction. Data work compounds; wasted training runs do not.
Building a training corpus from the web? Get a free link.sc key and turn any URL into clean, training-ready markdown.