Quick answer: Web scraping for machine learning means assembling training datasets (features, labels, and text or image corpora) from public web content. The hard part is not collection, it is quality: deduplication, label integrity, licensing, and PII removal decide whether the dataset helps or poisons your model. Below is how to build ML datasets from the web that are clean, legal, and fresh, with a focus on general supervised and structured ML rather than LLM pretraining.
Every ML engineer eventually learns the same lesson: the model is rarely the bottleneck, the data is. A mediocre model on a clean, well-labeled dataset beats a great model on a noisy one. So the web is attractive because it is enormous, and dangerous for exactly the same reason. Let me walk through building a dataset that helps.
What You Are Actually Collecting
ML datasets from the web usually fall into three shapes, and each has different collection and quality concerns.
| Dataset type | Example | Main quality risk |
|---|---|---|
| Structured features | Product attributes, prices, specs, ratings | Missing and inconsistent fields |
| Labeled examples | Category tags, star ratings as labels | Label noise and leakage |
| Text or image corpora | Articles, reviews, product photos | Duplication, licensing, PII |
Being clear about which shape you need changes everything downstream. A features table wants schema consistency. A labeled set wants label integrity. A corpus wants scale plus deduplication and licensing hygiene.
Structured features and labels
A lot of practical ML is tabular: predict demand from product attributes, classify listings, forecast from public signals. The web is a rich source of both features and, often, free labels. A product's own category is a label. A star rating is a label. The catch is that these "free" labels are noisy and can leak. If your label is derived from a field that also appears in your features, your model will look brilliant in training and fail in production.
Text and image corpora
For NLP and vision, you collect raw content: articles, reviews, captions, images. Scale matters here, but scale without deduplication and licensing discipline creates a dataset you cannot ship. This overlaps with, but is distinct from, gathering data specifically for language-model training, which I cover separately in collect LLM training data from the web. This post stays on general ML: classifiers, recommenders, forecasters, and vision models.
The Collection Pipeline
The pipeline mirrors any scraping project, with quality gates bolted onto the end because in ML the gates are where the value is.
1. Source -> pick sites with the signal you need
2. Fetch -> render JS, absorb blocking
3. Extract -> structured fields or clean text/images
4. Clean + dedup -> the step that decides dataset quality
5. Label + split -> integrity checks, no leakage across splits
6. Refresh -> keep it current as the world drifts
The fetch and extract steps are the easy part. You request a page and get clean content, and for structured extraction you pass a schema so fields come out consistently across differently built sites.
curl https://api.link.sc/v1/fetch \
-H "x-api-key: lsc_your_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://shop.example.com/products/model-x",
"format": "markdown",
"render_js": true
}'
The response is { "content": "..." }, which you parse into features or feed to an extractor. Because link.sc handles rendering and blocking, your collection code stays about the data rather than about proxy rotation, which matters a lot when you are pulling tens of thousands of pages.
Quality: Where Datasets Live or Die
This is the section that separates a dataset that helps from one that quietly poisons your model.
Deduplication
Web data is full of near-duplicates: the same article syndicated across sites, the same product listed by five resellers, the same image reposted. Duplicates inflate your dataset size while teaching the model nothing new, and worse, they leak across your train and test splits so your evaluation lies to you. Deduplicate before you split, using content hashing for exact matches and similarity (for example MinHash or embedding distance) for near-duplicates.
Label integrity
If your labels come from the page, audit them. Are star ratings consistent across sites, or does one site's four stars mean another's three. Does the label field leak into a feature. Hold out a hand-checked sample to measure label noise, and do not trust a label source you have not spot-checked.
Missing and inconsistent fields
Scraped structured data is ragged. One site lists weight in kilograms, another in pounds, a third not at all. Normalize units, decide how you handle missing values before training rather than during, and log field-level completeness so you know which features you can actually rely on. The general technique of getting consistent structured output is worth reading up on in extract structured JSON from any webpage.
Licensing and PII: The Non-Negotiables
A dataset you cannot legally use is worse than no dataset, because you may not find out until after you have trained and shipped on it.
- Content licensing. Text and images are copyrighted by default. A model trained on scraped content raises licensing questions that depend on jurisdiction, use, and how the content ends up in the model. Prefer content with clear licenses (public domain, Creative Commons, or your own data), and record the license and source for every item so you can prove provenance later.
- PII removal. Public web content is full of personal data: names, emails, faces, handles. Under GDPR and similar laws, that data is regulated even though it is public. Filter and redact PII before it enters a training set, and be able to delete a person's data if you are required to.
- Terms of service and robots.txt. Respect them. Some sites prohibit automated collection or reuse of their content. A dataset built by violating terms is a liability sitting in your pipeline.
- Rate limits. Crawl politely. You gain nothing from knocking a source offline, and aggressive crawling gets you blocked before your dataset is complete.
None of this is legal advice. If a dataset feeds a production or commercial model, get licensing and privacy reviewed before training, not after. The broader legal picture is in is web scraping legal.
Keeping It Fresh
A dataset scraped once starts rotting immediately, because the world it describes keeps moving. Prices change, catalogs turn over, language drifts. This is data drift, and it degrades a model quietly. Build refresh into the pipeline: re-scrape on a schedule, track how much the distribution has shifted, and retrain when drift crosses a threshold you set. A fetch API makes scheduled refresh cheap, since the collection code you already wrote just runs again. link.sc offers 500 free credits a month and paid plans from $19, with pricing at link.sc/pricing, which covers a prototype dataset before you scale.
The summary for ML: collection is the easy 20 percent, and cleaning, deduplication, label integrity, licensing, PII, and freshness are the 80 percent that decide whether your dataset makes the model better or worse.
Building a training dataset from the web? link.sc fetches and extracts clean, structured content at scale so you can spend your time on data quality. Get 500 free credits a month.