How to Automate Web Scraping: Scheduling, Retries, and Monitoring

Quick answer: Automating web scraping means three things: running your scraper on a schedule (cron, GitHub Actions, or serverless), making each run safe to repeat (idempotent writes, retries with backoff), and knowing when it breaks (alerting on failures and on suspiciously empty results). The scraping code is maybe 20 percent of a production pipeline. This post covers the other 80 percent.

A scraper that runs once on your laptop is a script. A scraper that runs every hour for six months without you thinking about it is a system. The gap between the two is where most scraping projects die, so let's close it properly.

Pick a Scheduler: Cron, GitHub Actions, or Serverless

You have three realistic options, and the right one depends on where your scraper already lives.

Option	Best for	Cost	Catch
Cron on a VPS	Long-running jobs, databases on the same box	~$5/mo	You maintain the box
GitHub Actions	Small jobs, free tier, results committed to the repo	Free for public repos	6-hour job limit, shared runner IPs get blocked often
Serverless (Lambda, Cloudflare Workers)	Bursty, event-driven fetching	Pennies	Timeouts, cold starts, no local disk

For most people I'd say: start with GitHub Actions because it's free and versioned, then graduate to a VPS with cron when you outgrow it. Here's a scheduled workflow that runs a scraper every 6 hours and commits the data:

# .github/workflows/scrape.yml
name: scrape
on:
  schedule:
    - cron: "0 */6 * * *"
  workflow_dispatch: {}

jobs:
  scrape:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - run: python scrape.py
        env:
          LINKSC_API_KEY: ${{ secrets.LINKSC_API_KEY }}
      - run: |
          git config user.name "scraper-bot"
          git config user.email "[email protected]"
          git add data/ && git diff --cached --quiet || git commit -m "data: $(date -u +%F)"
          git push

One honest warning about Actions: shared runner IPs are datacenter IPs that thousands of other scrapers also use, so sites block them constantly. If your target site works from your laptop but fails in CI, that's why. Routing the fetch through an API with its own unblocking (more on that below) fixes it.

Make Every Run Idempotent

Schedulers retry, machines reboot, and you will rerun jobs by hand. If a rerun duplicates rows or double-sends alerts, your pipeline is broken even when the scraper works. Two rules:

1. Every scraped item needs a stable key (URL, SKU, listing ID).
2. Writes are upserts, never blind inserts.

def upsert_item(conn, item):
    conn.execute("""
        INSERT INTO items (item_key, title, price, scraped_at)
        VALUES (:key, :title, :price, CURRENT_TIMESTAMP)
        ON CONFLICT(item_key) DO UPDATE SET
            title = excluded.title,
            price = excluded.price,
            scraped_at = excluded.scraped_at
    """, item)

Keep the raw response too (HTML or markdown, compressed, keyed by URL and timestamp). Disk is cheap, and when your parser has a bug you can re-parse history instead of re-scraping it.

Retries and Backoff, Done Right

Networks flake and sites rate-limit. Naive retries make both worse. The pattern that works: retry only retryable failures (timeouts, 429, 5xx), back off exponentially, add jitter, and cap attempts.

import random, time, requests

RETRYABLE = {429, 500, 502, 503, 504}

def fetch_with_retry(url, api_key, attempts=4):
    for i in range(attempts):
        try:
            r = requests.post(
                "https://link.sc/v1/fetch",
                headers={"Authorization": f"Bearer {api_key}"},
                json={"url": url, "format": "markdown"},
                timeout=60,
            )
            if r.status_code == 200:
                return r.json()
            if r.status_code not in RETRYABLE:
                raise RuntimeError(f"permanent failure {r.status_code} for {url}")
        except requests.Timeout:
            pass
        time.sleep((2 ** i) + random.uniform(0, 1))
    raise RuntimeError(f"gave up on {url} after {attempts} attempts")

Do not retry 404s or 403s in a loop. A 404 is an answer, and a 403 repeated five times in ten seconds is how you get your IP banned harder.

Monitor the Thing, Because It Will Break

Scrapers fail in two ways: loudly (exceptions, non-zero exit codes) and silently (the run "succeeds" but returns 3 items instead of 300 because the site changed its layout). You need alerts for both.

Loud failures are easy: exit non-zero and let the scheduler notify you. GitHub Actions emails you on failed runs by default; on a VPS, pipe cron output to a webhook.

Silent failures need sanity checks on the output:

def sanity_check(items, min_expected=50):
    assert len(items) >= min_expected, f"only {len(items)} items, expected >= {min_expected}"
    missing_price = sum(1 for i in items if i.get("price") is None)
    assert missing_price / len(items) < 0.1, "over 10% of items missing price"

If either assertion fires, fail the run. A scraper that quietly writes garbage for three weeks is worse than one that crashes on day one. If your pipeline's real goal is detecting changes on specific pages, that's its own discipline; I wrote up the full approach in monitor web page changes and get alerts.

Keep Selectors From Rotting

Layout changes are the number one maintenance cost in scraping. You can't prevent them, but you can soften them:

• Prefer semantic anchors: data-* attributes, ARIA roles, and JSON-LD blocks survive redesigns far better than div.col-md-3 > span:nth-child(2).
• Check for a JSON API first. If the page fetches its data from an endpoint, scrape the endpoint; JSON schemas change much less often than HTML.
• Scrape markdown, not HTML. If you fetch pages as clean markdown, small DOM reshuffles often don't change your extracted text at all, and an LLM extraction step on top of markdown is nearly layout-proof.

That last point is the direction the industry is moving: fetch clean content, then extract fields with a model instead of selectors. It costs more per page but the maintenance cost drops to near zero. For background on structured extraction generally, see what is data extraction.

When to Stop Self-Hosting the Fetch Layer

Everything above assumes you can actually get the pages. In practice, automated pipelines fail most often at the fetch step: datacenter IPs get blocked, JavaScript-heavy pages return empty shells, CAPTCHAs appear. You can build proxy rotation and headless browsers yourself (I covered what that takes in the self-hosted web scraping guide), but it's a part-time job.

The pragmatic split: keep your scheduler, storage, and monitoring, and delegate fetching to an API that handles rendering and blocking. With link.sc that's one HTTP call per page (as in the retry example above), it returns clean markdown or structured JSON, and pricing is credit-based so an hourly job costs a predictable amount. Your cron job stays five lines long forever.

A Minimal Production Pipeline, End to End

Putting it together, a pipeline I'd actually run:

1. GitHub Actions on a 6-hour cron, with workflow_dispatch for manual reruns.
2. Fetch each URL through an API with retries and jittered backoff.
3. Upsert into SQLite (or Postgres once you pass a few million rows), keep raw markdown alongside.
4. Sanity-check counts and field coverage; fail loudly if they regress.
5. Alert on failed runs via the scheduler's built-in notifications, plus a weekly "still alive" digest so silence never means uncertainty.

None of this is glamorous, and that's the point. Automation is what turns scraping from a demo into data you can trust.

---

Want the fetch layer handled for you? Sign up for link.sc and get 500 free credits a month: one API call fetches any URL as clean markdown, rendering and unblocking included.