Web Scraping With n8n: Build a Scraping Workflow Step by Step

Quick answer: To do web scraping with n8n, build a workflow that starts with a Schedule Trigger, uses an HTTP Request node to fetch a page (or call a fetch API for clean content), parses the result with a Code or HTML node, and writes it to storage or an alert. n8n fits when you want automation without maintaining a codebase; you graduate to real code when logic or scale outgrows the visual editor.

n8n is a workflow automation tool where you wire nodes together instead of writing a full program. It is a strong middle ground for scraping: more powerful than a spreadsheet, less overhead than a standalone service, and self-hostable if you care about that. Here is how to actually build a scraping workflow in it.

Why n8n for Scraping

n8n gives you three things that make scraping practical without code:

• Scheduling built in, so a workflow runs itself on a cron-like cadence.
• HTTP requests as a first-class node, so you can hit any URL or API.
• Connectors for hundreds of destinations (Sheets, databases, Slack, email), so the "what do I do with the data" step is a node, not a script.

It is a good fit when you want a scraping or monitoring job that a non-engineer can see and adjust, and when the logic is mostly "fetch, parse, store, notify" rather than something branching and stateful.

The Anatomy of a Scraping Workflow

Every scraping workflow in n8n follows the same shape. Four stages, one node (or a few) per stage.

Stage	Node	Job
Trigger	Schedule Trigger	Decide when the workflow runs
Fetch	HTTP Request	Get the page content
Parse	Code / HTML / Set	Turn raw content into fields
Store or alert	Sheets, Postgres, Slack, Email	Do something with the result

Let me walk through each.

Step 1: The Schedule Trigger

Drop a Schedule Trigger node as the workflow's entry point. It has an interval setting (every X minutes/hours) or a cron expression for precise timing. For most monitoring, an interval of an hour is plenty. For a one-off scrape you can swap it for a Manual Trigger while you build, then switch to the schedule when it works.

Pick a cadence you can actually act on and that respects the target site. Polling a page every 30 seconds is both wasteful and a good way to get blocked.

Step 2: The HTTP Request Node (the fetch)

This is the heart of the workflow. The HTTP Request node makes the call that retrieves your data. You have two ways to use it.

Option A: Fetch the page directly

Set the method to GET and the URL to your target. n8n returns the raw HTML. This works for simple, static, unprotected pages, and it hits the same walls those pages always hit: JavaScript-rendered content comes back empty, and protected sites return a block page. You also have to parse messy HTML yourself in the next step.

Option B: Call a fetch API for clean content

For anything real, point the HTTP Request node at a fetch API that handles rendering, proxies, and parsing, then returns clean markdown. Configure the node like this:

• Method: POST
• URL: https://api.link.sc/v1/fetch
• Authentication: Generic Credential, Header Auth, header name x-api-key, value your lsc_ key (stored as an n8n credential, not pasted in the node)
• Send Body: on, Body Content Type: JSON
• Body:

{
  "url": "https://example.com/pricing",
  "format": "markdown"
}

The node now returns { "content": "..." } with clean markdown regardless of how the page was rendered, because link.sc does the heavy lifting server-side. Storing the key as an n8n credential keeps it out of the workflow JSON, which matters if you export or share the workflow.

To collect by topic instead of a fixed URL, point a second HTTP Request node at the search endpoint with a POST body of { "q": "your query", "engine": "google" }. The search field is q.

Step 3: Parse the Result

Now you have content and need fields. n8n gives you a few options depending on what you fetched.

If you used the clean-markdown route, parsing is mostly string work in a Code node:

// Code node: pull a price line out of clean markdown
const content = $input.first().json.content;
const match = content.match(/\$[0-9]+(?:\.[0-9]{2})?/);

return [{
  json: {
    url: $('HTTP Request').first().json.url,
    price: match ? match[0] : null,
    checkedAt: new Date().toISOString(),
  },
}];

If you fetched raw HTML instead, use the built-in HTML node (Extraction operation) to pull values by CSS selector, for example .product-price into a price field. The HTML node is convenient but breaks when the site's markup changes, which is the usual downside of selector-based parsing.

A Set node is handy right after parsing to rename and shape fields into exactly the columns your destination expects.

Step 4: Store or Alert

The last stage is where n8n earns its keep, because the destination is just another node.

• Google Sheets node to append a row per run.
• Postgres or MySQL node to insert into a table.
• Slack or Email node to notify when something changed.

For a monitor, add an IF node before the alert: compare the new value against the previous one (pulled from your store) and only send a notification when they differ. That turns a plain scraper into a change monitor. The full change-detection pattern (what to compare and how to avoid false alarms) is in our guide on monitoring web page changes and getting alerts.

Putting It Together

The finished workflow reads left to right:

Schedule Trigger
  -> HTTP Request (POST https://api.link.sc/v1/fetch)
  -> Code (extract fields from content)
  -> IF (changed since last run?)
       true  -> Slack (send alert) -> Postgres (save new value)
       false -> Postgres (update last-checked timestamp)

Each node is configured once, and the schedule runs the whole chain on its own from then on. You can watch executions in n8n's history to debug when something returns unexpected data.

When n8n Fits vs When to Use Code

n8n is the right call when:

• The logic is a mostly linear pipeline (fetch, parse, store, notify).
• You want a visual workflow a teammate can read and tweak.
• You value built-in connectors over writing integration code.
• You are self-hosting and want automation you control.

Reach for real code when:

• Your logic gets deeply branching, stateful, or needs real testing.
• You are processing large volumes where per-node overhead adds up.
• You need version control, code review, and a proper deploy pipeline.
• You want to reuse scraping logic across many projects as a library.

The good news is the fetch call is portable. The POST to /v1/fetch with the x-api-key header is identical whether it lives in an n8n HTTP Request node or a Python script, so moving off n8n later does not mean rewriting your data access. For the broader picture of scheduling and running scrapers however you host them, see our guide on how to automate web scraping.

The Bottom Line

n8n turns "fetch, parse, store, alert" into four nodes and a schedule, and calling a fetch API from the HTTP Request node removes the two things that usually break no-code scraping: JavaScript rendering and anti-bot blocks. Start with a Manual Trigger to build, switch to the Schedule Trigger when it works, and only move to code when the workflow stops being mostly linear.

---

Building a scraping workflow in n8n? Grab a free link.sc key, 500 credits a month, and drop it into an HTTP Request node to get clean content from any page.