What Is an HTTP Request? A Practical Guide for Scrapers and API Users

Quick answer: An HTTP request is a message your program sends to a web server asking it to do something. It has four parts: a request line (method plus URL plus protocol version), headers (metadata like User-Agent and content type), an optional body (the data you send, common in POST), and it triggers a response carrying a status code (200 for success, 404 for not found, 403 when you are blocked). Every scraper, browser, and API call is built on this. Understanding it makes debugging obvious instead of mysterious.

If you scrape the web or call APIs, HTTP is the ground you stand on. You do not need the RFC memorized, but you do need a working mental model, because when a request fails, the answer is almost always sitting in one of these four parts. Let me walk through them with examples you can run.

The four parts of a request

Every HTTP request looks roughly like this on the wire:

GET /articles/web-scraping HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (compatible; MyScraper/1.0)
Accept: text/html

Breaking that down:

1. Request line: GET /articles/web-scraping HTTP/1.1. The method (GET), the path, and the protocol version.
2. Headers: Host, User-Agent, Accept, and so on. Metadata about the request and what you can accept back.
3. Body: empty here. A GET usually has no body; a POST carries one.
4. It is sent to a server, which replies with a response and a status code.

That is the whole shape. Everything else is detail.

Methods: GET, POST, and friends

The method tells the server what kind of action you want.

Method	Purpose	Has a body?
GET	Retrieve a resource (fetch a page)	No
POST	Send data to create or process something	Yes
PUT	Replace a resource	Yes
PATCH	Partially update a resource	Yes
DELETE	Remove a resource	Sometimes
HEAD	Like GET but headers only, no body	No

For scraping you mostly live in GET (fetch a page) and POST (submit a form or call a JSON API). HEAD is useful when you only want to check whether a URL exists or what type it is without downloading the whole body.

Headers: the metadata that decides your fate

Headers are where most scraping problems live. A few that matter constantly:

• User-Agent: identifies the client. Default library User-Agents get filtered, so set a realistic one.
• Accept: what content types you can handle, like text/html or application/json.
• Content-Type: what you are sending in the body. application/json when you POST JSON.
• Authorization or an API-specific key header: how you authenticate.
• Cookie: session state carried between requests.

When a request that works in your browser fails in your script, the difference is nearly always headers. The browser sends dozens; your script sends three.

The body: data you send

GET requests carry no body. POST requests do, and it holds the data you are sending. For APIs that is usually JSON:

{"query": "web scraping", "limit": 10}

The server reads the body according to the Content-Type header, which is why forgetting to set Content-Type: application/json on a JSON POST produces confusing parse errors on the server side.

Status codes: reading the response

The response comes back with a three-digit status code. You will meet these constantly:

Code	Meaning	What to do
200	OK, here is your content	Parse it
301 / 302	Redirect to another URL	Follow it
400	Bad request, your input is malformed	Fix the body or params
401 / 403	Unauthorized or forbidden	Check auth, or you are blocked
404	Not found	The URL is wrong or gone
429	Too many requests	Slow down, back off
500 / 503	Server error	Retry later, not your fault

The one that trips up scrapers most is 403. It often does not mean your URL is wrong; it means the site's anti-bot system decided you are a bot. A 429 is the polite version, asking you to slow down. Honor it: back off, and keep your request rate reasonable. Scraping public data is fine; ignoring a server that is asking you to slow down is not.

Seeing a real request with curl

The fastest way to internalize this is to watch a request happen. curl's -v flag prints the request and response headers:

curl -v https://example.com

You will see lines starting with > (your request) and < (the server's response), including the status line and every header. This one command answers most "why is my scraper failing" questions.

To send a POST with a body and headers, which is what calling an API looks like:

curl -X POST https://api.example.com/search \
  -H "Content-Type: application/json" \
  -d '{"query":"web scraping"}'

Making the same request in code

Here is the identical GET in Python, so the mapping from concept to code is clear:

import requests

response = requests.get(
    "https://example.com",
    headers={"User-Agent": "Mozilla/5.0 (compatible; MyScraper/1.0)"},
    timeout=15,
)

print(response.status_code)  # the status code, e.g. 200
print(response.text)         # the response body

response.status_code is the number from the table above, response.text is the body, and the headers argument is exactly the metadata section we covered. Same four parts, just in Python.

How a fetch API wraps all of this

Here is the practical payoff. A fetch API takes the messy parts of making requests (rendering JavaScript, rotating proxies so you avoid 403s, converting HTML to clean text) and wraps them behind a single, well-behaved HTTP request that you make. You send one POST, it makes the hard requests for you, and you get clean content back.

The linksc fetch API is a good example. You make an ordinary POST with an x-api-key header (note: x-api-key, not Authorization: Bearer):

curl -X POST https://api.link.sc/v1/fetch \
  -H "x-api-key: lsc_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","format":"markdown"}'

The response is JSON shaped like {"content":"..."}. Behind that single request the API executed the rendering, the proxying, and the HTML-to-markdown conversion that would otherwise be your problem. That is the whole idea of a fetch API: it turns a hard scraping job into one clean HTTP request. If you want the deeper version, what is a web scraping API covers it, and web scraping vs API: which to use helps you decide when to reach for one.

The bottom line

An HTTP request is four parts (line, headers, body) plus a status code on the way back. Learn to read those, especially headers and status codes, and most scraping and API mysteries turn into obvious fixes. When the requests get hard to make yourself, a fetch API is just a friendly HTTP request that makes the hard ones on your behalf.

You can start on the free tier (500 credits per month) and see the details at link.sc/pricing.

---

Want to turn a hard fetch into one clean HTTP request? Get a free link.sc API key and call the fetch API in a single POST.