Quick answer: The Python requests library is the standard way to make HTTP calls in Python. Install it with pip install requests, then use requests.get(url) and requests.post(url, json=...) to talk to websites and APIs. You add query parameters with params, custom headers with headers, reuse connections and cookies with a Session, and always set a timeout. It is perfect for APIs and static pages, but it does not run JavaScript, so pages that render in the browser need a different tool.
The requests library is one of the most used packages in the entire Python ecosystem, and for good reason: it makes HTTP feel simple. This tutorial walks through the pieces you will actually use every day, with runnable code, and it is honest about where requests stops and you need something heavier.
Install requests
One command. Requests is not in the standard library, but it is a tiny dependency.
pip install requests
Your First GET Request
A GET request fetches data. The response object holds the status code, headers, and body.
import requests
resp = requests.get("https://httpbin.org/get")
print(resp.status_code) # 200
print(resp.text[:200]) # body as a string
resp.text gives you the body as text. resp.content gives you raw bytes, which you want for images or files.
Query Parameters
Do not build query strings by hand. Pass a dict to params and requests encodes it correctly, including escaping special characters.
resp = requests.get(
"https://httpbin.org/get",
params={"q": "web scraping", "page": 2},
)
print(resp.url) # .../get?q=web+scraping&page=2
Working With JSON
Most APIs speak JSON. Requests parses a JSON response with one method call.
resp = requests.get("https://httpbin.org/json")
data = resp.json() # dict or list
print(data["slideshow"]["title"])
POST Requests
POST sends data. Use json= to send a JSON body (requests sets the Content-Type header for you) or data= for form-encoded data.
# JSON body
resp = requests.post(
"https://httpbin.org/post",
json={"name": "Ada", "role": "engineer"},
)
# form-encoded body
resp = requests.post(
"https://httpbin.org/post",
data={"name": "Ada", "role": "engineer"},
)
Custom Headers
Headers are a dict. A realistic User-Agent is the most common one you will set, since some servers reject the default requests agent.
headers = {
"User-Agent": "Mozilla/5.0 (compatible; MyBot/1.0; +https://example.com/bot)",
"Accept": "application/json",
}
resp = requests.get("https://httpbin.org/headers", headers=headers)
I like a User-Agent that says who I am and links to a page explaining the bot. It is honest and it gives site owners a way to reach me.
Sessions: Reuse Connections and Cookies
If you make more than one request to the same host, use a Session. It reuses the underlying TCP connection (faster) and persists cookies and headers across calls.
session = requests.Session()
session.headers.update({"User-Agent": "MyBot/1.0"})
session.get("https://httpbin.org/cookies/set/token/abc123")
resp = session.get("https://httpbin.org/cookies")
print(resp.json()) # {'cookies': {'token': 'abc123'}}
The cookie set in the first call is sent automatically in the second. That is how you stay logged in across requests to a service you are authorized to use.
Timeouts (Never Skip These)
By default, requests waits forever. A single hung server can freeze your whole script. Always pass a timeout.
try:
resp = requests.get("https://httpbin.org/delay/10", timeout=5)
except requests.Timeout:
print("Request timed out")
Error Handling
Two things go wrong: the network fails, or the server returns an error status. Handle both.
try:
resp = requests.get("https://httpbin.org/status/404", timeout=5)
resp.raise_for_status() # raises on 4xx and 5xx
data = resp.json()
except requests.HTTPError as err:
print(f"Bad status: {err}")
except requests.RequestException as err:
print(f"Request failed: {err}")
raise_for_status() turns a 404 or 500 into an exception so a bad response never sails through silently. RequestException is the base class that catches timeouts, connection errors, and everything else requests can raise.
A Full Runnable Example
Here is a small, complete script that pulls paginated JSON from an API with a session, timeouts, and error handling: the shape of most real requests code.
import requests
def fetch_all(base_url: str, pages: int) -> list[dict]:
results = []
with requests.Session() as session:
session.headers.update({"User-Agent": "MyBot/1.0"})
for page in range(1, pages + 1):
try:
resp = session.get(base_url, params={"page": page}, timeout=10)
resp.raise_for_status()
except requests.RequestException as err:
print(f"Page {page} failed: {err}")
continue
results.extend(resp.json().get("items", []))
return results
if __name__ == "__main__":
items = fetch_all("https://api.example.com/items", pages=3)
print(f"Fetched {len(items)} items")
When requests Is Not Enough
Requests speaks HTTP, and that is exactly its limit. It sends and receives; it does not run a browser. You will hit a wall in two situations.
| Situation | Why requests falls short | What to reach for |
|---|---|---|
| Page renders with JavaScript | requests gets the initial HTML, not the DOM the browser builds | A headless browser (Playwright, Selenium) or a rendering fetch API |
| Heavy anti-bot defenses | Some sites challenge automated clients regardless of headers | A managed fetch service that handles rendering and access |
| Infinite scroll / clicks | requests cannot interact with a page | A browser automation tool |
The tell is simple: fetch the page with requests, print resp.text, and search for the data you want. If it is not in there, the content is rendered client-side and requests alone will not see it. We map out the full toolkit in Python web scraping libraries in 2026.
When you would rather not run and maintain a browser, a hosted fetch API renders the page and hands back clean content. With link.sc you keep using requests; only the endpoint changes:
import requests
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_your_key_here"},
json={"url": "https://example.com/spa-page", "format": "markdown"},
timeout=60,
)
resp.raise_for_status()
print(resp.json()["content"]) # rendered page as clean markdown
The auth header is x-api-key, not Authorization: Bearer, and the rendered result comes back as { "content": "..." }. The link.sc docs cover options like waiting on a selector or choosing a country.
Scrape Responsibly
Requests makes it trivial to fire off thousands of calls, so restraint is on you. Stick to public data, respect robots.txt and any published rate limits, set a timeout and a delay between requests, and identify your bot with an honest User-Agent. Do not use cookies or credentials to reach data you are not authorized to see. A few well-spaced, well-labeled requests almost never cause trouble; a flood does.
The Bottom Line
The requests library is the tool for HTTP in Python: GET and POST, params, headers, Session for reuse, timeout on every call, and raise_for_status() for clean error handling. It is ideal for APIs and static pages. The moment a page needs JavaScript to render or throws anti-bot challenges, requests alone is not enough, and you move up to a browser or a rendering fetch API. Learn requests first; it is the foundation everything else builds on.
Hitting JavaScript walls with plain requests? link.sc renders and cleans any page, and you call it with the same requests code you already know. Start free with 500 credits a month.