How to Scrape Images and Media Files From a Website

Quick answer: To scrape images from a website, fetch the page HTML, extract every image reference (<img src>, srcset, CSS background-image, and og:image meta tags), resolve each relative URL against the page URL, then download the ones you want with the referring page set as the Referer. The tricky parts are not the download; they are finding assets hidden in srcset and inline styles, deduplicating the same image served at many sizes, and staying on the right side of copyright.

Downloading a file is trivial. Finding every image on a page is not, because modern sites scatter image URLs across at least four different places, and lazy-loading hides half of them behind JavaScript. This walks through finding them all, resolving them correctly, and storing them without collecting the same picture forty times.

Where Image URLs Actually Live

A single page can reference images in several ways, and a naive <img src> scraper misses most of them:

Source	Where it lives	Notes
`<img src>`	The obvious one	Often a tiny placeholder if lazy-loaded
`srcset`	On `<img>` and `<source>`	Multiple resolutions, comma-separated
`data-src` / `data-lazy`	Custom lazy-load attributes	Populated by JS on scroll
`background-image`	Inline `style` or CSS	`url(...)` inside the value
`og:image`	`<meta>` in the head	The social-share image, usually high quality
`<picture>` / `<source>`	Art-directed responsive images	Different crops per breakpoint

If you only read src, you miss the srcset variants, every lazy-loaded image, and every CSS background. That is why image scrapers that "work" on one site return nothing on the next.

Extracting the References

Here is a parser that covers the common cases. It uses BeautifulSoup for HTML and a small regex for CSS backgrounds.

import re
from urllib.parse import urljoin
from bs4 import BeautifulSoup

def find_image_urls(html: str, page_url: str) -> set[str]:
    soup = BeautifulSoup(html, "html.parser")
    urls: set[str] = set()

    # <img src> and lazy-load attributes
    for img in soup.find_all("img"):
        for attr in ("src", "data-src", "data-lazy", "data-original"):
            if img.get(attr):
                urls.add(img[attr])
        # srcset: "a.jpg 1x, b.jpg 2x" -> keep the URL tokens
        if img.get("srcset"):
            for part in img["srcset"].split(","):
                urls.add(part.strip().split(" ")[0])

    # <source srcset> inside <picture>
    for src in soup.find_all("source"):
        if src.get("srcset"):
            for part in src["srcset"].split(","):
                urls.add(part.strip().split(" ")[0])

    # og:image
    for meta in soup.find_all("meta", property="og:image"):
        if meta.get("content"):
            urls.add(meta["content"])

    # CSS background-image in inline styles
    for el in soup.find_all(style=True):
        for match in re.findall(r"url\(([^)]+)\)", el["style"]):
            urls.add(match.strip("'\""))

    # Resolve every relative URL against the page it came from
    return {urljoin(page_url, u) for u in urls if u and not u.startswith("data:")}

The urljoin call is the step people skip. Image URLs are frequently relative (/media/photo.jpg or even ../img/x.png), and they only make sense relative to the page that served them. Skip data: URIs unless you specifically want inline base64 images.

The JavaScript Problem

If a page lazy-loads images, the initial HTML holds placeholders and the real URLs appear only after the browser runs the page's scripts and you scroll. A plain HTTP fetch never sees them.

You need rendered HTML: the DOM after JavaScript has run. You can drive a headless browser yourself (the tradeoffs are in curl vs headless vs stealth browsers), or ask a fetch API to render for you. With link.sc, request the HTML format with rendering on and parse the result with the same function above:

import os, requests

def get_rendered_html(url: str) -> str:
    resp = requests.post(
        "https://api.link.sc/v1/fetch",
        headers={"x-api-key": os.environ["LINKSC_KEY"]},
        json={"url": url, "format": "html", "render_js": True},
        timeout=90,
    )
    resp.raise_for_status()
    return resp.json()["content"]

Rendered HTML has the lazy-loaded URLs filled in, so your extractor finds them without you writing any browser-driving code.

Downloading, With the Referer Set

Once you have absolute URLs, download the ones you want. Two practical details: send a Referer header (many servers reject image requests that do not come from a page on their own site, a cheap hotlink defense), and stream to disk so a large video does not sit in memory.

import requests, hashlib, pathlib

def download(url: str, referer: str, out_dir: str = "media") -> str | None:
    headers = {"Referer": referer, "User-Agent": "MyResearchBot/1.0"}
    resp = requests.get(url, headers=headers, stream=True, timeout=60)
    if resp.status_code != 200:
        return None
    # Name the file by content hash to dedupe automatically
    ext = pathlib.Path(url.split("?")[0]).suffix or ".bin"
    body = resp.content
    digest = hashlib.sha256(body).hexdigest()[:16]
    path = pathlib.Path(out_dir) / f"{digest}{ext}"
    path.parent.mkdir(exist_ok=True)
    path.write_bytes(body)
    return str(path)

Deduplicating Media

Responsive images are the dedupe trap. The same photo ships as photo-320.jpg, photo-640.jpg, and photo-1280.jpg, each a distinct URL, plus the og:image might be a fourth copy. URL-level dedupe treats them as four images; they are one.

Two layers fix it. First, hash the downloaded bytes (the code above names files by their SHA-256, so byte-identical files collapse to one file automatically). Second, for near-duplicates that differ only in resolution, a perceptual hash (pHash) groups visually identical images so you can keep just the largest. The general principle carries over from page dedupe in deduplicating pages when crawling: dedupe on content, not on the URL.

For storage, the same two-tier pattern from any scraping pipeline applies: put the binary files in a blob store or object storage, and keep a small index (source page, original URL, content hash, dimensions, license notes) in a database. That way you can find and re-serve an asset without a filesystem scan.

The Copyright and Ethics Note

This is the section that matters most for images, because unlike text facts, images are almost always somebody's copyrighted work. Downloading an image does not give you a license to use it.

Respect robots.txt and terms of service. If a site disallows crawling a media directory, do not crawl it.
Copyright stays with the creator. A photo being publicly viewable is not permission to republish, retrain on, or redistribute it. Check the license (Creative Commons, stock terms, or explicit permission) before any use beyond your own analysis.
Rate-limit and identify yourself. Bulk image downloads are bandwidth-heavy for the host. Throttle per domain and use an honest User-Agent, the same etiquette as avoiding IP bans when scraping.
No auth or paywall evasion. Public assets only. Do not scrape images behind a login you were not granted or a paid gallery.

When in doubt, treat someone else's images as their property, because legally they usually are. Scraping them for a private index or research is a very different act from republishing them, and only you know which one you are doing.

Get the extraction right (all four sources, resolved to absolute URLs, rendered when lazy-loaded), dedupe on bytes, and keep the licensing question in front of you. The download itself was never the hard part.

Need clean HTML with lazy-loaded images already resolved? Get a free link.sc key and fetch any page rendered, ready to parse.