How to Create an RSS Feed From Any Website (Even Without One)

Quick answer: To create an RSS feed from any website, first check whether the site already publishes one by looking for <link rel="alternate" type="application/rss+xml"> in the HTML head or common paths like /feed and /rss.xml. If none exists, fetch the page, extract the repeating list of items (title, link, date), and emit valid RSS or Atom XML yourself. A fetch API like link.sc handles the messy part: turning rendered HTML into clean, structured content you can parse.

RSS is not dead. It is the quiet backbone of newsletters, read-it-later apps, Slack bots, and half the automation people run through Zapier. The problem is that plenty of sites you care about, a company blog, a changelog page, a government notices board, never bothered to publish a feed. This post shows how to generate one for any page, cleanly and repeatably.

Step 1: Always Check for an Existing Feed First

Do not reinvent a feed that already exists. Many sites have one hidden in the HTML head even when there is no visible RSS icon.

Two places to look:

Autodiscovery links in the <head>: <link rel="alternate" type="application/rss+xml" href="..."> or type="application/atom+xml".
Conventional paths: /feed, /feed/, /rss, /rss.xml, /atom.xml, /index.xml, /feed.json.

Here is a detection pass using the link.sc fetch API to grab the HTML, then a quick scan:

import re
import requests

API = "https://api.link.sc/v1/fetch"
HEADERS = {"x-api-key": "lsc_your_key", "Content-Type": "application/json"}

def find_existing_feed(url):
    r = requests.post(API, headers=HEADERS, json={"url": url, "format": "html"})
    html = r.json()["content"]

    # 1. autodiscovery <link> tags
    links = re.findall(
        r'<link[^>]+type=["\']application/(?:rss|atom)\+xml["\'][^>]*>', html
    )
    for tag in links:
        href = re.search(r'href=["\']([^"\']+)["\']', tag)
        if href:
            return href.group(1)

    # 2. common paths
    from urllib.parse import urljoin
    for path in ("/feed", "/rss.xml", "/atom.xml", "/index.xml"):
        candidate = urljoin(url, path)
        head = requests.head(candidate, allow_redirects=True, timeout=10)
        if head.status_code == 200 and "xml" in head.headers.get("content-type", ""):
            return candidate
    return None

If this returns a URL, you are done. Point your reader at it.

Step 2: When There Is No Feed, Extract the Items

If detection comes back empty, you build the feed from the page content. The pattern on almost every listing page is the same: a repeating block of items, each with a title, a link, and usually a date.

Fetch the page as markdown so you get clean text without the wrapper cruft. link.sc renders JavaScript-heavy pages too, which matters because a lot of modern blogs hydrate their post lists client-side.

def fetch_markdown(url):
    r = requests.post(API, headers=HEADERS, json={
        "url": url,
        "format": "markdown",
        "render_js": True
    })
    return r.json()["content"]

Markdown makes extraction simpler because links come through as [title](href). For a blog index, the items are usually the links that point to article paths:

from urllib.parse import urljoin

def extract_items(markdown, base_url):
    items = []
    seen = set()
    for m in re.finditer(r'\[([^\]]{15,120})\]\(([^)]+)\)', markdown):
        title, href = m.group(1).strip(), m.group(2).strip()
        link = urljoin(base_url, href)
        # keep only same-site article-looking links, drop nav and footers
        if base_url.split("/")[2] in link and link not in seen:
            seen.add(link)
            items.append({"title": title, "link": link})
    return items[:25]

Tune the filter to the site. Some pages need a length floor on the title, others need a path prefix like /blog/ or /posts/. If a site is truly hostile to regex, ask link.sc for format: "json" and work with structured fields instead.

Step 3: Emit Valid RSS 2.0 XML

Once you have items, generating the feed is mechanical. Use a builder rather than string concatenation so escaping is handled for you.

from datetime import datetime, timezone
import xml.etree.ElementTree as ET

def build_rss(feed_title, site_url, items):
    rss = ET.Element("rss", version="2.0")
    channel = ET.SubElement(rss, "channel")
    ET.SubElement(channel, "title").text = feed_title
    ET.SubElement(channel, "link").text = site_url
    ET.SubElement(channel, "description").text = f"Generated feed for {feed_title}"
    ET.SubElement(channel, "lastBuildDate").text = datetime.now(
        timezone.utc
    ).strftime("%a, %d %b %Y %H:%M:%S +0000")

    for it in items:
        entry = ET.SubElement(channel, "item")
        ET.SubElement(entry, "title").text = it["title"]
        ET.SubElement(entry, "link").text = it["link"]
        ET.SubElement(entry, "guid").text = it["link"]
        if it.get("published"):
            ET.SubElement(entry, "pubDate").text = it["published"]

    return ET.tostring(rss, encoding="unicode", xml_declaration=True)

RSS 2.0 is the safest target because every reader supports it. Atom is stricter and slightly cleaner, but unless a consumer specifically requires Atom, RSS 2.0 gives you the widest reach.

Format	Best for	Notes
RSS 2.0	Maximum compatibility	Loose spec, universally supported
Atom 1.0	Strict validation, dates	Requires globally unique IDs and RFC 3339 dates
JSON Feed	Modern apps, easy parsing	Simple to emit, thinner reader support

Step 4: Schedule and Deduplicate

A feed is only useful if it stays current. Run your generator on a schedule (a cron job, a Cloudflare Worker on a cron trigger, a GitHub Action), and dedupe against what you have already published so the same item does not reappear as new.

import json, os

STATE = "seen_links.json"

def load_seen():
    return set(json.load(open(STATE))) if os.path.exists(STATE) else set()

def save_seen(seen):
    json.dump(sorted(seen), open(STATE, "w"))

def new_items(items):
    seen = load_seen()
    fresh = [it for it in items if it["link"] not in seen]
    seen.update(it["link"] for it in items)
    save_seen(seen)
    return fresh

Dedupe on the item link or guid, never on the title. Titles get edited after publish, links rarely change. A hash of the normalized link is a reliable key.

A sensible cadence for most sites is every 30 to 60 minutes. Check the page's own update frequency first: polling a monthly changelog every five minutes wastes credits and hammers a server that has nothing new for you.

A Note on Etiquette and Terms of Service

Generating a feed is reading a public page and reformatting it for your own consumption, which is generally fine. Stay on the right side of it:

Respect robots.txt and any stated crawl policy.
Keep polling intervals reasonable so you do not add load.
Do not republish full article bodies as if they were yours. Feeds should link back to the source.
If a site offers an official API or feed, use that instead.

If you want alerting layered on top of a generated feed, the same fetch-and-diff pattern powers change detection. We cover that in monitoring web page changes and getting alerts. And if your extraction step is fighting messy HTML, converting HTML to clean markdown for LLMs explains why a rendering fetch layer saves you hours of parser maintenance.

Why Use a Fetch API Instead of Raw Requests

You could do all of this with requests and BeautifulSoup. It works until it does not: the page needs JavaScript to render its list, the site blocks your data center IP, or the markup changes and your parser silently returns zero items. A managed fetch layer handles rendering, proxying, and clean output, so your feed generator stays a small script instead of a scraping platform. link.sc gives you 500 free credits a month at link.sc/pricing, which is plenty to run a handful of feeds on a schedule.

Build the detection step first, always. The best RSS feed is the one the site already publishes and you did not have to generate.

Ready to turn any website into a clean, structured feed? Get a free link.sc API key and start fetching in minutes.