Quick answer: Content gap analysis finds topics your audience searches for that your site does not cover well, then ranks them by opportunity. Do it with web data in three passes: pull the SERP and People Also Ask questions for your core terms to see what searchers ask, fetch competitor pages to see what they cover, then compare both against your own content to surface the gaps. A search-and-fetch API like link.sc runs all three passes from one pipeline.
Most content plans are guesswork dressed up as strategy. A gap analysis replaces the guessing with evidence: here is what people ask, here is what competitors answer, here is what you are missing. This post shows how to build that pipeline with code you can run today.
The Three Sources of a Gap
A real gap shows up when all three of these disagree with each other:
| Source | Question it answers | How you get it |
|---|---|---|
| SERP and People Also Ask | What are searchers actually asking? | Search API, PAA extraction |
| Competitor pages | What does the ranking content cover? | Fetch top results, extract topics |
| Your own content | What do you already cover? | Crawl your sitemap, extract topics |
A topic that searchers ask about, competitors cover, and you do not, is your highest-priority gap. A topic only you cover might be a differentiator or a dead end. The analysis is about lining up all three.
Step 1: Capture What Searchers Ask
Start with the SERP. A search that returns full results plus the People Also Ask box tells you the shape of demand around a term. Use the link.sc search API, where the search field is q:
import requests
SEARCH = "https://api.link.sc/v1/search"
FETCH = "https://api.link.sc/v1/fetch"
HEADERS = {"x-api-key": "lsc_your_key", "Content-Type": "application/json"}
def serp(query):
r = requests.post(SEARCH, headers=HEADERS, json={
"q": query,
"engine": "google"
})
r.raise_for_status()
return r.json()["serpData"]
The People Also Ask questions are the clearest signal of what a page on this topic should answer. We go deep on pulling and using them in web scraping for SEO: extract People Also Ask, which pairs directly with this workflow.
Step 2: See What Competitors Cover
Take the top-ranking URLs from the SERP, fetch each as markdown, and extract the topics they cover. Headings are the cheapest reliable proxy for topic coverage.
import re
def fetch_markdown(url):
r = requests.post(FETCH, headers=HEADERS, json={
"url": url,
"format": "markdown",
"render_js": True
})
r.raise_for_status()
return r.json()["content"]
def extract_topics(markdown):
# markdown headings are the coverage skeleton of a page
return [h.strip("# ").strip().lower()
for h in re.findall(r'^#{2,3}\s+.+$', markdown, re.MULTILINE)]
def competitor_topics(serp_data, top_n=5):
urls = [r["url"] for r in serp_data.get("organic", [])[:top_n]]
coverage = {}
for url in urls:
try:
coverage[url] = extract_topics(fetch_markdown(url))
except requests.RequestException:
continue
return coverage
This gathering step is the same primitive behind broader competitive intelligence with web scraping; a gap analysis is one focused application of it.
Step 3: Map Your Own Coverage
Do the same extraction against your own pages. Pull your sitemap, fetch each URL, and build a set of topics you already cover.
def my_coverage(my_urls):
covered = set()
for url in my_urls:
try:
for topic in extract_topics(fetch_markdown(url)):
covered.add(topic)
except requests.RequestException:
continue
return covered
For a large site, do this once and cache it. Your coverage does not change hour to hour, and re-fetching every page on each analysis burns credits for no reason.
Step 4: Compute and Prioritize the Gaps
Now line up all three sources. A gap is a topic that appears in searcher questions or competitor coverage but not in yours. Score each gap so you know what to write first.
from collections import Counter
def find_gaps(paa_questions, competitor_coverage, my_topics):
# count how many competitors cover each topic
demand = Counter()
for topics in competitor_coverage.values():
for t in set(topics):
demand[t] += 1
gaps = []
for topic, comp_count in demand.items():
if topic in my_topics:
continue # already covered
asked = any(topic in q.lower() for q in paa_questions)
score = comp_count * 2 + (5 if asked else 0)
gaps.append({
"topic": topic,
"competitors_covering": comp_count,
"searchers_ask": asked,
"priority_score": score,
})
return sorted(gaps, key=lambda g: g["priority_score"], reverse=True)
The scoring is deliberately simple and worth tuning. The logic: a topic that many competitors cover has proven demand, and a topic searchers explicitly ask about is worth extra weight because it maps to a real query. Adjust the weights to your strategy.
Reading the Output
The ranked list is your content brief in miniature. A few patterns to look for:
- High score, searchers ask, most competitors cover. Write this now. It is table-stakes coverage you are missing.
- Medium score, competitors cover, searchers do not ask. Useful supporting content, lower urgency.
- Topics only you cover. Not gaps, but check whether they still draw traffic or should be pruned.
Do not treat the list as gospel. It surfaces candidates; you still apply judgment about which gaps fit your business and which are noise from a competitor covering something irrelevant.
Keeping It Current
Search demand and competitor coverage drift. Rerun the analysis quarterly for a stable niche, monthly for a fast-moving one. Because the pipeline is just search plus fetch, scheduling it is trivial: run it, diff against last time, and you get a report of new gaps that opened up.
A managed search-and-fetch API keeps this to a small script instead of a scraping stack. link.sc returns full page content from search rather than snippets, which is what makes the competitor-coverage step accurate, and the free tier of 500 credits a month at link.sc/pricing covers a modest analysis. When your niche is large, the paid tiers scale the same pipeline without changing the code.
Start with one core term, run the three passes, and read the gap list. It will almost certainly show you something you did not know you were missing.
Ready to turn search and competitor data into a content plan? Start free on link.sc and run your first gap analysis today.