How to Audit Every URL in Your Sitemap and Find Orphan Pages

Your sitemap tells search engines which URLs you care about, but it does not guarantee those pages are healthy or linked from anywhere. Pages that sit only in the sitemap, with no internal links pointing at them, are orphans: they get little internal link equity and are often forgotten, outdated or duplicated.

Finding them by hand means downloading the sitemap, crawling the site, and diffing the two lists, then checking each page for titles, canonicals and status codes. The SEO Audit Crawler does both halves at once: it crawls links like a search engine, reads sitemap.xml and the sitemaps listed in robots.txt, audits the sitemap URLs nothing links to, and flags them.

To be clear about cost: every HTML page audited is charged at $0.006 ($6 per 1,000). If all you need is the list of URLs in a sitemap, parse the XML yourself for free, as shown in the last step.

Step by step

1. Estimate your site size and set maxPages

   Orphan detection is only conclusive when the crawl covers the whole site, because an orphan is a sitemap URL that no audited page links to. Count the URLs in your sitemap and set maxPages comfortably above that number; the default is 200. The run summary includes orphanCheckComplete, which tells you whether the result is conclusive.

   {
     "startUrls": ["https://example.com/"],
     "maxPages": 3000,
     "maxDepth": 20,
     "useSitemap": true,
     "respectRobotsTxt": true,
     "excludeGlobs": ["**/tag/**", "**/page/*"]
   }

2. Keep the sitemap on and trim noise

   Leave useSitemap on (the default). The actor reads sitemap.xml and any sitemaps listed in robots.txt, and after the pages reachable by links it audits sitemap URLs that no page links to. Use excludeGlobs to skip tag archives and pagination you do not want to pay for, but do not exclude sections that link to sitemap pages, or those pages may be wrongly reported as orphans.

3. List the orphan pages

   Each page row has an issues list; orphan sitemap URLs carry the orphan-page issue code. Pull the dataset and filter. The snippet accepts issues as plain codes or as objects with a code field, since you will see the exact structure in your first run.

   import os
   import pandas as pd
   from apify_client import ApifyClient
   
   client = ApifyClient(os.environ["APIFY_TOKEN"])
   run = client.actor("actor-forge/seo-audit").call(run_input={
       "startUrls": ["https://example.com/"],
       "maxPages": 3000,
       "useSitemap": True,
   })
   rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
   
   def codes(row):
       return {i if isinstance(i, str) else i.get("code") for i in row.get("issues") or []}
   
   orphans = [r for r in rows if "orphan-page" in codes(r)]
   pd.DataFrame(orphans)[["url", "statusCode", "indexable", "score", "wordCount"]] \
     .to_csv("orphan_pages.csv", index=False)
   print(len(orphans), "orphan pages of", len(rows), "audited")
   
   summary = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("OUTPUT")
   print("Orphan check complete:", (summary or {}).get("value", {}).get("orphanCheckComplete"))

4. Triage the rest of the sitemap URLs

   Orphans are just one signal. For every audited URL you also get status code, redirect chain, indexability (noindex and canonical), title and meta description problems, H1 count, thin content and broken internal links. A sitemap should list only indexable, 200-status, self-canonical pages, so filter for rows where indexable is false or statusCode is not 200 and remove or fix them in the sitemap.

   df = pd.DataFrame(rows)
   bad_in_sitemap = df[(df["indexable"] == False) | (df["statusCode"] != 200)]
   print(bad_in_sitemap[["url", "statusCode", "indexable"]].head(50))
   
   top = df.explode("issues")["issues"].map(lambda i: i if isinstance(i, str) else (i or {}).get("code"))
   print(top.value_counts().head(15))  # most common issues across the site

5. Decide what to do with each orphan

   For each orphan, choose: link to it from a relevant hub or category page if it still matters, merge it into a stronger page and redirect, or remove it from the sitemap if it should not be indexed. Re-run the audit afterwards; pages you linked should lose the orphan-page flag.

6. Audit only a fixed list of sitemap URLs (optional)

   If you only want the pages in one sitemap audited, without crawling the rest of the site, fetch and parse the sitemap yourself (free), then pass the URLs as startUrls with maxDepth 0, which audits only the start pages. Turn useSitemap off in this mode. Inlinks and orphan detection are not meaningful here, because the crawl does not follow links.

   import requests
   import xml.etree.ElementTree as ET
   
   NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
   xml = requests.get("https://example.com/sitemap.xml", timeout=30).content
   urls = [loc.text.strip() for loc in ET.fromstring(xml).findall(".//sm:loc", NS)]
   print(len(urls), "URLs in sitemap")  # a sitemap index lists child sitemaps instead
   
   run_input = {
       "startUrls": urls,
       "maxDepth": 0,
       "maxPages": len(urls),
       "useSitemap": False,
   }

Sample input

{
  "startUrls": [
    "https://crawlee.dev/js/docs/guides"
  ],
  "maxPages": 500,
  "maxDepth": 20,
  "useSitemap": true,
  "respectRobotsTxt": true
}

Sample output

The sample includes a page flagged orphan-page with zero inlinks next to well-linked pages from the same section.

Run it from code

cURL

curl -X POST "https://api.apify.com/v2/acts/actor-forge~seo-audit/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev/js/docs/guides"], "maxPages": 500, "maxDepth": 20, "useSitemap": true, "respectRobotsTxt": true}'

JavaScript (Node.js)

// Node.js 18+ (built-in fetch). Set APIFY_TOKEN in your environment.
const input = {
  "startUrls": [
    "https://crawlee.dev/js/docs/guides"
  ],
  "maxPages": 500,
  "maxDepth": 20,
  "useSitemap": true,
  "respectRobotsTxt": true
};

const res = await fetch(
  "https://api.apify.com/v2/acts/actor-forge~seo-audit/run-sync-get-dataset-items",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.APIFY_TOKEN}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(input),
  }
);
const items = await res.json();
console.log(items.length, items[0]);

Python

# pip install apify-client   (set APIFY_TOKEN in your environment)
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run_input = {
    "startUrls": [
        "https://crawlee.dev/js/docs/guides"
    ],
    "maxPages": 500,
    "maxDepth": 20,
    "useSitemap": True,
    "respectRobotsTxt": True
}

run = client.actor("actor-forge/seo-audit").call(run_input=run_input)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

The run-sync-get-dataset-items endpoint waits for the run (up to 5 minutes) and returns the dataset rows directly. For longer runs, start the run asynchronously and read the dataset when it finishes, as the Python example does. Replace APIFY_TOKEN with your own token from the Apify console.

What it costs

Auditing a site with 2,500 HTML pages costs 2,500 x $0.006 = $15.00; a 100-page site costs $0.60. Pages that answer 4xx/5xx, time out or are blocked are reported free and do not count.

Tips and pitfalls

• Orphan results are only reliable when orphanCheckComplete in the summary is true; if the crawl hit maxPages, raise it and re-run.
• Pages are fetched as static HTML. Links that only appear after JavaScript runs are not seen, so a JavaScript-rendered menu can make linked pages look orphaned.
• Starting at a sub-path such as /blog keeps the crawl under that path; use the homepage as the start URL for a site-wide orphan check.
• The crawler sends at most 3 requests at a time and pauses on HTTP 429, so large sites take a while; schedule big audits off-peak.

FAQ

Actors used in this guide

Related guides

All guides