SEO Audit Crawler: On-Page and Technical SEO Check for Every Page

What it does

Spot-checking a few pages misses the problems that only show up across a site: duplicate titles, pages nothing links to, internal links that hit 404s or redirect chains. The SEO Audit Crawler follows your links and your sitemap, respects robots.txt and runs more than 40 technical and on-page checks on every HTML page it fetches.

Each page gets HTTP status and redirect chain, indexability (meta robots, X-Robots-Tag, canonical), title and meta description checks including duplicates, H1 and heading outline, word count and thin content, images missing alt text, link counts, broken internal links, inlinks, hreflang, Open Graph and Twitter cards, schema.org JSON-LD, mixed content and URL hygiene. Issues are graded error, warning or notice, and the page score starts at 100 and drops 10, 5 or 1 point per distinct issue.

A site summary adds average score, top issues by pages affected, duplicate title and description groups, the most-linked broken URLs and orphan pages that are in your sitemap but linked from nowhere. It costs $6 per 1,000 pages audited ($0.60 for a 100-page site), and pages that answer an error are reported free. It suits SEOs, agencies and developers who want a scheduled audit without installing desktop software.

How it works

1. You give one or more startUrls; the crawl stays on that site and under each URL's path unless you set includeGlobs.
2. It follows links up to maxDepth clicks and, with useSitemap on, reads sitemap.xml and sitemaps listed in robots.txt, then audits sitemap URLs no page links to as orphan pages.
3. Pages are fetched as static HTML, at most 3 requests at a time, pausing on HTTP 429 for the Retry-After period.
4. Once the crawl finishes it computes the cross-page checks (duplicates, broken internal links, inlinks, hreflang return links) and writes one row per page.
5. The site-level summary is saved to the run's key-value store as the OUTPUT record.

Input example

A typical run input (JSON). Every field is also available as a form in the Apify console.

{
  "startUrls": [
    "https://crawlee.dev/js/docs/guides"
  ],
  "maxPages": 500,
  "maxDepth": 20,
  "includeSubdomains": false,
  "excludeGlobs": [
    "**/tag/**",
    "**/page/*"
  ],
  "useSitemap": true,
  "respectRobotsTxt": true
}

Output example

Each audited page is one row with its score, status, indexability, key lengths, inlinks and issue codes, as in the sample below.

Pricing

Pay per event: you are charged only for the results you receive, with no subscription. Set a maximum total charge on any run and it stops cleanly at the cap.

Prices are read from the live Apify Store listing when this page is built. The listing is always authoritative.

Use cases

How it compares

If your site renders content client-side or you need Core Web Vitals and external link checks, a desktop crawler with JavaScript rendering or a full SaaS platform covers more. This actor is for fast, cheap static-HTML audits you can schedule and pull through an API.

Limits and honest notes

• Pages are fetched as static HTML without a browser: fast and cheap, and it covers almost all websites. Content, links and tags that exist only after JavaScript runs (single-page apps that render client-side) are not seen, which is also how many search engine and AI crawlers see your site. Core Web Vitals and other browser performance metrics are not measured.
• Thresholds: titles 30-60 characters, meta descriptions 70-160 characters, thin content under 200 words (counted in the page body without navigation, header, footer, sidebars and scripts), URLs over 115 characters, responses slower than 2 seconds.
• The score starts at 100 and loses 10 points per distinct error, 5 per warning and 1 per notice. Duplicate titles and descriptions are compared across indexable pages only (noindex and canonicalised copies are expected to repeat), case and spacing aside.
• Broken internal links and links to redirects are found from the pages the crawl fetched; internal links beyond your page limit, and external links, are not checked (use a broken link checker for that). Orphan pages are sitemap URLs no audited page links to, so they are only conclusive when the crawl covered the whole site (orphanCheckComplete in the summary).
• Crawling respects robots.txt and stays on your site and under the start URL's path, unless you set include globs. Pages behind a login are not supported. Polite by design: at most 3 requests at a time to your site, and a 429 'Too Many Requests' pauses it for its Retry-After. Pages refused by bot protection are reported as 'fetch-blocked' rows, free.
• Rows are delivered when the crawl finishes, since duplicates, broken links, hreflang return links and inlinks need every page; runs resume after a platform migration without re-charging work already done. Pages larger than 5 MB and non-HTML files are skipped.
• Links to internal network addresses (localhost, private IPs, cloud metadata) are never requested and never charged.
• Respects your run budget: set a maximum total charge and the run stops cleanly at the cap and delivers every page audited so far.

FAQ

Use it from code

Run the actor through the Apify API with the input above.

cURL

curl -X POST "https://api.apify.com/v2/acts/actor-forge~seo-audit/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev/js/docs/guides"], "maxPages": 500, "maxDepth": 20, "includeSubdomains": false, "excludeGlobs": ["**/tag/**", "**/page/*"], "useSitemap": true, "respectRobotsTxt": true}'

JavaScript (Node.js)

// Node.js 18+ (built-in fetch). Set APIFY_TOKEN in your environment.
const input = {
  "startUrls": [
    "https://crawlee.dev/js/docs/guides"
  ],
  "maxPages": 500,
  "maxDepth": 20,
  "includeSubdomains": false,
  "excludeGlobs": [
    "**/tag/**",
    "**/page/*"
  ],
  "useSitemap": true,
  "respectRobotsTxt": true
};

const res = await fetch(
  "https://api.apify.com/v2/acts/actor-forge~seo-audit/run-sync-get-dataset-items",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.APIFY_TOKEN}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(input),
  }
);
const items = await res.json();
console.log(items.length, items[0]);

Python

# pip install apify-client   (set APIFY_TOKEN in your environment)
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run_input = {
    "startUrls": [
        "https://crawlee.dev/js/docs/guides"
    ],
    "maxPages": 500,
    "maxDepth": 20,
    "includeSubdomains": False,
    "excludeGlobs": [
        "**/tag/**",
        "**/page/*"
    ],
    "useSitemap": True,
    "respectRobotsTxt": True
}

run = client.actor("actor-forge/seo-audit").call(run_input=run_input)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

The run-sync-get-dataset-items endpoint waits for the run (up to 5 minutes) and returns the dataset rows directly. For longer runs, start the run asynchronously and read the dataset when it finishes, as the Python example does. Replace APIFY_TOKEN with your own token from the Apify console.

Step-by-step guides

Related actors

More SEO & Marketing Intelligence tools · All actors