Wayback Machine Scraper: Archive.org Snapshots and URL History

What it does

The Wayback Machine holds years of captures for most public websites, but browsing its calendar one day at a time is slow, and the raw CDX index is awkward to query. This actor turns the official Wayback CDX API into clean dataset rows: capture time, original URL, HTTP status, MIME type, size, content digest and archive links with and without the Wayback toolbar.

You can look up a single page, a folder (example.com/blog/*), a host or a whole domain with subdomains. By default it keeps only unique content versions and flags each with changedFromPrevious, so a page captured thousands of times collapses to the moments it really changed. Alternatively keep one capture per hour, day, month or year, or one row per distinct URL to list every page ever archived on a site.

Summary mode returns one row per site with first and last capture, totals, captures per year, missing years and longest gap. You can also fetch the archived HTML of captures as Markdown or text. Snapshots cost $2 per 1,000 rows, summaries $0.004 per exact URL or $0.01 per prefix, host or domain, and archived content $0.004 per page. The data comes from the Internet Archive; ActorForge is not affiliated with it.

How it works

1. You list pages, folders or domains in urls and choose a matchType: exact, prefix, host or domain.
2. The actor queries archive.org's CDX index, applying your from/to dates, statusFilter, mimeFilter and collapse mode.
3. Requests are paced (about one per second, two URLs in parallel) and retried with back-off on rate limits, server errors and timeouts.
4. Rows are written per capture (snapshots mode) or per input (summary mode), capped by maxSnapshotsPerUrl.
5. With fetchContent on, the first maxContentPages HTML captures with status 200 are downloaded and their title and main content added as Markdown or text.

Input example

A typical run input (JSON). Every field is also available as a form in the Apify console.

{
  "urls": [
    "https://apify.com/pricing",
    "example.com"
  ],
  "matchType": "exact",
  "outputMode": "snapshots",
  "collapse": "digest",
  "from": "2018",
  "statusFilter": "200",
  "mimeFilter": "text/html",
  "maxSnapshotsPerUrl": 300
}

Output example

Each row below is one archived capture with its timestamp, status, MIME type, change flag and a ready-to-open Wayback link.

Pricing

Pay per event: you are charged only for the results you receive, with no subscription. Set a maximum total charge on any run and it stops cleanly at the cap.

Prices are read from the live Apify Store listing when this page is built. The listing is always authoritative.

Use cases

How it compares

For one page and a quick look, the Wayback Machine website is free and perfectly fine, and developers comfortable with the CDX API can query it directly. This actor saves the paging, retries, change flags and summaries when you have many URLs or need the data in a spreadsheet or pipeline.

Limits and honest notes

• Data comes from the Internet Archive's public Wayback CDX API. Captures the archive doesn't have (never crawled, excluded by the site owner, or very recent) can't be returned; such inputs get a free row saying so.
• archive.org is a non-profit and sometimes slow: exact-URL lookups take about a second, but prefix/host/domain lookups take 20–40 s per index page (each page covers several thousand captures). The actor paces its requests (about one per second, two URLs in parallel) and retries rate limits (HTTP 429), server errors and timeouts with backoff; a URL that still fails gets a free error row.
• Prefix/host/domain results come in URL order (then time), not purely by date. Newest-first is available for exact URLs only. These queries stop after 10 index pages per input by default (raise Max index pages to go deeper). On multi-URL results, collapsing works per URL: "one per day" keeps one capture per day for each URL.
• Unique-versions mode compares each capture with the previous one (archive.org's digest collapse), so a page that changes back to an earlier version shows up again — that is a real change.
• Summary counts for exact URLs come from the Wayback Machine's capture calendar (all captures, monthly precision for date ranges). With status/MIME filters, or for prefix/host/domain, they are counted from the index (up to 100,000 captures or the index page cap; scanComplete says whether the count is complete).
• Archived page content is extracted from the raw capture as archived; pages that were built by JavaScript may come out nearly empty, and images/CSS are not downloaded. Content is fetched only for HTML captures with HTTP 200 and is capped at 200,000 characters.
• Respects your run budget: set a maximum total charge and the run stops cleanly at the cap, with no row charged twice.

FAQ

Use it from code

Run the actor through the Apify API with the input above.

cURL

curl -X POST "https://api.apify.com/v2/acts/actor-forge~wayback-snapshots/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://apify.com/pricing", "example.com"], "matchType": "exact", "outputMode": "snapshots", "collapse": "digest", "from": "2018", "statusFilter": "200", "mimeFilter": "text/html", "maxSnapshotsPerUrl": 300}'

JavaScript (Node.js)

// Node.js 18+ (built-in fetch). Set APIFY_TOKEN in your environment.
const input = {
  "urls": [
    "https://apify.com/pricing",
    "example.com"
  ],
  "matchType": "exact",
  "outputMode": "snapshots",
  "collapse": "digest",
  "from": "2018",
  "statusFilter": "200",
  "mimeFilter": "text/html",
  "maxSnapshotsPerUrl": 300
};

const res = await fetch(
  "https://api.apify.com/v2/acts/actor-forge~wayback-snapshots/run-sync-get-dataset-items",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.APIFY_TOKEN}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(input),
  }
);
const items = await res.json();
console.log(items.length, items[0]);

Python

# pip install apify-client   (set APIFY_TOKEN in your environment)
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run_input = {
    "urls": [
        "https://apify.com/pricing",
        "example.com"
    ],
    "matchType": "exact",
    "outputMode": "snapshots",
    "collapse": "digest",
    "from": "2018",
    "statusFilter": "200",
    "mimeFilter": "text/html",
    "maxSnapshotsPerUrl": 300
}

run = client.actor("actor-forge/wayback-snapshots").call(run_input=run_input)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

The run-sync-get-dataset-items endpoint waits for the run (up to 5 minutes) and returns the dataset rows directly. For longer runs, start the run asynchronously and read the dataset when it finishes, as the Python example does. Replace APIFY_TOKEN with your own token from the Apify console.

Step-by-step guides

Related actors

More SEO & Marketing Intelligence tools · All actors