How to Use a Wayback Machine Snapshots API and See When a Page Changed

The Wayback Machine is the largest public record of how websites looked over time, but its calendar view shows one capture at a time and a busy page can have thousands of near-identical captures. To answer "when did this pricing page change?" or "what URLs did this site have in 2016?", you need the capture index as data.

The Internet Archive's CDX API provides that index, but you have to handle paging, rate limits, slow domain queries and deduplication yourself. The Wayback Snapshots actor does that and adds a changedFromPrevious flag and per-site summaries. ActorForge is not affiliated with the Internet Archive.

This guide pulls snapshots for a URL, builds a change history, lists every archived URL of a domain, and explains the limits: a change only shows up if the archive happened to capture the page before and after it.

Step by step

1. Decide what you are looking up

   Put pages, folders or domains in urls and set matchType: exact for one page, prefix for a folder, host for one host name, domain for the host and all subdomains. Shortcuts work too: example.com/blog/* is a prefix and *.example.com is the whole domain. Exact lookups take about a second; prefix, host and domain lookups take 20-40 seconds per index page on archive.org's side.

2. Get a page's change history with unique versions

   The default collapse mode, digest, keeps a capture only when its content digest differs from the previous capture, and marks it with changedFromPrevious. Add statusFilter 200 and mimeFilter text/html to skip redirects and error captures, and use from and to to limit the period. Setting newestFirst to true (exact URLs only) returns the most recent versions first.

   {
     "urls": ["https://apify.com/pricing"],
     "matchType": "exact",
     "collapse": "digest",
     "statusFilter": "200",
     "mimeFilter": "text/html",
     "from": "2019",
     "to": "2025",
     "maxSnapshotsPerUrl": 500
   }

3. Turn the versions into a change timeline

   Load the rows into pandas, sort by capturedAt and look at how often the page changed per month and how its size moved between versions. Big jumps in lengthBytes often mark redesigns, while small ones tend to be copy or price edits. Each row's archiveUrl opens that exact version in the Wayback Machine.

   import os
   import pandas as pd
   from apify_client import ApifyClient
   
   client = ApifyClient(os.environ["APIFY_TOKEN"])
   run = client.actor("actor-forge/wayback-snapshots").call(run_input={
       "urls": ["https://apify.com/pricing"],
       "collapse": "digest",
       "statusFilter": "200",
       "mimeFilter": "text/html",
   })
   rows = [r for r in client.dataset(run["defaultDatasetId"]).iterate_items() if not r.get("error")]
   
   df = pd.DataFrame(rows)
   df["capturedAt"] = pd.to_datetime(df["capturedAt"])
   df = df.sort_values("capturedAt")
   df["sizeChange"] = df["lengthBytes"].diff()
   
   print(df.groupby(df["capturedAt"].dt.to_period("M")).size().tail(24))  # versions per month
   print(df[["capturedAt", "lengthBytes", "sizeChange", "archiveUrl"]]
         .loc[df["sizeChange"].abs() > 2000])

4. Read what the page said in each version

   Set fetchContent to true to download the archived HTML of captures and add the page title and main content as Markdown or text (contentFormat). Only HTML captures with status 200 are fetched, up to maxContentPages per run (10 by default), at $0.004 each. Pages that were built by JavaScript may come out nearly empty, because only the raw archived HTML is read.

5. List every archived URL of a site

   To recover lost pages or plan redirects, use matchType domain or host with collapse set to unique-urls, which returns one row per distinct URL. These queries stop after maxIndexPagesPerUrl index pages per input (10 by default); raise it for large sites, keeping in mind each page takes 20-40 seconds.

   {
     "urls": ["*.example.com"],
     "matchType": "domain",
     "collapse": "unique-urls",
     "statusFilter": "200",
     "mimeFilter": "text/html",
     "maxSnapshotsPerUrl": 5000,
     "maxIndexPagesPerUrl": 30
   }

6. Check a domain's history at a glance

   Set outputMode to summary for one row per input with first and last capture, total captures, captures per year, missing years and longest gap. It is charged per summary rather than per capture: $0.004 for an exact URL and $0.01 for a prefix, host or domain.

7. Know what this cannot tell you

   This is archived history, not live monitoring. A change appears only if archive.org captured the page before and after it, so infrequently crawled pages can hide many edits between captures, and very recent changes may not be archived yet. A page that reverts to earlier content is flagged as changed again, which is accurate. To be alerted to future changes as they happen you need a separate monitoring tool that fetches the live page on a schedule.

Sample input

{
  "urls": [
    "https://apify.com/pricing"
  ],
  "matchType": "exact",
  "collapse": "digest",
  "statusFilter": "200",
  "mimeFilter": "text/html",
  "maxSnapshotsPerUrl": 300
}

Sample output

Each row is one archived capture with its time, status, MIME type, change flag and a direct Wayback link.

Run it from code

cURL

curl -X POST "https://api.apify.com/v2/acts/actor-forge~wayback-snapshots/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://apify.com/pricing"], "matchType": "exact", "collapse": "digest", "statusFilter": "200", "mimeFilter": "text/html", "maxSnapshotsPerUrl": 300}'

JavaScript (Node.js)

// Node.js 18+ (built-in fetch). Set APIFY_TOKEN in your environment.
const input = {
  "urls": [
    "https://apify.com/pricing"
  ],
  "matchType": "exact",
  "collapse": "digest",
  "statusFilter": "200",
  "mimeFilter": "text/html",
  "maxSnapshotsPerUrl": 300
};

const res = await fetch(
  "https://api.apify.com/v2/acts/actor-forge~wayback-snapshots/run-sync-get-dataset-items",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.APIFY_TOKEN}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(input),
  }
);
const items = await res.json();
console.log(items.length, items[0]);

Python

# pip install apify-client   (set APIFY_TOKEN in your environment)
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run_input = {
    "urls": [
        "https://apify.com/pricing"
    ],
    "matchType": "exact",
    "collapse": "digest",
    "statusFilter": "200",
    "mimeFilter": "text/html",
    "maxSnapshotsPerUrl": 300
}

run = client.actor("actor-forge/wayback-snapshots").call(run_input=run_input)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

The run-sync-get-dataset-items endpoint waits for the run (up to 5 minutes) and returns the dataset rows directly. For longer runs, start the run asynchronously and read the dataset when it finishes, as the Python example does. Replace APIFY_TOKEN with your own token from the Apify console.

What it costs

Tracking the change history of 20 competitor pricing pages at up to 300 unique versions each is at most 6,000 snapshot rows, or $12.00; fetching the content of 50 of those versions adds 50 x $0.004 = $0.20. A summary of 100 whole domains costs 100 x $0.01 = $1.00.

Tips and pitfalls

• Every snapshot row is charged, so maxSnapshotsPerUrl doubles as your cost cap per input.
• On multi-URL results (prefix, host, domain) collapsing works per URL: daily keeps one capture per day for each URL, not for the whole site.
• Prefix, host and domain results come in URL order and then time, not purely by date; sort by capturedAt yourself.
• archive.org's CDX API is public, so proxyConfiguration is normally not needed. Turn on Apify Proxy only if runs keep hitting rate limits.

FAQ

Actors used in this guide

Related guides

All guides