How to Convert PDF to Markdown for RAG Pipelines

Retrieval quality depends heavily on what you feed the embedder. Plain text extracted from a PDF loses the headings that tell you where one topic ends and the next begins, and naive fixed-size splitting cuts sections in half. Markdown keeps that structure, so you can chunk along it.

The usual approach is a stack of libraries: one PDF parser for digital files, an OCR engine for scans, and glue code for the cases where they disagree. The Any URL to Markdown actor reads the PDF text layer with heading and paragraph detection, falls back to Tesseract OCR for pages without text, and returns Markdown plus a token estimate in one row per file.

This guide converts a batch of PDFs, filters out failures, splits the Markdown into heading-aware chunks of a target size in Python, and writes JSONL ready for your vector store.

Step by step

1. Collect the PDF links

   List the PDFs you want in your knowledge base as URLs. The actor fetches each link itself, so the files must be reachable without a login; password-protected PDFs cannot be read. You can pass up to 1,000 links per run in urls, and duplicates are removed. If your PDFs are on your own storage, use pre-signed or public links.

2. Set PDF and OCR limits

   maxPdfPages (500 by default) caps how many pages of each PDF are read. Scanned pages are read by OCR when ocr is on, up to maxOcrPages per PDF (20 by default, $0.003 each); pages beyond that are left out and the row is marked truncated. If your corpus is all digital, set ocr to false so scanned files are reported for free instead of being charged.

   {
     "urls": [
       "https://arxiv.org/pdf/1706.03762",
       "https://example.com/reports/annual-report-2025.pdf"
     ],
     "ocr": true,
     "ocrLanguages": ["eng"],
     "maxOcrPages": 50,
     "maxPdfPages": 500
   }

3. Run the conversion and keep the good rows

   Each result has url, title, markdown, tokenEstimate, route and a status. Failed or unsupported files come back as rows with an error and are not charged. Keep only rows that have Markdown, and log the rest so you can follow up on them.

   import os
   from apify_client import ApifyClient
   
   client = ApifyClient(os.environ["APIFY_TOKEN"])
   run = client.actor("actor-forge/any-url-to-markdown").call(run_input={
       "urls": [
           "https://arxiv.org/pdf/1706.03762",
       ],
       "maxOcrPages": 50,
   })
   
   docs, failed = [], []
   for item in client.dataset(run["defaultDatasetId"]).iterate_items():
       if item.get("markdown"):
           docs.append(item)
       else:
           failed.append((item.get("url"), item.get("error")))
   
   print(f"{len(docs)} converted, {len(failed)} failed")
   for url, err in failed:
       print("FAILED", url, err)

4. Split the Markdown by heading into token-sized chunks

   Split each document at Markdown headings first, so a chunk never straddles two sections, then pack paragraphs into chunks up to a target size. The actor's tokenEstimate is characters divided by 4, so the same rule is used here to stay consistent. Each chunk keeps the heading path as context, which helps retrieval when a paragraph on its own is ambiguous.

   import re
   
   HEADING = re.compile(r"^(#{1,6})\s+(.*)$")
   
   def est_tokens(text):
       return len(text) // 4  # same rule as tokenEstimate
   
   def split_sections(markdown):
       """Yield (heading_path, body) pairs, one per Markdown section."""
       path, body = [], []
       for line in markdown.splitlines():
           m = HEADING.match(line)
           if m:
               if body:
                   yield " > ".join(path), "\n".join(body).strip()
               level = len(m.group(1))
               path = path[: level - 1] + [m.group(2).strip()]
               body = []
           else:
               body.append(line)
       if body:
           yield " > ".join(path), "\n".join(body).strip()
   
   def chunk_markdown(markdown, max_tokens=500):
       chunks = []
       for heading, body in split_sections(markdown):
           if not body:
               continue
           current = ""
           for para in re.split(r"\n\s*\n", body):
               candidate = (current + "\n\n" + para).strip()
               if current and est_tokens(candidate) > max_tokens:
                   chunks.append({"heading": heading, "text": current})
                   current = para
               else:
                   current = candidate
           if current:
               chunks.append({"heading": heading, "text": current})
       return chunks

5. Write JSONL for your vector store

   Attach the source URL, title and heading path to every chunk so answers can cite where they came from. JSONL loads easily into most vector databases and into LangChain or LlamaIndex document loaders.

   import json
   
   with open("chunks.jsonl", "w", encoding="utf-8") as f:
       for doc in docs:
           for i, ch in enumerate(chunk_markdown(doc["markdown"], max_tokens=500)):
               f.write(json.dumps({
                   "id": f"{doc['url']}#{i}",
                   "source": doc["url"],
                   "title": doc.get("title"),
                   "heading": ch["heading"],
                   "text": ch["text"],
               }, ensure_ascii=False) + "\n")

6. Handle heavily scanned or non-English PDFs

   OCR in this actor covers 7 languages (English, German, French, Spanish, Italian, Portuguese and Dutch) and up to maxOcrPages pages per PDF. For long scanned documents, other languages or per-page confidence scores, run those files through PDF OCR instead and chunk its page text the same way. Note that OCR output is plain text without Markdown headings, so the chunker falls back to paragraph packing.

Sample input

{
  "url": "https://arxiv.org/pdf/1706.03762",
  "ocr": true,
  "ocrLanguages": [
    "eng"
  ],
  "maxOcrPages": 20,
  "maxPdfPages": 500
}

Sample output

Each converted URL becomes one row; for PDFs, the markdown column holds the full document and tokenEstimate tells you how much context it will use.

Run it from code

cURL

curl -X POST "https://api.apify.com/v2/acts/actor-forge~any-url-to-markdown/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://arxiv.org/pdf/1706.03762", "ocr": true, "ocrLanguages": ["eng"], "maxOcrPages": 20, "maxPdfPages": 500}'

JavaScript (Node.js)

// Node.js 18+ (built-in fetch). Set APIFY_TOKEN in your environment.
const input = {
  "url": "https://arxiv.org/pdf/1706.03762",
  "ocr": true,
  "ocrLanguages": [
    "eng"
  ],
  "maxOcrPages": 20,
  "maxPdfPages": 500
};

const res = await fetch(
  "https://api.apify.com/v2/acts/actor-forge~any-url-to-markdown/run-sync-get-dataset-items",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.APIFY_TOKEN}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(input),
  }
);
const items = await res.json();
console.log(items.length, items[0]);

Python

# pip install apify-client   (set APIFY_TOKEN in your environment)
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run_input = {
    "url": "https://arxiv.org/pdf/1706.03762",
    "ocr": True,
    "ocrLanguages": [
        "eng"
    ],
    "maxOcrPages": 20,
    "maxPdfPages": 500
}

run = client.actor("actor-forge/any-url-to-markdown").call(run_input=run_input)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

The run-sync-get-dataset-items endpoint waits for the run (up to 5 minutes) and returns the dataset rows directly. For longer runs, start the run asynchronously and read the dataset when it finishes, as the Python example does. Replace APIFY_TOKEN with your own token from the Apify console.

What it costs

Converting 1,000 digital PDFs of up to 50 pages each costs 1,000 x $0.002 = $2.00. If 100 of those were 120-page reports instead, each counts three times, adding 100 x 2 x $0.002 = $0.40, and every scanned page read by OCR adds $0.003.

Tips and pitfalls

• Check tokenEstimate before chunking to spot unusually large files; very large results are cut at about 3.5 million characters and marked truncated.
• Tables in digital PDFs are detected as layout, but complex tables can still come out imperfect; spot-check a few before indexing financial reports.
• Keep respectRobotsTxt on for public sites. If a host answers 429 or 403, the remaining URLs on that host are skipped as free error rows, so spread big batches over time.
• Files are limited to 50 MB. Split or compress larger PDFs before converting.

FAQ

Actors used in this guide

Related guides

All guides