Web page to Markdown for LLMs and RAG

Before a web page goes into a prompt or a vector store it has to lose its HTML. We measured the four usual ways to do that on four real pages, counting tokens with OpenAI's o200k_base tokenizer (tiktoken), and checked what each one keeps. Every snippet below was run as shown on 22 October 2026.

What we measured

Tokens per page. "Text" is BeautifulSoup(html).get_text() after dropping scripts and styles; "markdownify" converts the whole HTML; the last two are the Shotmint content API with its defaults and with links=false.

Page	Tokens
Wikipedia: Origami Long article, 470 links	Raw HTML 134,361 Text 11,406 markdownify 36,842 API Markdown 18,975 API, links=false 9,488
MDN: Using the Fetch API Docs page with code blocks	Raw HTML 49,236 Text 5,339 markdownify 10,042 API Markdown 4,913 API, links=false 3,191
simonwillison.net: LLMs in 2024 Long blog post	Raw HTML 22,517 Text 9,680 API Markdown 15,423 API, links=false 9,619
quotes.toscrape.com/js Content rendered by JavaScript	Raw HTML 1,484 Text 23 (no quotes at all) markdownify 68 API, links=false 246 (all 10 quotes)

What the numbers say:

Raw HTML is 7 to 14 times bigger than the readable content. Sending it to a model wastes most of the context window on markup.
Plain text looks cheap but carries junk. On Wikipedia it still had "Jump to content", "Privacy policy" and "Log in"; on MDN "Skip to main content" and the cookie notice. It also loses the headings, so you can't chunk by section.
markdownify keeps everything, menus and footers included, so it was the biggest of the non-HTML options.
Links are a third to a half of good Markdown. On these pages, dropping link URLs and images (keeping the link text) cut 35-50% of the tokens. Keep them if your app cites sources inline; drop them for embeddings and summaries.
A plain HTTP fetch sees nothing on JavaScript pages. The quotes page sends an empty shell and builds the quotes in the browser, so requests plus BeautifulSoup got 23 tokens of page chrome. Many docs sites, shops and dashboards work this way.

The one-call version

The content API loads the page in Chromium (so JavaScript runs), finds the main article with Mozilla Readability (the library behind Firefox Reader View) and converts it to Markdown with headings, lists, code blocks and tables. Get a free key (25 calls, no signup) and try:

curl "https://shotmint.creasework.com/v1/content?url=https://quotes.toscrape.com/js/&links=false" \
  -H "X-API-Key: $SHOTMINT_KEY"

The JSON also carries title, byline, published_time, lang, words and truncated, which make good chunk metadata. Use format=text for plain text, and main_content=false for home pages and lists that have no single article. No code needed to try it: website to Markdown runs the same conversion in the browser.

Python: fetch, chunk by heading, write JSONL

Section-sized chunks embed better than fixed-size windows, because each one is about one thing. This splits on #, ## and ### headings, then on paragraphs if a section is still longer than 2,000 characters, and keeps the section title with each chunk. Only requests is needed:

import json, os, re, requests

API = "https://shotmint.creasework.com/v1/content"

def fetch_markdown(url):
    r = requests.get(API, params={"url": url, "links": "false"},
                     headers={"X-API-Key": os.environ["SHOTMINT_KEY"]}, timeout=90)
    r.raise_for_status()  # 400 = bad or unreachable URL, 402 = out of credits
    return r.json()

def chunk(markdown, max_chars=2000):
    """Split on headings, then on paragraphs if a section is still too long."""
    chunks, heading = [], ""
    for section in re.split(r"\n(?=#{1,3} )", markdown):
        if section.startswith("#"):
            heading = section.split("\n", 1)[0].lstrip("# ").strip()
        buf = ""
        for para in section.split("\n\n"):
            if buf and len(buf) + len(para) > max_chars:
                chunks.append((heading, buf.strip()))
                buf = ""
            buf += para + "\n\n"
        if buf.strip():
            chunks.append((heading, buf.strip()))
    return chunks

page = fetch_markdown("https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch")
with open("chunks.jsonl", "w") as f:
    for i, (heading, text) in enumerate(chunk(page["content"])):
        f.write(json.dumps({"id": f"{page['url']}#{i}", "title": page["title"],
                            "section": heading, "text": text}) + "\n")
print(page["title"], "-", page["words"], "words")

On the MDN page this wrote 20 chunks, the longest 1,639 characters, each tagged with its section (empty for the intro, then "Making a request", "Setting the method" and so on). Feed text to your embedding model and store id, title and section next to the vector so answers can cite them.

Node.js: a batch of URLs to Markdown files

For a knowledge base built from many pages, this converts a list of URLs three at a time and writes one Markdown file per page with YAML front matter. Node 18+ (built-in fetch), saved as batch.mjs:

import { mkdir, writeFile } from "node:fs/promises";

const urls = [
  "https://en.wikipedia.org/wiki/Origami",
  "https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch",
  "https://quotes.toscrape.com/js/",
];

async function toMarkdown(url) {
  const params = new URLSearchParams({ url, links: "false" });
  const res = await fetch("https://shotmint.creasework.com/v1/content?" + params, {
    headers: { "X-API-Key": process.env.SHOTMINT_KEY },
  });
  if (!res.ok) throw new Error(`${url}: ${res.status} ${await res.text()}`);
  return res.json();
}

await mkdir("pages", { recursive: true });
// 3 at a time: enough to be fast, few enough to stay clear of rate limits.
for (let i = 0; i < urls.length; i += 3) {
  const batch = urls.slice(i, i + 3);
  const results = await Promise.allSettled(batch.map(toMarkdown));
  for (const [j, r] of results.entries()) {
    if (r.status === "rejected") { console.error(r.reason.message); continue; }
    const p = r.value;
    const front = `---\nurl: ${p.url}\ntitle: ${JSON.stringify(p.title)}\nwords: ${p.words}\n---\n\n`;
    await writeFile(`pages/${i + j}.md`, front + p.content);
    console.log(`pages/${i + j}.md`, p.words, "words", p.title);
  }
}

A failed page is logged and skipped, and a failed call never costs a credit. Status 400 means a bad or unreachable URL, 402 means the key is out of credits.

Pages behind a login, cookie walls, slow pages

Logged-in pages: POST the same parameters as JSON with headers (e.g. an Authorization header) or cookies; see pages behind a login, which works the same way for content.
Cookie banners and ads: block_ads=true, or hide_selectors for one stubborn element.
Content that loads late: wait_until=networkidle or delay (milliseconds).
Agents: the Shotmint MCP server exposes the same conversion as a read_page tool for Claude, Cursor and other MCP clients.

Doing it yourself

The API runs open-source parts you can assemble: Playwright or Puppeteer to render, @mozilla/readability to find the article, Turndown with the GFM plugin for Markdown. Two things we had to fix on top of them: Readability takes the first element with an "author" class out of the content as the byline, which on list pages deleted the first quote's author; and pages with layout tables (Hacker News) come out as raw HTML under the GFM plugin unless you flatten them. If you only convert static pages, Python's trafilatura is a solid self-hosted choice; it doesn't run JavaScript.

What it costs

One page = one credit. €5 buys 2,000 pages, credits never expire and there's no subscription. Repeat calls with cache_ttl set are free for as long as you ask, up to 30 days.

Try it with 25 free renders a month

No signup, no card. Get a key in one click, then buy credits only if you need more (from €5 for 2,000, never expire). From a script, an AI agent, Make or Zapier, POST /v1/keys/free (no body) returns a key as JSON (openapi.json, no-code recipes).

Get a free API key Full API reference

More guides: Use cases

All guides: guides · integrations · comparisons