LangChain web page loader: what WebBaseLoader gets wrong

LangChain's built-in web loaders are the first thing most RAG tutorials reach for. We ran them on four real pages and counted what ends up in the vector store, with OpenAI's o200k_base tokenizer. Three problems came up: menus and footers in every document, nothing at all from JavaScript pages, and error pages indexed as if they were content. Below are the numbers, then a 30-line loader that fixes all three. Everything was run on 22 October 2026 with langchain-community 0.4.2 and langchain-core 1.6.13.

What the built-in loaders return

WebBaseLoader fetches with requests and calls BeautifulSoup's get_text() on the whole page. AsyncHtmlLoader + Html2TextTransformer fetches with aiohttp and converts the HTML to Markdown-ish text. The last column is the Shotmint content API with links=false, the call the loader below makes.

Page	Tokens and what's in them
Wikipedia: Origami	WebBaseLoader 12,157, with "Jump to content", "Privacy policy", "Log in" AsyncHtmlLoader 38: Wikipedia's "Please respect our robot policy" notice (HTTP 403), stored as the document API 9,526, article only
MDN: Using the Fetch API	WebBaseLoader 5,553, with "Skip to main content" and the cookie notice AsyncHtmlLoader 6,553 API 3,191, starting with the page's own intro
quotes.toscrape.com/js Quotes built by JavaScript	WebBaseLoader 34: no quotes, and the heart in the footer became mojibake ("‚ù§") AsyncHtmlLoader 28: no quotes API 246: all 10 quotes and authors
A Wikipedia page that doesn't exist HTTP 404	WebBaseLoader 302 words of menus, titled "This page does not exist 42 - Wikipedia", returned as a normal document AsyncHtmlLoader: the 403 robot notice again Loader below: skipped with a warning

What that means for a RAG pipeline:

Boilerplate gets embedded. Every Wikipedia chunk set carries the same menu and footer text, so a question about "privacy" or "log in" retrieves chunks from every page you loaded.
JavaScript pages come back empty without an error. Both loaders send a plain HTTP request and never run the page's scripts, so docs sites, shops and dashboards built in React or Vue give you their loading shell.
Failures look like successes. WebBaseLoader has a raise_for_status option, but it is off by default, so a 404 page lands in the index. AsyncHtmlLoader stored Wikipedia's bot-blocking notice as the page text: aiohttp's default user agent is blocked, and nothing in the result says so.
langchain-community is being sunset. Importing it now prints a DeprecationWarning pointing to standalone integration packages. A custom loader only needs langchain-core.

A loader that fixes all three

The content API loads each URL in Chromium (so JavaScript runs), keeps the main article with Mozilla Readability (Firefox Reader View's engine), returns Markdown with headings, lists, tables and code blocks, and reports the page's own HTTP status. Save this as shotmint_loader.py. It needs langchain-core and requests, and a free key (25 calls, no signup) in SHOTMINT_KEY:

import logging, os, requests
from langchain_core.document_loaders import BaseLoader
from langchain_core.documents import Document

API = "https://shotmint.creasework.com/v1/content"
META = ("title", "description", "byline", "site_name", "published_time", "lang", "words")


class ShotmintLoader(BaseLoader):
    """Load web pages as clean Markdown (main content only) with the Shotmint content API."""

    def __init__(self, urls, links=False, api_key=None):
        self.urls = [urls] if isinstance(urls, str) else list(urls)
        self.links = links  # False drops link URLs and images: fewer tokens
        self.api_key = api_key or os.environ["SHOTMINT_KEY"]

    def lazy_load(self):
        for url in self.urls:
            r = requests.get(API, params={"url": url, "links": str(self.links).lower()},
                             headers={"X-API-Key": self.api_key}, timeout=90)
            if r.status_code == 400:  # the page could not be loaded (bad URL, DNS, timeout)
                logging.warning("skipped %s: %s", url, r.json().get("error"))
                continue
            r.raise_for_status()  # 401 bad key, 402 out of credits, 429 rate limit
            d = r.json()
            if d["status"] and d["status"] >= 400:  # a 404 or a bot wall, not the page you wanted
                logging.warning("skipped %s: the page answered HTTP %s", url, d["status"])
                continue
            meta = {k: d[k] for k in META if d.get(k) is not None}
            yield Document(page_content=d["content"], metadata={"source": d["final_url"] or url, **meta})

Each document's metadata holds source (the URL after redirects), title, description, byline, site_name, published_time, lang and words, so you can filter or cite by them later. Set links=True if your answers should link to sources inline: on these pages, link URLs and images were a third to a half of the Markdown tokens.

Load, split by heading, then by size

Because the output is real Markdown, MarkdownHeaderTextSplitter can cut each page into sections, and each chunk keeps its section titles as metadata. Sections longer than 2,000 characters are split again:

from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
from shotmint_loader import ShotmintLoader

docs = ShotmintLoader([
    "https://en.wikipedia.org/wiki/Origami",
    "https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch",
    "https://quotes.toscrape.com/js/",
    "https://en.wikipedia.org/wiki/This_page_does_not_exist_42",
]).load()

by_heading = MarkdownHeaderTextSplitter([("#", "h1"), ("##", "h2"), ("###", "h3")])
by_size = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
chunks = []
for doc in docs:
    for part in by_size.split_documents(by_heading.split_text(doc.page_content)):
        part.metadata = {**doc.metadata, **part.metadata}
        chunks.append(part)

for doc in docs:
    print(doc.metadata["source"], doc.metadata["words"], "words")
print(len(chunks), "chunks; first:", chunks[0].metadata)

Output from our run:

WARNING:root:skipped https://en.wikipedia.org/wiki/This_page_does_not_exist_42: the page answered HTTP 404
https://en.wikipedia.org/wiki/Origami 5882 words
https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch 2045 words
https://quotes.toscrape.com/js/ 183 words
58 chunks; first: {'source': 'https://en.wikipedia.org/wiki/Origami', 'title': 'Origami', ...}

From here the chunks go to any vector store, e.g. Chroma.from_documents(chunks, embeddings).

Errors and what the loader does

Case	What happens
Page answers 404, 403, 500	API returns 200 with `status` set to the page's code The loader logs a warning and skips it
Host unreachable or bad URL	API returns 400 with an `error` message Skipped with a warning
Bad key, no credits, rate limit	401, 402, 429 Raised: fix the key or top up
Login-only pages	POST the same call with `cookies` or `headers` See the API reference

When the built-in loaders are fine

Static pages you control (your own docs, a sitemap of plain HTML): WebBaseLoader with raise_for_status=True and a bs_kwargs parse filter for your main element costs nothing per page.
Thousands of pages from one static site: crawl with a SitemapLoader and a custom parsing function for that site.
Pages with JavaScript content, many different sites, or pages you don't control: render them, or you index shells and menus.

Related: web page to Markdown for LLMs (token counts for HTML, text, markdownify), the free website to Markdown tool, and the MCP server if an agent should read pages itself.

What it costs

One page = one credit. €5 buys 2,000 pages, credits never expire and there's no subscription.

Try it with 25 free renders a month

No signup, no card. Get a key in one click, then buy credits only if you need more (from €5 for 2,000, never expire). From a script, an AI agent, Make or Zapier, POST /v1/keys/free (no body) returns a key as JSON (openapi.json, no-code recipes).

Get a free API key Full API reference

More guides: Use cases

All guides: guides · integrations · comparisons