sdxc

[ Content & formats ]

@sdxc/distill

Extract the article from a web page: bounded fetch, scoring and sanitizing

npm add @sdxc/distill
pnpm add @sdxc/distill
yarn add @sdxc/distill
bun add @sdxc/distill
Installs with
@sdxc/html@sdxc/outbound@sdxc/result@sdxc/robots
Used by
reader, blog
Source
packages/distill

Extract the article from a web page: bounded fetch, scoring and sanitizing.

A page wraps its article in navigation, share rails, comments and related-post blocks. This package scores the page's containers to find the article, reads its title, byline and excerpt, and sanitizes the body, treating every page as untrusted markup.

Installation

npm add @sdxc/distill

Every function answers a Result from @sdxc/result. To honour robots.txt, also install @sdxc/robots.

Usage

Distill a URL

import { distill } from "@sdxc/distill";
import { isFailure } from "@sdxc/result";

let article = await distill("https://example.com/post", {
	userAgent: "MyReader/1.0 (+https://example.com/about)",
});

if (isFailure(article)) {
	article.error.outcome; // "refused" | "timeout" | "empty"
} else {
	article.data.html; // sanitized body
	article.data.title; // string | null
	article.data.byline; // string | null
	article.data.excerpt; // string | null
}

userAgent is required, so a publisher can tell who is asking.

Honour robots.txt

import { distill } from "@sdxc/distill";
import { fetchRobots } from "@sdxc/robots/fetch";

let url = "https://example.com/post";
let userAgent = "MyReader/1.0 (+https://example.com/about)";

let robots = await fetchRobots(url, { userAgent });
let article = await distill(url, { userAgent, robots });

A path the rules disallow fails with DistillRefusedError before any request goes out.

Distill markup you already have

import { distillFrom } from "@sdxc/distill";

let article = distillFrom(html, "https://example.com/post");

The URL is the address the page was served from; every relative URL resolves against it.

Summarize any page

import { summaryFrom } from "@sdxc/distill";

let summary = summaryFrom(html, "https://example.com/watch?v=1");
// success({ title: "A Video", excerpt: "Five minutes on harbours.", url: "https://example.com/watch?v=1" })

A landing page or a video page has no article but still has a headline and a summary.

API

distill(url: string, options: Distill.Options): Promise<Result<Distill.Retrieved, DistillError>>

Fetches a page under the bounds below and reads the article out of it. A response that is not text/html fails as empty.

  • options.userAgent: the User-Agent the request is sent with. Required.

  • options.robots: the outcome fetchRobots from @sdxc/robots/fetch answered for the origin. Parsed rules are evaluated for userAgent, an unavailable file (4xx) allows everything, and an unreachable one (5xx, 429, network failure) refuses everything.

  • options.maxBytes, options.maxRedirects, options.timeoutMs: move the default bounds.

  • options.signal: aborts the retrieval alongside its deadline.

Bounds every request runs under:

  • HTTP(S) only, checked on the first request and every redirect hop. URLs with credentials, IP literals, single-label hosts and reserved suffixes (localhost, local, internal, test) are refused.

  • MAX_REDIRECTS (5) redirects, MAX_BYTES (2 MB) of body counted off the stream, and TIMEOUT_MS (8 seconds) for the whole chain plus the body read.

  • No cookies or credentials; only Accept and User-Agent headers.

A 401, 402, 403, 429 or 451 fails as refused; any other non-2xx status, like a spent bound, fails as timeout.

distillFrom(source: string, url: string): Result<Distill.Article, DistillEmptyError>

Scores, sanitizes and reads metadata from markup in hand. Fails when the source has no markup or no article.

summaryFrom(source: string, url: string): Result<Distill.Summary, DistillEmptyError>

The page's title, excerpt and canonical url, read as distillFrom reads them. Fails only when the source carries no markup.

addressable(url: string): Result<URL, DistillRefusedError>

Applies the same host rule as distill without making a request, so a link can be refused before it is queued.

MAX_BYTES, MAX_REDIRECTS, TIMEOUT_MS

The default bounds: 2097152, 5 and 8000.

EXCERPT_LENGTH

300. An excerpt is the page's og:description, description or twitter:description, else the article's first paragraph, cut at the last whole word within this many characters.

Errors

Each error carries an outcome to render copy from.

ErroroutcomeMeaning
DistillRefusedError"refused"The site said no, robots.txt disallows it, or the host is refused
DistillLimitError"timeout"Time, bytes or redirects ran out, or the site failed
DistillEmptyError"empty"The page carried no article or was not HTML

DistillError is the union of the three.

Sanitization

The body is cleaned by an allow-list: prose, lists, tables, figures, links and images survive. href and src keep only http:, https: and mailto: URLs, relative URLs resolve against the article's canonical address, and every image gets referrerpolicy="no-referrer" and loading="lazy". Scripts, iframes, forms, on* handlers, style, class and id are removed. Images still load from the publisher; route them through an image proxy when that matters.

Types

  • Distill.Article: html, title, byline, excerpt, canonical url, chars (length of the readable text) and sanitized (the SanitizeReport from @sdxc/html: removed elements, attributes and URLs).

  • Distill.Retrieved: an Article plus bytes read off the wire and mayCache, false when the response's X-Robots-Tag carries noarchive for every agent or for userAgent.

  • Distill.Summary: title, excerpt and url.

  • Distill.Options: the options distill takes.

  • Distill.Outcome: "extracted" | "refused" | "timeout" | "empty", for recording what an attempt amounted to.

Pattern: Copy For Every Outcome

Map each failure to what a reader sees, falling back to the excerpt the link came with.

import type { Distill } from "@sdxc/distill";

import { distill } from "@sdxc/distill";
import { isFailure } from "@sdxc/result";

let MESSAGES: Record<Exclude<Distill.Outcome, "extracted">, string> = {
	refused: "This site does not allow reading here.",
	timeout: "The page took too long to load.",
	empty: "There is no article on this page.",
};

async function readerView(url: string, excerpt: string) {
	let article = await distill(url, { userAgent: "MyReader/1.0 (+https://example.com/about)" });
	if (isFailure(article)) {
		return { html: null, excerpt, notice: MESSAGES[article.error.outcome] };
	}
	return { html: article.data.html, excerpt: article.data.excerpt ?? excerpt, notice: null };
}

Pattern: Caching By URL

A distilled article is the same for everyone, so the URL is the whole cache key. Cache failures for less time than successes, and skip storing a page whose mayCache is false.

import { distill } from "@sdxc/distill";
import { isFailure } from "@sdxc/result";

interface ArticleStore {
	read(url: string): Promise<{ html: string | null } | null>;
	write(url: string, entry: { html: string | null }, options: { ttl: number }): Promise<void>;
}

async function articleFor(url: string, store: ArticleStore): Promise<string | null> {
	let hit = await store.read(url);
	if (hit !== null) return hit.html;

	let article = await distill(url, { userAgent: "MyReader/1.0 (+https://example.com/about)" });

	if (isFailure(article)) {
		await store.write(url, { html: null }, { ttl: 60 * 60 });
		return null;
	}

	if (article.data.mayCache) {
		await store.write(url, { html: article.data.html }, { ttl: 7 * 24 * 60 * 60 });
	}

	return article.data.html;
}

Pattern: Keeping The Article Only When It Adds Something

A teaser, a consent wall or an error page can distill cleanly and still carry less than the excerpt you already have. Compare chars against it.

import { distill } from "@sdxc/distill";
import { isFailure } from "@sdxc/result";

async function betterThanExcerpt(url: string, excerpt: string): Promise<string | null> {
	let article = await distill(url, { userAgent: "MyReader/1.0 (+https://example.com/about)" });
	if (isFailure(article)) return null;
	return article.data.chars > excerpt.length ? article.data.html : null;
}

Pattern: Robots Rules Cached Per Origin

fetchRobots answers an outcome with a lifetimeMs, so one fetch serves every article on the origin until it expires.

import type { RobotsFetch } from "@sdxc/robots/fetch";

import { distill } from "@sdxc/distill";
import { fetchRobots } from "@sdxc/robots/fetch";

let USER_AGENT = "MyReader/1.0 (+https://example.com/about)";
let ROBOTS = new Map<string, { outcome: RobotsFetch.Outcome; expiresAt: number }>();

async function robotsFor(url: string): Promise<RobotsFetch.Outcome> {
	let origin = new URL(url).origin;
	let cached = ROBOTS.get(origin);
	if (cached !== undefined && cached.expiresAt > Date.now()) return cached.outcome;

	let outcome = await fetchRobots(url, { userAgent: USER_AGENT });
	ROBOTS.set(origin, { outcome, expiresAt: Date.now() + outcome.lifetimeMs });
	return outcome;
}

async function politeDistill(url: string) {
	return distill(url, { userAgent: USER_AGENT, robots: await robotsFor(url) });
}

Pattern: Background Distillation With Tighter Bounds

Refuse an unaddressable link before it reaches the queue, then give the job its own budget and cancel it with the job.

import { addressable, distill } from "@sdxc/distill";
import { isFailure } from "@sdxc/result";

function enqueueIfAddressable(link: string, queue: { push(url: string): void }): boolean {
	let url = addressable(link);
	if (isFailure(url)) return false;
	queue.push(url.data.href);
	return true;
}

async function runJob(url: string, signal: AbortSignal) {
	return distill(url, {
		userAgent: "MyReader/1.0 (+https://example.com/about)",
		maxBytes: 512 * 1024,
		maxRedirects: 2,
		timeoutMs: 20_000,
		signal,
	});
}

Pattern: Testing Against A Fixture

distillFrom runs the same scoring and sanitizing as distill, so a saved page tests extraction without a network.

import { readFile } from "node:fs/promises";

import { distillFrom, summaryFrom } from "@sdxc/distill";
import { isSuccess } from "@sdxc/result";
import { expect, test } from "vitest";

test("extracts the article from a saved page", async () => {
	let html = await readFile("fixtures/post.html", "utf8");
	let article = distillFrom(html, "https://example.com/post");

	expect(isSuccess(article)).toBe(true);
	if (!isSuccess(article)) return;
	expect(article.data.html).not.toContain("<script");
	expect(article.data.sanitized.removedElements).toBeGreaterThan(0);

	let summary = summaryFrom(html, "https://example.com/post");
	expect(isSuccess(summary) && summary.data.title).toBe(article.data.title);
});

Written by Sergio Xalambrí. Follow @sergiodxa for new packages, or sponsor the work.