@sdxc/spam

Spam scoring for user content: local rules, reputation checks and a classifier add weighted signals, and the total becomes a ham, unsure or spam verdict with every reason attached.

Installation

npm add @sdxc/spam

Remote checks and stores answer @sdxc/result values, and the classifier's database store runs on remix/data-table from remix.

Usage

Score a submission

import { createSpamFilter, DEFAULT_RULES } from "@sdxc/spam";

let filter = createSpamFilter({ checks: DEFAULT_RULES });

let assessment = await filter.check({
	content: "cheap watches https://a.example https://b.example",
});
assessment.verdict; // "ham" | "unsure" | "spam"
assessment.score; // 5
assessment.signals; // [{ check: "links.count", score: 2, detail: "2 links" }, …]

Describe the submission

Only content is required. Every other field lets more rules fire.

let assessment = await filter.check({
	content: String(form.get("comment") ?? ""),
	format: "markdown",
	author: {
		name: String(form.get("name") ?? ""),
		email: String(form.get("email") ?? ""),
		url: String(form.get("website") ?? ""),
		ip: request.headers.get("CF-Connecting-IP") ?? undefined,
		userAgent: request.headers.get("User-Agent") ?? undefined,
	},
	renderedAt, // from a signed field the visitor cannot forge
	languages: ["en", "es"],
});

Add reputation checks

import { createSpamFilter, DEFAULT_RULES } from "@sdxc/spam";
import { akismet } from "@sdxc/spam/akismet";
import { authorEmail } from "@sdxc/spam/author-email";
import { linkBlocklist } from "@sdxc/spam/link-blocklist";
import { stopForumSpam } from "@sdxc/spam/stop-forum-spam";

let filter = createSpamFilter({
	checks: [
		...DEFAULT_RULES,
		authorEmail(),
		stopForumSpam(),
		linkBlocklist(),
		akismet({ apiKey: env.AKISMET_KEY, blog: "https://example.com" }),
	],
	thresholds: { unsure: 5, spam: 10 },
	timeout: 1500,
});

Local checks run first, in order. Remote checks run in parallel only while the score is below the spam threshold, and a check that fails or overruns timeout adds nothing and is listed in assessment.failures.

Learn from moderators

let failures = await filter.report(submission, "ham");

Every check with a report method receives the decision; failures lists the ones that could not record it.

API

@sdxc/spam

createSpamFilter(options: SpamFilter.Options): SpamFilter

Builds a filter over options.checks, with thresholds (default DEFAULT_THRESHOLDS) and a timeout in milliseconds for each remote and escalation check (default 1500).

filter.check(submission: Submission): Promise<SpamFilter.Assessment>

Runs local checks in order, then remote checks in parallel while the score is below spam, then escalation checks in parallel while it sits in the unsure band. A check that fails, throws or times out adds nothing and is listed in failures.

filter.report(submission: Submission, label: Label): Promise<SpamFilter.Failure[]>

Sends a moderator's decision to every check with a report method, in parallel, and resolves the ones that failed.

verdictFor(score: number, thresholds: SpamFilter.Thresholds): SpamFilter.Verdict

The mapping check uses: at or above spam is spam, at or above unsure is unsure, below is ham. Use it to re-score stored signals under new thresholds.

DEFAULT_THRESHOLDS

{ unsure: 5, spam: 10 }.

SpamCheckError

A check's failure. code is unavailable, timeout, misconfigured (a key or option the provider refused) or invalid-response; the message carries the provider's detail.

Rules

Each rule is a function taking an options object of weights and limits and returning a local SpamCheck. DEFAULT_RULES holds every rule at its defaults, tuned together so a submission needs more than one kind of evidence to reach the spam threshold.

URL_SHORTENERS and ABUSED_TLDS are the lists linkTargets() uses unless shorteners or abusedTlds replace them.

Types

• Submission: { content, format?, author?, renderedAt?, submittedAt?, languages? }. format is "text" (default), "markdown" or "html"; author holds name, email, url, ip and userAgent; submittedAt defaults to the time of the check; languages are BCP 47 tags.

• Signal: { check, score, detail? }. A positive score points to spam and a negative one to ham, so a provider can vouch for an author.

• SpamCheck: name, stage ("local" | "remote" | "escalation"), a check(submission, options) answering Signal[] or Promise<Result<Signal[], SpamCheckError>>, and an optional report(submission, label). options.signal aborts at the timeout; options.score and options.signals are what earlier stages found.

• SpamFilter: the filter, with SpamFilter.Options, Thresholds, Verdict, Failure ({ check, error }) and Assessment ({ verdict, score, signals, failures }).

• Label: "spam" | "ham".

@sdxc/spam/author-email

authorEmail(options?: { disposableScore?: number }): SpamCheck

A local rule scoring an author.email on a disposable-mail domain, subdomains included (author-email.disposable, default 3). It ships apart from DEFAULT_RULES because the domain list it bundles is large.

@sdxc/spam/stop-forum-spam

stopForumSpam(options?: stopForumSpam.Options): SpamCheck

A remote check looking the author's IP, email and name up in StopForumSpam. Each reported field scores its weight times the reported confidence (ipScore 5, emailScore 6, usernameScore 2), reports older than maxAgeDays (90) are ignored, and the total caps at maxScore (10). Signals are stop-forum-spam.ip, .email and .username. endpoint (default STOP_FORUM_SPAM_ENDPOINT) pins a regional mirror.

@sdxc/spam/link-blocklist

linkBlocklist(options?: linkBlocklist.Options): SpamCheck

A remote check looking every linked domain and author.url up in DNS URI blocklists over DNS-over-HTTPS. zones defaults to ["surbl", "uribl"]; a Spamhaus Data Query Service dqsKey adds "spamhaus". A listing scores listedScore (6), a low-confidence one weakScore (2), across at most maxHosts (10) domains, capped at maxScore (12). Signals are link-blocklist.spamhaus, .surbl and .uribl. A blocklist refusing the query fails as misconfigured; resolver routes queries through a DoH resolver the zones accept.

@sdxc/spam/akismet

akismet(options: akismet.Options): SpamCheck

A remote check sending each submission to Akismet with apiKey and blog. It adds akismet.spam (spamScore 8), akismet.discard for blatant spam (discardScore 12) or akismet.ham (hamScore -2). Akismet requires author.ip; a submission without one draws nothing. commentType (default "comment"), blogLang and isTest pass through, and report forwards decisions to submit-spam / submit-ham within reportTimeout (10000 ms).

@sdxc/spam/bayes

bayes(options: bayes.Options): SpamCheck

A local naive Bayes classifier trained by every filter.report. It adds bayes.spam (spamScore 6) at or above spamCut (0.9) and bayes.ham (hamScore -4) at or below hamCut (0.2), naming the deciding tokens in the detail, and stays silent until store holds minDocuments (20) reports of each label. strength, prior, interestingTokens and minDeviation tune the combination; the tokenize options shape the tokens.

tokenize(submission: Submission, options?: tokenize.Options): string[]

The tokens the classifier counts: content words between minLength (3) and maxLength (24), a host: token per linked host and the author's website, and an email-domain: token unless emailDomain is false, up to maxTokens (200).

TokenStore

Where counts live: read(tokens) and increment(tokens, label), both answering a Result. TokenStore.Counts and TokenStore.Tally are the shapes read answers.

MemoryTokenStore

Keeps counts in the instance, for tests and local development.

new DataTableTokenStore(db: Database)

Stores counts in the spamTokens table of a remix/data-table database, one upsert per write, so concurrent reports never lose a count. SPAM_TOKENS_SCHEMA_SQL creates the table in a migration; SpamTokenRow is its row type.

@sdxc/spam/workers-ai

workersAi(options: workersAi.Options): SpamCheck

An escalation check asking a Workers AI model about submissions left in the unsure band. Pass the binding as ai (any object with a matching run, workersAi.Binding) and describe the audience in site. A spam answer adds workers-ai.spam at spamScore × confidence (default 5), a ham answer workers-ai.ham at -(hamScore × confidence) (default 5), with the model's reason cut to maxReasonLength (200) as the detail. Content is cut to maxContentLength (4000). model defaults to DEFAULT_WORKERS_AI_MODEL, and takes any text model with JSON mode.

@sdxc/spam/memory

new MemoryCheck(options?: MemoryCheck.Options)

A scripted check for tests, named "memory" in the "remote" stage unless name and stage say otherwise. A call answers the next queued outcome, else signals.

• signalNext(signals) / failNext(code, message?) queue the next answer; both chain.

• failReports(code) makes every later report fail.

• calls, last and reports record what the check received.

• reset() forgets calls, reports and queued outcomes.

Pattern: Refuse bots before scoring

@sdxc/honeypot refuses what no person produces before any rule runs, and hands the signed render time the timing rule reads.

import { Honeypot } from "@sdxc/honeypot";
import { honeypot } from "@sdxc/honeypot/middleware";
import { unwrap } from "@sdxc/result";
import { createSpamFilter, DEFAULT_RULES } from "@sdxc/spam";

let trap = new Honeypot({ secret: env.HONEYPOT_SECRET });
let filter = createSpamFilter({ checks: DEFAULT_RULES });

router.post(routes.comments.create, {
	middleware: [honeypot(trap)],
	async handler(ctx) {
		let form = await ctx.request.formData();
		let assessment = await filter.check({
			content: String(form.get("content") ?? ""),
			renderedAt: unwrap(ctx.honeypotOutcome).renderedAt,
		});
		return saveComment(form, assessment);
	},
});

Pattern: Hold, discard or publish

The verdict is a recommendation; the app decides what each one does. Answering spam like a success teaches a bot nothing.

import type { Submission } from "@sdxc/spam";

import { createSpamFilter, DEFAULT_RULES } from "@sdxc/spam";

let filter = createSpamFilter({ checks: DEFAULT_RULES });

async function submitComment(submission: Submission) {
	let assessment = await filter.check(submission);
	if (assessment.verdict === "spam") return { status: "received" };
	let comment = await saveComment(submission, {
		published: assessment.verdict === "ham",
		reasons: assessment.signals,
	});
	return { status: comment.published ? "published" : "received" };
}

Pattern: Train a classifier on your moderation queue

The classifier reads the site's own history, so build it per request from that tenant's database, and send every moderator decision back through report.

import type { Database } from "remix/data-table";
import type { Label, Submission } from "@sdxc/spam";

import { createSpamFilter, DEFAULT_RULES } from "@sdxc/spam";
import { bayes, DataTableTokenStore } from "@sdxc/spam/bayes";

function spamFilter(db: Database) {
	return createSpamFilter({
		checks: [...DEFAULT_RULES, bayes({ store: new DataTableTokenStore(db) })],
	});
}

async function moderate(db: Database, submission: Submission, label: Label) {
	let failures = await spamFilter(db).report(submission, label);
	return failures.map(({ check, error }) => `${check}: ${error.code}`);
}

Create the table in a migration from SPAM_TOKENS_SCHEMA_SQL, and store each submission exactly as it was checked, so training sees the same tokens.

Pattern: Escalate the unsure band to a model

Rules settle most submissions; a model call runs only on the ones they leave unsure.

import { createSpamFilter, DEFAULT_RULES } from "@sdxc/spam";
import { workersAi } from "@sdxc/spam/workers-ai";

let filter = createSpamFilter({
	checks: [...DEFAULT_RULES, workersAi({ ai: env.AI, site: "comments on a cooking blog" })],
	timeout: 3000,
});

let assessment = await filter.check({ content: "Great recipe! Visit my store for cheap pans" });
assessment.signals.find((signal) => signal.check.startsWith("workers-ai."))?.detail;

Pattern: Write a rule for your site

A check is an object with a name, a stage and a check function; its signals add to every other rule's.

import type { SpamCheck } from "@sdxc/spam";

import { createSpamFilter, DEFAULT_RULES } from "@sdxc/spam";

let competitorLinks: SpamCheck = {
	name: "competitor-links",
	stage: "local",
	check: (submission) =>
		/rival\.example/i.test(submission.content)
			? [{ check: "competitor-links", score: 4, detail: "links to rival.example" }]
			: [],
};

let filter = createSpamFilter({ checks: [...DEFAULT_RULES, competitorLinks] });

Pattern: Reweight a rule

DEFAULT_RULES is the rule functions called with defaults. Swap one for your own call.

import { contactBait, createSpamFilter, DEFAULT_RULES, links } from "@sdxc/spam";

let filter = createSpamFilter({
	checks: [
		...DEFAULT_RULES.filter((rule) => rule.name !== "contact-bait" && rule.name !== "links"),
		contactBait({ phoneScore: 0 }), // the form asks for a phone number
		links({ freeLinks: 3 }),
	],
});

Pattern: Test code that filters

MemoryCheck fixes the evidence, so a test reaches every verdict and failure without a provider. MemoryTokenStore does the same for bayes.

import { createSpamFilter } from "@sdxc/spam";
import { MemoryCheck } from "@sdxc/spam/memory";
import { expect, test } from "vitest";

test("holds an unsure comment and survives a provider outage", async () => {
	let rules = new MemoryCheck({ stage: "local", signals: [{ check: "test", score: 6 }] });
	let provider = new MemoryCheck().failNext("unavailable");
	let filter = createSpamFilter({ checks: [rules, provider] });

	let assessment = await filter.check({ content: "hello" });

	expect(assessment.verdict).toBe("unsure");
	expect(assessment.failures[0]?.error.code).toBe("unavailable");
	expect(provider.last).toEqual({ content: "hello" });
});