profanity-checker

A Cloudflare Worker that checks text and usernames for profanity using TypeSafe's Jev (typesafe/jev on Workers AI).

Jev is a structured evaluation model: instead of generating text, it answers typed yes/no questions with calibrated probabilities. This Worker asks it one or two narrow questions per request and turns the probabilities into a boolean verdict in code. OpenAPI docs are generated from the schema via chanfana + Hono.

API

POST /api/profanity-check

{
  "text": "This movie was damn good.",
  "kind": "text"
}

Field	Type	Default	Description
text	string (required)	—	The text or username to check
kind	"text" | "username"	"text"	username additionally catches disguised profanity (phonetic gags, look-alike spellings)

Response:

{
  "success": true,
  "text": "This movie was damn good.",
  "kind": "text",
  "is_profane": false,
  "probability": 0.18,
  "probabilities": { "literal": 0.18 },
  "threshold": 0.5
}

• kind: "text" asks one question: does the text contain profanity? Mild words like "damn" or "hell" used non-aggressively are allowed.
• kind: "username" asks two questions in parallel (one inference call): literal profanity in the handle, and profanity disguised phonetically or with look-alike characters (mike_hunt, a55h0le). The handle is flagged when either question clears the threshold.
• probability is the max across questions; probabilities keeps the raw per-question values so you can apply your own policy.

Using it from another Worker

Deploy this Worker once, then call it from your other Workers.

Option A — Service Binding (recommended, same account). No public URL, no latency through the edge, no auth needed. In the consumer Worker's wrangler.jsonc:

{
  "services": [
    { "binding": "PROFANITY", "service": "profanity-checker" }
  ]
}

Then in the consumer Worker:

interface Env {
  PROFANITY: Fetcher;
}

const res = await env.PROFANITY.fetch(
  "https://profanity-checker/api/profanity-check",
  {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ text: username, kind: "username" }),
  },
);
const verdict = await res.json(); // { is_profane, probability, ... }

Option B — Plain fetch to the deployed *.workers.dev URL. Works across accounts, but the endpoint is public — put auth (e.g. a shared-secret header) in front of it if you go this route.

Building your own checker instead

This repo is also a minimal reference for calling Jev from any Worker:

const envelope = await env.AI.run("typesafe/jev", {
  state: { username: "a55h0le" },
  questions: {
    disguised_profanity: {
      type: "noul",
      instructions: "Does the handle in `username` disguise profanity?",
      criteria: {
        true: "Phonetic or look-alike profanity (e.g. 'a55h0le')",
        false: "An ordinary name or handle",
      },
    },
  },
});
// Note: third-party models return { state, result, gatewayMetadata } —
// the answers live under envelope.result.answers.
const p = envelope.result.answers.disguised_profanity.noul;
const flagged = p >= 0.5; // threshold lives in your code

Requires an ai binding in wrangler.jsonc:

{ "ai": { "binding": "AI" } }

Prerequisites

• A Cloudflare account with Workers enabled.
• AI Gateway credits. typesafe/jev is a third-party model on Workers AI, billed through AI Gateway Unified Billing — prepaid credits, not the postpaid per-neuron billing that native @cf/* models use. Without credits, inference fails with 2021: Insufficient AI Gateway credits. Load them in the dashboard under AI Gateway → Credits Available → Manage → Top-up credits.
• Node.js 18+ and npm.

Develop

npm install
npx wrangler dev

Open the printed local URL for the Swagger UI where you can try the endpoint. Note: Workers AI bindings run against your real Cloudflare account even in local dev, so local requests consume AI Gateway credits.

Deploy

npx wrangler deploy

Tuning

Behavior is controlled by two knobs in src/endpoints/profanityCheck.ts:

• PROFANITY_THRESHOLD (default 0.5) — raise it when false positives are expensive (rejecting a legit username), lower it when misses are expensive. You can also treat a middle band (e.g. 0.3–0.7) as "send to human review" instead of a hard verdict.
• The question criteria — plain-language definitions of what counts as profane. Rewording these is usually more effective than moving the threshold. See the Noul docs for guidance.

Known limitations

• Two-hop phonetic gags that resolve to explicit anatomy (e.g. mike_hunt) score low and stable (~0.16) even when listed in the criteria — a model calibration limit, not a wording bug. Look-alike spellings (a55h0le) and one-hop puns (ben_dover) are caught reliably. For production, pair this with a user-report flow or human review band.
• Jev's context window is 32,000 tokens; truncate very long inputs.

License

MIT