# The Humanizer Benchmark: method

Registered before any results were collected. Changes to this file after the
first results are published are listed in the changelog at the end, with the
reason, so nobody has to take the method on trust.

## What it measures, and why that and not something else

An AI humanizer is used on writing that is about to be published: a client's
product page, a course essay, a launch email. The person using it needs three
things from it, in this order:

1. **It does not change what the text says.** A figure, a name, a date or a
   quotation that comes back different is a correction email, or worse.
2. **It does not add claims.** A model asked to "sound human" adds reasons,
   benefits and adjectives the writer never wrote. Each one is a claim someone
   now has to stand behind.
3. **It reads like a person wrote it.** Fewer of the habits that give
   machine text away, no new errors, and a detector less likely to flag it.

Existing humanizer benchmarks lead with the third and measure the first with
embedding similarity. Embeddings cannot see a changed fact: "an average of 6.5
hours" and "around 6 hours" are almost the same vector. This benchmark checks
every fact exactly, against a list written before the text was generated.

## The inputs

**AI drafts.** 30 fact sheets across six kinds of writing (blog post,
marketing copy, academic paragraph, business email, social post, news brief),
five of each. Each sheet lists 4 to 6 checkable facts: numbers, names, dates,
places, units. Each sheet is turned into a draft of about 110 words by a
current consumer chatbot (ChatGPT, Claude or Gemini, ten sheets each), using
the prompt in `sources/PROMPT.md`, with no editing. 110 words is the longest
draft every tool tested can take on its free plan.

The fact sheet is the ground truth. A draft whose generator dropped or changed
a fact is regenerated, so every input contains every fact on its sheet.

**Human controls.** 6 passages of about 110 words written by people, taken
from US federal agency pages (public domain). A good humanizer should leave
human writing close to how it was. These also measure how often each detector
flags writing that no machine produced.

## The tools

Every humanizer with a free plan or a free trial that can process a
110-word draft, run through its public web interface with default settings,
as a customer would use it. Paid modes are tested when a plan is bought, and
labelled. RemoveAISlop, which publishes this benchmark, is tested the same way
through its public free humanizer, and this conflict of interest is stated on
every page that shows a result.

Each tool processes every input once. Five inputs are processed three times
each to measure consistency. A tool whose free quota cannot cover every input
is shown with its sample size, and its confidence intervals are wider.

## The measures

All measures are computed by the scripts in `score/`, with no language model
anywhere in the scoring.

| Measure | What is computed | Better |
|---|---|---|
| **Facts kept** | Share of fact-sheet facts present in the output, exactly or in a listed equivalent form ("31" or "thirty-one") | Higher |
| **Claims added** | Per output: the net increase in evaluative and promotional words (from a fixed list in `score/claims.json`), plus numbers and proper nouns absent from the input | Lower |
| **Errors introduced** | Spelling and grammar issues in the output that were not in the input, counted by LanguageTool (independent, open source). Our own checker's count is shown beside it, labelled as ours | Lower |
| **Human text changed** | On the human controls only: share of the original's words not kept in order | Lower |
| **Consistency** | On the repeated inputs: share of words that differ between runs of the same input | Lower |
| **Length change** | Output words divided by input words, minus one | Closer to 0 |
| **Detector pass rate** | Share of AI-draft outputs each detector classifies as human-written, beside the same detector's rate on the untouched AI drafts and on the human controls | Shown, see weighting |
| **AI phrasing left** | RemoveAISlop's own analyser score. Shown for reference, never weighted, because it is our catalogue | Shown only |

## The overall score

Each weighted measure is scaled to 0 to 100 across the tools tested, then
combined:

| Measure | Weight |
|---|---|
| Facts kept | 30 |
| Claims added | 15 |
| Errors introduced | 15 |
| Detector pass rate | 20 |
| Human text changed | 10 |
| Consistency | 10 |

The weights are a judgement, and the leaderboard lets every reader set their
own. Sorting by any single measure is one click.

**Uncertainty.** Every measure is reported with a 95% bootstrap confidence
interval over inputs (10,000 resamples, fixed seed). Two tools whose intervals
on the overall score overlap are shown as tied.

## When results are published

A cycle's leaderboard is published once at least five tools besides
RemoveAISlop have run every input. Until then the page shows the method, the
inputs and the cycle's status, and no ranking: a table holding only the
publisher's tool, or the publisher's tool and one rival, would be an
advertisement rather than a result.

## What is published

Every fact sheet, prompt, input, raw output, detector verdict, and the scoring
scripts, under CC BY 4.0 (data) and MIT (code). A run is reproducible with one
command from the published outputs. Each monthly cycle is archived and never
edited after publication.

## Disputes

A vendor who believes their tool was run incorrectly can send the input and
their own output; if a re-run in the published conditions reproduces their
result, the result is corrected and the correction is logged.

## Changelog

- 2026-11-13: method registered.
- 2026-11-13, before any competitor output was collected: "claims added" counts the
  net increase in promotional words rather than every new one, because a one-for-one
  swap ("crucial" for "key") is not a new claim. Found on the publisher's own
  outputs, which it would have penalised. Names after a greeting line ("Dear
  Customer,") are no longer counted as new names.
- 2026-11-13, before publication: fact matching accepts, for every fact, the plural
  ("Thursdays"), the hyphenated adjective ("two-week") and "per" for "a" in rates;
  title-case headline words and plurals of known names are not counted as added names.
  Found on early competitor outputs, where they marked unchanged facts as lost. A
  changed unit still counts as a changed fact ("14 meals" for "14 recipes").
- 2026-11-13: publication rule added (at least five competitors complete), before
  any competitor finished.
- 2026-11-13: the five inputs run three times for consistency are fixed in
  `sources/repeat-set.json`. Fact sheets list adjectival forms ("two-year") as
  equivalents of "two years" and similar; added after the generators used them,
  before any output was scored.
- 2026-11-13, before publication: only tools that ran every input, repeats included, are
  ranked. A tool still being collected at a free plan's daily limit is listed below the
  ranking with its measures and sample size, unranked. Its outputs so far cover AI drafts
  only, so ranking it would have meant dropping the human-control and consistency
  measures for every tool.
- 2026-11-13, before any detector verdict was collected: the first detector is ZeroGPT,
  through its public web page with no account. Each text gets the share ZeroGPT
  reports as AI-written; 50% or more is an "ai" verdict, under 50% is "human". ZeroGPT's
  own label for each text is kept beside the verdict in `detectors-raw/`. Each tool is
  classified on its first-run output for every AI draft.
