|

How LLM text watermarking works

Free-form text has no file container to carry metadata, so a text watermark lives in the statistics of word choice made during generation.

Google, OpenAI and Anthropic now mark generated text. Google announced SynthID Text in the Gemini app in May 2024, OpenAI marks eligible ChatGPT and Codex output in the EU with textGrain, and Anthropic applies a SynthID-Text-based mark to newer Claude models. The schemes differ in how they turn a secret key into a token choice and in who may run the detector.

TL;DR

Generation-time text watermarks link token or sentence choices to secret-key-derived values. A detector reconstructs those values and tests the accumulated evidence against an unmarked-text null hypothesis. Token-based schemes can bias probabilities or use keyed sampling randomness; semantic schemes mark sentence-embedding regions. Detection depends on usable length, available word choices, marking strength and calibration, and coverage and detector access vary by provider.

OpenAI reports approximately 80% detection at 200 tokens and 95% at 400 tokens (vendor claim, 1% target FPR). DIPPER paraphrasing reduced detection from 99.9% to 52.8% (OPT-13B, 300 tokens, 1% FPR). Milis et al. report 0.4%, 1.1% and 78.6% forgery success with no detector access, yes/no feedback and white-box access, respectively (open-weight KGW implementation, 0.1% FPR).

How a text watermark is embedded and detected How a text watermark is embedded and detected Generation Recent tokens Secret key + Seed Keyed choice: bias logits or keyed sampling Next token becomes context Detection Suspect text Same tokenizer + key Per-token keyed scores Statistic: z or Gamma p-value vs FPR target No model needed for green-count or textGrain detection
How a text watermark is embedded and detected (open full size)

The common pattern in token-based schemes

A language model produces probabilities for its next token. A watermarking layer changes how a token is selected, using secret-key-derived pseudorandom values that the detector can reproduce. In context-seeded schemes, recent tokens and the key determine a seed. Position-indexed schemes instead associate randomness with positions in a key sequence.

The detector converts text back into tokens, reconstructs keyed values and tests the accumulated evidence against an unmarked-text null hypothesis. Green-count detection can do this without the generating model, as illustrated by the Transformers generation utilities. Matching the tokenizer and seeding rule matters because different token boundaries or contexts produce different keyed trials.

The false-positive rate, or FPR, describes how often the test flags text under its unmarked null. A p-value measures how extreme the observed statistic is under that null. Neither quantity gives the probability that AI wrote a document. The null also has conditions: repeated contexts can create dependent scores, and text deliberately chosen using the key is not independent unmarked text.

Green lists: biasing the logits

In Kirchenbauer et al.’s KGW scheme, hashing the preceding token seeds a pseudorandom partition of the vocabulary. A fraction gamma enters the green list. Soft watermarking adds delta to those tokens’ logits, yielding p'(v) ∝ p(v) exp(delta × 1[v in G]). Red tokens remain possible; hard watermarking forbids them. The bias acts before the next token is emitted. After sampling, that token becomes part of the context for subsequent seeds.

Take four tokens with probabilities p(A,B,C,D)=(.25,.25,.25,.25), and green tokens {A,C}. With delta=ln(3), each green token’s unnormalised weight triples. Normalisation produces p'=(.375,.125,.375,.125): the green probability mass rises from .5 to .75. Gamma describes vocabulary membership, not the green probability mass after biasing.

The detector reconstructs each partition and counts green hits. Under the unmarked null, the expected count is gamma×T for T scored tokens. With G observed hits, z=(G-gamma×T)/sqrt(T×gamma×(1-gamma)) measures the excess relative to the null standard deviation, using the formula in the authors’ implementation documentation. For gamma=.5, T=100 and G=65, the expected count is 50, so z=(65-50)/5=3: the count is three null standard deviations higher than expected.

Green-list bias and the z-score Green-list bias and the z-score Green-list bias A and C are green Before After delta = ln(3) Green mass .5 -> .75 Detection .25 A .25 B .25 C .25 D .375 A .125 B .375 C .125 D gamma = .5, T = 100 tokens, G = 65 green expected green = 50 z = (65 – 50) / 5 = 3 T = 200, G = 130: z = 30 / 7.07 = 4.24
Green-list bias and the z-score (open full size)

Seeding width changes how edits propagate. The authors’ tuning guidance connects smaller context widths with edit robustness and smaller delta with quality. The reliability study distinguishes LeftHash, seeded from preceding tokens, from SelfHash, whose minimum-based rule includes the candidate token itself; its experiments use h=1 and h=4, respectively.

Repeated n-grams reuse seeds, making their scores dependent. The implementation recommends excluding them from green-count detection.

Keyed sampling instead of bias

Keyed sampling embeds a signal by replacing the randomness used to select tokens, rather than by adding a logit bias.

Scott Aaronson’s proposal substitutes keyed pseudorandom choices for sampling randomness and describes accumulating keyed evidence across n-grams. The exponential-minimum formulation in Fernandez et al.’s Three Bricks assigns each candidate an independent uniform value r_v and selects the smallest -ln(r_v)/p_v, excluding zero-probability tokens. Equivalently, it maximises r_v^(1/p_v).

For p(A,B)=(.8,.2) and keyed values r=(.5,.9), the costs are approximately (.866,.527). B wins despite its lower model probability. Across fresh independent random values, the race preserves the model’s token probabilities; a particular fixed key still determines a particular choice.

One detector accumulates S=-sum ln(1-r_observed). Under independent uniform null scores, S follows Gamma(T,1), giving an upper-tail p-value. Generation favours large keyed values relative to token probability; detection tests accumulated compatibility with those values. Repetition correction matters because the Gamma null assumes independent trials.

Kuditipudi et al. also describe inverse-transform sampling. A keyed random vocabulary ordering defines cumulative probability intervals, and a uniform draw selects an interval. With p(A,B)=(.8,.2), ordering [B,A] puts B in [0,.2); u=.1 selects B. Changing the ordering changes the keyed choice without changing the interval lengths. Their exponential-minimum variant uses the same race principle.

Three ways a key picks the token Three ways a key picks the token Exponential minimum Inverse transform Tournament (SynthID Text) p(A,B) = (.8, .2) keys r = (.5, .9) cost = -ln(r)/p = (.866, .527) A .866 B .527 B wins (lowest cost) key order [B, A] B occupies [0, .2) A occupies [.2, 1) u = .1 B A 0 .2 1 B wins candidates A and B g-values 1 and 0 1 A 0 B A wins ties broken at random
Three ways a key picks the token (open full size)

Detecting through insertions and deletions

Position-indexed sampling allows detection by aligning text with a key sequence. Kuditipudi et al.’s detector searches alignments and key shifts, combining token-matching costs with insertion and deletion penalties. This search accommodates positions disrupted by edits. A randomisation test compares the observed cost with costs from independently randomised keys: p_hat=(1+count[cost_random<=cost_observed])/(R+1). Lower cost means stronger evidence. Validity requires text independent of the key; ties make the test conservative, while cached reference distributions are an experimental approximation.

For example, if R=99 random keys yield no cost as low as the observed one, the formula gives p_hat=1/100=.01.

Distribution preservation and undetectability

Preserving the sampling distribution and hiding a watermark from keyless observers are different properties. Distortion-free refers to preservation averaged over the relevant fresh randomness. A separate cryptographic notion appears in Christ, Gunn and Zamir: efficient keyless observers cannot distinguish marked from original sampling, even with adaptively chosen prompts. Their construction assumes one-way functions, and detection completeness requires sufficient realised surprisal, including informative contiguous excerpts. Publishing the key forfeits that undetectability guarantee.

Carrying a message, not just a yes/no

A watermark can also encode a payload: MPAC assigns token positions pseudorandomly to message digits, then boosts the vocabulary colour corresponding to each digit. For binary payload 10, positions assigned to the first bit favour colour 1; those assigned to the second favour colour 0. Repeated votes recover each digit by majority.

Unknown-message decoding selects the most frequent colour and changes the null distribution. Searching multiple payload hypotheses likewise changes significance: Three Bricks gives 1-(1-p_min)^M for M independent, calibrated null tests. The smallest individual p-value is not automatically a valid global p-value.

SynthID Text: tournament sampling

Google’s SynthID Text paper derives seeds from recent tokens and a secret key, then computes pseudorandom g-values at several layers. Conceptually, a depth-m tournament draws 2^m candidates from the model distribution, compares pairs using the layer’s g-values and advances winners until one token remains. Repeated candidates are allowed; ties are broken randomly. Distribution updates can implement the conceptual tournament.

In a one-layer example, candidates A and B have g-values 1 and 0. A wins. Equal values require random tie-breaking. With two competitors per match, the method preserves single-token probabilities averaged over seeds; masking repeated contexts extends sequence-level guarantees. More than two competitors gives distortionary marking, with stronger detection potentially costing quality.

The mean detector averages the observed token’s g-values across the tournament layers: Score=sum_t sum_l g_l(x_t,r_t)/(mT), where x_t is the token at position t, r_t is its context-derived seed, m is the number of layers and T is the number of scored tokens. For two tokens with layer values (1,1) and (1,0), the sum is 3 and mT=4, giving a score of .75. A learned Bayesian detector instead combines evidence using labelled data.

The Hugging Face and Google integration uses model.generate(watermarking_config=...). Its ngram_len setting controls the context used for seeding: longer n-grams improve detectability but make the mark more brittle under edits. Transformers also exposes keys per tournament depth and sampling-table settings.

Repetition handling tracks previously seen contexts through context_history_size, whose default is 1024. The processor skips watermarking repeated contexts and supplies matching masks and g-value utilities for detection.

The released repository is reference software, explicitly not intended for production use, and offers no cryptographic security guarantee.

textGrain: block transport under an entropy budget

textGrain links vocabulary-block choices to keyed randomness while limiting how much that link reduces sampling entropy. OpenAI and university collaborators’ technical report describes an optimal-transport coupling: a joint distribution of block choices and keyed random values that preserves each block’s overall probability and the relative token probabilities within it.

For a toy distribution p(A,B,C,D)=(.4,.2,.3,.1), blocks {A,B} and {C,D} have probabilities .6 and .4. Those probabilities are preserved when averaged over the keyed randomness, while individual block choices are associated with particular keyed values. Within the blocks, the token ratios remain 2:1 and 3:1.

The entropy budget limits how strongly block selection depends on the keyed randomness. Let P be the original token distribution and Q the distribution conditional on that randomness. The average entropy loss is H(P)-E[H(Q|keyed randomness)]. This is the mutual information between the token choice and the keyed randomness, also expressed as a KL divergence from an independent coupling. The ideal constraint is KL<=beta H(P): with H(P)=2 bits and beta=.1, the budget is .2 bits.

Detection uses the association between the observed block and its keyed value. The detector reconstructs the block’s keyed cost, then applies a Gumbel cumulative-distribution-function transform to obtain an exponential score. It sums scores over distinct contexts and compares the total with Gamma(n,1), where n is the number of scored contexts. This requires matching tokenisation, context rules and block configuration, but not the generating model, coupling or embedding budget. The theoretical null assumes ideal independent keys; fixed keys and finite precision affect empirical calibration.

Deployed systems

Deployment combines the marking scheme with provider-specific coverage and detector access. Google announced Gemini deployment in May 2024. Microsoft’s Azure guidance, updated 3 November 2026, labels selected text watermarking support as planned.

ProviderSchemeWhere appliedWho can detect
GoogleSynthID TextGemini app and web experience, announced May 2024A detector with matching key and configuration; reference code is public
OpenAItextGrainEligible EU ChatGPT/Codex rollout; supported API opt-in globallySeparately approved provenance researchers and similar institutions
AnthropicVersion of SynthID-TextGlobal rollout on newer Claude models; older models to follow over monthsPrivate-preview API for eligible organisations
Microsoft, plannedSelected Azure OpenAI GPT-5.1–5.6 text marking supportPlanned support in Azure guidanceNot stated

OpenAI says API marking starts off by default; coverage depends on supported models and settings. Enabling marking does not grant detector access, which requires separate approval. Its help documentation describes free text checks for research, academic and similar provenance studies, with other applications outside that scope.

Anthropic’s September update lists authorities, researchers, media, education and qualifying enterprises as eligible for its preview. Claude uses its own key, so its detector cannot identify other providers’ watermarks.

What decides whether detection works

Detection depends on the amount of usable evidence, the choices available during generation, marking strength and the calibration of the statistical test.

Length

More usable tokens can accumulate more evidence. At the same 65% green rate with gamma=.5, T=200 and G=130 give z=(130-100)/sqrt(50)=30/sqrt(50), about 4.24, versus z=3 at T=100. OpenAI reports approximately 80% detection at 200 tokens and 95% at 400 tokens (psychology-like ELI5 responses, 1% target FPR, generating model undisclosed); mathematics performs worse in that evaluation.

Longer observations can also accumulate evidence after paraphrasing. In the reliability study, human paraphrases became detectable after about 800 observed tokens on average, with passages aggregated per writer. This used Vicuna/LFQA, FPR 10^-5, gamma=.25 and delta=4; the original passages had been preselected at z>9.

Length is measured differently across evaluations. Mark My Words defines watermark size as the median shortest detected prefix at 2% FPR, assigning infinity to undetected outputs. WaterBench compares five length settings across nine tasks and finds short outputs generally harder to detect, using Llama2-7B-chat and Internlm-7B-8k with method-specific strengths.

Entropy and code

Low-entropy continuations leave few plausible choices. Forcing alternatives can change likely wording or break code. SWEET applies green-logit bias only where next-token entropy exceeds a threshold, then scores only eligible positions: z=(G_H-gamma×N_H)/sqrt(N_H×gamma×(1-gamma)).

That detector needs model-based entropy estimates; the paper also allows a smaller surrogate. Its StarCoder experiments evaluate code quality and detection jointly (HumanEval, MBPP and DS-1000, varying completion lengths, pass@k, AUROC and TPR at FPR below 5%).

Anthropic says code and factual passages provide fewer opportunities to embed its watermark, and proofreading may change too few words for detection.

Strength and quality

Marking strength controls how token selection is tied to the keyed signal, with different consequences for the sampling distribution. KGW changes that distribution through delta; textGrain limits average entropy loss; SynthID distinguishes distortionary from non-distortionary tournaments.

KGW’s perplexity experiment examines strength alongside model-based text quality. It uses OPT-1.3B on C4 RealNewsLike, approximately 500 sequences of 200±5 tokens and an OPT-2.7B perplexity oracle, without a fixed detection FPR.

Google’s quality evaluations measure user feedback and human preference. Google reports no significant feedback change across approximately 20 million marked and unmarked Gemini responses in a random traffic split, with variable lengths and an undisclosed model snapshot. A separate evaluation on 3,000 ELI5 questions reports no significant human preference change for Gemma 7B-IT, also with variable response lengths.

False positives and calibration

Repeated contexts violate independent-trial assumptions. Three Bricks found empirical FPR exceeding theoretical predictions on human Wikipedia (100,000×10 keys, 256 tokens, LLaMA tokenizer, gamma=.25, varying context widths and thresholds). Analytical tests plus repetition correction closed the gap.

Searching windows or payloads adds another calibration problem. WinMax maximises the green-count z-score over contiguous spans and empirically calibrates the resulting multiple tests. A single-window threshold does not automatically control a document scan. WaterBench matches target TPR before comparing quality and reports true-negative rates separately; equal TPR does not imply equal FPR.

OpenAI’s provenance guidance states that a negative result does not establish human authorship: the signal can be unsupported, absent or degraded.

Threats to account for

Attacks can weaken an existing mark or make other text trigger its detector. Rewriting and dilution target detection of marked text; forgery creates a matching signal in text from another source. The available attack paths depend on access to generated samples, detector feedback, keys or model probabilities.

Paraphrasing

An attacker with an 11B paraphraser or equivalent rewriting service can replace lexical choices and reorder text without the watermark key. Those changes alter both scored tokens and the contexts used to derive seeds. DIPPER reduced detection from 99.9% to 52.8% (OPT-13B/Wikipedia, 300 generated tokens, lexical and order diversity both set to 60, 1% FPR).

SemStamp marks sentence representations rather than individual token choices. It maps sentence embeddings into regions using locality-sensitive hashing, then rejection-samples sentences until they fall in permitted regions. Disrupting that signal requires a paraphraser to change the sequence of semantic regions occupied by the text.

Its detection rates differed with the rewriting tool. After Pegasus-bigram rewriting, SemStamp retained a 76.7% true-positive rate (TPR), compared with KGW’s 42.7%. After GPT-3.5-bigram rewriting, the rates were 31.3% and 5.9%, respectively. Both comparisons used RealNews, fine-tuned OPT-1.3B, 32-token prompts, 200 generated tokens and 1% FPR. The generation temperature was .7; SemStamp used LSH dimension 3, valid ratio .25 and margin .02.

An editor can instead replace a controlled fraction of words with synonyms. OpenAI reports textGrain detection falling from 92% to 66% and 17% after 10% and 25% synonym replacement, respectively (400-token English ELI5, model and operating FPR unspecified for this editing result). Mark My Words excludes paraphrasing from its aggregate robustness score and reports those attacks separately.

Residual detection is worth measuring against the rewriting tool and usable length, including aggregation across documents. Controlled word replacement and full paraphrasing change different portions of the token sequence, while semantic marking also depends on the embedding and sentence-selection configuration.

Translation

An attacker can translate marked output, or translate a prompt into a pivot language, obtain marked output there and translate it back. The CWRA construction in He et al. uses that latter capability. Translation replaces the tokens and contexts the detector scores.

Direct translation changed KGW’s watermark strength with a mean relative error of 92.99% (Baichuan-7B, 500 mC4 prompts, multinomial sampling, 200-token source responses, GPT-3.5-turbo-0613 translating English into Chinese, Japanese, French and German; strength metric, FPR inapplicable).

Source-to-target language paths and newly marked translations require separate tests. Anthropic says Claude-produced translations acquire a mark through new word choices, while complete rewriting can remove its mark.

Dilution inside human text

An attacker able to assemble a document can place marked spans inside unrelated human text. Whole-document scoring dilutes the excess green hits. AUC was below .7 at 200±25 tokens and below .85 at 600±25 tokens (C4/LLaMA-7B, temperature .7, gamma=.25, delta=2, 25% marked content, one or three insertions, LeftHash/SelfHash, no fixed FPR). AUC summarises discrimination across thresholds, not the true-positive rate at one chosen false-positive target.

Window search changes the detection unit and introduces multiple testing. For WinMax, scan-level calibration and measurements over marked-span length and surrounding human text determine the appropriate threshold.

Token-level tricks

An attacker who controls prompting and post-processing can request an emoji after every token, then strip the emojis. The KGW paper illustrates how removal changes subsequent seeding contexts (ChatGPT demonstration, 15 December 2022, no controlled bypass rate).

Testing the exact tokenizer and normalisation path against inserted and removed tokens reveals how many independently scored contexts survive. Context width matters because the seed uses recent tokens; visible wording alone does not describe what the detector reconstructs.

Learning the key’s behaviour

Repeated generation queries can reveal hidden token preferences without revealing the key. Watermark stealing acquired 30,000 responses of at most 800 tokens each, then enabled 81% quality-qualified spoofing without detector feedback (Llama-7B provider, Mistral-7B attacker, a KGW SelfHash variant, Dolly-CW, average 450-token outputs, nominal .1% FPR, GPT-4 quality at least 6.5/10). The learned preferences guide token choices that the victim’s detector recognises.

Sadasivan et al. estimated green lists for 181 common words using one million watermarked OPT-1.3B queries. Those estimates enabled people to compose text that resembled watermarked output to the detector.

Distillation provides another route: a student trained on the victim’s generated samples can inherit its watermark. In Gu et al., harmful text from the student triggered the victim’s watermark detector, creating an attribution error. The Alpaca-7B student answered 71% of HarmfulQ prompts harmfully, compared with 0% for the Llama-2-Chat-7B victim. The harmful outputs had median length 593 tokens and median p=.002; the setup used KGW k=1, gamma=.25, delta=2 and GPT-4 judgements.

Randomised key selection chooses one key per response and rejects suspicious texts triggering multiple keys. Four-key selection cut distillation forgery from 82% to 6% on AdvBench and 74% to 1% on RealHarmfulQ (Llama-7B/Mistral-7B, C4 training with 10,000 samples per key, response length 800, 100 evaluations per dataset, family-wise FPR 1%). Its statistical theorem assumes independent key-symmetric outcomes.

Forgery models need to account for generation access even when detector access is restricted. Acquisition budgets define the scope of spoofing and student-transfer measurements. Key count and joint rejection calibration affect the protection provided by randomised selection.

Detector access

An attacker with feedback can search for changes that increase or decrease the detector’s statistic; white-box access additionally exposes its scoring rule. Using human ELI5 answers as carrier text, Milis et al. measured forgery success of 0.4% with no detector access, 1.1% with binary feedback and 78.6% with white-box access (a single-key KGW implementation, nominal .1% FPR, Qwen2.5-7B framework/3B rewriter, the 1,000 longest answers, averaging 220 tokens, open-weight implementations). Their proposed public/private channel split tests disproportionate strengthening of the exposed signal (combined baseline null accounting .2%).

Publicly detectable constructions can embed digital signatures through rejection sampling, separating a public verification key from secret marking authority. That construction distinguishes cryptographic forgery resistance from removal resistance and explicitly limits usefulness against unrestricted erasure.

Limiting key exposure and measuring repeated-query attacks against the actual feedback surface address these access paths; the KGW authors propose rate limiting and monitoring slightly edited variants. Public verification also calls for separate measurements of erasure and verification behaviour.

A second marked model and averaging

An attacker can use a second watermarked paraphraser to rewrite the first model’s output. Lost in Overlap finds that this can erode the first signal more than unmarked paraphrasing.

An attacker with accessible probabilities or logits from multiple marked model instances has another capability. WASH averages output distributions to cancel independently structured watermark perturbations, using vocabulary alignment and context resynchronisation; text-only API access is insufficient.

Marked-to-marked transformations warrant separate evaluation. Access to probability distributions and the independence of marking configurations determine whether the averaging mechanism is available, making distribution-access restrictions relevant.

Limits shown in theory

An attacker with a quality oracle and local perturbation oracle can repeatedly edit while retaining acceptable quality. Watermarks in the Sand proves conditional removability when the perturbation process mixes efficiently; the attacker requires no knowledge of the key or scheme identity.

Its implemented attack lowered mean KGW z-score from 6.236 to 1.628 (successful attacks, Llama2-7B, C4 RealNews, 20-token prompts, default 200-token outputs, nominal 5% threshold z=1.645, reward-model and GPT-3.5 quality checks, GPT-4 evaluation).

An explicit edit budget and quality criterion make removal-resistance claims testable. The theorem’s mixing and oracle assumptions define which attacks its result covers.

Dated milestones

The EU’s voluntary Code of Practice on Transparency of AI-Generated Content treats one imperceptible watermark layer as sufficient for free-form text. Article 50(2) of the AI Act requires machine-readable marking and detectability subject to technical feasibility, without naming an algorithm. The Code exempts very short text but states that marking still applies to free-form text longer than 200 tokens.

DateItem
23 October 2024Google and Hugging Face announce the open Transformers SynthID Text release, distinct from May’s Gemini deployment.
10 June 2026Commission publishes the final voluntary Code.
9 July 2026Commission adequacy assessment, as recorded by the AKOS transparency FAQ.
2 August 2026AI Act general application date under Article 113, subject to transitional provisions.
5 October 2026OpenAI announces its textGrain rollout and a plan to open-source it.
2 December 2026Article 111(4) deadline for Article 50(2) compliance by systems placed on the market before 2 August 2026.

References

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *