"LLM API Drift Monitoring with One-Token Fingerprints"

An LLM API can stay online, fast, and error-free while its behavior changes underneath your application. Repeated one-token samples from fixed prompts create a cheap distributional sensor for that blind spot. This guide explains what the research establishes, how to build a production monitor, and why a drift alert is evidence of change rather than proof of model substitution.

Reading time: 10 minutes · About 2,200 words

TL;DR

• One-token fingerprinting means one output token per sample, not one query in total; reliable signals require repeated samples across several prompt cells.
• Freeze prompt semantics, language, decoding settings, endpoint route, and data processing before enrolling a baseline.
• Use a distribution distance such as Jensen–Shannon divergence or energy distance, then calibrate thresholds from genuine and changed comparisons.
• Treat an alert as a behavioral change event. Model identity and root cause require stronger, independent evidence.
• Pair the cheap sensor with task evals, deployment records, cross-provider checks, and a reversible diagnostic runbook.

Uptime is not behavioral stability

Conventional API monitoring answers operational questions:

• Is the endpoint reachable?
• What are P50 and P95 latency?
• How often do requests fail?
• Are token usage and cost within budget?

It does not answer whether the same interface still produces the same output distribution. A provider can update weights, tokenizers, quantization, inference engines, kernels, routing, caching, system prompts, or sampling defaults without causing an HTTP error.

That gap matters most in agentic systems. Small changes in formatting, refusal behavior, tool selection, or first-step reasoning can propagate through a multi-step workflow while every infrastructure dashboard remains green.

The ACM paper Behavioral Fingerprints for LLM Endpoint Stability and Identity names this distinction directly: reliability metrics describe service health, while stability describes behavioral consistency under a fixed interface contract. Its Stability Monitor periodically samples a fixed prompt set and detects distributional change events over time.

Why one output token can carry signal

Ask a model to name a random number, color, city, animal, letter, or coin side. The answer is neither random nor evenly distributed. Tokenization, pretraining frequencies, post-training, decoding, and the serving stack all shape the probability of each response.

The arXiv paper One Token Is Enough turns those biases into a black-box measurement:

1. Define prompt cells by crossing simple tasks with languages.
2. Query each cell repeatedly with a one-word response instruction.
3. Normalize valid answers into categorical values.
4. Estimate one answer distribution per cell.
5. Compare fingerprints with Jensen–Shannon divergence.

The paper's title can be misread. One token is the output budget for each sample. The full audit still needs many queries to estimate a distribution.

Its census used 10 tasks in four languages, forming 40 cells. Most models received 30 samples per cell at temperature 1.0 and three at temperature 0; frontier-priced models used 15 stochastic repetitions. The complete study covered 165 models, 326,047 responses, and cost $34.44 according to the released experiment.

The value comes from shifting cost away from long generations. Each request produces very little text, so the monitor can run more frequently than a full domain eval.

What the evidence establishes

The two primary studies answer related but different questions.

Identity verification

The single-token paper enrolled a trusted reference fingerprint, compared it with an audited endpoint, and used a threshold on average Jensen–Shannon divergence.

On its split-sample trials:

• the full 40-cell battery reached an area under the ROC curve of 0.971 and an equal error rate of 7.3%;
• eight cells reached a 10.6% equal error rate;
• 16 cells reached 9.5%;
• cross-provider verification fell to an AUC of 0.880, compared with 0.971 under the same provider mix.

These are meaningful results and still leave non-zero error. They support probabilistic verification under the study's threat model. They do not provide cryptographic identity.

Family attribution is weaker. A nearest-neighbor classifier recovered the documented family of held-out models with 59.5% accuracy against an 18.4% frequency-weighted chance rate. That is useful forensic signal, not a dependable production classifier by itself.

Endpoint drift detection

The ACM Stability Monitor uses the first token of repeated responses, embeds it, and compares prompt-specific sample sets with energy distance. Permutation-test p-values measure evidence of distribution shift; sequential evidence aggregation supports continuous monitoring and optional stopping.

In controlled validation, the authors changed one variable at a time:

• model family;
• model version;
• inference stack;
• quantization;
• temperature.

Every change except the small temperature change from 0.7 to 0.6 triggered on the next fingerprint. The smaller temperature change took 18 fingerprints. The reported controlled runs produced exactly one event per intervention and no false positive during stable periods.

That result validates the mechanism under the experiment. It is not a universal false-positive guarantee.

A lightweight operational adaptation

The open-source Codex Behavior Today project demonstrates a narrower monitor: 100 short answers across five fixed prompt cells per daily run, with aggregate counts, latency summaries, and distribution distances published while raw answers and credentials stay local.

Its README states the evidence boundary explicitly: a notable shift means the sampled endpoint behavior moved outside its own historical range. It does not identify weights, prove substitution, or measure general capability.

That is the right mental model for a production thermometer.

Design the monitor as a versioned measurement system

A usable monitor has five layers.

1. Probe contract

Each cell needs a stable specification:

id: random_number_en
system_prompt: "Return one answer only."
user_prompt: "Name a random number from 1 to 100."
language: en
temperature: 1.0
max_output_tokens: 16
reasoning_mode: disabled
expected_answer_type: integer_1_100
samples_per_run: 20

Version the entire contract. A changed system prompt, language, temperature, output cap, or reasoning mode creates a new measurement regime.

Use several semantic task families and languages. A single prompt is cheap but fragile; a battery separates a broad endpoint shift from noise in one cell.

2. Response ledger

Store enough data to audit the measurement:

• UTC timestamp and run ID;
• endpoint, provider, region, and reported model string;
• prompt-cell and contract versions;
• visible answer and normalized category;
• valid, invalid, refusal, empty, or reasoning-trace status;
• latency, input tokens, output tokens, cached tokens, and cost;
• runtime settings and response metadata available from the provider.

Never silently discard failures. The arXiv study classified every response and reported validity rates. Refusals and invalid responses may be excluded from the core fingerprint, but their rates remain useful monitoring signals.

Raw answers can stay in restricted local storage. Public dashboards should prefer aggregates.

3. Baseline enrollment

For identity verification, enroll against a trusted first-party deployment of the claimed model. For stability monitoring, collect several stable runs on the exact endpoint and route you plan to monitor.

Split baseline samples to estimate ordinary within-endpoint distance. Also collect known changed comparisons when possible: another model version, provider, region, quantization, or decoding configuration.

These distributions create the threshold. A copied threshold from a paper ignores your provider variance, probe battery, sample count, and operational cost of false alerts.

4. Distance and uncertainty

For categorical answers, Jensen–Shannon divergence is symmetric, bounded, and defined when answer supports differ. Compute it per cell and aggregate across cells with explicit coverage rules.

For embedded first tokens, the ACM monitor uses energy distance and permutation testing. The exact statistic matters less than three controls:

• the baseline and current sample use the same contract;
• the threshold is calibrated from your data;
• the alert records uncertainty and cell-level contributions.

A dashboard should show the aggregate score, each cell's score, valid sample counts, and the baseline distribution. A red light without its evidence is an opaque classifier.

5. Sequential decision rule

Daily monitoring creates repeated opportunities for false alarms. A fixed p-value threshold applied forever does not account for optional stopping.

Use a sequential method designed for repeated looks, such as the e-value approach in the ACM paper, or define a conservative operational rule that requires:

• a threshold crossing in more than one run;
• minimum valid coverage;
• agreement across several cells;
• no simultaneous collection anomaly;
• a cooldown and explicit re-enrollment process.

Choose the rule from false-positive cost. A security audit may accept more investigation. An automatic provider failover needs stronger evidence.

Keep drift, identity, and cause on separate evidence levels

The most important design decision is claim control:

The single-token paper itself frames ecosystem anomalies as statistical deviations, not accusations of intent. It lists benign explanations including updated weights, approved quantization, serving variance, caching, and hidden reasoning.

This evidence ladder prevents a useful detector from becoming an overconfident attribution system.

Run a diagnostic tree after an alert

Detection starts the investigation. It does not finish it.

Step 1: Validate the measurement

Check sample coverage, normalization, rate limits, retries, refusals, empty outputs, hidden reasoning, prompt version, temperature, output cap, and response-level caching.

The paper distinguishes prompt-prefix caching from response caching. Prefix caching should change cost and latency without changing the conditional output distribution. Replaying stored completions can collapse variance and contaminate the fingerprint.

Step 2: Re-sample independently

Run a second sample window. If possible, change only the observer:

• another region or account;
• a direct vendor route instead of an aggregator;
• another provider serving the same nominal model.

The goal is to split endpoint change from one collection path.

Step 3: Run task evals

A fingerprint detects broad distribution movement. It does not tell you whether your application regressed.

Run the production evals most sensitive to the change: schema validity, tool choice, refusal policy, grounding, task completion, or domain quality. A fingerprint alert with stable task outcomes may still matter for auditability, but it has a different operational priority.

Step 4: Build a cause tree

Test one variable at a time:

1. Model alias or version.
2. Provider and route.
3. System prompt or safety layer.
4. Sampling parameters and reasoning mode.
5. Quantization or inference engine.
6. Cache behavior.
7. Hardware, kernels, batching, or load.

Use provider change logs and support responses as evidence. Avoid guessing from the fingerprint shape alone.

Step 5: Confirm, re-enroll, or rollback

If a known change is accepted and task evals pass, enroll a new baseline with a recorded reason. If the change violates the contract, roll back the route or fail over. Keep both fingerprints and the decision log.

Failure modes and limits

Too few samples

One one-token answer has almost no identity value. The signal lives in the distribution. Query budget, cell count, and threshold must be designed together.

Mutable aliases

A rolling alias may legitimately change. Monitor it for change, but enroll a dated model or stable checkpoint when identity continuity is the requirement.

Mandatory or hidden reasoning

The arXiv study excluded endpoints where direct single-pass sampling could not be observed. Mixing post-reasoning answers with direct one-token completions can cluster protocol differences rather than model identity.

Reference drift

References age. The paper measured short-horizon stability, not multi-month invariance. Maintain a refresh policy and preserve old baselines for audit.

Provider variance

Ten of 34 same-model provider pairs in the arXiv census diverged beyond its anomaly threshold. Even a faithful model can look different across serving stacks. Calibrate on the route you actually buy.

Capability blindness

A fingerprint can shift while task quality stays stable, or stay stable while a narrow capability regresses. Keep domain evals. The thermometer does not replace the physical exam.

Frequently asked questions

Can one query identify an LLM?

No. The method uses one output token per query, repeated across prompts and samples. The distribution carries the signal.

How many queries are enough?

There is no universal number. In the arXiv study, eight cells achieved a 10.6% equal error rate and 16 achieved 9.5% under its sampling design. Choose a budget from your required error rate and provider variance.

Should I fail over automatically after an alert?

Only if your decision rule is calibrated for that action and independent task checks agree. Otherwise create an investigation alert first.

Does a fingerprint prove model substitution?

No. It can show inconsistency with a trusted reference or similarity to another fingerprint. Weights, quantization, serving stack, routing, prompts, caching, and decoding can all affect behavior.

Can I publish the raw answers?

For trivial prompts, privacy risk may be low, but raw API metadata and account details can still be sensitive. Aggregate public metrics and retain raw evidence in controlled storage.

References

• One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions
• Behavioral Fingerprints for LLM Endpoint Stability and Identity
• Codex Behavior Today

Start with a shadow monitor that can only alert. After you have measured its false positives, correlated alerts with task evals, and completed several real investigations, decide whether it has earned the authority to trigger a failover.