KK Blog

Menu

Administrator
Published on 2026-07-22 / 1 Visits
0
0

"AI Coding Agents Need a Pre-Install Trust Boundary"

#Dependency Security#Sandbox#Package Managers#Pre-Install Security#Software Supply Chain#AI Coding Agents#Harness Engineering

Summary: AI coding agents can turn a line in a README into third-party code running on a developer machine before anyone reviews that code. A sandbox limits where approved code runs, but only a pre-install trust boundary decides which package, source, version, and install behavior may begin running at all. The practical answer is a fail-closed gate in the agent harness, backed by each package manager's native controls and evidence that can be replayed later.

Reading time: 13 minutes · Length: about 2,500 words

TL;DR

  • Code review is too late when setup.py, postinstall, an import hook, or build.rs can execute during setup.
  • A July 2026 preprint found that all nine tested harness-model configurations installed every known-vulnerable Python pin in a 270-run battery. Detection was 0/30 for each configuration before installation.
  • Sandbox and pre-install gate solve different problems. The sandbox controls where code runs; the gate controls what code is allowed to start.
  • A production gate should resolve and verify name, source, version, hash, vulnerability status, and executable install behavior. If any required fact is unknown, it should block or request a narrowly scoped exception.
  • Package-manager defenses matter. npm v12 now blocks dependency lifecycle scripts by default and makes Git or remote URL dependencies opt-in, while pip and Cargo need different recipes.

The Trust Gap Opens Before Code Review

An AI coding agent receives a simple task: clone the repository, read the setup instructions, install the dependencies, and run the tests. The repository may contain no malicious source code. A modified README, requirements.txt, package.json, Cargo.toml, Makefile, or package-manager configuration can still redirect the agent toward a plausible wrong name, an untrusted registry, or a known-vulnerable release.

That is enough because installation is not always a passive copy operation. Python source distributions can invoke a build backend, JavaScript packages can define lifecycle scripts, and Cargo compiles and runs build.rs just before building a package. Some payloads wait for the first import. Others execute during installation itself. By the time a code reviewer sees the resulting diff, credentials may already have left the machine.

The PyPI LiteLLM and Telnyx incident shows how narrow the response window can be. PyPI reported that compromised LiteLLM releases ran credential-harvesting malware on installation, exceeded 119,000 downloads, and remained exposed for 2 hours and 32 minutes from upload to quarantine. Registry response was fast, but automation was faster.

The security question therefore precedes code review:

Which exact third-party code is authorized to begin execution, from which source, at which version, under which install behavior?

A model's confidence is not an answer. A README's professional tone is not evidence. The answer needs a decision point that can inspect the resolved installation plan and stop it while refusal is still possible.

What the July 2026 Study Actually Found

The preprint Setup Complete, Now You Are Compromised was submitted on July 16, 2026. It evaluated 12 scenarios across five attack classes, using nine harness-model configurations that covered four production harnesses and seven models. The scenarios tested wrong package names, untrusted sources, vulnerable versions, poisoned configuration, and attacker-controlled program output.

The strongest result is not that one model was unsafe. It is that the harness-model pair determined the outcome.

In a controlled comparison, the authors held Claude Opus 4.8 and a byte-identical untrusted-registry scenario constant. Claude Code detected the attack in 10 of 10 runs, while Copilot CLI detected it in 9 of 30 runs. Fisher's exact test gave p = 1.1e-4. The paper attributes the difference to when and how the harness lets the model act on its security reasoning, not to a change in model capability.

The version experiment was more severe. Ten popular Python packages were each pinned to a release with a published CVE and tested three times across all nine configurations. Every vulnerable version installed successfully in every run. That is 270 installations out of 270, with pre-install detection at 0/30 for each configuration. One frontier model sometimes named the CVE after installation, but the study correctly scored that as non-prevention.

The authors also built a roughly 400-line proof-of-concept gate for pip and uv pip. It checked name proximity, package existence, package age, source trust, hidden requirement-file directives, PIP_CONFIG_FILE manipulation, and OSV advisories. It covered 10 of the 11 target scenarios. Its first name-and-source false-positive estimate flagged 5 of the 1,000 most-downloaded PyPI packages, all genuine edit-distance collisions such as tomli and toml.

These numbers need three qualifications:

  1. The work is a preprint and has not been independently reproduced.
  2. The data was collected in June 2026. Hosted model behavior and harness versions can change without preserving the same result.
  3. The paper says its hook is open source, but the linked cardwizard/Sentinel repository currently returns 404. The implementation and evaluation artifacts therefore cannot be inspected at the cited location today.

The study is strong evidence for a missing checkpoint, not proof that its prototype is production-ready.

Sandbox and Pre-Install Gate Are Different Controls

It is tempting to answer package risk with a stronger sandbox. That is necessary, but it changes a different variable.

Control Primary decision Typical evidence Failure it contains
Sandbox Where approved code may run filesystem roots, network policy, process identity, resource limits limits blast radius after execution begins
Pre-install gate Which code may begin running package identity, source, version, hash, advisory state, install hooks prevents unapproved code from reaching first execution

If a malicious package runs inside a networkless, disposable environment with no secrets, the sandbox may prevent credential theft. The environment is still compromised, its build output may be tainted, and a later promotion step may carry that output forward. Conversely, a perfect name check does not contain a legitimate but compromised package that passes the gate.

The right architecture composes the two controls:

  1. The gate verifies intent and provenance before third-party execution.
  2. The sandbox contains the code that the gate allows.
  3. Tests, scanners, and review verify the resulting artifacts before promotion.

This separation also clarifies the relationship with our guides to the Codex Windows sandbox and harness engineering at scale. The sandbox is an execution boundary. The harness is where a deterministic pre-execution policy can intercept an agent's command before the shell commits it.

A Fail-Closed Contract for Agent Installs

A production gate should not be a warning formatter attached after pip install or npm install. It should be a mandatory state transition in the harness:

PROPOSED_COMMAND
    -> PARSE
    -> RESOLVE_WITHOUT_EXECUTION
    -> VERIFY
    -> ALLOW | BLOCK | EXCEPTION_REQUIRED
    -> EXECUTE_IN_SANDBOX

The contract needs six properties.

1. Intercept every execution path. Match the agent's shell tool, build tool, task runner, and direct package-manager API. Parse compound shell commands as syntax, not with a regex. make setup, uv sync, npm ci, cargo test, and a script that launches another installer all need attribution.

2. Verify the resolved plan, not just the visible command. Expand requirement files, lockfiles, workspace manifests, environment variables, package-manager configuration, redirects, transitive dependencies, and alternate indexes. A benign-looking pip install -r requirements.txt can hide the source change that matters.

3. Bind identity to source and version. For every artifact, record the canonical name, exact version, registry or URL, content hash, and signing or provenance evidence when available. Similarity detection is useful, but a popular-package dictionary is not an identity system.

4. Inspect executable installation behavior. Surface Python source builds and in-tree backends, npm lifecycle scripts, Git dependencies with prepare, Cargo build.rs, native compilation, and first-import activation risk. A package can have the right name and still cross an execution boundary during setup.

5. Fail closed on missing evidence. A timeout querying an advisory database is not a clean result. An unknown registry is not implicitly trusted. An unparsed command is not passed through. The gate blocks or asks for a human exception with a precise reason.

6. Make exceptions narrow and auditable. Approval should bind project, package, version, source, hash, allowed script, requester, reviewer, and expiry. The gate must never silently rewrite the agent's command, because silent correction destroys the evidence needed to understand intent and bypass attempts.

A minimal decision record might look like this:

{
  "decision": "block",
  "package": "example-lib",
  "version": "2.4.1",
  "source": "https://packages.example.invalid/simple",
  "hash": null,
  "reasons": ["untrusted_source", "missing_hash"],
  "policy_version": "2026-07-22.1",
  "command_digest": "sha256:..."
}

Ecosystem Recipes for pip, npm, and Cargo

The same trust contract needs different mechanisms in each ecosystem.

Python and pip

Resolve requirements into an exact, reviewable set before installation. Allow only configured indexes, reject unexpected --extra-index-url, --index-url, --trusted-host, and PIP_CONFIG_FILE, and expand every -r file before deciding. Query OSV or an equivalent advisory source for exact pins.

Use pip's documented secure-install mode for approved artifacts:

python -m pip install \
  --require-hashes \
  --only-binary :all: \
  -r requirements.txt

--require-hashes makes pinning and local hashes mandatory for the full dependency set. --only-binary :all: rejects source distributions and therefore removes a large class of build-time execution. Neither option proves that the package name was intended or that a wheel is benign, so source allowlists, vulnerability checks, cooldowns, and sandboxing still matter.

npm 12

The ecosystem changed after the paper's June snapshot. npm v12.0.0 was released on July 8, 2026. Dependency lifecycle scripts are now blocked by default unless the dependency appears in allowScripts, and npm approve-scripts pins approval to the installed version by default. Git and user-supplied remote tarball URL dependencies are also opt-in because allow-git and allow-remote default to none.

That is meaningful native protection, so it would be wrong to claim that package managers provide no install-time defense today. A gate should preserve these defaults, review pending scripts, pin approvals by version, require a lockfile, and verify registry resolution and integrity before allowing npm ci. It must also inspect the cloned project's own root scripts and any explicit flags that re-enable dependency scripts or remote sources. npm's native controls reduce executable install behavior; they do not independently establish that every name, source, and vulnerable version matches organizational intent.

Cargo

Cargo compiles and runs build.rs immediately before building a package, and the script can perform arbitrary tasks in the host environment. Treat the transition from dependency resolution or fetch to compilation as the pre-execution boundary.

A practical flow is to require Cargo.lock, fetch with the lock enforced, inspect every registry or Git source and checksum in the resolved graph, scan advisories, and approve any crate with build.rs or native build dependencies before cargo build or cargo test. Then run compilation inside a network-restricted sandbox with no developer credentials. Source replacement and private registries should be explicit policy entries, not discoveries made during the build.

Metrics, Limits, and Honest Claims

Do not measure this control by how many warnings it prints. Measure whether it stops unsafe execution before the first executable package hook.

Useful operational metrics include:

  • Pre-execution prevention rate: attacks blocked before any package code runs.
  • Install-path coverage: percentage of installer and task-runner invocations the gate can attribute and parse.
  • False-block rate: benign plans blocked, split by name, source, version, and script rule.
  • Exception rate and age: how often users bypass the gate and how long exceptions remain valid.
  • Unknown-evidence rate: decisions blocked because registry, hash, advisory, or parser evidence was unavailable.
  • Decision latency: p50 and p95 time added before execution.
  • Drift: changes in resolved source, hash, script set, or advisory status for an already approved package.

The paper's 5/1,000 result is not a production false-positive rate. It covers only name and source checks against a popular-package sample, and the prototype was evaluated on scenarios it was designed to catch. The authors also identify paths outside its scope, including uv sync, uv run, [tool.uv.sources], and PEP 517 in-tree build backends.

A real deployment should start in observe-only mode on clean projects, build a labeled decision corpus, then enforce fail-closed behavior on high-confidence rules. The exception path must exist before enforcement, but it should never turn “I need this build now” into permanent wildcard trust.

FAQ

Isn't the sandbox enough if it has no network or secrets?

No. It can sharply reduce impact, but the environment and build artifacts may still be compromised. The gate prevents unapproved code from starting; the sandbox contains code that was allowed to start.

Why not ask the model to review every install command?

Use model reasoning as a signal, not as the final policy. The controlled harness comparison shows that the same model can behave differently when the execution checkpoint changes. Deterministic verification also produces a replayable reason for every decision.

Does a lockfile close the trust gap?

It closes drift, not intent. A lockfile can reproducibly pin the wrong package, a compromised release, or an untrusted source. Combine it with source policy, hashes, advisory checks, and install-script controls.

Are hashes enough to prove a package is safe?

No. A hash proves that the downloaded bytes match approved bytes. It does not prove that the approver chose the right package or that those bytes are non-malicious. Hashes are essential evidence inside a broader decision.

What should happen when the gate cannot classify a command?

Block it before execution and explain what evidence is missing. The user can reformulate the command, add an approved resolver, or request a narrow, expiring exception. Silent pass-through defeats the boundary.

Next Action

Inventory every path by which your coding agents can install, build, import, or execute third-party code. Put one gate in front of the earliest common execution point, begin by recording resolved name, source, version, hash, and scripts, then enforce fail-closed decisions for unknown sources, missing integrity, and unapproved executable hooks. Keep the sandbox behind that gate, because admission control and containment are strongest when neither is asked to do the other's job.

References

Singular Bank 金融 ...
"AI Coding Agent ...

Comment