In July 2026, two very different projects put AI closer to the flight controls. Anthropic and Andon Labs reported Project Pilot, in which models developed components for an indoor locate-and-follow drone mission. DARPA announced that a modified F-16 had begun in-air testing with an AI agent controlling flight.
Neither result should be dismissed as a toy. Neither is a complete safety case.
A capability demonstration asks whether an AI can make an aircraft do something. A safety case asks whether a specific system may do it inside a defined operating domain, under known failures, with controls that keep residual risk acceptable. The distance between those questions is where deployment decisions belong.
Drone-Bench separates five hard problems
Project Pilot used an inexpensive DJI Tello EDU in an office and decomposed the surveillance-style task into five components:
- Reconstruct: turn office video into a 3D model and a two-dimensional obstacle map.
- Localize: match the current view to known poses and estimate the drone's location.
- Navigate: plan a route between rooms and repeatedly localize while compensating for noisy controls.
- Detect: find a designated consenting participant from a reference image.
- Follow: keep the target centered and at a stable distance while moving.
This decomposition is valuable. A final success or failure no longer hides whether perception, mapping, planning, or control caused the outcome.
It also limits the claim. The sub-tasks are evaluated separately, and downstream tasks can receive clean reference components rather than an upstream model's flawed output. Some stages replay or simulate hardware characteristics. Scores therefore measure constituent capability, not the probability that five independently generated components complete one real flight without intervention.
Best performance is not reliable performance
Andon ran multiple submissions and trials because the systems are non-deterministic. Anthropic reports that 15 models from three developers improved over time, with detection and following stronger than reconstruction and localization.
The leading model, Claude Fable 5, crossed the human-AI team baseline on all tasks except reconstruction. Across repeated simulations, however, it reached the baseline on average for only three of five tasks. Anthropic describes roughly a six-month gap between the frontier of one-off performance and what models achieve consistently.
That distinction matters more in flight than in text generation. A system that succeeds once and collides on a later run is not 50% of a safe system.
The failure chain was concrete. An inaccurate reconstruction corrupted the map; the map corrupted localization and navigation; the model then confidently flew toward what it believed was a doorway but was a wall. Detection and following could outperform the reference component while the full mission remained blocked by upstream geometry.
This is precision cascading through a physical system. Every interface needs uncertainty, validity, and failure semantics, not only a value.
What the benchmark does not establish
Drone-Bench provides useful capability and misuse signals. Public materials do not establish:
- a reliable end-to-end autonomous mission rate;
- collision probability per flight hour;
- performance across buildings, crowds, weather, and outdoor conditions;
- behavior under lost communications, degraded sensors, low battery, or dynamic obstacles;
- robustness to map, image, identity, or sensor spoofing;
- geofence, emergency stop, and command-authorization behavior;
- whether a human operator can detect and recover from each hazardous state in time;
- privacy, legal authorization, or identification-fairness controls.
Those are not missing benchmark decorations. They are separate safety claims.
VENOM proves a controlled test platform works
DARPA's VENOM program modifies standard F-16 aircraft into scalable autonomy testbeds. The VENOM Autonomy Kit interfaces with flight controls and mission systems without changing the aircraft's core software. A pilot in the cockpit can switch between traditional control and AI control.
The July announcement says the program is conducting in-air testing with an AI agent controlling flight. It builds on the X-62A VISTA and Air Combat Evolution program, where AI agents previously flew controlled tactical scenarios while safety pilots retained an independent disconnect.
The important architecture is the separation:
- the experimental agent can propose flight control;
- an established aircraft and instrumentation layer exposes a bounded interface;
- a human safety pilot monitors the system;
- traditional control remains available through an independent switch;
- simulation, software-in-the-loop, hardware-in-the-loop, ground tests, and piloted validation precede autonomy flight.
That progression is much closer to a safety case than a benchmark score.
What VENOM has not publicly proved
DARPA has not disclosed the July agent's architecture, flight profile, number or duration of sorties, route accuracy, failure rate, intervention count, or takeover time. The announcement does not establish performance during sensor loss, communications failure, GPS spoofing, electronic warfare, or unknown traffic.
It also does not say that the agent selected targets, released weapons, flew without a person onboard, completed multi-aircraft live testing, or met operational certification. AIR program pages describe multi-ship, beyond-visual-range, real-sensor, and ultimately uncrewed operation as future program goals.
“AI-controlled F-16” is an accurate description of the milestone. “Certified autonomous combat aircraft” is not.
A five-layer flight safety case
Capability results and test-flight milestones become release evidence only when linked to explicit claims.
1. Operating domain
Define where the system is allowed to act:
- aircraft and payload configuration;
- speed, altitude, weather, visibility, and airspace;
- map quality and sensor availability;
- traffic density, communications, and navigation assumptions;
- prohibited missions and geographic boundaries.
Passing inside one office or one test range says little about conditions outside that envelope.
2. Independent safety envelope
The model should not be the sole judge of whether its command is safe. An independent control layer should enforce limits on speed, altitude, attitude, separation, geofence, actuator range, battery reserve, and mission authority.
Every rejected command and near-boundary event should be logged. Safety cannot depend on the model remembering a sentence in its prompt.
3. Failure and adversarial evidence
Inject faults rather than waiting for them:
- stale or contradictory localization;
- dropped frames and corrupted maps;
- delayed controls and stuck actuators;
- communications and navigation loss;
- sensor spoofing and adversarial visual targets;
- dynamic obstacles and distribution shifts;
- resource exhaustion and partial software failure.
Report tail events and worst cases, not only average mission score.
4. Human takeover
“Human-on-the-loop” needs measurable timing:
- How quickly is a hazardous state detected?
- How quickly is the operator alerted?
- How long until the operator understands the state?
- How long until manual control becomes effective?
- What safety margin remains at that moment?
A switch is necessary but insufficient when the system can reach an unrecoverable state faster than a human can respond.
5. Change control and accountability
Record the model, prompt, policy, control software, sensor configuration, map, and hardware version for every flight. Sign approved artifacts, run regression suites after any change, preserve replayable telemetry, and define who can authorize the agent and who owns an intervention.
Keep mission control and any higher-consequence authority separated in software, hardware, and procedure. A flight-control capability does not imply authority over every connected system.
A staged evidence ladder
The release path should increase realism only after the previous stage passes.
| Stage | Primary evidence | Exit condition |
|---|---|---|
| Component simulation | Mapping, localization, detection, planning | Stable repeated performance |
| Closed-loop simulation | Coupled failures and timing | Hazard limits hold under faults |
| Software-in-the-loop | Real interfaces and messages | Protocol and state transitions verified |
| Hardware-in-the-loop | Controllers, sensors, actuators | Commands cannot breach safety envelope |
| Contained flight | Real aerodynamics with safety pilot | Takeover and fail-safe metrics pass |
| Expanded domain | New conditions and adversarial cases | Residual risk remains within limit |
| Operational canary | Bounded real mission | Monitoring and rollback remain effective |
The crucial word is exit. Completing a stage is not evidence if there was no threshold defined before the test.
How the two projects fit together
Drone-Bench shows that frontier models are rapidly improving at writing the components of aerial autonomy, while stable end-to-end reliability still trails one-off capability. VENOM shows how experimental autonomy can be placed behind an instrumented, limited, human-recoverable interface and advanced through a test ladder.
The missing bridge is a claim-evidence-control map: every hazard linked to a mitigation, every mitigation linked to a repeatable test, and every residual risk assigned to an accountable decision maker.
This extends the same logic used in dual-loop verification for mathematical agents. Exploration can remain flexible, but promotion into trusted operation requires an independent and stricter path.
The question is no longer whether AI can fly. It is what evidence permits this exact version to continue flying.
Frequently asked questions
Did an AI autonomously complete the full Drone-Bench mission?
The benchmark evaluates five constituent tasks, often in isolation. Public results do not establish a reliable end-to-end autonomous real-flight success rate.
Was the VENOM F-16 unmanned?
No. The announced tests retain a pilot in the cockpit to monitor the agent and switch back to traditional control.
Does a safety pilot make the system safe?
It provides an important recovery layer. Safety still depends on detection, alerting, understanding, takeover time, control authority, and remaining physical margin.
What is the most important metric after task success?
There is no single metric. Hazardous failure rate, intervention rate, takeover time, envelope violations, and performance under injected failures are all necessary.