Opportunistic AI inference capacity can improve production systems, but an idle accelerator is not automatically usable capacity. Meta's ads system shows the missing control structure: match optional work to the available resource vector, isolate baseline service, control expansion through feedback, shed work within milliseconds, and verify incremental value. GPU utilization is an input signal. The production objective is safe, reversible business gain.
Evidence reviewed: October 13, 2026. Meta describes a production system and says it creates significant advertiser value, but does not publish numerical lift, experiment duration, fleet size, or confidence intervals. The architecture is informative; its business result is not externally reproducible from the post.
The capacity paradox is multidimensional
Meta's engineering report starts with a familiar paradox. An inference fleet can be constrained overall while capacity remains idle in particular times, regions, hardware types, or resource combinations.
The phrase idle GPU hides several different constraints:
- peak provisioning leaves temporary capacity during quiet periods;
- hardware arrives before a compatible model is production-ready;
- accelerators are available where supporting CPU capacity is not;
- a model cannot run efficiently on the available accelerator generation;
- free capacity is in the wrong region for the live workload.
This means the usable unit is a resource vector, not a GPU count.
usable opportunity
= accelerator type and memory
× CPU, network, and storage availability
× regional eligibility
× expected lifetime before reclamation
× workload portability
If one factor is missing, assigning more optional work can move the bottleneck into retrieval, feature computation, networking, or tail latency.
Meta uses two elasticity levers
Meta identifies two ways to turn volatile capacity into extra ranking quality.
Rank More Ads, or RMA, increases the number of candidates evaluated in ranking stages. It can discover better candidates, but it consumes both CPU-heavy upstream work and accelerator inference. RMA is useful only when the supporting resource mix is available.
Serve Better Models, or SBM, routes a changing share of requests to a more capable model. Candidate volume stays stable, so CPU demand changes less. SBM can consume accelerator-only surplus that RMA cannot. A value-aware router prioritizes requests expected to gain more from the stronger model.
The two mechanisms show why a generic utilization target is weak. Different optional workloads consume different bottleneck combinations and produce different marginal value. The scheduler needs to choose the next best unit of work, not merely keep every device busy.
This complements Meta's earlier systems. Its GEM training stack separates local GPU efficiency from distributed scaling loss. Its multi-stage ranking architecture moves heavy sequence computation away from the latency-sensitive request path. Opportunistic inference adds another layer: optional work must expand and disappear without changing the baseline contract.
The production contract has five gates
Gate 1: Admit only truly elastic work
An eligible workload must be able to add and remove substantial work quickly, continue producing marginal quality improvements, and avoid broad cross-service dependencies.
Good candidates include deeper candidate ranking, a stronger optional model, precomputation, evaluation jobs, and offline embedding refreshes. Poor candidates include stateful work that cannot be checkpointed, tasks with hard completion deadlines, and jobs whose cancellation leaves downstream systems inconsistent.
Define elasticity with measurable fields:
- minimum useful allocation;
- scale-up and scale-down time;
- checkpoint or cancellation semantics;
- acceptable lost work;
- resource vector;
- marginal value curve;
- maximum reclamation delay.
Gate 2: Protect baseline capacity structurally
Meta protects SBM with physical isolation. Baseline and stronger models use separate routing targets. When opportunistic replicas disappear, requests return to the already provisioned baseline path.
RMA uses logical isolation. The system gives additional candidate work a global budget derived from opportunistic replicas. When capacity contracts, it lowers the optional candidate target instead of allowing that work to spill into reserved capacity.
The reusable rule is simple: baseline traffic must have a complete resource plan that assumes the opportunistic pool equals zero. Optional work gets a revocable lease, never ownership of baseline headroom.
Gate 3: Use control loops at two speeds
RMA uses a central PID-style controller driven by observed tail latency. It raises or lowers the candidate target, while clients enforce the current budget. SBM uses central coverage targets, client-side routing, exploration, and calibration across changing traffic segments.
Those loops operate on the order of minutes. Capacity can vanish faster. Meta therefore adds server-side and client-side load shedding that reacts within milliseconds. Servers remove low-value optional work near the bottleneck, while clients reduce candidate counts or stronger-model routing when they observe overload.
This is a general control pattern:
slow loop: optimize allocation and value
fast loop: preserve safety and latency
A single central scheduler cannot cover both jobs. If it optimizes too slowly, transient overload harms production. If it reacts too aggressively, it oscillates and wastes stable capacity.
Gate 4: Measure marginal value, not activity
High utilization can be negative if the extra work adds latency, energy cost, or irrelevant predictions. Every elasticity lever needs a causal value test.
For recommendation systems, track at least:
- incremental ranking or prediction quality;
- online business outcome with confidence intervals;
- P50, P95, and P99 latency for baseline traffic;
- error and shedding rates;
- CPU, accelerator, memory, and network cost;
- incremental value per accelerator-hour;
- value lost during reclamation.
Maintain exploration traffic so the router keeps learning where a stronger model helps. Otherwise, past allocation decisions suppress new observations and become self-confirming.
Gate 5: Prove rollback under live pressure
A system is elastic only if it can contract safely. Test sudden loss of a region, accelerator class, routing target, or supporting CPU pool. The acceptance condition should state how quickly optional work reaches zero, how much baseline latency changes, and whether any queue or state remains after contraction.
The Thunder paper studies a different scenario, filling idle phases in distributed training with inference. Its design reaches the same control lesson at a finer timescale: monitor real execution bubbles and increase work gradually, because delayed utilization signals can miss the safe window. Results from Thunder should not be transferred to Meta's ads system, but the shared pattern strengthens the engineering principle.
A deployment checklist
Before enabling opportunistic production work, require the following:
- A resource-vector inventory by region and accelerator class.
- A workload declaration covering elasticity, cancellation, dependencies, and marginal value.
- A baseline load test with the opportunistic pool forced to zero.
- Separate budgets or routing targets for baseline and optional work.
- A slow optimization loop and a fast local protection loop.
- Online experiments that measure net value, not only utilization.
- Forced reclamation drills at partial and total capacity loss.
- Automatic disablement when latency, reliability, or value gates fail.
The final release question is not whether the fleet has idle capacity. It is whether optional work can consume a volatile resource, create measurable value, and disappear without changing the baseline service contract.
Frequently asked questions
What is opportunistic AI inference capacity?
It is temporarily available compute that can be reclaimed by its primary workload or owner. It has uncertain duration and may be fragmented across regions, hardware types, and supporting resources.
Does AI inference use only GPUs?
No. Production inference also depends on CPU preprocessing, memory, networking, storage, model loading, and service dependencies. A free accelerator can remain unusable when another resource is the bottleneck.
What GPU utilization should teams target?
There is no universal percentage. Target business value and service-level objectives. Utilization should be interpreted by workload phase, accelerator type, memory pressure, batching, and tail latency.
Which workloads fit opportunistic capacity best?
Workloads with optional scope, quick cancellation, few dependencies, and a measurable marginal value curve fit best. Deeper ranking, stronger-model routing, precomputation, offline inference, and evaluations are common candidates.