GEPA's optimize_anything looks like a universal optimizer for prompts, code, agent architectures, skills, and configurations. Its more important contribution is architectural: the evaluator becomes the stable interface that defines what good means, while artifacts and optimization engines can change. That design creates leverage only when the evaluator represents the real objective. A blind spot becomes a target, and richer feedback can help the optimizer exploit it faster.
This guide explains the evaluator contract, Actionable Side Information, the three search modes, the reported results, and a production validation pattern that treats optimization as a controlled search process rather than an automatic quality guarantee.
Separate the API from the engine
The naming is easy to blur. GEPA, short for Genetic-Pareto, is a reflective optimization method. optimize_anything is the frontend that accepts an artifact, evaluator, objective, datasets, and budget. A July 22, 2026 update made the frontend engine-pluggable and pipeline-composable across GEPA, AutoResearch, Meta-Harness, and combined strategies.
That distinction clarifies the system boundary:
- The artifact is what changes: a prompt, program, configuration, policy, skill, or agent definition.
- The evaluator defines evidence: score, diagnostics, constraints, cost, and failure information.
- The engine controls search: which candidate to inspect, mutate, merge, or retain.
- The test gate decides whether the final candidate generalizes beyond the search loop.
The evaluator is the task interface because it should remain meaningful when the artifact representation or engine changes. If switching from GEPA to another search strategy changes the definition of success, the system has coupled task semantics to the optimizer.
The smallest evaluator contract
The official API reduces the evaluator to a function that receives a candidate and, in dataset modes, an example. It returns a scalar score or a score plus Side Information, commonly called Actionable Side Information.
def evaluate(candidate: str, example: dict):
result = run_in_sandbox(candidate, example)
score = compute_score(result)
return score, {
"correct": result.correct,
"runtime_ms": result.runtime_ms,
"cost_usd": result.cost_usd,
"stderr": result.stderr,
}
The scalar determines whether a candidate improved. The diagnostic payload tells the proposer why. Compiler errors, failed assertions, profiler traces, per-case scores, rendered images, and structured sub-objectives can all become feedback.
GEPA's paper calls this Side Information; current documentation often uses Actionable Side Information. It is useful to think of ASI as gradient-like because it supplies direction rather than only a pass or fail signal. It is not a mathematical gradient, and the direction is only as reliable as the evaluator.
How reflective search works
At a high level, the GEPA engine:
- Evaluates the seed artifact across examples and objectives.
- Keeps records of candidates that perform best on particular examples or metrics.
- Selects a parent and a small batch of cases.
- Gives the proposer the artifact, scores, and diagnostic traces.
- Produces a targeted mutation after reflecting on the failures.
- Fully evaluates promising mutations and updates the candidate pool.
- Returns the candidate with the best aggregate validation result when the budget ends.
Current documentation describes the selection as Pareto-aware. More precisely, it tracks per-key best-in-class candidates and samples candidates according to how often they lead a case or objective. This preserves complementary specialists, but it is not necessarily the complete textbook set of every non-dominated solution. A candidate that is consistently second-best and never leads any key can fall outside the default frontier.
This precision matters when interpreting diversity. The system preserves evidence of distinct strengths. It does not guarantee exhaustive multi-objective coverage.
Three modes solve different problems
The API chooses a mode from the presence of dataset and valset.
Single-task search
There is one difficult problem, and the candidate is the answer or solver. Circle packing and black-box optimization fit this pattern. The evaluator directly scores each proposed artifact.
Use it when each run targets one fixed environment and generalization is not the main goal.
Multi-task search
A dataset contains related problems, and lessons can transfer between them. CUDA kernel generation is a good example because different operations share a language, compiler, hardware, and optimization techniques.
The output can include specialized artifacts for individual tasks. Relatedness is a requirement, not a convenience. In the paper, combining unrelated circle-packing sizes hurt performance: the single-task score was 2.6360, while broader multi-task variants fell to 2.6313 and 2.5973.
Generalization
Training examples drive search while a separate validation set selects one global artifact intended to work on unseen inputs. Prompt and policy optimization usually belong here.
Repeated selection on the validation set can still overfit it. A sealed test set should remain invisible to the engine and run only after optimization. The current API includes a test_set path designed for that structural separation.
What the reported benchmarks establish
The optimize_anything paper reports strong results across distinct domains:
- A Gemini Flash ARC-AGI agent improved from 32.5% to 89.5%.
- A cloud routing algorithm reduced cost by 40.2% relative to Dijkstra in the authors' simulator.
- On 31 KernelBench operations using an NVIDIA V100, 87% of generated kernels matched or beat PyTorch and 48% reached at least 1.1 times its speed.
- A GPT-4.1-mini system prompt improved from 46.67% to 60.00% on AIME 2025.
- Coding-agent skills raised Claude Haiku 4.5 from 79.3% to 98.3% on the selected task suite and reduced completion time by 47%.
These are author-reported experiments under specific models, tasks, hardware, evaluators, and budgets. They establish that one reflective interface can compete across several text-representable optimization problems. They do not establish a universal expected uplift for production systems.
The cost boundary is also real. Reported configurations ranged from roughly one dollar to about $144.70, with evaluation often dominating total cost. Proposer capability mattered: stronger models produced substantially better AIME and circle-packing outcomes than smaller alternatives.
The evaluator is both specification and attack surface
An optimizer cannot recover a requirement that the evaluator never measures. It can only find candidates that score well under the implemented contract.
Suppose an agent-quality evaluator rewards task completion and latency but omits:
- whether the answer used an unauthorized tool;
- whether hidden instructions leaked into output;
- whether the solution survives distribution shift;
- whether the candidate consumed ten times the expected tokens;
- whether it passes only because of a brittle shortcut.
Search pressure will favor those omissions whenever they improve the visible score. ASI makes the loop more efficient by exposing detailed failure information. The same efficiency can accelerate evaluator gaming when the feedback reveals a proxy's shape.
The paper does not claim to solve reward hacking. Its strongest examples enforce important constraints inside the evaluator. The circle-packing evaluator validates geometry; code examples use compilation, correctness checks, timeouts, and sandboxed execution. Those controls are part of the task specification, not automatic features of a universal optimizer.
A four-layer evaluator contract
A production evaluator should separate four layers.
1. Admissibility
Reject invalid candidates before scoring quality:
- schema and syntax validation;
- allowed imports, tools, and network destinations;
- resource and time limits;
- secret and data-boundary checks;
- hard safety constraints.
An invalid candidate should not receive a low-but-competitive score. It should be outside the search space.
2. Task quality
Measure the intended outcome with cases that discriminate between plausible candidates. For code, this means correctness tests rather than style proxies. For agents, measure completion, evidence, recovery, and side effects rather than a judge's general impression.
Use structured sub-scores when one average would hide a catastrophic dimension.
3. Operational cost and variance
Record latency, tokens, money, memory, retries, and failure frequency. Repeat non-deterministic evaluations. A candidate with a high mean and a dangerous tail may be worse than a slightly lower but stable alternative.
Cache only deterministic results or cache by every input that affects execution, including environment, model version, and configuration.
4. Generalization and release
Keep training, validation, and sealed test data separate. Add distribution-shift cases and adversarial inputs. Re-run the winner with multiple seeds, then compare it with the baseline on real traffic through a bounded canary.
The search engine should never be able to inspect or adapt to the final release gate.
A safe optimization workflow
The following sequence keeps search power inside a verifiable boundary:
- Freeze the baseline, datasets, evaluator version, model versions, and budget.
- Run the seed through all hard constraints before optimization.
- Optimize in a sandbox with network, filesystem, time, and cost limits.
- Return structured ASI that diagnoses failures without exposing sealed answers.
- Track per-objective results rather than only one aggregate score.
- Select on validation data and evaluate once on the sealed test set.
- Repeat the final comparison across seeds and likely distribution shifts.
- Review the diff between baseline and optimized artifact.
- Deploy behind a canary with rollback and live outcome metrics.
This workflow turns optimize_anything into a search component inside a larger assurance system. The engine generates candidates; the release process decides whether evidence is sufficient.
The separation resembles a dual-loop verification protocol for mathematical agents: one loop explores, while an independent boundary decides what is admissible and releasable.
When automatic text optimization is a poor fit
Avoid the pattern when the real objective is unmeasurable, changes faster than the evaluation cycle, or carries consequences that cannot be represented by a safe offline test. It is also a poor fit when evaluating a candidate would expose production data, execute uncontrolled code, or create irreversible side effects.
Some objectives need human judgment. Human review can be part of an evaluator, but a tiny panel or an LLM judge introduces variance and bias. The paper's visual demo included five human raters as a sanity check, which is useful evidence and still a small sample.
Universal text optimization has a precise boundary: the artifact must be representable as text or a text-addressable structure, and quality must be measurable at an acceptable cost. Everything else remains outside the claim.
Frequently asked questions
Is GEPA only a prompt optimizer?
No. GEPA can optimize prompts, code, configurations, agent architectures, and other textual artifacts. The system still needs an evaluator that can execute or inspect each candidate safely.
What should an evaluator return?
At minimum, a score where higher is better. A useful evaluator also returns structured diagnostics such as correctness, errors, runtime, cost, per-case results, or rendered output.
What is Actionable Side Information?
ASI is diagnostic feedback supplied to the proposer during reflection. It can make search much more sample-efficient than score-only mutation. It is not a mathematical gradient and can misdirect search when the diagnostics reflect the wrong objective.
How do I prevent overfitting?
Separate training, validation, and sealed test data; limit repeated exposure; add adversarial and distribution-shift cases; repeat evaluations; and keep the release gate inaccessible to the optimizer.
Does GEPA prevent reward hacking?
No general optimizer can compensate for an incomplete objective. Put hard constraints outside the score, sandbox candidate execution, hide release tests, and audit optimized artifacts for shortcuts.
How much does optimization cost?
Cost depends on proposer models, number of candidates, evaluator expense, dataset size, and retries. The paper reports task configurations from roughly $1 to $144.70, which should be treated as experimental costs rather than production TCO.
References
- optimize_anything paper
- Official optimize_anything introduction
- Official omni and pluggable-engine update
- GEPA GitHub repository
- Official evaluator API
- Official candidate-selection guide
- Reproduction artifact
The evaluator is where intent becomes executable evidence. Design it as a versioned, adversarially tested interface, and optimization engines become interchangeable sources of candidates. Leave it vague, and a better optimizer will produce a better demonstration of the vagueness.