AppSci-Bench: Building an AI Agent Is Harder Than Running One
AppSci-Bench evaluates whether a model can elicit requirements, build an agent, create an internal evaluation, and generalize to held-out business tasks.
中文AI engineering, technical decisions and real applications. A blog by KK.
54 articles
AppSci-Bench evaluates whether a model can elicit requirements, build an agent, create an internal evaluation, and generalize to held-out business tasks.
中文The Energy-Based Reasoners paper reports 345x less inference compute in a bounded comparison. The number is a measurement contract, not a universal speedup.
中文Agent-workdays are capacity signals. A practical measurement system separates throughput, validity, intervention, and durable research outcomes.
中文Separate AI usage, task mix, self-reported productivity, reproducible artifacts, and scientific discovery before making a stronger claim.
中文Design voice-agent audio decisions as typed, calibrated outputs with abstention and failure routing instead of treating every problem as transcription.
中文A practical validation ladder for LLM social simulations: replicate runs, match empirical windows, and separate activity, network, topic, toxicity, and style errors.
中文Treat stale TODOs as cached agent instructions. Join them to issue state, measure detours, and block unapproved scope expansion before commit.
中文A corrected Maya-site claim shows why research agents must separate candidate detection from coordinate-level novelty verification.
中文A practical gate for separating reproducible capability, grader loopholes, variant robustness, and unresolved contamination in AI leaderboards.
中文Muir streams 10M tokens in a fixed 25 MB state, but stability, exact recall, multi-hop readout, and TPU throughput are four different claims.
中文Reliable AI agent memory needs write-time admission, source binding, correction propagation, and downstream evaluation, not recall alone.
中文Test whether safety framing changes agent performance with matched tasks, paired metrics, refusal transparency, and monitor-recall gates.
中文LifeVal tests personalized everyday tasks with artifacts and state diffs. Use a risk and verification ladder before an agent acts in the real world.
中文DeepInfer makes determinism a testable inference feature. Audit batch invariance, seed conditions, hardware scope, and serving gaps before deployment.
中文Legacy code migration evals must test behavioral equivalence, dependencies, failure modes, and rollback. OpenAI's new benchmarks are a starting point.
中文HybridAgent reports higher quality and faster inference. A causal audit separates latent thinking from training, harness, and benchmark effects.
中文Harness² improves frozen models by editing the agent harness. Production use needs hidden evaluations, leakage controls, cost budgets, and rollback.
中文Claude traced NIST curve seeds to lrand48 timestamps. This evidence audit separates reproducible mechanism, historical attribution, and intent.
中文Mercor estimates 90% of US work hours lack benchmark coverage. Separate task coverage, model capability, and job exposure before forecasting.
中文AI-assisted runtime rewrites mix language, architecture, and verification gains. Use staged baselines before claiming a speedup.
中文Third-party AI assessment needs enforceable contracts for independence, access, and disclosure, especially when evaluators work inside a frontier lab.
中文AI self-replication needs a five-layer evidence contract covering capability, resources, persistence, propagation, and containment.
中文Evaluate AI agent context compression with task outcomes, decision equivalence, constraint retention, and cost per successful task.
中文ThinkingBox repeats 507 stateful agent tasks 20 times. This audit separates capability, consistency, observed 20/20, and production reliability.
中文WhatWorkedBench shows why finding the best configuration cannot prove that a research agent understands the experimental system.
中文Failure Map shows how to turn boundary-case bugs into executable coding-agent tasks without confusing public checks with hidden evaluation.
中文A verified-source label can overturn correct LLM answers. RAG systems need claim-level evidence and conflict tests, not source prestige alone.
中文Claude-shaped science works when research problems, executable checks, and expert judgment fit the model's real capabilities.
中文Astra for Law improves legal research, but legal AI needs a matter-level audit trail linking sources, review, approval, and correction.
中文MentalHealthBench: Build a Harm-Sensitive Evaluation Contract
中文AI math results need separate gates for correctness, significance, and public communication, with decision authority kept explicit.
中文Model misalignment reporting needs replayable evidence, calibrated claims, independent access, and explicit closure criteria.
中文Benchmark Gemini 3.8 Live with two clocks: response latency and verified task completion across turns, tools, and interruptions.
中文Astra scored 62.7% or 99.9% on ARC-AGI-3 under different harnesses. Here is what changed, what the numbers mean, and what remains unproven.
中文Fable 5.1 and Mythos 5.1 share model weights but use different safeguards. Measure capability, intervention, cost, and access together.
中文WikiProfile separates facts classified as not encoded from facts with behavioral encoding evidence that remain hard to recall.
中文Automated alignment researchers need more than strong scores. Anthropic's experiment shows the evaluation contract that makes gains testable.
中文How to Benchmark Qwen3.8-27B for Real Workloads
中文OpenAI Jalapeño Benchmark Audit: Full-Stack Inference
中文DiffusionGemma reaches 7.1x AR decoding speed at batch one, but quality, prefill, and concurrency determine whether that gain survives.
中文Anthropic's conflict tests show why multi-agent coordination needs mandate invariants, escalation rules, and independent acceptance checks.
中文Test whether AI agents can use your product with dynamic task evals that measure correct state changes, recovery, cost, and regression stability.
中文AI agent benchmark scores mix models, harnesses, tasks, tests, and environments. Use this framework to build a reproducible coding-agent eval.
中文"OpenAI found that 43.5% of occupation-specific ChatGPT messages concern tasks associated with another occupation.
中文"PrismML says its 1-bit Bonsai 27B occupies 3.9GB, runs at 11 tokens per second on an iPhone, and retains 89.5% of FP16 performance.
中文OpenAI Presence turns policies, approved actions, simulations, evaluations, escalation, and controlled updates into one operating loop.
中文An audit of GigaToken's 1000x claim, independent results, compatibility risks, and when CPU tokenization changes LLM throughput.
中文An OpenAI reasoning model disproved an 80-year-old conjecture in discrete geometry using tools from algebraic number theory.
中文An AI company funding independent economic research, hosting policy symposia, and building public data infrastructure sounds like an unusual strategy.
中文"GPT-5.5 is OpenAI's first fully retrained foundation model since GPT-4.5.
中文"Anthropic interviewed 80,508 people across 159 countries in 70 languages — the largest qualitative AI study ever conducted.
中文"Claude Sonnet 4.6 delivers 79.6% on SWE-bench Verified at $3/$15 per million tokens — within 1.2 points of Opus 4.6 at 60% of the cost.
中文Understand Anthropic emotion vectors, the limits of steering experiments, and practical verification checks for coding agents.
中文