Measure the pairing.
Preserve the authority.

An agent benchmark measures a model–harness system, not the model alone.

The authors compare OpenCode, Codex CLI, Pi CLI and DeepAgents across four core models, with native references and stress conditions. Their 111 settings yield 52,168 trajectories across six benchmarks. Rankings vary by model and task; finishing normally with an evaluable result is not the same as solving the task correctly.

Evidence & limits. This is an author-reported preprint under review, not a universal harness leaderboard. Valid Completion Rate measures operational validity, not answer quality. Pairing-level observations do not isolate a single causal component; versions, budgets, benchmark panels and task mix matter.

A later tool receipt may confirm a decision already made, rather than require a fresh reasoning call.

Executable Decision Graphs retain exact actions, literal arguments, dependencies and guards. Observed receipts activate eligible continuations; new semantic decisions or unsupported evidence return to the original model. Actor weights stay fixed.

On the paper’s 98-task HumanEval test split, the Codex comparison preserves 90.82% success while reducing actor tokens by 29.54%.

Evidence & limits. Author-reported preprint results, not guaranteed savings for arbitrary workloads. Companion configurations and task-specific adapters affect comparisons. The main quality–cost result does not attribute every saved token to reuse alone. Model-call, cache/context accounting, configuration search and recovery costs must be distinguished. In an MBPP ablation, the no-graph condition solved four more tasks than full HarnessMM: savings are not universally quality-neutral.

A similarity score is a starting point, not necessarily the final evidence ranking.

ActionRAG reconciles entity types and identities, retrieves entity and triple seeds, then uses bounded graph diffusion to refine evidence. Seed protection and hub penalties aim to recover relevant connections without letting popular nodes dominate. Across eight deterministic MemoryAgentBench columns, the authors report a 55.1 unweighted mean, 5.4 points above the strongest compared retrieval-based baseline.

Evidence & limits. Results come from the arXiv manuscript; independent replication was not checked. Baselines largely come from the benchmark paper, and LLM-judged tasks are excluded. Cost accounting excludes graph construction, storage and local embedding and uses November 2025 prices. Table 4 includes three query costs above $0.001 despite the abstract’s below-$0.001 wording. The 440.7 ms figure adds separately measured component p99s; it is not observed end-to-end p99 latency.

A lost acknowledgment does not prove that an action failed to happen.

The study executes 336 controlled histories across native LangGraph and Temporal hosts. Per host, retained-ledger reconciliation conforms in all 42 cases: 21 completions, 15 denials and six unresolved outcomes with reservations retained. Cached retry, permission revalidation and stable-key retry policies conform in 30, 36 and 31 cases respectively.

Evidence & limits. This preprint isolates application recovery policies, not LLM behavior or framework defaults. Paired host outcomes are identical. The finite schedules and provider contracts bound the result; “42 passes” includes correct denials and unresolved cases, not 42 successful actions. The separate InfinityOS component study uses repository fakes, not a production deployment evaluation. There are no real provider/model trials or public execution repository; released tables alone do not reproduce the experiment.