In this briefing
- 01Performance benchmarks begin to cover complete workflows
- 02Average success rates and repeated success rates answer different questions
- 03Runtime failures need distinct semantics for stopping and recovery
- 04Recovery extends to Agent state and its dependent resources
- →What to watch next
- ↗Sources and verification
- MLPerf Inference v6.1 adds End-to-End RAG and Edge Agentic Inference; the latter includes multi-turn coding workloads and latency and accuracy thresholds. The datacentre Agentic benchmark remains forthcoming.
- Claude Code v2.1.274 fixes MCP configurations for longer timeouts not taking effect and endless retries in corrupted sessions, and provides a limit on startup waiting. These are vendor release records; no independent reliability retesting under matched conditions is available yet.
- Cohesity Agent Resilience is currently available only to selected customers. It initially supports Amazon Bedrock AgentCore and Bedrock Agents, with general availability targeted for the end of 2026. Support for Microsoft and Google platforms remains on the roadmap.
Performance benchmarks begin to cover complete workflows
MLCommons released MLPerf Inference v6.1 on 16 September. The new End-to-End RAG benchmark brings embedding, vector retrieval, reranking and LLM responses into a single pipeline, measuring both ingestion of a document corpus and question answering over an existing vector database. It covers several components and no longer measures only an isolated generation step.
The Edge Agentic Inference benchmark introduced in the same round targets a single-user edge environment. Its single-stream coding workload represents history that grows from turn to turn, combining latency measures with statistical accuracy thresholds. Its two tasks examine accuracy within a time limit and the performance of a deterministic workload with built-in correctness checks, respectively. The announcement explicitly says the corresponding datacentre Agentic performance benchmark remains forthcoming.
Complete pipelines and multi-turn workloads bring performance comparisons closer to the dependencies in real applications, but tasks, models, quantisation and scenarios still define the measurement scope. These results can supplement capacity assessments; they cannot directly establish that enterprise data permissions are correct, long tasks remain reliable throughout, or recovery after a failure is complete. The performance of a deterministic workload is not the same as the success rate of all real tasks.
Average success rates and repeated success rates answer different questions
As earlier background, IBM Research introduced the Consistency Analyzer in ALTK-Evolve on 15 September; the associated paper was submitted on 8 September. The authors tested ReAct with GPT-4.1 on the 168 tasks in AppWorld test_normal. The average success rate across five runs, Mean@5, was 77.4%, while the proportion of tasks successful in all five runs, Pass^5, was only 53.0%. Pass^5 differs from Pass@5, which measures success at least once.
The authors perform controlled resampling of decision steps in recorded trajectories, then generate guidance for future runs. In tests on the same tasks, Pass^5 rose to 69.0% and Mean@5 to 81.0%. The diagnostic steps reuse recorded context without making new tool calls or rerunning complete tasks. These figures come from the authors’ experiments, rather than an independent evaluation of enterprise business workflows.
Multi-turn inference benchmarks and research on repeated runs offer complementary perspectives: the former describe the performance of a given workload, while the latter describe whether the same task succeeds consistently. In workflows involving writes, repeated execution may also create additional side effects, giving trajectory analysis potential value. However, offline diagnosis of decisions cannot replace verification of outcomes in the environment, and the effectiveness of guidance still depends on the actual tasks and models.
Runtime failures need distinct semantics for stopping and recovery
Claude Code v2.1.274 was released on 17 September. It adds a limit on waiting for MCP connections in the first non-interactive turn and fixes Streamable HTTP tool calls still timing out after around five minutes despite a longer configured timeout. For loops of 400 errors caused by unexpected tool_use_id, the release notes say a corrupted transcript will heal itself where possible; otherwise, retries end with a clear error and a rewind prompt.
The same version also fixes the loss of the active goal when resuming a compacted session, and insufficient_scope 403 errors being misreported as expired logins. The release notes confirm that these fixes are included in this version, but provide no independent test results on recovery completeness, repeated side effects or production failure rates.
Connections that are not ready, timeouts, insufficient permissions and corrupted sessions each call for different handling; they cannot all be classified as temporary failures suitable for automatic retry. Clearer errors and waiting limits help explain failures, but whether a resumed session retains all its state, and whether external actions have already occurred, still depends on tool results and persistent records. Ending an error loop is not the same as restoring business operations.
Recovery extends to Agent state and its dependent resources
Cohesity announced Agent Resilience on 16 September. It initially protects Agent memory and configuration, provides point-in-time recovery, and includes databases, file systems and other services managed by Agents within its protection scope. A topology view links memory stores, applications, databases and underlying infrastructure. The capability is currently available only to selected customers and supports Amazon Bedrock AgentCore and Bedrock Agents, with general availability targeted for the end of 2026. Microsoft and Google platforms remain on the roadmap.
CISO material published by Google on the same day provides another type of background on state protection: in the second quarter of 2026, Mandiant investigated multiple extortion incidents involving the theft of proprietary models, skills, prompts, source code and research material. Google proposes dynamic relationship graphs connecting code, models, data lineage and runtime identities. This is vendor investigation and defensive design; it cannot establish the exposure of each enterprise, nor is it an independent test validating all protection mechanisms.
Dependencies between Agent memory, configuration and business resources mean recovery from failure may involve several versions and points in time. Topology and relationship graphs can help define the protection scope, but a backup alone cannot establish that the state after recovery is trustworthy. Rolling back cannot retrieve material that has already leaked, and actions already executed may not be reversible. The current material provides no common guarantee of consistent recovery across platforms, credential revocation or verification of business operations after recovery.
What to watch next
- When the MLPerf datacentre Agentic benchmark is released, and whether complete-workflow performance and repeated success rates can be presented together under the same task conditions.
- Whether Claude Code publishes reproducible tests of session repair and recovery that specify the handling boundaries for interrupted tool calls and external side effects.
- Whether Cohesity specifies protected fields, consistency of recovery across resources, recovery time objectives and recovery point objectives at general availability, and publishes the results of independent exercises.
Sources and verification
Golden Data has edited this briefing from the public materials listed above. The original sources govern facts and figures. The enterprise relevance sections are Golden Data editorial analysis and do not constitute an endorsement of any third-party product.
← Back to AI Daily Briefing