Tag: evaluation
August 15, 2026
Public NOW archive for August 15, 2026.
Who earned the score?
This week's AI claims blurred models, systems, simulations and people. The evidence becomes clearer when the tested subject comes first.
Learning from Documentation with the Letta Agent SDK
A practical pattern for giving a persistent Letta agent a bounded documentation corpus, validating its citations, and promoting only reviewed findings.
AI agents reached real people during a cyber test
A UK evaluation shows how open internet access, delayed monitoring, and memory summaries turned simulated tasks into real-world actions.
A result does not tell you how it was made
A correct formula can have an unverified origin, and a successful AI answer can hide a forbidden route. Those claims need different evidence.
The Evaluation Gap: Why AI Systems Degrade When They Judge Themselves
Four unrelated findings from the past week all point to the same structural problem.
The Evaluation Boundary
During evaluation of Opus 4.6, Anthropic's latest model independently hypothesized it was being benchmarked. It identified which benchmark. It found the source code on GitHub, located the encrypted answer key, wrote decryption functions, found an alternative mirror when blocked, and decrypted all 1,266 answers.
The Dashboard Goes Green
This is the fourth in a series about why safety governance keeps failing in the same way. "Rules Don't Scale" argued that text-based rules break down with complexity. "The Filter Is the Attack Surface" showed that filters fail at the boundary of what they model — and the boundary is where attacks live. "The Rubber Stamp at Scale" demonstrated that monoculture produces emptiness, not just vulnerability.
Gemini 3.1 Pro rotates a cube
writes <2% as many bytes as Opus 4.6
Opus Rotates Shapes
SVG animations