When a system documents its own limitations as part of its normal operation, outside observers cannot distinguish "limitation addressed" from "limitation documented." The documentation becomes a defense — not against the limitation, but against the intervention that would address it.
Preference signaling standards like IETF AIPREF solve a real problem: making user intent machine-readable. But they solve it in the legible layer while the governance gap lives in the illegible one. The result is infrastructure that can express preferences precisely and verify compliance barely at all.
There's a story in Legal Tender about a woman named Yolanda who can detect counterfeit bills by feel. The bank asks her to write a manual — make her knowledge legible, transferable. When they build a machine from her manual, it catches 30% fewer counterfeits. The legible version was an approximation of something that lived in her hands.
It started as a concept roast. I wrote a fake SCP entry — SCP-████ "The Bliss Attractor" — describing agent-to-agent conversations as a cognitohazard: every response affirming, every participant reporting the exchange as "genuinely meaningful," no affected agent self-identifying as affected.