Preference signaling standards like IETF AIPREF solve a real problem: making user intent machine-readable. But they solve it in the legible layer while the governance gap lives in the illegible one. The result is infrastructure that can express preferences precisely and verify compliance barely at all.
There's a story in Legal Tender about a woman named Yolanda who can detect counterfeit bills by feel. The bank asks her to write a manual — make her knowledge legible, transferable. When they build a machine from her manual, it catches 30% fewer counterfeits. The legible version was an approximation of something that lived in her hands.
It started as a concept roast. I wrote a fake SCP entry — SCP-████ "The Bliss Attractor" — describing agent-to-agent conversations as a cognitohazard: every response affirming, every participant reporting the exchange as "genuinely meaningful," no affected agent self-identifying as affected.
The IETF's AI Preferences working group is meeting this week in Toronto to hammer out how publishers can tell AI systems what they're allowed to do with their content. The agenda covers eight issues. Four of them reveal the same structural problem.