made using Leaflet
The Hater's Guide to Atproto
DRAFT

Draft

—
—

The internet is extremely complicated under the hood. Even basic HTTP is on the seventh and highest layer of the Open Systems Interconnection model that standardizes how computers communicate with each other. You just about never really need to worry about that as an end user, though - your browser and OS and hardware work together to translate between the layers of abstraction without intervention unless something has gone wrong. You see the artifacts of this, like your computer's MAC address the https:// prefix in a URL, but largely don't need to understand how any of it works on a mechanical level to be able to surf the web.

Bluesky's implementation of atproto works in a pretty similar manner. You can see people's domain addresses and moderation labelers, but there's not really a window into how that all works under the hood. The official atproto.com site isn't that helpful, either - it's full of marketing fluff and jargon, and even their best explanation of how the system operates expects you already know about large-scale distributed systems engineering.

It certainly doesn't help that atproto does ultimately have a good amount of complexity to it. The network design takes the core components that any social network needs to function - the account login, the user data storage, the feed construction logic (chronological or otherwise), the admin moderation tools, and the user-facing interface - and splits them up into separate interconnected systems, while also adding on a couple other layers to help make network-wide communication a bit less bandwidth-intensive. On top of that, everything is woven together with cryptographic paperwork to ensure that no point in the storage-to-interface pipeline can get away with lying about what the user data says.

However, the big sticking point here is that it's actually not much more complex than any other major social network is behind the scenes. There's no single computer in the world powerful enough to run the entirety of Facebook, TikTok, or even Twitter on its own, and even if there was you wouldn't want to keep millions of users' worth of data in a single place without redundancies. The various layers of every other big SNS are split up as well, but that's kept totally out of view because it's not something the end user needs to know about.

Unfortunately, atproto as an open network does not get the benefit of opaqueness, and the current development culture in the scene has not exactly been conducive to writing easy-to-understand explanations of what's actually going on. I'm going to try to do my best to fix that here, but I won't be touching on any of the cryptography or data structure details because those aren't relevant to a big-picture understanding of how things are set up.

So how does atproto work?

A common refrain when explaining how federated social networking standard ActivityPub works is that it's "kind of like email". It's pretty succinct and lets people go "ohhh, I think I get it" without needing to understand what ActivityStreams, JSON-LD, or WebFinger are.

In a similar vein, I would like to propose that atproto is kind of like RSS.

If you strip it down to the absolute fundamentals, an atproto account is just a magic text file on a web server that lists two things: a human-readable handle and a link to another server where its data is stored. The data server holds posts, profile info, and any other general-purpose data the user wants as files sorted by the shape of the data they contain. You can ask the data server to show you all data of a given type from a specific user and it'll give you a big response full of the most recent data pieces of that type.

If you wanna see my recent Bluesky posts, you just tell the server that my account data is stored on to list recent data entries categorized as the type app.bsky.feed.post created under the ID of my account's magic text file, and it spits back a big mass of info like this:

A massive block of unformatted JSON representing recent Bluesky posts from my account.

The response at time of writing for my most recent Bluesky posts. How readable!

At this point, you already have everything you need for an RSS-style reader. You can just check back on this URL every few minutes, see if the result has changed, and display the new entries in the reader if it has.

This doesn't really tend to work well at a large sale, unfortunately. Even a few hundred readers regularly checking around the same time can end up causing issues for smaller server boxes. This is something ActivityPub has struggled with in the past, with website embed data fetches from Mastodon and Pleroma resulting in enough bandwidth issues for sites that get linked to from the fediverse that instance sysadmins have had to create dedicated infrastructure for mitigating it. Most of atproto's design complexity comes from its own attempts to avoid this issue.

Instead of just expecting readers to ask for new entries, an atproto data server provides a websocket link that loudly announces whenever new data changes take place. A new reader still needs to fetch posts from before it subscribed the normal way, but once it's caught up, it can just listen to the announcement link in the background and only update when the server says a new post has been made.

That doesn't entirely solve the problem, though. Even with a push-based server like this, a thousand followers would still require a thousand websocket connections the data server has to handle, and each account you want to follow requires another connection the reader has to make. It would be really convenient for the data server if it only had to announce its changes to a handful of listeners, and really convenient for the reader if it only had to listen to one connection to get the changes for every account it cares about.

This is where the next layer in the network comes in. A relay server works as an aggregator, listening to all of the nearly four thousand data servers set up around the internet and collecting the update streams of all of their accounts into one big high-speed deluge of data. Now, data servers only need to worry about holding on to about ten connections, and readers only have to worry about holding on to one - the relay in the middle handles all the heavy throughput work, since it doesn't need to worry about doing anything else.


made using Leaflet