Meta recently revealed Owl, its new hot content distribution system that provides high-throughput distribution of large data objects to hosts in Meta’s private cloud.
Owl consists of a decentralized data plane based on peer-to-peer distribution trees with a centralized control plane – tracking services that maintain metadata about peers, their cache status, and ongoing downloads. It also provides a configurable policy interface that customizes various distribution use cases.
Before Owl, Meta had tried three different systems to solve this problem.
The first implementation used hierarchical caching to provide centralized content distribution. This system was easy to use but had various problems. This required many machines for caching. Sudden load spikes caused requests to be throttled. Provisioning was difficult due to the system’s high infrastructure requirements.
In the following set of implementations, Meta engineers used two highly decentralized solutions: a location-aware BitTorrent implementation and a hash-based static peer-to-peer distribution tree.
These highly decentralized systems scaled better than hierarchical caching. But they introduced a new set of challenges.
In these systems, peers decided where to fetch and what to cache. Each peer made independent decisions, leading to suboptimal results on where to fetch. To get system status and health, engineers had to collect and aggregate data from their peers.
As a result, highly decentralized systems were inefficient and difficult to maintain, while highly centralized systems did not scale well.
To solve this problem, Meta engineers decided to use the split approach. In the new design, peers are simple and provide the mechanism for caching and forwarding blocks of data, while the central control plane is made up of trackers that identify the sources from which peers should obtain each block of data. content, when and how to cache fetched content, and how to retry failed downloads.
A request is sent to the tracker when a client requests a song. The tracker asks the Superpeer to get the chunk from external storage, cache it, and then ask the client to download it.
Super peers are tasks running the Owl Peer Library as a standalone process without any clients. They have large, long-lived caches. They can also recover data from external storage systems that a regular peer cannot recover.
The team had anticipated that the capacity of a single tracker would not be sufficient as Owl usage increased. So, they added the ability to share peers across multiple trackers.
To ensure security, all communication between Owl components is encrypted and all RPCs are checked against access control lists. When reading blocks from external sources, Owl generates an internal checksum, passes it along with the block data, and validates it before sending it back to clients. Thus, maintaining integrity throughout the process.
Image source: the original Engineering at Meta blog post
Another challenge was that some customers needed low latency, while others wanted to reduce the load on external storage to avoid throttling.
Because distribution policies are at the tracker level, teams now have the ability to update policies quickly and customize content distribution for each type of customer.
As mentioned in the article, Owl distributes over 700PB of hot content daily to millions of peers at Meta with a cache hit rate of over 90%.