Building a Scalable News Feed System with Caching and Rate Limiting

Designing a newsfeed system is a popular interview question, with similar questions being asked for designing Facebook Newsfeeds, Instagram Feeds, Twitter Timelines, and more. In order to design a newsfeed system, the first step is to understand the problem and establish the design scope. This includes figuring out what features need to be supported, such as whether the system is a mobile app, web app, or both, and what important features are necessary.

The newsfeed system is divided into two parts: feed publishing and newsfeed building. Feed publishing involves writing corresponding data into the cache and database when a user publishes a post, and this data is populated to the user’s friends’ newsfeeds. The newsfeed APIs allow clients to communicate with the service, including posting a status, retrieving news feeds, and adding friends.

The newsfeed building section discusses how newsfeed is built behind the scenes. When a user sends a request to retrieve their newsfeed, the system retrieves newsfeed data from the cache and generates the newsfeed in real time. For users with many friends, the system adopts a hybrid approach to precompute the newsfeed during writing time and generate the newsfeed in real time for other users.

The fanout service fetches friends IDs from the graph database and filters out friends based on user settings. The message queue sends friend list and new post ID to the found-out workers, which fetches data from the…