Skip to main content

How does Disney Hotstar capture 5 Billion Emojis during a tournament?

 Here is my understanding of how the system works.


1. Clients send emojis through standard HTTP requests. You can think of Golang Service as a typical Web Server. Golang is chosen because it supports concurrency well. Threads in GoLang are lightweight. 

2. Since the write volume is very high, Kafka (message queue) is used as a buffer.

3. Emoji data are aggregated by a streaming processing service called Spark. It aggregates data every 2 seconds, which is configurable. There is a trade-off to be made based on the interval. A shorter interval means emojis are delivered to other clients faster but it also means more computing resources are needed.

4. Aggregated data is written to another Kafka. 

5. The PubSub consumers pull aggregated emoji data from Kafka. 

6. Emojis are delivered to other clients in real-time through the PubSub infrastructure. 

The PubSub infrastructure is interesting. Hotstar considered the following protocols: Socketio, NATS, MQTT, and gRPC, and settled with MQTT. For those who are interested in the tradeoff discussion, see [2].

A similar design is adopted by LinkedIn which streams a million likes/sec [3].

Over to you: What are some of the off-the-shelf Pub-Sub services available? Is there anything you would do differently in this design?

Sources: 
[1] Capturing A Billion Emo(j)i-ons: https://lnkd.in/e24qZK2s
[2] Building Pubsub for 50M concurrent socket connections: https://lnkd.in/eKHqFeef
[3] Streaming a Million Likes/Second: Real-Time Interactions on Live Video https://lnkd.in/eUthHjv4







Comments

Post a Comment

Popular posts from this blog

Use of PartOf in Hybris

Definition PartOf modifier is used to define the aggregation relationship between Parent and Child objects. To explain it better I would say PartOf is used to defining cascade delete.  When we delete a Parent object then all its child objects(partOf) will be deleted automatically. Why or When to use PartOf? PartOf may make sense when the relationship involves "is part of" description. For example, an OrderEntry record(AbstractOrderEntry) is part of its parent Order(AbstractOrder), and older entries will never be shared between multiple orders. If the Order were to vanish, the OrderEntry should as well, and an order entry without an Order would be a problem. How to use PartOf? Looking into the following items.xml snippet we can able to understand the importance of PartOf This also can be defined at the attribute level. Like <itemtype code="User" extends="Principal" jaloclass="de.hybris.platform.jalo...
Post a Comment
Read more

World of Drool Engine

  Rule Engine Within mission-critical applications, the process of maintaining business logic within the source code can become too complicated. Business rules can be used to ease the development and maintenance by separating business logic from the source code. In the Java world, most of the rules engines libraries implement JSR94 standard known as  Java Rule API Engine .     The following table highlights the top five major Rule Engine based technologies: Popularity Rule Engine Summary 1 Drools A Business Rules Management System (BRMS) solution which provides a core business rules engine (BRE), a web authoring and rules management application (Drools Workbench) and an Eclipse IDE plugin for core development 2 OpenRules A general-purpose business rules and decision management system 3 EasyRules EasyRules pro...
Post a Comment
Read more