Skip to main content

How does Disney Hotstar capture 5 Billion Emojis during a tournament?

 Here is my understanding of how the system works.


1. Clients send emojis through standard HTTP requests. You can think of Golang Service as a typical Web Server. Golang is chosen because it supports concurrency well. Threads in GoLang are lightweight. 

2. Since the write volume is very high, Kafka (message queue) is used as a buffer.

3. Emoji data are aggregated by a streaming processing service called Spark. It aggregates data every 2 seconds, which is configurable. There is a trade-off to be made based on the interval. A shorter interval means emojis are delivered to other clients faster but it also means more computing resources are needed.

4. Aggregated data is written to another Kafka. 

5. The PubSub consumers pull aggregated emoji data from Kafka. 

6. Emojis are delivered to other clients in real-time through the PubSub infrastructure. 

The PubSub infrastructure is interesting. Hotstar considered the following protocols: Socketio, NATS, MQTT, and gRPC, and settled with MQTT. For those who are interested in the tradeoff discussion, see [2].

A similar design is adopted by LinkedIn which streams a million likes/sec [3].

Over to you: What are some of the off-the-shelf Pub-Sub services available? Is there anything you would do differently in this design?

Sources: 
[1] Capturing A Billion Emo(j)i-ons: https://lnkd.in/e24qZK2s
[2] Building Pubsub for 50M concurrent socket connections: https://lnkd.in/eKHqFeef
[3] Streaming a Million Likes/Second: Real-Time Interactions on Live Video https://lnkd.in/eUthHjv4







Comments

Post a Comment

Popular posts from this blog

Performance Improvement in Hybris eCommerce

Below points will specially help   grocery   based projects where adding   50-60 products per cart   is a common trend and where   complex promotions   are used using   drools   engine. Add/Update to cart     à   In OOB Hybris whenever any new product is added to cart or quantity of existing product is updated in cart (from PLP, PDP, Search page etc) then calculateCart() method is called in order to calculate total price of the cart. This call can be prevented (to increase   performance ) in case the business requirement is to just show notification of add to cart without total price in minicart. View Cart   à   In case 50-60 products are added in cart and approx. 30% of the products have promotions applied then OOB Hybris view cart can take between 10-15 seconds to load (based on complexity of promotion). In order to avoid custom waiting 10-15 sec to see view cart this operation of can be split into below 3 steps so that every 3-4 seconds rendering of data starts and custom
Post a Comment
Read more

SmartEdit & its Personalization in Hybris

SmartEdit introduced  in 6.0 Hybris version and  Personalization (based on SmartEdit)  enabled in 6.1 Hybris version Onwards . SAP Commerce SmartEdit allows content managers to easily create and manage their website content on-the-fly in different inflection points and make it available to their customers with the click of a button. Personalization (based on SmartEdit) provides an integrated, user-friendly way of building experiences that are relevant to your customers, which is key to driving engagement and conversion.  Personalization encompasses modules and extensions that facilitate using Personalization Mode in SmartEdit to create a fully customized user experience. Personalization capabilities work across both content and commerce in an integrated way, so that you can build customer experiences consistently across channels and functionality. Using SmartEdit for Customer Experience, you can see the end-customer experience as you build it, all the while working in one set o
Post a Comment
Read more

Use of PartOf in Hybris

Definition PartOf modifier is used to define the aggregation relationship between Parent and Child objects. To explain it better I would say PartOf is used to defining cascade delete.  When we delete a Parent object then all its child objects(partOf) will be deleted automatically. Why or When to use PartOf? PartOf may make sense when the relationship involves "is part of" description. For example, an OrderEntry record(AbstractOrderEntry) is part of its parent Order(AbstractOrder), and older entries will never be shared between multiple orders. If the Order were to vanish, the OrderEntry should as well, and an order entry without an Order would be a problem. How to use PartOf? Looking into the following items.xml snippet we can able to understand the importance of PartOf This also can be defined at the attribute level. Like <itemtype code="User" extends="Principal" jaloclass="de.hybris.platform.jalo
Post a Comment
Read more