Learning Websub (Part 4) : Connecting a Message Broker
In a previous article, we covered the integration of a 3rd party API with the hub implementation. In this article, we’ll focus on enhancing the scalability, reliability, and stability of the hub implementation by incorporating a message broker. Specifically, we’ll be utilizing the Apache Kafka message broker for this example.
Message brokers are well-suited for implementing WebSub hubs as they fulfill a comparable role. But, before diving into the message broker integration we should identify what are the limitations of the current implementation.
Following flow diagram depicts the current notification dispatching flow.
As we can see there is a Notification Sender component which distributes weather alerts to News Receivers (News Channels). However there are few issues with this approach.
- Every time Notification Sender runs, it has to query the internal state of the hub to retrieve the valid News Receivers for a particular location.
- And if there is an error while distributing content to a News Receiver the particular weather alert will be lost.
- No way to query/view already distributed weather alerts.
The incorporation of Apache Kafka into our notification dispatching process will introduce a publish/subscribe model, which will offer numerous benefits.
- Notification generation and notification distribution will be separated.
- Kafka will provide a persistence layer for weather alerts.
- If there is an error while distributing content to a News Receiver we can implement a proper retry mechanism because the notification will be persisted in Kafka.
The integration of Kafka will result in a modified notification dispatching flow which aligns with the following format.
According to the flow diagram, there are two components, namely Notification Sender and Notification Receiver available within the hub. The Notification Sender component periodically publishes weather reports into a Kafka topic. The Notification Receiver component listens to the Kafka topic and upon receiving an event, it forwards the message to the News Receiver (News channel). Each Notification Receiver component is directly associated with a corresponding News Receiver and each Notification Sender is directly associated with a particular location for which the weather reports are generated.
The diagram below illustrates how the components within the hub are mapped to Kafka topics and consumer groups.
As every Notification Sender is linked to a location, a corresponding Kafka topic will be assigned to each location. Additionally, since each Notification Receiver is connected to a News Receiver, a unique Kafka consumer group will be allocated to every Notification Receiver.
Kafka message publishing logic can be implemented easily.
Since our Notification Sender required publish weather alerts periodically, we will be using a scheduled task to implement it.
In order to set up the News Receiver, it is necessary to have the capability to fetch messages from the Kafka server. Hence, we need a Kafka consumer.
Notification Receiver will be implemented using a Ballerina async function.
When analyzing the code above, we can see that we have disabled automatic offset commit for the Kafka consumer and replaced it with a manual offset commit mechanism. We intentionally made this change because with manual offset commit, we are able to inform Kafka whether a given message has been successfully processed or not, and also enable a retry mechanism for failed messages.
With these changes the state management of the hub implementation is changed as well.
The onSubscriptionIntentVerified function of the hub service will now be updated as follows.
In this article we discussed how we could integrate a message broker to our hub implementation. With this article I will conclude the WebSub article series which I have been writing. Happy coding….!
