What is ExpressWebJs:
ExpressWebJs is a powerful and flexible Node FrameWork for building efficient and scalable backend services for modern applications. It is designed to be easy to use and flexible, allowing you to build a wide range of backend services.
It includes support for enterprise-level features such as building multi-tenant systems, distributed systems, multi-threading, real-time events, queues, dependency injection, testing, and more. With ExpressWebJs, you can easily build efficient and reliable backend services that can handle a large volume of traffic and data.
What is Kafka:
Apache Kafka is a powerful distributed event streaming platform widely used for building real-time, scalable, and fault-tolerant applications. Kafka allows seamless communication between various applications, systems, and services. In this tutorial, we will explore how to use Kafka in ExpressWebJs by using kafkajs library, a popular Kafka client for Node.js.
Prerequisites:
Before diving into Kafka with ExpressWebJs using kafkajs, ensure you have the following prerequisites installed:
Node.js and npm: Make sure you have Node.js (v16 or higher) and npm (Node Package Manager) installed. You can download them from the official Node.js website.
Kafka Broker: Have a Kafka broker up and running. If you haven't set up Kafka, follow the official Apache Kafka quickstart guide to install and start Kafka.
Setting up:
To install ExpressWeb, Make sure you have Nodejs and ts-node installed. Once that is done, you can use npx expresswebcli new command, followed by your project name and --ts or --typescript to install your ExpressWeb Project.
--ts or --typescript flag enables Expresswebjs to generate the typescript version.
npx expresswebcli new myProjectName --typescript
Or you can use ts in place of typescript:
npx expresswebcli new myProjectName --ts
You can decide to install expresswebcli globally via npm like so:
npm install -g expresswebcli
Then run the below command to create your project.
expresswebcli new ExpressWebJsKafka --ts
Once that is done, move into your project directory or use the command below in your terminal.
cd ExpressWebJsKafka
Then open the project in your code editor:
Then install all dependencies by running yarn install.
yarn install
Finally, install Kafkajs library
yarn add kafkajs
Once that is done, run the following command to generate your .env file from example.env
cp example.env .env
In the env file, add the following variables:
KAFKA_CLIENT_ID=Test
KAFKA_BROKERS=localhost:9092
Because I want this tutorial to only focus on Kafka integration with ExpressWebJs, I will not be connecting to HTTP server. To do this, I will update the app.ts file in the root directory.
import { StartApp } from "expresswebcorets";
StartApp.withOutServiceBrokerAndHttpServer();
This tells our application to start without HTTP server and Service Broker.
Creating Kafka Client:
Next is to create the Kafka client which handles the connection to Kafka. In App directory, create a file called KafkaClient.ts and paste the code below.
import { Kafka } from "kafkajs";
export class KafkaClient {
private client: Kafka;
constructor() {
this.client = new Kafka({
clientId: process.env["KAFKA_CLIENT_ID"],
brokers: [process.env["KAFKA_BROKERS"] as string],
});
}
public getClient(): Kafka {
return this.client;
}
}
Register KafkaClient In AppServiceProvider:
Now we need to register KafkaCkient in AppServiceProvider class in the App/Providers directory. Here I pasted the complete code, and will go on to explain each section.
import { ServiceProvider } from "Elucidate/Support/ServiceProvider";
import { Authenticator } from "Elucidate/Auth/Authenticator";
import { KafkaClient } from "App/KafkaClient";
import { MessageProcessorServiceImpl } from "App/Service/Impl/MessageProcessorServiceImpl";
import { MessageProcessorService } from "App/Service/MessageProcessorService";
export class AppServiceProvider extends ServiceProvider {
/**
* Register any application services.
* @return void
*/
public register() {
this.singleton(Authenticator);
this.singleton(KafkaClient);
this.bindAsSingletonClass(MessageProcessorService, MessageProcessorServiceImpl);
}
/**
* Bootstrap any application services.
* @return void
*/
public async boot() {
const kafkaClient = this.use(KafkaClient);
const consumer = kafkaClient.getClient().consumer({ groupId: "test-group" });
await consumer.subscribe({ topic: "test-topic", fromBeginning: true });
await consumer.subscribe({ topic: "test-topic2", fromBeginning: true });
await consumer.subscribe({ topic: "test-topic3", fromBeginning: true });
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
if (message.value == null) return;
let offset = message.offset;
let value = message.value.toString();
this.use(MessageProcessorService).processMessage(value, offset, partition, topic);
},
});
}
/**
* Load any service after application boot stage
* @return void
*/
public async booted() {
const messageProcessorService = this.use(MessageProcessorService);
messageProcessorService.publishMessage("test-topic", "Hello and welcome to ExpressWebJs !");
messageProcessorService.publishMessage("test-topic2", "Hello Kafka");
messageProcessorService.publishMessage("test-topic3", "In this tutorial you will learn about Kafka and ExpressWebjs");
}
}
In the register method, we have this.singleton(Authenticator). This comes by default with ExpressWebJs, to auto-resolve authenticator handler
Next, we have this.singleton(KafkaClient). Here we are registering KafkaClient to ExpressWebJs IOC.
The third is this.bindAsSingletonClass(MessageProcessorService,MessageProcessorServiceImpl). This we will create later in the Service folder in App directory. This is binding the abstract class to the implementation.
Next Section is the boot method:
This section is involved in bootstrapping any application service. So this is the perfect place to connect to our consumer and subscribe to topics.
To do this, we use the this.use method, while passing in KafkaClient which was already registered as a singleton, and assigning it to a constant called kafkaClient (k in lowercase). The this.use method resolves and returns the resolved KafkaClient.
Now that we have the resolved KafkaClient, we can access the client via the getClient method while also calling the consumer to assign the groupId
const consumer = kafkaClient.getClient().consumer({ groupId: "test-group" })
Next is to let the consumer subscribe to topics. In this tutorial, we are subscribing to test-topic, test-topic2, and test-topic3 topics.
await consumer.subscribe({ topic: "test-topic", fromBeginning: true });
await consumer.subscribe({ topic: "test-topic2", fromBeginning: true });
await consumer.subscribe({ topic: "test-topic3", fromBeginning: true });
Once that is done, we now run the consumer while passing the value, offset, partition, and topic to our message handler, in this case, MessageProcessorService.processMessage method.
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
if (message.value == null) return;
let offset = message.offset;
let value = message.value.toString();
this.use(MessageProcessorService).processMessage(value, offset, partition, topic);
},
});
The final section in the AppServiceProvider is the booted method. This method loads any service after the application boot stage. You are free to publish your topic from anywhere in the application. But to keep this tutorial simple, I decided to do it here, so that it will be triggered once the application starts.
public async booted() {
const messageProcessorService = this.use(MessageProcessorService);
messageProcessorService.publishMessage("test-topic", "Hello and welcome to ExpressWebJs !");
messageProcessorService.publishMessage("test-topic2", "Hello Kafka");
messageProcessorService.publishMessage("test-topic3", "In this tutorial you will learn about Kafka and ExpressWebjs");
}
Here we are publishing to the topics we subscribed to.
Now let's create our MessageProcessorService abstract class and MessageProcessorServiceImpl class in the App/Service directory.
App/Service/MessageProcessorService.ts
export abstract class MessageProcessorService {
abstract processMessage(message: string, offset: string, partition: number, topic: string): Promise<void>;
abstract publishMessage(topic: string, message: string): Promise<void>;
}
App/Service/Impl/MessageProcessorServiceImpl.ts
import { KafkaClient } from "App/KafkaClient";
import { MessageProcessorService } from "../MessageProcessorService";
import { Producer } from "kafkajs";
export class MessageProcessorServiceImpl implements MessageProcessorService {
producer: Producer;
constructor(private kafkaClient: KafkaClient) {
this.producer = this.kafkaClient.getClient().producer();
}
public async processMessage(message: string, offset: string, partition: number, topic: string): Promise<void> {
console.log({ partition, offset, message });
}
public async publishMessage(topic: string, message: string): Promise<void> {
//to push to a topic
await this.producer.connect();
await this.producer.send({
topic,
messages: [{ value: message }],
});
}
}
Once done run the below code to start your application.
yarn dev
Conclusion:
Congratulations! You have successfully set up a Kafka producer and consumer using the kafkajs library in ExpressWebJs.
This tutorial provides a fundamental introduction to using Kafka in ExpressWebJs with the kafkajs library. From here, you can explore more advanced features and configurations to suit your specific use case. Happy coding!
Kindly:
Star to support ExpressWebJs on GitHub