While developing your application, you may have some tasks, such as sending emails to clients, which take too long to perform during a typical web request.
ExpressWebJs allows you to easily create queued jobs that will be processed in the background. By moving time-intensive tasks to a queue, your application can respond to web requests with blazing speed and provide a better user experience to your clients.
ExpressWebJs streamlines queue configuration in Config/Queue.ts, providing preset driver connections. Leveraging these queueing features optimizes your application, offering a smoother user experience, particularly for resource-heavy tasks like email dispatch.
Currently, ExpressWebJs version 4.2 supports RabbitMQ and Redis with BullMQ.
What is a Queue?
Using an explanation that is very easy to understand. Imagine you are in a bank and have to wait in line for you to be attended to. Each line represents a queue. Each person waiting in that line represents a job. And there is a worker that processes each person (job).
The banker (worker) has to process each person (job), in the order of first in first out (FIFO).
A queue is a data structure used in computer science and programming to store and manage a collection of items or tasks in an ordered and linear manner. It operates on the principle of "first in, first out" (FIFO), meaning that the item or task that is added to the queue first is the one that will be removed and processed first.
What is Redis?
Redis, which stands for "REmote DIctionary Server," is an open-source, in-memory data storage system. It is often referred to as a data structure server because it provides a way to store, manipulate, and retrieve data structures in real time. Redis is known for its high performance and versatility, and it's commonly used as a caching mechanism, message broker, and data store in various applications.
Redis with bullmq
In order to use the Redis queue driver, you should first download the provider. You will need to install bullMQ.
yarn add bullmq
Once the provider is installed, you can configure a Redis connection in your Config/Queue.ts configuration file.
import { env, queueProviders } from "expresswebcorets/lib/Env";
import * as bullmq from "bullmq"; 👈
export default {
/*
|--------------------------------------------------------------------------
| Default Queue Connection Name
|--------------------------------------------------------------------------
*/
default: env("QUEUE_CONNECTION", null),
/*
|--------------------------------------------------------------------------
| Queue Connections
|--------------------------------------------------------------------------
*/
connections: {
redis: {
driver: "redis",
provider: bullmq, 👈
provider_name: queueProviders.BullMQ,
connection: "default",
queue: env("REDIS_QUEUE", "default"),
password: env("REDIS_PASSWORD"),
host: env("REDIS_HOST"),
port: env("REDIS_PORT"),
},
},
};
We imported Bullmq and set it as the provider in connections redis provider.
And finally, set your configuration variables in .env
QUEUE_CONNECTION=redis
REDIS_CLIENT=default
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=null
REDIS_DB=0
Creating Jobs
By default, all of the queueable jobs for your application are stored in the App/Jobs directory. If the App/Jobs directory doesn't exist, it will be created when you run the make-job Maker command:
yarn maker make-job SendMailJob
The above command will generate SendMailJob class in the App/Jobs directory.
Here is the generated class:
import ShouldQueue from "expresswebcorets/lib/Queue/ShouldQueue";
export class SendMailJob extends ShouldQueue {
constructor() {
super("SendMailJob");
}
/**
* Handle method executes job.
* @return void
*/
public handle<T>(data: T): void {
//
}
}
Job classes are very simple, normally containing only a handle method that is invoked when the job is processed by the queue. You can write your email sending code in the handle method.
Dispatching Jobs
Once you have written your job class, you may dispatch it using the dispatch method on the job itself. The arguments passed to the dispatch method will be given to the job's handle method:
import { Request, Response, NextFunction } from "Elucidate/HttpContext";
import HttpResponse from "Elucidate/HttpContext/ResponseType";
import sendMailJob from "App/Jobs/sendMail_job";
import ClientRepository from "App/Repository/ClientRepository";
export class ClientController{
private clientRepository = Autowire(ClientRepository);
public async store(req: Request, res: Response){
const client = await this.clientRepository.save({
firstName: req.body.firstname,
lastName: req.body.lastname,
email: req.body.email,
});
new sendMailJob().dispatch(client); 👈 You dispatch your job
this.response.OK(res, client);
}
}
If you would like to conditionally dispatch a job, you may use the dispatchIf method:
const condition = client.email? true : false;
new sendMailJob().dispatchIf(condition, client);
The above code will dispatch the job only if the client has an email.
Using BullMQ options:
You can still pass Bullmq job options when dispatching your job.
The code below is not related to the SendEmailJob.
import { JobOptions } from "bullmq";
new lookUpJob().dispatch<IUser, JobOptions>(user, {
repeat: {
every: 10000,
limit: 20,
},
});
This repeats a job every 10 seconds but no more than 20 times. You can read more on the BullMQ website for more options to use.
Dispatching To A Particular Queue:
By pushing jobs to different queues, you may "categorize" your queued jobs. Keep in mind, this does not push jobs to different queue "connections" as defined by your queue configuration file, but only to specific queues within a single connection. To specify the queue, use the onQueue method when dispatching the job:
import { Request, Response, NextFunction } from "Elucidate/HttpContext";
import HttpResponse from "Elucidate/HttpContext/ResponseType";
import sendMailJob from "App/Jobs/sendMail_job";
import ClientRepository from "App/Repository/ClientRepository";
export class ClientController{
private clientRepository = Autowire(ClientRepository);
public async store(req: Request, res: Response){
const client = await this.clientRepository.save({
firstName: req.body.firstname,
lastName: req.body.lastname,
email: req.body.email,
});
new sendMailJob().onQueue("email").dispatch(client); 👈
this.response.OK(res, client);
}
}
Running The Queue Worker:
Queue worker is a process that runs in the background and is responsible for processing unprocessed jobs in a queue.
ExpressWebjs includes a Maker command that will start a queue worker and process new jobs as they are pushed into the queue. You may run the worker using the queue-work Maker command. Note that once the queue-work command has started, it will continue to run until it is manually stopped or you close your terminal:
yarn maker queue-work
If you are running the queue with a name, then you need to add the name after queue-work:
yarn maker queue-work [QUEUE-NAME]
In production, you need a way to keep your queue-work processes running. A queue-work process may stop running for a variety of reasons, such as an exceeded worker timeout.
For this reason, you need to configure a process monitor that can detect when your queue:work processes exit and automatically restart them.
Conclusion
Congratulations! You have seen how queues handle jobs in ExpressWebJs making it easy to process heavy tasks in the background without the user having to wait for them to be completed.
This was just an example of where and how you could use queues. Also, we have covered dispatch, dispatchIf, and onQueue.
Hope this article was helpful, Kindly:
Star to support ExpressWebJs on GitHub