Efficient Background Processing in NodeJS with BullMQ

Efficient Background Processing in NodeJS with BullMQ — MP4 to HLS.

In one of our recent projects, we had to design a scalable distributed system to process incoming videos and serve those videos efficiently to large amount of concurrent users. Like netflix, but not complex from engineer perspective.

Two of the major concerns we had while designing the system was efficiently converting the video into HLS format, so that user can have better experience and the video adapts with change of the bandwidth, other concern is regarding tracking the watch time and recommending other videos to the user.

The second one is a lot easier because of prior experience of developers in our team with recommendation system on python and websockets for tracking everything the user does.

For converting the video into HLS format, we had a major hassle as this requires a lot of domain knowledge about video encoding and conversions.

Luckily, ffmpeg acted as a saviour for all of our issues. We could have used node-ffmpeg but we went with native ffmpeg in order to keep it simple, as we’re going distributed we may also have an extra endpoint layer of ffmpeg in aws to handle the video conversion in and out from AWS S3.

$ ffmpeg -i filename.mp4 -codec: copy -start_number 0 -hls_time 10 -hls_list_size 0 -f hls filename.m3u8

We used this script in order to convert a file from mp4 format to HLS.

Currently, we have a bit longer script which re-encodes the video into multiple resolutions for seamless and lag-free streaming experience.

In order to let this script to be executed, we needed a message broker which takes incoming video and consumes it to convert it to HLS.

For that problem, we used bull to act as message broker.

We created a queue at first called VideoQueue which will route our messages into the consumer.

Then, we wrote a very simple consumer function which just encodes the raw video file. Keep in mind, if you’re planning to do the same, you must validate all the inputs. Don’t throw native packages randomly as the file may not be what you want and unsanitized may lead to RCE.

This is our internal video converter which uses the shell script in order to convert the file and save it into movies folder.

Bull is extremely easy to use and configure.

It just stores your message into redis and consumer pulls from it. We’ve had a pretty good response time and conversion status with it as of now. We’re currently experimenting other video conversion formats to serve best traffic under certain video bandwidth.

Thanks. Too busy these days. No time to write articles.

Efficient Background Processing in NodeJS with BullMQ — MP4 to HLS.

Comments (1)

More from this blog

Is your code really good?

From Dumb and Boring YAML to Dumb and Boring YAML; IaC Experience

Structuring Infrastructure as Code with a Layered Approach

Hands-On: I wrote my auth logic with JS in Nginx; Will you?

Dev Blog 1; Scratching the Kube API Server to get TTY

Command Palette

Comments (1)

More from this blog