Background jobs in a monolith are less about reliability and more about isolating failure.
Here’s how you’d set up a basic background job processing system within a monolithic application, using a simple queue and worker pattern.
Imagine you have a user registration process. Sending a welcome email, creating a profile picture thumbnail, and adding the user to a newsletter mailing list are all tasks that don’t need to happen synchronously during the user’s signup. Blocking the user’s signup while these tasks complete would lead to a poor user experience.
This is where background jobs shine. We’ll use a simple in-memory queue for demonstration, but in production, you’d use a robust message broker like Redis, RabbitMQ, or Kafka.
# app/jobs/welcome_email_job.rb
class WelcomeEmailJob < ApplicationJob
queue_as :default
def perform(user_id)
user = User.find(user_id)
WelcomeMailer.send_welcome_email(user).deliver_now
Rails.logger.info "Sent welcome email to user #{user_id}"
end
end
# app/controllers/users_controller.rb
class UsersController < ApplicationController
def create
@user = User.new(user_params)
if @user.save
WelcomeEmailJob.perform_later(@user.id) # Enqueue the job
flash[:notice] = "User created successfully. Welcome email is on its way!"
redirect_to @user
else
render :new
end
end
private
def user_params
params.require(:user).permit(:email, :password, :name)
end
end
In this snippet, WelcomeEmailJob defines the work to be done: finding a user by ID and sending them an email. The perform_later method enqueues this job. The queue_as :default line tells Active Job which queue this job belongs to.
Now, we need a worker process to actually run these jobs. This worker is typically a separate process, even within a monolith, that polls the queue for new jobs and executes them.
# In your terminal, within your Rails application directory
bundle exec rails jobs:work
This command starts a single worker process. It will continuously check the default queue (and any other queues you specify) for jobs. When it finds one, it deserializes the job data, calls the perform method with the provided arguments, and marks the job as completed. If an error occurs during perform, Active Job has built-in retry mechanisms.
The mental model here is a producer-consumer pattern. Your application code (the controller in this case) produces jobs by enqueuing them. A separate worker process consumes these jobs from the queue and executes them. This decoupling is key: if the worker crashes, the jobs aren’t lost (if using a persistent queue) and can be picked up by another worker later. The main application thread remains unblocked, allowing it to handle more incoming requests.
The queue_as method is more than just a label; it dictates which queue the job is placed into. You can define multiple queues with different priorities. For example, a critical job like processing a payment might be queue_as :critical, while sending a newsletter could be queue_as :low_priority. You can then configure your workers to poll these queues in a specific order of priority.
# app/jobs/process_payment_job.rb
class ProcessPaymentJob < ApplicationJob
queue_as :critical # This job goes into the 'critical' queue
def perform(order_id)
order = Order.find(order_id)
PaymentProcessor.process(order)
Rails.logger.info "Payment processed for order #{order_id}"
end
end
# app/jobs/send_newsletter_job.rb
class SendNewsletterJob < ApplicationJob
queue_as :low_priority # This job goes into the 'low_priority' queue
def perform(newsletter_id)
newsletter = Newsletter.find(newsletter_id)
User.find_each { |user| NewsletterMailer.send_to(user, newsletter).deliver_later }
Rails.logger.info "Newsletter #{newsletter_id} sent to all users"
end
end
When starting your workers, you can specify which queues they should listen to and in what order:
# Worker that only processes critical jobs
bundle exec rails jobs:work --queue critical
# Worker that processes default and low_priority jobs, giving default higher precedence
bundle exec rails jobs:work --queue default,low_priority
This allows you to manage resource allocation. You might have dedicated workers for high-priority tasks and a larger pool of workers for less critical ones.
The real power of background jobs in a monolith isn’t just about offloading work; it’s about managing load and resilience. By deferring non-essential tasks, you prevent your primary application threads from being bogged down by I/O-bound operations like network requests or disk writes. This means your application can handle more concurrent user requests, leading to a snappier user experience. Furthermore, if a background job fails, it doesn’t necessarily bring down the entire request-response cycle. Active Job’s built-in retry mechanisms (which you can configure with retry_on or attempts) ensure that transient failures are handled gracefully. You can also set up dead-letter queues to capture jobs that consistently fail, allowing for manual inspection and intervention.
The most overlooked aspect of background job configuration is often the serialization format. By default, Active Job uses YAML. While convenient for development, YAML can be a security risk if your queue is ever exposed or if you’re processing jobs from untrusted sources, as it can execute arbitrary code. For production environments, especially with external message brokers, it’s safer to configure JSON serialization. You’d do this in an initializer:
# config/initializers/active_job.rb
Rails.application.config.active_job.serializer = :json
This change means job arguments and serialized job data will be in JSON format, which is generally safer and more interoperable.
The next challenge you’ll face is scaling your workers to handle increased job volume.