Rails 7.1 Adds ActiveJob#perform_all_later To Enqueue Multiple Jobs At Once

Traditionally, enqueuing multiple jobs involved individual calls, leading to potential performance bottlenecks when dealing with large volumes of tasks.

Before

Below is an example of using sidekiq as adapter and to enqueue multiple jobs it makes lot of round trips to Redis

# app/jobs/email_notification_job.rb

class EmailNotificationJob < ApplicationJob
  queue_as :default

  def perform(user_id)
    # Code to send an email notification to the user
    puts "Sending email notification to user #{user_id}"
  end
end

# Enqueuing EmailNotificationJob for multiple users
User.find_each do |user|
  EmailNotificationJob.perform_later(user.id)
end

After

Enter perform_all_later method tailored to streamline the process of enqueuing multiple jobs simultaneously within ActiveJob.

perform_all_later method leverages Sidekiq’s push_bulk functionality, enabling a significant optimization in the enqueuing process. Eliminating repeated Redis round-trips, drastically reduces the overhead associated with queuing numerous jobs, thereby enhancing overall application performance.

The perform_all_later method accepts an array of job instances and doesn’t run any callbacks like ActiveRecord bulk methods.

ActiveJob.perform_all_later([<job1>, <job2>])

Let’s use perform_all_later to Enqueue EmailNotificationJob for multiple users

email_notification_jobs = users.map do |user|
  EmailNotificationJob.new(user.id)
end

ActiveJob.perform_all_later(email_notification_jobs)

For scenarios involving enqueuing over 1000 jobs and implementing delays for specific tasks, passing instantiated jobs directly allows adding individual delays for each job.

email_notification_jobs = users.map.with_index do |user, index|
  EmailNotificationJob.new(user.id).set(wait: index.seconds)
end

ActiveJob.perform_all_later(email_notification_jobs)

The perform_all_later method allows us to enqueue multiple jobs that belong to different classes.

ActiveJob.perform_all_later(
  [
    EmailNotificationJob.new(user.id),
    SignUpNotificationJob.new(user.id)
  ]
)

Few things to consider

• If the queuing backend lacks support for bulk enqueuing, perform_all_later gracefully reverts to sequential job enqueuing.

• perform_all_later method refrains from executing any callbacks (e.g., before_enqueue, before_perform, after_perform), aligning with ActiveRecord’s bulk import APIs.

• At present, no explicit restriction on the quantity of jobs enqueued at once via perform_all_later. Internally, Sidekiq’s push_bulk method adeptly manages this scenario.

• Differing from perform_later that returns an instance of the queued job class, perform_all_later returns nil upon execution.

• perform_all_later method introduces a separate event, enqueue_all.active_job, and does not utilize the existing enqueue.active_job

• For queue adapters that support the new enqueue_all method, such as the Sidekiq adapter, the enqueuing process is further optimized using push_bulk.

Summary

ActiveJob’s perform_all_later brings efficiency and flexibility to job enqueuing in Rails 7.1. This addition streamlines handling diverse job arguments, supports bulk enqueuing, and offers enhanced control over job execution.

However, it’s essential to note the current limitations in handling return values, lack of callbacks, and considerations regarding batch sizes. These aspects provide insights into potential areas for improvement and future enhancements within the ActiveJob framework.

In conclusion, perform_all_later offers a promising solution to accelerate job enqueuing processes, paving the way for enhanced performance and scalability in Rails applications.