Sidekiq To SolidQueue Migration

As part of optimizing our Miru application deployed on Fly, we decided to migrate from sidekiq to solid_queue. The main reason behind this migration was the additional cost incurred from using Upstash Redis for Sidekiq. SolidQueue allows us to utilize PostgreSQL for job storage, eliminating the need for Redis and simplifying our infrastructure.

In this blog post, we will walk you through the steps of this migration, using examples from our application to illustrate the process.

Getting Started

First, we will update our Gemfile to include SolidQueue and run bundle install

# Gemfile

gem "solid_queue", "~> 0.3"

Now, we need to install the required migrations and config file using the below command

bin/rails generate solid_queue:install
bin/rails db:migrate

We will now set solid_queue as Active Job’s adapter in our environment config file

# config/application.rb

config.active_job.queue_adapter = :solid_queue

Configurations

Uncomment the config/solid_queue.yml file and edit it as per our needs. Our config file looks like this

# config/solid_queue.yml
default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
      recurring_tasks:
        update_invoice_status:
          class: UpdateInvoiceStatusToOverdueJob
          schedule: '0 0 * * *' # Runs every day at 12AM UTC
        weekly_reminder:
          class: WeeklyReminderToUserJob
          schedule: '0 14 * * 1 Asia/Kolkata' # Runs every Monday at 14:00
  workers:
    - queues: "*"
      threads: 5
      processes: 1
      polling_interval: 0.1

development:
  <<: *default

test:
  <<: *default

production:
  <<: *default

The solid_queue.yml file is pivotal in configuring how it manages background jobs. It includes a default section that specifies dispatcher and worker settings, applicable across all environments unless overridden.

Dispatchers handle job scheduling, with settings like polling_interval and batch_size. Recurring tasks, such as update_invoice_status and weekly_reminder, are also defined here with their respective schedules.

Workers settings define how jobs are processed, including the number of threads and processes, and the polling_interval for checking new jobs.

Worker Configuration

We will now update the worker configuration to use solid_queue

# fly.production.toml
[processes]
  web = "bin/rails fly:server"
  worker = "bundle exec rake solid_queue:start"

# Procfile.dev

web: bin/rails s -b 0.0.0.0 -p 3000
webpacker: bin/webpacker-dev-server
worker: bundle exec rake solid_queue:start

The worker command will start the SolidQueue’s supervisor and it will start processing the enqueued jobs.

The supervisor in SolidQueue is responsible for creating and managing worker and dispatcher processes based on the configuration. It monitors their status through heartbeats and controls their operation, sending signals to start or stop them as necessary.

Monitoring and Retries

SolidQueue does not have an automatic retry mechanism, it relies on Active Job for that. Failed jobs are moved to a designated table.

We can use mission_control_jobs to monitor jobs and for retrying failed jobs.

# Gemfile

gem "mission-control-jobs"

Now run bundle install and mount the mission control in th routes.rb file.

# routes.rb

Rails.application.routes.draw do
    mount MissionControl::Jobs::Engine, at: "/jobs"
end

Now, on visiting jobs will load a dashboard where we can monitor our jobs.

To prevent exposing jobs route to unauthorized users we can add authentication for the production environment by creating a new controller and specifying it as base class for the Mission Control’s controller.

# application.rb

config.mission_control.jobs.base_controller_class = "MissionControlAuthController"

class MissionControlAuthController < ApplicationController
  before_action :authenticate!, if: :production_env?
  skip_after_action :verify_authorized

  private

    def authenticate!
      authenticate_or_request_with_http_basic do |username, password|
        username == ENV.fetch("SOLID_QUEUE_USERNAME") && password == ENV.fetch("SOLID_QUEUE_PASSWORD")
      end
    end

    def production_env?
      Rails.env.production?
    end
end

Migrating from Sidekiq to SolidQueue was a significant step towards simplifying our application’s background job processing. The transition was smooth, and straightforward. By leveraging PostgreSQL, we reduced our dependency on Redis, making our stack easier to manage.