Rails 8 Solid Queue: Database-Backed Background Jobs

Background job processing has always required external dependencies like Redis or Memcached in Rails applications. With Rails 8, that changes. Solid Queue is now the default Active Job backend, and it stores jobs directly in the database.

This eliminates the need for Redis in many applications, simplifying deployment and reducing infrastructure costs.

In this post, we will explore Solid Queue in depth, covering installation, configuration, real world usage patterns, and important caveats to consider.

Why Solid Queue?

Traditional job backends like Sidekiq and Resque rely on Redis for job storage. While Redis is fast, it adds operational complexity:

• Additional infrastructure to manage
• Memory constraints limiting job history
• Separate backup and monitoring requirements
• Extra costs for managed Redis services

Solid Queue leverages modern SSD performance and database features like FOR UPDATE SKIP LOCKED to achieve comparable throughput without Redis.

Installation

Solid Queue comes preconfigured in new Rails 8 applications. For existing applications on Rails 7.1 or later, we can add it manually:

bundle add solid_queue
bin/rails solid_queue:install

This creates several files:

• config/queue.yml for worker and dispatcher configuration
• config/recurring.yml for scheduled tasks
• db/queue_schema.rb for the database schema
• bin/jobs executable to start the queue processor

Database Configuration

Solid Queue works best with a dedicated database. Here is how to configure it with SQLite:

# config/database.yml



production:

  primary:

    <<: *default

    database: storage/production.sqlite3

  queue:

    <<: *default

    database: storage/production_queue.sqlite3

    migrations_paths: db/queue_migrate

For PostgreSQL or MySQL:

# config/database.yml



production:

  primary:

    <<: *default

    database: myapp_production

    username: myapp

    password: <%= ENV["DATABASE_PASSWORD"] %>

  queue:

    <<: *default

    database: myapp_queue_production

    username: myapp

    password: <%= ENV["DATABASE_PASSWORD"] %>

    migrations_paths: db/queue_migrate

Then run the database setup:

bin/rails db:prepare

Real World Configuration Example

Let us look at a configuration for an e-commerce application that processes orders, sends notifications, and generates reports:

# config/queue.yml



default: &default

  dispatchers:

    - polling_interval: 1

      batch_size: 500

      concurrency_maintenance_interval: 300



  workers:

    - queues: [critical, default]

      threads: 5

      processes: 2

      polling_interval: 0.1



    - queues: [reports]

      threads: 2

      processes: 1

      polling_interval: 1



development:

  <<: *default



production:

  <<: *default

This configuration creates:

• One dispatcher that checks for scheduled jobs every second
• Two worker processes handling critical and default queues with 5 threads each
• One dedicated worker for report generation with lower concurrency

Queue Priority and Order

Solid Queue processes queues in the order specified. In our example, critical jobs always run before default jobs.

We can also use numeric priorities within a queue:

# app/jobs/order_confirmation_job.rb


class OrderConfirmationJob < ApplicationJob
  queue_as :critical
  queue_with_priority 1

  def perform(order_id)
    order = Order.find(order_id)
    OrderMailer.confirmation(order).deliver_now
    order.update!(confirmation_sent_at: Time.current)
  end
end

# app/jobs/inventory_sync_job.rb


class InventorySyncJob < ApplicationJob
  queue_as :critical
  queue_with_priority 10

  def perform(product_id)
    product = Product.find(product_id)
    InventoryService.sync(product)
  end
end

Lower priority numbers run first. Order confirmations (priority 1) will process before inventory syncs (priority 10).

Concurrency Controls

Solid Queue provides built in concurrency controls to prevent resource contention. This is useful when jobs access external APIs with rate limits or shared resources:

# app/jobs/payment_processing_job.rb


class PaymentProcessingJob < ApplicationJob
  queue_as :critical

  limits_concurrency to: 5, key: ->(order_id) { "payments" }, duration: 2.minutes

  def perform(order_id)
    order = Order.find(order_id)
    PaymentGateway.process(order)
  end
end

This ensures only 5 payment jobs run simultaneously. Other jobs wait until a slot becomes available.

For per record concurrency:

# app/jobs/user_sync_job.rb


class UserSyncJob < ApplicationJob
  queue_as :default

  limits_concurrency to: 1, key: ->(user_id) { "user_sync_#{user_id}" }, duration: 5.minutes

  def perform(user_id)
    user = User.find(user_id)
    ExternalCRM.sync_user(user)
  end
end

This prevents multiple sync jobs for the same user from running concurrently.

Recurring Tasks

Solid Queue handles scheduled tasks without external tools like cron:

# config/recurring.yml



production:

  daily_report:

    class: DailyReportJob

    schedule: "0 6 * * *"

    args: ["sales"]



  hourly_cleanup:

    class: CleanupExpiredSessionsJob

    schedule: "0 * * * *"



  weekly_digest:

    class: WeeklyDigestJob

    schedule: "0 9 * * 1"

    queue: reports

The scheduler ensures tasks run exactly once, even with multiple scheduler processes running for redundancy.

Handling Failed Jobs

Solid Queue relies on Active Job for retries. Configure retry behavior in the jobs:

# app/jobs/webhook_delivery_job.rb


class WebhookDeliveryJob < ApplicationJob
  queue_as :default

  retry_on Net::OpenTimeout, wait: :polynomially_longer, attempts: 5
  retry_on Faraday::ConnectionFailed, wait: 30.seconds, attempts: 3
  discard_on ActiveRecord::RecordNotFound

  def perform(webhook_id, payload)
    webhook = Webhook.find(webhook_id)
    WebhookService.deliver(webhook, payload)
  end
end

The polynomially_longer wait strategy increases delay between retries using the formula (executions ** 4) + 2 seconds. This means retries happen at approximately 3s, 18s, 83s, 258s, and so on, giving transient issues time to resolve without overwhelming the failing service.

Failed jobs that exhaust retries are stored in solid_queue_failed_executions. Use Mission Control Jobs for monitoring:

# Gemfile


gem "mission_control-jobs"

# config/routes.rb


Rails.application.routes.draw do
  mount MissionControl::Jobs::Engine, at: "/jobs"
end

Running in Production

Start the job processor with:

bin/jobs

For Puma integration, add to the configuration:

# config/puma.rb


plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"]

Then set SOLID_QUEUE_IN_PUMA=true in production.

Caveats and Considerations

Database Load

Solid Queue adds read and write load to the database. For high volume applications:

• Use a dedicated queue database
• Monitor database connections carefully
• Consider connection pooling settings

Polling Overhead

Unlike Redis pub/sub, Solid Queue polls the database. Very low polling intervals increase database load. Start with 0.1 seconds for workers and adjust based on the latency requirements.

Transaction Behavior

Jobs enqueued within a transaction are visible only after commit. This can cause issues if we expect immediate processing:

# This job might not be visible immediately

ActiveRecord::Base.transaction do
  order = Order.create!(params)
  OrderConfirmationJob.perform_later(order.id)
end

Use enqueue_after_transaction_commit for predictable behavior:

# app/jobs/application_job.rb


class ApplicationJob < ActiveJob::Base
  self.enqueue_after_transaction_commit = :always
end

No Automatic Retry by Default

Unlike Sidekiq, Solid Queue does not retry failed jobs automatically. We must configure retry_on in each job class or in ApplicationJob.

Memory Usage

Long running jobs with large payloads can increase memory usage. Keep job arguments small and load data within the job:

# Avoid this

LargeDataJob.perform_later(huge_array)

# Do this instead

LargeDataJob.perform_later(record_id)

Concurrency Control Duration

The duration parameter in concurrency controls is a failsafe, not a job timeout. If a job crashes without releasing its lock, other jobs wait until duration expires. Set it longer than the longest expected job runtime.

Migration from Sidekiq

Migrating from Sidekiq is straightforward since both use Active Job:

1. Add Solid Queue and run the installer
2. Update queue_adapter configuration
3. Convert Sidekiq specific features to Solid Queue equivalents
4. Test thoroughly in staging
5. Deploy and monitor

Sidekiq specific features like sidekiq_options need conversion:

# Before (Sidekiq)

class MyJob
  include Sidekiq::Job
  sidekiq_options retry: 5, queue: "critical"
end

# After (Solid Queue)

class MyJob < ApplicationJob
  queue_as :critical
  retry_on StandardError, attempts: 5
end

Conclusion

Solid Queue brings database backed job processing to Rails 8, eliminating Redis dependencies for many applications. It offers excellent performance with modern databases, built in concurrency controls, and seamless Active Job integration.

Consider the caveats around database load and polling overhead when planning the deployment. For most applications, Solid Queue provides a simpler, more maintainable solution than traditional Redis backed queues.

Check the official Solid Queue repository for the latest documentation and updates.