Queues, Workers, and Horizon Under Load

Queues, Workers, and Horizon Under Load takes the pool from Chapter 2 and gives it a service time measured in seconds rather than milliseconds. Everything in that model still applies, and two things get worse: the queue is durable, so it remembers, and the work is retried, so it can multiply.

Key takeaways

• Laravel's queue documentation states that a job timeout must be shorter than the connection's retry_after value, because otherwise jobs may be processed twice.
• Backlog age is the alarm and backlog depth is only a graph. Horizon's waits configuration sets a long-wait threshold per connection and queue, defaulting to 60 seconds, with 0 disabling the notification.
• Horizon documents that under the auto balancing strategy it does not enforce strict priority between queues, and recommends multiple supervisors where you need relative priority.
• Google's SRE book, chapter 22, argues that retries need limits, randomised exponential backoff and a service-wide retry budget, because retries at multiple layers multiply.
• Horizon requires Redis, is not compatible with Redis Cluster at this time, and defaults a supervisor to one attempt per job unless the job class sets its own tries.

Read this after Chapter 2, whose utilisation arithmetic is the whole of queue capacity planning, and after Chapter 5, because moving work to a worker moves the lock contention with it rather than removing it. Chapter 8 applies the same pool model to long-lived web workers, where the service time is short again and the state is not.

An import feature is added. It queues one job per uploaded file, and each job takes minutes.

It shares a queue with password reset emails. On a quiet day nobody notices. On the day a customer uploads two hundred files, password resets stop arriving, and the support team is told the email system is broken.

The email system is fine. It is behind two hundred imports, in a line.

A worker pool is Chapter 2's pool with a longer service time

The arithmetic transfers without modification. Throughput is worker count divided by average job duration, arrival rate is jobs dispatched per second, and utilisation is the ratio of the two.

What changes is the consequence of exceeding one. A web request that cannot be served waits in a socket backlog and eventually times out, which is unpleasant and self-limiting. A job that cannot be served waits in Redis, and Redis has no opinion about how long. The backlog grows for as long as the imbalance lasts, and it is still there tomorrow.

That durability is a feature and it hides the failure. A saturated queue looks exactly like a healthy queue from the outside, because in both cases jobs go in and jobs come out. The difference is only visible in how old the oldest waiting job is.

Two numbers per queue, then. Jobs per second in, and seconds of work per job.

Backlog age is the alarm; depth is only a graph

Depth is the number everybody dashboards and it is close to useless on its own. A depth of five thousand is healthy for a queue of ten millisecond jobs and catastrophic for a queue of ten-minute ones.

Age answers the question the business actually asked. How long has the oldest thing in this queue been waiting, and is that within what we promised. It is dimensionally correct, comparable across queues, and it moves before anything else does.

Horizon gives you this directly. Its documentation records a waits configuration setting long-wait thresholds per connection and queue, defaulting to 60 seconds, where a value of zero disables the notification for that queue. Horizon's metrics dashboard also needs horizon:snapshot scheduled every five minutes to populate, which is a step teams miss and then conclude the metrics are broken.

Set a different threshold per queue, deliberately. A queue of user-facing notifications and a queue of nightly report builds have no business sharing an alarm.

retry_after and the job timeout are one setting with two names

This is the most consequential pair of numbers in Laravel's queue configuration, and getting the relationship backwards causes duplicate side effects rather than slowness.

retry_after, configured per connection, is how long the queue waits before deciding a reserved job was abandoned and making it available again. The worker timeout is how long a worker allows a single job to run before killing it. Laravel's documentation states the constraint plainly: the timeout value should always be shorter than the retry after value, otherwise the job may be processed twice.

Follow the mechanism once and you will never misconfigure it again. If the timeout is longer, the queue releases the job for a second worker while the first is still running it. Two workers now hold the same job. Whatever it was going to do, it does twice.

Laravel also notes that the pcntl extension must be installed for job timeouts to work at all. Without it the setting is inert, which is worth checking on a container image somebody else built.

Head-of-line blocking is a topology problem

One queue serving job classes with wildly different durations is the most common queue design error in Laravel applications, and no worker count fixes it.

The mechanism is the scene above. A short job dispatched behind a long one waits for the long one, regardless of how urgent it is, because a queue is ordered and a worker takes the next thing. Adding workers raises the number of long jobs in flight and leaves the short job behind whichever one it landed after.

Horizon's documentation closes the obvious escape route. Under the auto balancing strategy it does not enforce strict priority between queues, and where you need relative priority the recommendation is to run multiple supervisors. Listing queues in a particular order in the config does not create a priority order.

So the fix is topology. Separate queues by service-time class, not by feature area, and give each its own supervisor with its own process floor. Fast queues get a guaranteed minimum. Slow queues get a ceiling.

Horizon's balancing knobs, and the defaults you inherit

Horizon requires Redis to power the queue and its documentation states it is not compatible with Redis Cluster at this time. That is a constraint on your infrastructure choice rather than a tuning parameter, and it is better known before a migration than during one.

Balancing takes three values. auto is the default and adjusts worker processes per queue based on workload, simple splits processes evenly across queues, and false processes queues in order. Under auto, autoScalingStrategy accepts time, which assigns workers by estimated time to clear the queue, or size, which assigns by job count.

The scaling behaviour is bounded by four settings worth naming: minProcesses as the floor per queue, maxProcesses as the total ceiling across queues, balanceMaxShift as how many processes may be created or destroyed per cycle, and balanceCooldown as the seconds between adjustments. Those last two are the difference between scaling and oscillating.

Two documented defaults cause real incidents. A supervisor's tries defaults to a single attempt unless the job class sets its own, so a transient failure is a permanent one by default. And middleware such as WithoutOverlapping and RateLimited consume attempts, meaning a job released by rate limiting has spent a try without ever running its body.

Retries multiply at every layer that has them

Google's SRE book, chapter 22, is the best short treatment of this and its conclusions apply directly to a Laravel queue.

That chapter argues retries need three things: a limit, randomised exponential backoff, and a service-wide retry budget rather than a per-caller one. Its central warning is that retries at multiple layers multiply. A job retried three times, calling a client that retries three times, against a dependency that is already overloaded, is nine calls where one was intended.

The failure this produces is a positive feedback loop. The dependency is slow because it is overloaded, slowness triggers retries, retries add load, and the system converges on the worst state available to it. Adding workers accelerates it.

Laravel 13 gives you declarative control at the job level through the #[Tries] and #[Backoff] attributes, alongside #[Timeout] and #[FailOnTimeout]. Use backoff with jitter rather than a fixed delay, because a fixed delay synchronises every failed job into a single retry wave.

One more documented trap sits nearby. Laravel warns that setting block_for to zero makes queue workers block indefinitely until a job is available, which also prevents signals such as SIGTERM from being handled until the next job has been processed. That turns a deploy into a wait.

Where jobs actually wait, and when to dispatch them

Most long-running jobs are not computing. They are waiting on somebody else's service, which makes the outbound call the thing to instrument first, and Chapter 1's Pulse recorder for slow outgoing requests is where it shows up.

Laravel's HTTP client documentation records a default connect timeout of ten seconds and lets you set both a connect timeout and a total timeout explicitly. It also documents retry with an attempt count and a delay, and a throw: false form, plus pool for issuing requests concurrently. An outbound call inside a job with no explicit timeout is an unbounded lease on a worker.

There is a dispatch-time trap that looks unrelated and is not. A job dispatched inside a database transaction can be picked up by a worker before the transaction commits, at which point the row it was told about does not exist yet. Laravel documents after_commit, which waits until open parent transactions have committed before dispatching.

Set it on the connection rather than remembering it per call site. A convention beats a habit.

A worked reading, composited and labelled

The details here are composited from ordinary production shapes rather than one system, and the shape is exact.

A queue is reported as backed up. The first reading plots three series per queue on one axis: jobs dispatched per minute, jobs completed per minute, and the age of the oldest waiting job. The shape tells you which problem you have. Completions flat while dispatches rise means you are simply under-provisioned against arrival rate. Completions falling while dispatches hold means service time per job increased, which is a change in the job rather than in the traffic.

The second reading splits the same three series by job class. This is the reading that finds head-of-line blocking, because it shows a fast class whose completion rate collapsed without its own dispatch rate changing at all.

Neither reading contains a duration target. What each contains is a shape that eliminates possibilities, which is what you want before you change a process count. Chapter 12 is where you generate these shapes on purpose rather than waiting for them.

The objection: Horizon's auto balancing will handle it

Autoscaling is genuinely good and it invites a belief it cannot support, which is that capacity planning is now automatic. Two limits are documented and one is structural.

The documented ones are already above. Auto balancing does not enforce strict priority between queues, so it cannot fix head-of-line blocking, and its behaviour is bounded by maxProcesses, balanceMaxShift and balanceCooldown, which are numbers you chose.

The structural limit is Chapter 2's point restated. Autoscaling workers cannot create database connections, Redis capacity or third-party rate limit headroom. Scaling a pool whose downstream pool is already saturated converts a queue you could see into contention you cannot, and the jobs begin failing on lock waits or connection errors instead of waiting politely.

So autoscale between a floor and a ceiling you derived from the pool downstream. That derivation is the work. The scaling strategy is a detail.

Chapter summary

A queue worker pool is Chapter 2's fixed pool with a service time measured in seconds, and the crucial difference is that the queue is durable, so a saturated queue looks identical to a healthy one until you measure the age of the oldest waiting job rather than the count of waiting jobs. Horizon supports this directly through waits thresholds per connection and queue, defaulting to 60 seconds with zero disabling them, and its metrics dashboard needs horizon:snapshot scheduled every five minutes. The single most damaging misconfiguration is a job timeout longer than the connection's retry_after, which Laravel documents as causing jobs to be processed twice, since the queue releases a job that is still running; the ordering is retry_after longest, supervisor timeout shorter, job timeout shortest, and none of it works without pcntl installed. Head-of-line blocking is a topology problem that worker count cannot solve, and Horizon documents that its auto balancing strategy does not enforce strict priority between queues, recommending separate supervisors instead, so queues should be split by service-time class rather than by feature. Horizon requires Redis, is documented as incompatible with Redis Cluster, bounds scaling with minProcesses, maxProcesses, balanceMaxShift and balanceCooldown, defaults a supervisor to one attempt, and consumes attempts through WithoutOverlapping and RateLimited middleware. Google's SRE book, chapter 22, is the reason to cap retries, randomise backoff and hold a service-wide retry budget, because retries at multiple layers multiply into a feedback loop. And an outbound call without an explicit timeout is an unbounded lease on a worker.

Workers that stay alive between units of work are the other half of this idea, and web traffic has its own version. Chapter 8 is Octane and Long-Lived Processes, where the application stops dying between requests and starts remembering things you did not intend it to.