Your Laravel jobs are running twice: retry_after vs timeout

NewSumm reads news from hundreds of outlets and groups articles about the same event into one story. All of that happens in queued jobs: fetching feeds, grouping articles, writing summaries. Some of those jobs take several minutes.

For a while, our failed jobs table kept filling up. 726 failures, nearly all of them TimeoutExceededException, spread across fetching, grouping and summarising. Nothing was obviously broken. The jobs worked when we ran them by hand. They just kept failing in production, and the summaries kept costing more than they should.

The cause was two numbers in two different files.

Two timers that sound like the same thing

Laravel queues have two settings that are easy to mix up.

The job timeout is how long a single job is allowed to run before the worker stops it. You set it on the job class, or with --timeout on queue:work (the default is 60 seconds):

class ClusterArticlesJob implements ShouldQueue
{
    public int $timeout = 600;
}

retry_after is set per connection in config/queue.php. It is how long the queue waits for a job it has handed out before deciding the worker must have died, and handing the job to someone else:

'database' => [
    'driver' => 'database',
    'queue' => 'default',
    'retry_after' => 90,
],

90 is the value a new Laravel app ships with. It is fine while every job finishes in well under a minute and a half.

What happens when retry_after is shorter

Our grouping job could run for up to 600 seconds. With retry_after at 90, this is what one job looked like:

1. Worker 1 picks up the job and starts working.
2. At 90 seconds the job is still running, which is normal. The queue assumes worker 1 has died and makes the job available again.
3. Worker 2 picks it up and starts the same work from the beginning. So does worker 3, 90 seconds later.
4. Each pick-up counts as an attempt. Eventually the copies either hit their timeout or run out of attempts, and the job lands in failed_jobs.

Meanwhile every copy was doing real work. Stories were grouped twice, and summaries were written two or three times over, each one a paid API call. The failures were the visible part. The duplicate work was the expensive part.

Why nobody spots it

• The two numbers live far apart. One is a property on a job class in app/Jobs, the other a line in config/queue.php. Nothing connects them, so no code review will notice.
• It works locally. In development you often use the sync driver, or only one worker, so there is nobody to hand the job to.
• The error is misleading. TimeoutExceededException sends you looking for slow code. The job wasn't too slow; it was being run several times at once.
• Unique jobs don't help. ShouldBeUnique stops a second copy being dispatched. Here nothing new was dispatched. The queue handed out the same job again.

The rule

We raised retry_after to 1200 seconds on every connection that has one, and made it configurable per environment:

'retry_after' => (int) env('DB_QUEUE_RETRY_AFTER', 1200),

There is a trade-off. If a worker really does die (an out-of-memory kill, a server restart), its job now waits up to 20 minutes before it is retried. For us that is fine: the scheduled jobs run again on their own soon after. If your app has both quick jobs that must recover fast and slow jobs that must not be duplicated, put the slow ones on their own connection with its own retry_after.

(SQS doesn't use retry_after; the queue's visibility timeout plays the same role, and the same rule applies.)

Making sure it can't come back

The fix was one line. The real risk was someone adding a slow job next year and bringing the bug straight back, because nothing connects the two numbers. So we wrote a test that does:

it('keeps retry_after above every job timeout', function (string $job) {
    $timeout = (new $job)->timeout;

    collect(config('queue.connections'))
        ->filter(fn (array $connection) => isset($connection['retry_after']))
        ->each(fn (array $connection, string $name) => expect($connection['retry_after'])
            ->toBeGreaterThan($timeout, "{$job} timeout must be below the '{$name}' retry_after"));
})->with([
    ClusterArticlesJob::class,
    GenerateEmbeddingsJob::class,
    SummarisePendingClustersJob::class,
]);

Now raising a job's timeout past retry_after fails the build with a message that says exactly what to change.

One more trap in the same place

While we were in there, we found a second one. A worker started with:

php artisan queue:work --queue=default,news

does not share its time between the two queues. The list is a priority order: the worker empties default completely before it takes a single job from news. If default is refilled every few minutes and holds long jobs, news can wait forever. Nothing fails and nothing is logged; the jobs just sit there.

If two queues matter equally, give each one its own worker.

Checklist

• Find your longest job timeout ($timeout on the class, or the worker's --timeout).
• Make sure every connection's retry_after is comfortably above it.
• Add a test that checks this, so the next slow job can't quietly break it.
• Check what your --queue= lists actually mean: they are priorities, not a fair share.

NewSumm groups news from hundreds of outlets into single stories, with a short summary and a link to every source. You can see it in action on today's top stories.