Jobs causing memory overflow do not consider $maxExceptions (and are retried endlessly if $tries=0)

[pingencom]

Laravel Version

12.40.1

PHP Version

8.4.15

Database Driver & Version

No response

Description

A job which causes a memory overflow, will be retried endlessly if $tries=0, despite having $maxExceptions=1
We also tried out what happens to jobs which timeout, however there it works as expected.

We'd expect a job which dies because of memory overflow to fail on second pickup by the worker, because $maxExceptions is set to 1 (and i'd argue that a memory overflow constitutes an exception, or is our assumption wrong here?)

This happens for redis driver & database only as for example SQS (as per comment below) has a native protection.

Steps To Reproduce

Create a job with:

timeout=10
tries=0
maxExceptions=1
Set 'retry_after' to something low, so you don't have to wait endlessly until workers pick up the job again

the handle() method of the job should look like this

ini_set('memory_limit', '72M');

$leak = [];
$ticks = 0;
while(true){
    $leak[$ticks] = file_get_contents(__FILE__);
    $ticks++;
    unset($leak[$ticks]);
}

in our stack the initial memory used by a queue worker is approximately 70MB. Maybe you'll need to adjust the memory limit to something closer to your 'initial' memory usage by the worker.

If you dispatch the job and see what horizon does, is that it will

• run the job
• report PHP Fatal error 'Allowed memory size of xxxx bytes exhausted'
• after the 'retry_after' period the same job will be retried again and run into the same issue. This will go on forever

[marius-ciclistu]

From my experince, jobs max memory is not enforced to generate an error but rather the memory consumption is read after each job. There were some issues opened and closed on this topic. #56077

You setting this

ini_set('memory_limit', '72M');

are changing how max memory works in Laravel because this generates a fatal error that is not counted.

A solution to not retry the job that exceeds the memory would be interesting to see.

[pingencom]

Thank you for your insight. Indeed that parameter does not solve the issue. We understand that here laravel doesn't have a way to set the 'job-exceptions' cache key which is used to determine if $maxExceptions has been reached as there is no graceful exit from the job.

There is a method 'markJobAsFailedIfAlreadyExceedsMaxAttempts' and we were trying to figure out if there is a possibility for 'markJobAsFailedIfAlreadyExceedsMaxExceptions' however did not come up with any efficient solution that would make sense (playing around with using $maxExceptions=0)

Our workaround for now is that we monitor ungraceful exits. When a job start it sets a cache key which is removed on graceful exit (which includes timeout). If a job starts and that key is still present, we fail the job. This effectively prevents endless-retries of memory overflow (which is a rare case indeed) but the cost is very high when running millions of jobs just to catch the one job that went rogue due to programmer error.

Any other smart solutions warmely welcomed.

[marius-ciclistu]

SQS from AWS has a default of 3 retries so in that case the jobs will not be retied indefinitely. DB driver for queues is not for production usage.

When a worker processes a message and fails (e.g., crashes, throws an error, or doesn't delete the message), the message becomes visible again after its visibility timeout expires and can be received again by any consumer. Each subsequent delivery increments the ApproximateReceiveCount.

Once this count reaches the maxReceiveCount defined in the redrive policy, SQS moves the message to a dead-letter queue (DLQ) for further inspection or manual handling.

[pingencom]

Thats a fantastic feature of SQS. In our case with redis, this is not natively possible.

I've updated the ticket description to reflect that this is mostly considering redis (and would happen with db driver as well, but as you mentioned, this is not used in large production environments, except in rare cases when it is crucial that job dispatching happens within transactions - all-or-nothing)

[marius-ciclistu]

Laravel can't fix this @pingencom . The max memory the queue supports does not involve ini_set. But if you set a low enough memory this can happen to any job. It is the same for timeout jobs that should be unique. #57772 Their lock is never released because the timeout stops the process in the middle of it. That is why setting an expiration for those locks instead of 0 s (indefinite) is a must that Laravel does not enforce.

Update
Laravel CAN fix it.

[graemlourens]

Hi @marius-ciclistu. I will takover directly: not sure if you have misunderstood the description. We do not use ini_set in production, this was just used in the example how to reproduce the error. there are many other ways.

We have indeed fixed it with our workaround so there is definitively a way to fix it, the question just being if there is an elgant generic way to do this in laravel directly in order to prevent other users of experiencing endlessly retrying jobs if the case arises that a rogue job or library causes a memory overflow.

furthermore, due to circuit breaker pattern, maintenance automation etc it is possible for jobs to retry dozens or hundreds of times, therefore we mostly use $tries=0 in combination with a retryUntil-Date as well as Maxexceptions. However also the retry-until date is mostly set quite liberally to prevent longer lasting popped circuit breakers to then cause hundreds if not thousands of failed jobs, therefore this is not a feasable solution.

We hope to have clarified sufficiently

[marius-ciclistu]

the question just being if there is an elgant generic way to do this in laravel directly in order to prevent other users of experiencing endlessly retrying jobs if the case arises that a rogue job or library causes a memory overflow.

I missunderstood first. Thank you.

Your fix could be included in laravel for messaging systems that don't have a builtin solution for this.

But also SQS's logic can be cloned for certain queue systems. The only possible issue could be the job identifier and its footprint on the used cache memory.

[macropay-solutions]

@graemlourens @pingencom We created a POC for this but we still need to recreate the job instance in the background command

[macropay-solutions]

@pingencom @graemlourens FYI \Illuminate\Queue\Jobs\RedisJob::attempts is analog with sqs attempts

    public function attempts()
    {
        return (int)$this->job['Attributes']['ApproximateReceiveCount'];
    }

Maybe you can use that

[marius-ciclistu]

@pingencom @graemlourens
The POC is ready for testing.
Could you apply the patch on your local vendor and test it? I don't use redis.