Multiple queue workers: some restart, some don't with ERROR (spawn error)

Our application provides a separate database for each of our users. I have set up an emailing job which users may dispatch to run in background via a Laravel 5.3 Queue. Some users succeed in evoking this facility (so it does work - same codebase) but some users fail.

The users who fail to generate the email are all characterised by the following error when trying to restart all user queues using sudo supervisor start all, eg:

shenstone-worker:shenstone-worker_02: ERROR (spawn error)
shenstone-worker:shenstone-worker_00: ERROR (spawn error)
shenstone-worker:shenstone-worker_01: ERROR (spawn error)

An example of a user who's email facility works:

meadowwood-worker:meadowwood-worker_02: started
meadowwood-worker:meadowwood-worker_00: started
meadowwood-worker:meadowwood-worker_01: started

The log of all attempted restarts has a load of these spawn errors at the beginning then all the successful queue restarts at the end.

The worker config files for these two users are:

[program:shenstone-worker]
process_name=%(program_name)s_%(process_num)02d
directory=/var/www/solar3/current
environment=APP_ENV="shenstone"
command=php artisan queue:work --tries=1 --timeout=300
autostart=true
autorestart=true
user=root
numprocs=3
redirect_stderr=true
stdout-logfiles=/var/www/solar3/storage/logs/shenstone-worker.log

and

[program:meadowwood-worker]
process_name=%(program_name)s_%(process_num)02d
directory=/var/www/solar3/current
environment=APP_ENV="meadowwood"
command=php artisan queue:work --tries=1 --timeout=300
autostart=true
autorestart=true
user=root
numprocs=3
redirect_stderr=true
stdout-logfiles=/var/www/solar3/storage/logs/meadowwood-worker.log

As you see, generically identical. Yet shenstone does not restart its queues to capture requests from its jobs table, but meadowwood does. No logfiles appear in storage.

So why do some of these queues restart successfully, and a load don't?

Looking at the stackoverflow issue Running multiple Laravel queue workers using Supervisor inspired me to run sudo supervisorctl status and yes I can see a more elaborate explanation of my problem:

shenstone-worker:shenstone-worker_00                               FATAL     too many open files to spawn 'shenstone-worker_00'
shenstone-worker:shenstone-worker_01                               FATAL     too many open files to spawn 'shenstone-worker_01'
shenstone-worker:shenstone-worker_02                               FATAL     too many open files to spawn 'shenstone-worker_02'

As opposed to:

meadowwood-worker:meadowwood-worker_00                             RUNNING   pid 32459, uptime 0:51:52
meadowwood-worker:meadowwood-worker_01                             RUNNING   pid 32460, uptime 0:51:52
meadowwood-worker:meadowwood-worker_02                             RUNNING   pid 32457, uptime 0:51:52

But I still cannot see what I can do to resolve the issue.

from Newest questions tagged laravel-5 - Stack Overflow https://ift.tt/fLPze5p
via IFTTT