Running multiple queues on the same server using Laravel horizon

Running queues for multiple applications on the same server with Laravel Horizon

Queues are an essential part of modern web applications. By handling long running processes as background tasks, apps become much faster and your users get a better experience. Examples are sending emails, processing file uploads and generating documents. With Laravel Horizon it’s easy to manage queues. Setting up horizon is easy and clearly described in the documentation. However, I ran into some issues when I tried to run horizon for multiple apps on the same server.

Use case

We have a sports app where users can predict the outcome of cycling events like the Tour de France. Each stage, all participants receive an email with the results from the current stage and their standing in the general classification. In the past, we used a simple php script that ran in the browser to send the emails one be one. This worked fine in the beginning, because there were only a few participants (the app originates from 2013). Meanwhile, the number of participants was growing, the script to send the emails took longer, until in the end the script timed out and not all participant received their daily email. It was time for a different setup.

We updated the code base to the latest Laravel version and started using queues and Horizon to handle the emails. On our server, we run two versions of the app: a staging version and a production version. This caused some issues, but in the end it was easy to fix them using the correct configuration.

Configuring Horizon

Start with setting up Horizon by following the documentation. You basically have to install the package using composer and then publish its assets. Then open the config/horizon.php file.

Here, you can set the horizon prefix. This is important when you want to run multiple horizon instances on the same server. Add an entry called ‘HORIZON_PREFIX’ to your .env file for each app on the server and make sure they have different values.

Back in the config/horizon.php file, scroll down to environments. By default, there are 2 entries: production and local. When you have a staging environment on the same server, add another entry for this staging environment. The most important thing is that the key of the environment (e.g. stage), corresponds to the value for APP_ENV in your .env file.

For each entry, there’s an attribute ‘queue’. This is the name of the queue that horizon should handle for this specific application. It’s good practice to use different queues for each app. Make sure that you put jobs in a queue that is actually handled by Horizon. I know this sounds obvious, but I initially overlooked this and it took me quite some time to figure out why my jobs weren’t running.

Configuring Queues

To use Horizon, you have to use Redis as a queue driver. The configuration for queues can be found in config/queue.php. First, you have to set the queue connection, which is set to ‘sync’ by default. Change this by adding the following entry to the .env file: QUEUE_CONNECTION=redis.

Next, go to the connections/redis section. Here you can specify the default queue for redis, by adding a REDIS_QUEUE variable to the .env file. Set this variable to (one of) the queue names that horizon will handle for that application.

Configuring databases

Redis makes use of databases and it would be good to use a different database for each application on the server. Open the config/database.php file and scroll down to the redis section. Here, you can change the redis database that is used. Just add a REDIS_DB parameter to the .env file. A redis database is defined by a number ranging from 0 to 15. Choose a number that is not in use by one of the other apps on the same server.

Now, you’re ready to safely run this app on a server with multiple apps that all use Horizon. The next step is to deploy the app.

Deployment using Laravel Forge

In order to run Horizon on a server that is managed by Laravel Forge, you have to do two things. First, go to the daemons section and create a new daemon. The command will be ‘php artisan horizon’, the user will be ‘forge’ and the directory should be the root of your project.

Forge will then configure supervisor to make sure that this command always keeps running. This also means that after deployment of a new version, you have to terminate horizon to make sure that changes in the code are picked up by the daemon. Therefore, add the following line to your deploy script: php artisan horizon:terminate.

Deployment using Envoyer

If you deploy your app using Envoyer, the setup is slightly different. The horizon:terminate command should be added as a deployment hook. Go to the deployment hooks section and add a new hook after the ‘activate new release’ hook. Run the hook as ‘forge’ and use the following script:

cd {{release}}
php artisan horizon:terminate

Next, go to Forge and navigate to the daemons section of the corresponding server. The command that should be run is ‘php artisan horizon’, but the path is different compared to deployment without Envoyer. It should be the path where the artisan file is located, which is likely: /home/forge/domainname/envoyer/current. Save the daemon and you’re done.

Summary

Running queues for multiple applications on the same server is easy using Laravel Horizon. All you need is the correct configuration and you’re ready to go.



Mijn Twitter profiel Mijn Facebook profiel
Leonie Derendorp Webdeveloper and co-owner of PLint-sites in Sittard, The Netherlands. I love to create complex webapplications using Laravel! All posts
View all posts by Leonie Derendorp

6 thoughts on “Running queues for multiple applications on the same server with Laravel Horizon

  1. Ashish Singh

    Hi,
    I’m in the same situation and have the same setup. I just wanted to clear that do I need to have two separate configurations for supervisor?

    [program:horizon]
    process_name=%(program_name)s
    command=php /home/forge/production.com/artisan horizon
    autostart=true
    autorestart=true
    user=forge
    redirect_stderr=true
    stdout_logfile=/home/forge/production.com/horizon.log

    and for staging

    [program:horizon]
    process_name=%(program_name)s
    command=php /home/forge/staging.com/artisan horizon
    autostart=true
    autorestart=true
    user=forge
    redirect_stderr=true
    stdout_logfile=/home/forge/staging.com/horizon.log

    or just one would be fine?

    Reply
    1. Leonie Derendorp Post author

      Hi Ashish,

      I’m using Laravel Forge to manage supervisor, but I think what you write is correct:

      You need supervisor to manage the ‘artisan horizon’ process on both production.com and staging.com, so you need both configurations.

      Reply
  2. Will

    Really nice set of clear articles which helped me a lot – thanks.

    I have a question though – if the queue worker needs to also queue a job, i.e. a job triggers another job, what queue will the queue worker use?

    Reply
    1. will

      Because I assumed I could use itself, but there was a complaint when deploying the worker about not being able to connect to redis (Connection refused [tcp://127.0.0.1:6379]) but connected fine to non worker.

      Reply
  3. Marcellus Barrus

    I have three environments.

    Production
    Test
    Dev

    Within the ‘Configuring Queues’ section you mention the REDIS_QUEUE variable to the .env file. Does the REDIS_QUEUE need to be different. Right now, my redis_queue is set to default. Do all three environments need to use a different queue…..so I would have production_default, test_default and dev_default. Doing so, I would need to specify a queue depending on the environment that a job is triggered from.

    If my job is dispatched from the dev environment, I would have to add ->onQueue( config(‘app.env’) . ‘_default’ );

    Reply
    1. Leonie Derendorp Post author

      Hi Marcellus,

      This REDIS_QUEUE variable refers to the default queue for a specific environment/application, so the queue to which jobs are dispatched when you don’t explicitly specify a queue. It’s not required to use different value for each environment, but it is advisable, because it better separates your different environments.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *