Configure Celery to run as Daemon service

by Krish V, posted in distributed-computing, Python, Ubuntu-Stuff

I am writing this post after consolidating my today’s deep search on How to configure celery to run as a Daemon?. I found many articles on the internet which are not enough and clear. After going through all the steps finally, I planned to write a blog which will have complete step-by-step procedure to make celery worker run as a Daemon.celery=as=daemon

1. Init-script:cleryd

/etc/init.d/celeryd:

In order to make the celery run at upstart we need to have executable file which has all the commands to handle worker to start. Just follow these steps,

(i) Create /etc/init.d/celeryd with the content from celery repo

(ii) Make celeryd executable with sudo chmod +x /etc/init.d/celeryd

sudo chmod 755 /etc/init.d/celeryd
sudo chown root:root /etc/init.d/celeryd
sudo chown -R root:root /var/log/celery/
sudo chown -R root:root /var/run/celery/

2. Configuration:

The configuration file should be places /etc/default, In your terminal just type in the following command,

sudo gedit /etc/default/celeryd

and save the following example configuration, or you could grab from the template here.

My example configuration:

CELERY_BIN="/home/user/.local/bin/celery"

# App instance to use
CELERY_APP="django-project"

# Where to chdir at start.
CELERYD_CHDIR="/home/user/django-project"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=4"

# %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>
# Workers should run as an unprivileged user.
#   You need to create this user manually (or you can choose
#   a user/group combination that already exists (e.g., nobody).
CELERYD_USER="user"
CELERYD_GROUP="user"

# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1

export SECRET_KEY="foobar"

For your case the CELERY_BIN may be varied, this can be checked with the following command on your terminal and replace the user as per your computer’s username.

which celery

Reference:

  1. https://pythad.github.io/articles/2016-12/how-to-run-celery-as-a-daemon-in-production

Leave a comment