2 Amazon AWS Elastic Beanstalk extensions you need to work with Sidekiq
Subscribe to my newsletter and never miss my upcoming articles
The first script ensures your Sidekiq instances would act properly on each application deployment:
commands:
create_post_dir:
command: "mkdir -p /opt/elasticbeanstalk/hooks/appdeploy/post"
ignoreErrors: true
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_sidekiq.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
. /opt/elasticbeanstalk/support/envvars
EB_APP_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
EB_APP_PID_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)
EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
. $EB_SUPPORT_DIR/envvars
. $EB_SCRIPT_DIR/use-app-ruby.sh
SIDEKIQ_PID=$EB_APP_PID_DIR/sidekiq.pid
SIDEKIQ_CONFIG=$EB_APP_DEPLOY_DIR/config/sidekiq.yml
SIDEKIQ_LOG=$EB_APP_DEPLOY_DIR/log/sidekiq.log
cd $EB_APP_DEPLOY_DIR
if [ -f $SIDEKIQ_PID ]
then
su -s /bin/bash -c "kill -TERM `cat $SIDEKIQ_PID`" $EB_APP_USER
su -s /bin/bash -c "rm -rf $SIDEKIQ_PID" $EB_APP_USER
fi
. /opt/elasticbeanstalk/support/envvars.d/sysenv
sleep 10
su -s /bin/bash -c "bundle exec sidekiq MALLOC_ARENA_MAX = 2 \
-e $RACK_ENV \
-P $SIDEKIQ_PID \
-C $SIDEKIQ_CONFIG \
-L $SIDEKIQ_LOG \
-d" $EB_APP_USER
"/opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_sidekiq.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
. /opt/elasticbeanstalk/support/envvars
EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
. $EB_SUPPORT_DIR/envvars
. $EB_SCRIPT_DIR/use-app-ruby.sh
SIDEKIQ_PID=$EB_APP_PID_DIR/sidekiq.pid
if [ -f $SIDEKIQ_PID ]
then
su -s /bin/bash -c "kill -USR1 `cat $SIDEKIQ_PID`" $EB_APP_USER
fi
The Second script is optional. You should apply it if your Sidekiq processes tend to memory bloating. It can be noticeable i.e. if your jobs process large files or initialize thousands of ActiveRecord objects each. In such situation your Sidekiq workers are constantly consuming more memory. In this configuration, monit script will restart them if its memory usage stays above 20%:
packages:
yum:
monit: []
files:
"/etc/monit.d/sidekiq":
mode: "000644"
owner: root
group: root
content: |
check process sidekiq
with pidfile /var/app/containerfiles/pids/sidekiq.pid every 2 cycles
if memory usage > 20% then exec "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_sidekiq.sh"
commands:
remove_bak:
command: "rm /etc/monit.d/sidekiq.bak"
ignoreErrors: true
services:
sysvinit:
monit:
ensureRunning: true
enabled: true
Software Engineer and Father. Love to learn, love to teach, love to build. CTO at recorrido.cl -> Ruby, Elixir, Vue / Stimulus
What happens if the process dies? you are not using a process manager like upstart, systemd or supervisord.
We have al these scripts to manage sidekiq properly.
commands:
02_create_sidekiq_log_file:
command: touch /var/log/sidekiq_worker.log
files:
"/opt/elasticbeanstalk/support/conf/sidekiq_worker.conf":
mode: "000755"
owner: root
group: root
content: |
description "Elastic Beanstalk Sidekiq Background Worker"
# Greatly reduce Ruby memory fragmentation and heap usage
# https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
env MALLOC_ARENA_MAX=2
respawn
respawn limit 3 30
# TERM is sent by sidekiqctl when stopping sidekiq. Without declaring these as
# normal exit codes, it just respawns.
normal exit 0 TERM
# Upstart waits 5 seconds by default to kill a process. Increase timeout to
# give sidekiq process enough time to exit.
kill timeout 30
script
exec /bin/bash <<"EOT"
EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
. $EB_SUPPORT_DIR/envvars
. $EB_SCRIPT_DIR/use-app-ruby.sh
EB_APP_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
cd $EB_APP_DEPLOY_DIR
echo "Bundle exec sidekiq worker"
exec bundle exec sidekiq -e ${RACK_ENV} \
-C ${EB_APP_DEPLOY_DIR}/config/sidekiq.yml >/var/log/sidekiq_worker.log 2>&1
EOT
end script
files:
"/etc/init/sidekiq_worker.conf":
mode: "120400"
content: "/opt/elasticbeanstalk/support/conf/sidekiq_worker.conf"
"/etc/init/workers.conf":
mode: "000755"
owner: root
group: root
content: |
description "Manages the Set of Sidekiq Processes"
# Use "sudo stop workers" to stop all Sidekiq instances.
# Use "sudo start workers" to start all instances.
# Use "sudo restart workers" to restart all instances.
# This starts upon bootup and stops on shutdown
start on runlevel [2345]
stop on runlevel [06]
pre-start script
sudo start sidekiq_worker
end script
post-stop script
sudo stop sidekiq_worker
end script
commands:
reload_initctl_for_sidekiq:
command: "sudo initctl reload-configuration"
files:
"/tmp/sidekiq_mute.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
skwpid=$(initctl status sidekiq_worker | grep /running | awk '{print $NF}')
if [ -f skwpid ]
then
echo "TSTP/quieting sidekiq worker"
sudo kill -TSTP skwpid
fi
files:
"/tmp/sidekiq_restart.sh":
mode: "000755"
owner: root
group: root
content: |
#!/bin/bash
skwpid=$(initctl status sidekiq_worker | grep /running | awk '{print $NF}')
if [ -f skwpid ]
then
echo "TERM/terminating sidekiq worker"
sudo kill -TERM skwpid
fi
echo "STOPPING all sidekiq workers"
sudo stop workers
echo "STARTING all sidekiq workers"
sudo start workers
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_sidekiq.sh":
mode: "120755"
owner: root
group: root
content: "/tmp/sidekiq_restart.sh"
"/opt/elasticbeanstalk/hooks/configdeploy/post/50_restart_sidekiq.sh":
mode: "120755"
owner: root
group: root
content: "/tmp/sidekiq_restart.sh"
"/opt/elasticbeanstalk/hooks/restartappserver/post/50_restart_sidekiq.sh":
mode: "120755"
owner: root
group: root
content: "/tmp/sidekiq_restart.sh"
"/opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_sidekiq.sh":
mode: "120755"
owner: root
group: root
content: "/tmp/sidekiq_mute.sh"
"/opt/elasticbeanstalk/hooks/configdeploy/pre/03_mute_sidekiq.sh":
mode: "120755"
owner: root
group: root
content: "/tmp/sidekiq_mute.sh"
"/opt/elasticbeanstalk/hooks/restartappserver/pre/03_mute_sidekiq.sh":
mode: "120755"
owner: root
group: root
content: "/tmp/sidekiq_mute.sh"
Hi Benjamin. Since I've been using monit to control memory consumption, I have no longer problems with system killing Sidekiq processes. However, thanks a lot for sharing your scripts. They can be very useful.
Piotr Jurewicz Yes that's a good way to tackle down "most" of possible sources that can cause a process to die. But of course there might be other. For instance, if you have your rails app in the same instance, it can grow in ram too, so even having sidekiq with low memory usage, the system can enter a "resource starvation" state and the Kernel would kill proceses to safeguard the system state. The way the Kernel chooses this proceses is based in many factors, so there's still chance your sidekiq dies.
Comments (3)