2 Amazon AWS Elastic Beanstalk extensions you need to work with Sidekiq

Subscribe to my newsletter and never miss my upcoming articles

The first script ensures your Sidekiq instances would act properly on each application deployment:

commands:
  create_post_dir:
    command: "mkdir -p /opt/elasticbeanstalk/hooks/appdeploy/post"
    ignoreErrors: true
files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_sidekiq.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      . /opt/elasticbeanstalk/support/envvars

      EB_APP_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
      EB_APP_PID_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)
      EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)

      . $EB_SUPPORT_DIR/envvars
      . $EB_SCRIPT_DIR/use-app-ruby.sh

      SIDEKIQ_PID=$EB_APP_PID_DIR/sidekiq.pid
      SIDEKIQ_CONFIG=$EB_APP_DEPLOY_DIR/config/sidekiq.yml
      SIDEKIQ_LOG=$EB_APP_DEPLOY_DIR/log/sidekiq.log

      cd $EB_APP_DEPLOY_DIR

      if [ -f $SIDEKIQ_PID ]
      then
        su -s /bin/bash -c "kill -TERM `cat $SIDEKIQ_PID`" $EB_APP_USER
        su -s /bin/bash -c "rm -rf $SIDEKIQ_PID" $EB_APP_USER
      fi

      . /opt/elasticbeanstalk/support/envvars.d/sysenv

      sleep 10

      su -s /bin/bash -c "bundle exec sidekiq MALLOC_ARENA_MAX = 2 \
        -e $RACK_ENV \
        -P $SIDEKIQ_PID \
        -C $SIDEKIQ_CONFIG \
        -L $SIDEKIQ_LOG \
        -d" $EB_APP_USER

  "/opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_sidekiq.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      . /opt/elasticbeanstalk/support/envvars

      EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)

      . $EB_SUPPORT_DIR/envvars
      . $EB_SCRIPT_DIR/use-app-ruby.sh

      SIDEKIQ_PID=$EB_APP_PID_DIR/sidekiq.pid
      if [ -f $SIDEKIQ_PID ]
      then
        su -s /bin/bash -c "kill -USR1 `cat $SIDEKIQ_PID`" $EB_APP_USER
      fi

The Second script is optional. You should apply it if your Sidekiq processes tend to memory bloating. It can be noticeable i.e. if your jobs process large files or initialize thousands of ActiveRecord objects each. In such situation your Sidekiq workers are constantly consuming more memory. In this configuration, monit script will restart them if its memory usage stays above 20%:

packages:
  yum:
    monit: []
files:
  "/etc/monit.d/sidekiq":
    mode: "000644"
    owner: root
    group: root
    content: |
      check process sidekiq
        with pidfile /var/app/containerfiles/pids/sidekiq.pid every 2 cycles
        if memory usage > 20% then exec "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_sidekiq.sh"
commands:
  remove_bak:
    command: "rm /etc/monit.d/sidekiq.bak"
    ignoreErrors: true
services:
  sysvinit:
    monit:
      ensureRunning: true
      enabled: true

Comments (3)

Benjamin Silva Herrera's photo

What happens if the process dies? you are not using a process manager like upstart, systemd or supervisord.

We have al these scripts to manage sidekiq properly.

image.png

commands:
  02_create_sidekiq_log_file:
    command: touch /var/log/sidekiq_worker.log
files:
  "/opt/elasticbeanstalk/support/conf/sidekiq_worker.conf":
    mode: "000755"
    owner: root
    group: root
    content: |
      description "Elastic Beanstalk Sidekiq Background Worker"

      # Greatly reduce Ruby memory fragmentation and heap usage
      # https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
      env MALLOC_ARENA_MAX=2

      respawn
      respawn limit 3 30

      # TERM is sent by sidekiqctl when stopping sidekiq. Without declaring these as
      # normal exit codes, it just respawns.
      normal exit 0 TERM

      # Upstart waits 5 seconds by default to kill a process. Increase timeout to
      # give sidekiq process enough time to exit.
      kill timeout 30

      script
      exec /bin/bash <<"EOT"
        EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
        EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)

        . $EB_SUPPORT_DIR/envvars
        . $EB_SCRIPT_DIR/use-app-ruby.sh

        EB_APP_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)

        cd $EB_APP_DEPLOY_DIR
        echo "Bundle exec sidekiq worker"
        exec bundle exec sidekiq -e ${RACK_ENV} \
          -C ${EB_APP_DEPLOY_DIR}/config/sidekiq.yml >/var/log/sidekiq_worker.log 2>&1
      EOT
      end script
files:
  "/etc/init/sidekiq_worker.conf":
    mode: "120400"
    content: "/opt/elasticbeanstalk/support/conf/sidekiq_worker.conf"
  "/etc/init/workers.conf":
    mode: "000755"
    owner: root
    group: root
    content: |
      description "Manages the Set of Sidekiq Processes"

      # Use "sudo stop workers" to stop all Sidekiq instances.
      # Use "sudo start workers" to start all instances.
      # Use "sudo restart workers" to restart all instances.

      # This starts upon bootup and stops on shutdown
      start on runlevel [2345]
      stop on runlevel [06]

      pre-start script
        sudo start sidekiq_worker
      end script

      post-stop script
        sudo stop sidekiq_worker
      end script

commands:
  reload_initctl_for_sidekiq:
    command: "sudo initctl reload-configuration"
files:
  "/tmp/sidekiq_mute.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/bin/bash
      skwpid=$(initctl status sidekiq_worker | grep /running | awk '{print $NF}')
      if [ -f skwpid ]
      then
        echo "TSTP/quieting sidekiq worker"
        sudo kill -TSTP skwpid
      fi
files:
  "/tmp/sidekiq_restart.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/bin/bash
      skwpid=$(initctl status sidekiq_worker | grep /running | awk '{print $NF}')
      if [ -f skwpid ]
      then
        echo "TERM/terminating sidekiq worker"
        sudo kill -TERM skwpid
      fi

      echo "STOPPING all sidekiq workers"
      sudo stop workers

      echo "STARTING all sidekiq workers"
      sudo start workers
files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_sidekiq.sh":
    mode: "120755"
    owner: root
    group: root
    content: "/tmp/sidekiq_restart.sh"

  "/opt/elasticbeanstalk/hooks/configdeploy/post/50_restart_sidekiq.sh":
    mode: "120755"
    owner: root
    group: root
    content: "/tmp/sidekiq_restart.sh"

  "/opt/elasticbeanstalk/hooks/restartappserver/post/50_restart_sidekiq.sh":
    mode: "120755"
    owner: root
    group: root
    content: "/tmp/sidekiq_restart.sh"

  "/opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_sidekiq.sh":
    mode: "120755"
    owner: root
    group: root
    content: "/tmp/sidekiq_mute.sh"

  "/opt/elasticbeanstalk/hooks/configdeploy/pre/03_mute_sidekiq.sh":
    mode: "120755"
    owner: root
    group: root
    content: "/tmp/sidekiq_mute.sh"

  "/opt/elasticbeanstalk/hooks/restartappserver/pre/03_mute_sidekiq.sh":
    mode: "120755"
    owner: root
    group: root
    content: "/tmp/sidekiq_mute.sh"
Piotr Jurewicz's photo

Hi Benjamin. Since I've been using monit to control memory consumption, I have no longer problems with system killing Sidekiq processes. However, thanks a lot for sharing your scripts. They can be very useful.

Benjamin Silva Herrera's photo

Piotr Jurewicz Yes that's a good way to tackle down "most" of possible sources that can cause a process to die. But of course there might be other. For instance, if you have your rails app in the same instance, it can grow in ram too, so even having sidekiq with low memory usage, the system can enter a "resource starvation" state and the Kernel would kill proceses to safeguard the system state. The way the Kernel chooses this proceses is based in many factors, so there's still chance your sidekiq dies.