Example app for deploying Sinatra to AWS?

[suranyami]

I've been going around in circles with the AWS documentation for an EB Ruby web server (Sinatra) + SQS worker.

The shoryuken docs seem to imply that if you have a Rails app, just add config, now you have workers.

Yet, with my Sinatra app, I have to launch a separate process. I've tested that shoryuken runs and works locally and by running eb ssh and connecting to my EB box:

cd /var/app/current
bundle exec shoryuken -r ./config/shoryuken_init.rb -C config/shoryuken.yml

All works fine. But when I tried to configure the workers to be launched by a .ebextensions/xxx.config file, I get grief with a AWSEBAutoscalingGroup error.

• Should I just fork a child process in my config.ru?
• Do I need a separate Worker Tier application created just to run the Workers? If so, how does the webapp launch the workers?

Does anyone have a canonical app/config for deploying a PORO/Sinatra webapp to AWS/SQS?

I'm happy to provide a minimal example if none to date.

[phstc]

Hey @suranyami

I run shoryuken in a standalone application (no Rails) with foreman:

Procfile

shoryuken: bundle exec shoryuken -C config/shoryuken.yml -r ./lib/shoryuken_init.rb

But I don't use it on AWS EB.

I found something for sidekiq: http://blog.noizeramp.com/2013/04/21/using-sidekiq-with-elastic-beanstalk/ shoryuken is based on sidekiq, so the same steps should work with shoryuken.

[suranyami]

Thanks. It's AWS EB in particular that's I'm struggling with.

My current workaround is to add this to config.ru:

worker_cmd = 'bundle exec shoryuken -r ./config/shoryuken_init.rb -C config/shoryuken.yml'
pid = Process.spawn worker_cmd

It works, but my only concern is it's not being monitored by EB, so if the worker crashes it won't be automatically restarted. It'll do for the moment, but when I go to production, it's gotta be more robust than that.

I'll take a look at the Sidekiq/EB article. Thanks!

[phstc]

If you create a recipe for shoryuken on EB, I would love to add to the project's wiki. 😄

I'll take a look at the Sidekiq/EB article. Thanks!

Anything that works for Sidekiq, should work for Shoryuken

[CyborgMaster]

@phstc I don't mean to re-open an old issue, but I just went through this process (Getting shoryuken running on an Elastic Beanstalk Rails server), and I have a config file that you could add to the wiki.

It's a very minor adaptation of the Sidekiq config from the blog post you linked to.

# Based on http://blog.noizeramp.com/2013/04/21/using-sidekiq-with-elastic-beanstalk/.
# If we want to try to improve it, we can attempt to add some of the techniques
# included in https://gist.github.com/gcarrion-gfrmedia/11396682.

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_shoryuken":
    mode: "000777"
    content: |
      cd /var/app/current

      if [ -f /var/app/support/pids/shoryuken.pid ]
      then
        kill -TERM `cat /var/app/support/pids/shoryuken.pid`
        rm -rf /var/app/support/pids/shoryuken.pid
      fi

      . /opt/elasticbeanstalk/support/envvars

      sleep 10

      bundle exec shoryuken \
        -R \
        -P /var/app/support/pids/shoryuken.pid \
        -C /var/app/current/config/shoryuken.yml \
        -L /var/app/support/logs/shoryuken.log \
        -d

  "/opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_shoryuken":
    mode: "000777"
    content: |
      if [ -f /var/app/support/pids/shoryuken.pid ]
      then
        kill -USR1 `cat /var/app/support/pids/shoryuken.pid`
      fi

[jamilbk]

Thanks @CyborgMaster !

For anyone else coming across this script, I was getting the error bundler: command not found: shoryuken. The app's ruby env wasn't loading. Was also getting an error about missing directories. Changing to this seemed to fix it:

# Based on http://blog.noizeramp.com/2013/04/21/using-sidekiq-with-elastic-beanstalk/.
# If we want to try to improve it, we can attempt to add some of the techniques
# included in https://gist.github.com/gcarrion-gfrmedia/11396682.

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_shoryuken":
    mode: "000777"
    content: |
      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir) 
      . $EB_SUPPORT_DIR/envvars
      . $EB_SCRIPT_DIR/use-app-ruby.sh

      mkdir -p /var/app/support/pids
      mkdir -p /var/app/support/logs

      cd /var/app/current

      if [ -f /var/app/support/pids/shoryuken.pid ]
      then
        kill -TERM `cat /var/app/support/pids/shoryuken.pid`
        rm -rf /var/app/support/pids/shoryuken.pid
      fi

      sleep 10

      bundle exec shoryuken \
        -R \
        -P /var/app/support/pids/shoryuken.pid \
        -C /var/app/current/config/shoryuken.yml \
        -L /var/app/support/logs/shoryuken.log \
        -d

  "/opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_shoryuken":
    mode: "000777"
    content: |
      if [ -f /var/app/support/pids/shoryuken.pid ]
      then
        kill -USR1 `cat /var/app/support/pids/shoryuken.pid`
      fi

[erwinkarim]

this is exactly what i'm looking for. should be in the documentation regrading deployment to AWS elastic beanstalk...

[phstc]

@erwinkarim good call. I added a link to this issue here.

[MatayoshiMariano]

Hi guys, I have realized that for shoryuken to work in Elastic Beantalk Worker Tier, I had to stop the aws-sqsd service, do you have to do it too?

The commando: sudo service aws-sqsd stop

[januszm]

@MatayoshiMariano see this thread, especially the first message from @farski #169

and this article about ElasticBeanstalk Worker Environments http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html

Looks like Shoryuken and the EB Worker Environment SQS Daemon do the same job, they poll SQS queue and trigger jobs (aws-sqsd by making POST requests to defined endpoints in your app). So it seems that it makes sense to disable the aws sqs daemon in your case. But it actually makes more sense to not use Shoryuken in EB Worker Env

[MatayoshiMariano]

@januszm yes, the best solution is not to use Shoryuken service, but I am using Shoryuken to send the message to SQS. I am developing the option to use Shoryuken with EB WorkerTier

[januszm]

But stopping that daemon defeats the purpose of using EB Workers right? Perhaps you should send messages to SQS from your Web Environment?

[MatayoshiMariano]

Yes, now I am using the SQS daemon in the EB Worker tier, but I use Shoryuken to send the message to the SQS from the Web Tier. The only problem to use Shoryuken is that it send the class name in a message attribute but the Amazon SQS daemon does not retrieve any message attribute so I have to make a little tweak in the gem.

Here is the commit in my fork.

[bettysteger]

I was just redeploying my app with the .ebextensions config above, had anyone had this error?:

Return code: 1 Output: /opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_shoryuken: line 3: kill: (14036) - No such process. 

[reneweteling]

Dude you saved my day, was looking for this yesterday for multple hours! Thanks!!

[tomharvey]

FYI - I've updated the config file to use AWS environment variables to dictate the location of the log, pid and working directories. Working well on Puma with Ruby 2.5 running on 64bit Amazon Linux/2.9.1

https://gist.github.com/tomharvey/a248800e492305bbf5ba3d56fa1e0513

# .ebextensions/shoryuken.config
# Based on the conversation in https://github.com/phstc/shoryuken/issues/48

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_shoryuken":
    mode: "000777"
    content: |
      APP_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
      LOG_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_log_dir)
      PID_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)

      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir) 
      . $EB_SUPPORT_DIR/envvars
      . $EB_SCRIPT_DIR/use-app-ruby.sh

      cd $APP_DEPLOY_DIR

      if [ -f $PID_DIR/shoryuken.pid ]
      then
        kill -TERM `cat $PID_DIR/shoryuken.pid`
        rm -rf $PID_DIR/shoryuken.pid
      fi

      sleep 10

      bundle exec shoryuken \
        -R \
        -P $PID_DIR/shoryuken.pid \
        -C $APP_DEPLOY_DIR/config/shoryuken.yml \
        -L $LOG_DIR/shoryuken.log \
        -d

  "/opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_shoryuken":
    mode: "000777"
    content: |
      PID_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)
      if [ -f $PID_DIR/shoryuken.pid ]
      then
        kill -USR1 `cat $PID_DIR/shoryuken.pid`
      fi