port from swaddle to honcho

[chadwhitacre]

There's a swaddle utility in Aspen/Gittip that duplicates some of the behavior of foreman. Let's try to use foreman, or maybe honcho.

[chadwhitacre]

Afaict, the one feature that swaddle has that foreman doesn't is to take environment definition from stdin (ddollar/foreman#308). This is used in payday.sh to take production configuration piped from heroku config. We could work around this with a file, but that's ugly. Using honcho would mean that we need to move this feature through both foreman and honcho.

Unfortunately, there's 21 open pull requests, and it sounds like foreman is rather neglected. :-/

[chadwhitacre]

Can we kludge this use case using env(1)?

[chadwhitacre]

Got a response from @ddollar, thankfully:

https://twitter.com/whit537/status/290646577352171520

[chadwhitacre]

One good reason to do this is to suppress dev/production mismatch bugs. I just hit one on DataDirect (YouGov project) where I have a similar setup. I made a change to the Makefile run target, but forgot to change the Procfile. It'd be better to just use the Procfile.

[joonas]

Since foreman supports the use environment files via -e, couldn't we already port over to it?

I'm not sure ddollar/foreman#308 is really a blocker in any way, because you could just redirect the output of heroku config to an env file that you then load via foreman start -e, or alternatively do:

  export `heroku config -s` | foreman [run|start] ...

(heroku config -s outputs the heroku config in shell format)

[chadwhitacre]

@joonas Does it really work to pipe export like that?

$ heroku config -s | grep ASPEN_LOGGING_THRESHOLD
ASPEN_LOGGING_THRESHOLD=1
$ export `heroku config -s` | echo $ASPEN_LOGGING_THRESHOLD

$

If so let's do it! This is the sort of solution I was grasping at above with the "kludge using env" comment.

[joonas]

Hmm, actually you're right, it doesn't work like that. I guess you'd still have to redirect the output to a file and then run the command loading that file.. I'll see if I can come up with something else :)

[chadwhitacre]

I'd prefer not to risk having production configuration in a local file, that's why I'm looking for a piping solution.

[joonas]

Hmm, alright, I'll see if I can figure something out in that case.

[joonas]

Ah yes, I'm not sure what I was thinking before...

So what you could do is this:

export `heroku config -s` && foreman [run|start] ...

Just to prove that it works, run for example:

unset DATABASE_URL && export `heroku config -s` && echo $DATABASE_URL

[chadwhitacre]

$ unset ASPEN_LOGGING_THRESHOLD && export `heroku config -s` && foreman run echo $ASPEN_LOGGING_THRESHOLD
1

!m @joonas

Let's proceed with this.

[chadwhitacre]

I've withdrawn ddollar/foreman#308.

[chadwhitacre]

Hrm ... the problem with this is that it puts sensitive configuration in your current shell environment, which is almost as bad as leaving it lying around in a file. Surely there's a way to do this properly using Unix tools.

[pjz]

mkfifo /tmp/foremanconfig 
heroku config -s >> /tmp/foremanconfig &
foreman run mycmd -e /tmp/foremanconfig

will work as long as foreman only reads the input file once.

[chadwhitacre]

@pjz That's a little better. You can control the permissions on /tmp/foremanconfig, at least. I wouldn't call it elegant or beautiful, though. I'm thinking of reviving ddollar/foreman#308.

[pjz]

foreman_stdincfg script is:

#!/bin/bash
foreman -e /dev/stdin "$@"

then call heroku config -s | foreman_stdincfg somecommand

[chadwhitacre]

Or just do:

heroku config -s | foreman run -e /dev/stdin foo.sh

!m @pjz. :)

[chadwhitacre]

Okay, full steam ahead!

[joonas]

👍

[chadwhitacre]

I expect that to maintain Windows portability we're going to want to go with Honcho.

[chadwhitacre]

Can we just use env for this somehow?

[pjz]

I don't see how. Is Windows support a necessity? If so, we could go back to
export heroku config -s && foreman run echo $FOO
and your worries (in ddollar/foreman#308 (comment) ) about exposing the environment can possibly be allayed by spawning another shell:
bash -c "export heroku config -s && foreman run echo $FOO"

[zbynekwinkler]

I have assigned this to myself. Take it as "intent to work" on this in the future. If anyone beats me to it, feel free. As we are moving to all python with #1769, I'd go with honcho.

[chadwhitacre]

@zwn The one caveat is that Heroku uses foreman (they wrote it), so we'll have a difference between dev and production there.

[chadwhitacre]

@zwn Also, make sure we that foreman honcho supports a -e option to take configuration from a file. We use that a lot.

[zbynekwinkler]

@whit537 Actually, why don't we run payday on heroku? Heroku has the ability to run 'workers' and it picks up the production configuration 'by itself'. We can use either one-off-dynos or regular workers (https://devcenter.heroku.com/articles/process-model, https://devcenter.heroku.com/articles/procfile).

[chadwhitacre]

Payday isn't the only place we use -e, we also use it for other one-off scripts (cf. #1771). I'm not seeing that running payday on Heroku would address a pain point for us unless it were part of automating payday further (#9). Though with the rumblings about moving off of Heroku, would we want to move that direction anyway?

Anyway, I checked, and honcho does support -e. Full steam ahead? :-)

[chadwhitacre]

I'm raising this to DevX ★. @D-Land and Alex (what's his GitHub?) hit this the other night at Open Hack, and it shouldn't actually be that much work to resolve.

[chadwhitacre]

Updated title and description to dictate honcho, to keep with a Python dev stack. Tradeoff: some variance from production, but a cleaner (Python) dev story.

[zbynekwinkler]

I don't see why it should be DevX ★. It works the way it is. We have a lots
of broken things right now. This is 'nice to have'.

[D-Land]

@alexvallejo

[chadwhitacre]

@zwn On Monday night @joeyespo and I were trying to onboard a couple new devs, and this was one of the issues we ran up against. The more important one was #1887.

That said, I take it that DevX ★ < ★★★.

[zbynekwinkler]

@zwn https://github.com/zwn On Monday night @joeyespohttps://github.com/joeyespoand I were trying to onboard a couple new devs, and this was one of the
issues we ran up against.

That much I figured from the previous comment. What I failed to understand
is what exactly was the issue - swaddle is bundled and make run and
make tests hides it from your view. So, what was the issue that it would
be fixed by bundling honcho instead of swaddle?

[chadwhitacre]

@zwn Both @D-Land and @alexvallejo are in a software testing class with @laboon. I was explaining to them how to run tests using py.test instead of make test, and found myself wishing I didn't have to explain swaddle and foreman. Admittedly, this was at the end of an exercise in frustration, trying to get Vagrant going with Postgres 9.3 (#1887).

I've dropped this back to DevX. :-)

[chadwhitacre]

... and have raised #1887 to DevX★.

[joeyespo]

@zwn It's a win for two reasons.

1. Honcho is cross-platform. Make has issues running on Windows, so this lowers the barrier for new hackers and brings us closer to a pure-Python (read: cross-platform) local development environment.
2. Honcho lets you re-use the same Procfile used in production. The Procfile becomes the project's unified entry point.

And like @whit537 said, testing with test.py makes sense. No need to explain anything.

[zbynekwinkler]

Done by #1974.