Master can't currently be deployed

[seanlinsley]

I was eager to get #2307 onto production, but for some reason the deploy succeeded... but the app crashed. Literally all heroku logs would give me is "App Crashed". I rolled back to the previous deploy, which works fine. Here's the compare link between the two releases: 12.5.1...12.6.0

[seanlinsley]

My guess is that is has to do with #2313

[clone1018]

You'll probably need to add MANDRILL_KEY to the env, you should be able to set it to a blank value until we get a real one.

[chadwhitacre]

This is affecting payday as well, because payday.sh uses the heroku config env.

[chadwhitacre]

I've added MANDRILL_KEY= to the production environment.

[clone1018]

@whit537 Did that fix it?

[chadwhitacre]

@clone1018 Yes, that fixed payday. Didn't try redeploying tho.

[seanlinsley]

Redeployed with MANDRILL_KEY=

[seanlinsley]

Not sure if it's Heroku's fault or our own, but there really should have been a usable error message somewhere. Or is heroku logs not the right place to look?

[clone1018]

We do show a helpful message on dev, not sure about heroku though.

[chadwhitacre]

Yeah, we definitely show a good error message for missing envvars on startup. Perhaps this was simply drowned out? Do we need to raise an exception that will make it to Sentry?

[seanlinsley]

I'm certain that ./release.sh didn't show the error. It showed a ton of output from e.g. compiling libsass on production, but no error.

[chadwhitacre]

I wouldn't expect release.sh to show the error, because release.sh doesn't show the output of the aspen process, only the output of the slug-building process.

[chadwhitacre]

I meant maybe it was drowned out in heroku logs.

[seanlinsley]

heroku logs is a huge async fire hose. Is there really no better source?

[chadwhitacre]

@seanlinsley I've had good luck piping redirecting heroku logs to a logfile locally while provoking the behavior I'm looking for (in this case, with a heroku restart), and then grepping through the logfile.