Tech Radar 3 #612

gratipay-bot · 2016-05-08T09:00:05Z

← Tech Radar 2

Docs

http://inside.gratipay.com/howto/sweep-the-radar

Mission

The mission of the tech team is to support the product team in solving problems in the market.

Scope

dev
ops
PagerDuty
UptimeRobot
Heroku
Digital Ocean
AWS
Payday
Librato
https://github.com/gratipay/postgres.py
https://github.com/gratipay/gip.rocks
https://github.com/gratipay/honcho
https://github.com/gratipay/environment.py

Roadmap

stop using immutable ids as primary key (use id as primary key instead of username gratipay.com#835)
other embarrassments

chadwhitacre · 2016-05-10T10:23:07Z

[gratipay] $ git branch --merged
* master
  symmetric-encryption
[gratipay] $

😳

Deploying gratipay/gratipay.com#3998!

chadwhitacre · 2016-05-10T10:25:54Z

[gratipay] $ ./bin/keygen.py | pbcopy

I set CRYPTO_KEYS in the Heroku UI to avoid trails in my bash history.

chadwhitacre · 2016-05-10T10:39:25Z

Hmmm ... build error? And it didn't halt deployment? :-/

remote:        copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi
remote:        
remote:        copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi
remote:        
remote:        copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi
remote:        
remote:        copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi
remote:        
remote:        running build_ext
remote:        
remote:        building '_cffi_backend' extension
remote:        
remote:        creating build/temp.linux-x86_64-2.7
remote:        
remote:        creating build/temp.linux-x86_64-2.7/c
remote:        
remote:        gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/app/.heroku/python/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
remote:        
remote:        c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
remote:        
remote:         #include <ffi.h>
remote:        
remote:                         ^
remote:        
remote:        compilation terminated.
remote:        
remote:        error: command 'gcc' failed with exit status 1
remote:        
remote:        ----------------------------------------
remote:        Cleaning up...
remote:        Command /app/.heroku/python/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-FCRh4o-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-inx0Dz-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip-FCRh4o-build
remote:        Storing debug log for failure in /app/.pip/pip.log
remote: 
remote: 
remote: -----> Discovering process types
remote:        Procfile declares types -> web
remote: 
remote: -----> Compressing...
remote:        Done: 62.2M
remote: -----> Launching...
remote:        Released v1592
remote:        https://gratipay.herokuapp.com/ deployed to Heroku
remote: 
remote: Verifying deploy.... done.
To [email protected]:gratipay.git
   279b7ca..3da5ce0  master -> master
Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 350 bytes | 0 bytes/s, done.
Total 4 (delta 2), reused 0 (delta 0)
To [email protected]:gratipay/gratipay.com.git
   1bb41d3..3da5ce0  master -> master
Total 0 (delta 0), reused 0 (delta 0)
To [email protected]:gratipay/gratipay.com.git
 * [new tag]         1953 -> 1953
[gratipay] $

chadwhitacre · 2016-05-10T10:39:33Z

Blech, and we're 500 ... :-/

Rolling back ...

chadwhitacre · 2016-05-10T10:41:40Z

Rolled back. Hmm ...

chadwhitacre · 2016-05-10T10:42:19Z

Possibly relevant: heroku/heroku-buildpack-python#231.

chadwhitacre · 2016-05-10T10:49:16Z

Looks like the Python buildpack makes libffi available when certain packages are named in requirements.txt. I believe we're pegged to an old buildpack, though ...

chadwhitacre · 2016-05-10T10:52:04Z

Here's the failure I captured in the heroku logs:

2016-05-10T10:40:29.538670+00:00 heroku[web.1]: Starting process with command `gunicorn gratipay.main:website --bind :14054 --workers=1`
2016-05-10T10:40:32.161043+00:00 app[web.1]: 2016-05-10 10:40:32 [3] [INFO] Starting gunicorn 18.0
2016-05-10T10:40:32.161718+00:00 app[web.1]: 2016-05-10 10:40:32 [3] [INFO] Listening at: http://0.0.0.0:14054 (3)
2016-05-10T10:40:32.161842+00:00 app[web.1]: 2016-05-10 10:40:32 [3] [INFO] Using worker: sync
2016-05-10T10:40:32.173027+00:00 app[web.1]: 2016-05-10 10:40:32 [8] [INFO] Booting worker with pid: 8
2016-05-10T10:40:33.288964+00:00 heroku[web.1]: State changed from starting to up
2016-05-10T10:40:33.239856+00:00 app[web.1]: 2016-05-10 10:40:33 [8] [ERROR] Exception in worker process:
2016-05-10T10:40:33.239865+00:00 app[web.1]: Traceback (most recent call last):
2016-05-10T10:40:33.239866+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker
2016-05-10T10:40:33.239867+00:00 app[web.1]:     worker.init_process()
2016-05-10T10:40:33.239867+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process
2016-05-10T10:40:33.239868+00:00 app[web.1]:     self.wsgi = self.app.wsgi()
2016-05-10T10:40:33.239869+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi
2016-05-10T10:40:33.239869+00:00 app[web.1]:     self.callable = self.load()
2016-05-10T10:40:33.239870+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load
2016-05-10T10:40:33.239870+00:00 app[web.1]:     return self.load_wsgiapp()
2016-05-10T10:40:33.239871+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp
2016-05-10T10:40:33.239872+00:00 app[web.1]:     return util.import_app(self.app_uri)
2016-05-10T10:40:33.239872+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app
2016-05-10T10:40:33.239873+00:00 app[web.1]:     __import__(module)
2016-05-10T10:40:33.239873+00:00 app[web.1]:   File "/app/gratipay/main.py", line 6, in <module>
2016-05-10T10:40:33.239874+00:00 app[web.1]:     import gratipay.wireup
2016-05-10T10:40:33.239875+00:00 app[web.1]:   File "/app/gratipay/wireup.py", line 32, in <module>
2016-05-10T10:40:33.239875+00:00 app[web.1]:     from gratipay.models.account_elsewhere import AccountElsewhere
2016-05-10T10:40:33.239876+00:00 app[web.1]:   File "/app/gratipay/models/account_elsewhere.py", line 17, in <module>
2016-05-10T10:40:33.239877+00:00 app[web.1]:     from gratipay.security.crypto import constant_time_compare
2016-05-10T10:40:33.239877+00:00 app[web.1]:   File "/app/gratipay/security/crypto.py", line 9, in <module>
2016-05-10T10:40:33.239878+00:00 app[web.1]:     from cryptography.fernet import Fernet, MultiFernet
2016-05-10T10:40:33.239878+00:00 app[web.1]: ImportError: No module named cryptography.fernet
2016-05-10T10:40:33.239879+00:00 app[web.1]: Traceback (most recent call last):
2016-05-10T10:40:33.239879+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker
2016-05-10T10:40:33.239880+00:00 app[web.1]:     worker.init_process()
2016-05-10T10:40:33.239881+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process
2016-05-10T10:40:33.239881+00:00 app[web.1]:     self.wsgi = self.app.wsgi()
2016-05-10T10:40:33.239882+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi
2016-05-10T10:40:33.239882+00:00 app[web.1]:     self.callable = self.load()
2016-05-10T10:40:33.239883+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load
2016-05-10T10:40:33.239883+00:00 app[web.1]:     return self.load_wsgiapp()
2016-05-10T10:40:33.239884+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp
2016-05-10T10:40:33.239884+00:00 app[web.1]:     return util.import_app(self.app_uri)
2016-05-10T10:40:33.239885+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app
2016-05-10T10:40:33.239885+00:00 app[web.1]:     __import__(module)
2016-05-10T10:40:33.239886+00:00 app[web.1]:   File "/app/gratipay/main.py", line 6, in <module>
2016-05-10T10:40:33.239886+00:00 app[web.1]:     import gratipay.wireup
2016-05-10T10:40:33.239887+00:00 app[web.1]:   File "/app/gratipay/wireup.py", line 32, in <module>
2016-05-10T10:40:33.239888+00:00 app[web.1]:     from gratipay.models.account_elsewhere import AccountElsewhere
2016-05-10T10:40:33.239888+00:00 app[web.1]:   File "/app/gratipay/models/account_elsewhere.py", line 17, in <module>
2016-05-10T10:40:33.239889+00:00 app[web.1]:     from gratipay.security.crypto import constant_time_compare
2016-05-10T10:40:33.239890+00:00 app[web.1]:   File "/app/gratipay/security/crypto.py", line 9, in <module>
2016-05-10T10:40:33.239891+00:00 app[web.1]:     from cryptography.fernet import Fernet, MultiFernet
2016-05-10T10:40:33.239891+00:00 app[web.1]: ImportError: No module named cryptography.fernet
2016-05-10T10:40:33.240346+00:00 app[web.1]: 2016-05-10 10:40:33 [8] [INFO] Worker exiting (pid: 8)
2016-05-10T10:40:33.383017+00:00 app[web.1]: 2016-05-10 10:40:33 [3] [INFO] Shutting down: Master
2016-05-10T10:40:33.383423+00:00 app[web.1]: 2016-05-10 10:40:33 [3] [INFO] Reason: Worker failed to boot.
2016-05-10T10:40:34.238289+00:00 heroku[web.1]: State changed from up to crashed
2016-05-10T10:40:34.220789+00:00 heroku[web.1]: Process exited with status 3

chadwhitacre · 2016-05-10T10:53:52Z

I believe we're pegged to an old buildpack, though ...

Yup:

[gratipay] $ heroku config -s | grep BUILDPACK
BUILDPACK_URL='https://github.com/heroku/heroku-buildpack-python.git#v54'

chadwhitacre · 2016-05-10T10:58:37Z

So I guess the thing to try is upgrading the buildpack. However, it'd be nice to know why the failure failed to abort the deployment.

chadwhitacre · 2016-05-10T10:59:27Z

Here's the archived build log.

chadwhitacre · 2016-05-10T11:03:04Z

it'd be nice to know why the failure failed to abort the deployment.

Seems likely to be related to using an old buildpack as well.

[gratipay] $ heroku config:set BUILDPACK_URL="https://github.com/heroku/heroku-buildpack-python.git#v80"
Setting BUILDPACK_URL and restarting ⬢ gratipay... done, v1594
BUILDPACK_URL: https://github.com/heroku/heroku-buildpack-python.git#v80
[gratipay] $

chadwhitacre · 2016-05-10T11:08:00Z

Alright, the build failed ... and succeeded to fail. :-)

remote:            copying cffi/model.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/ffiplatform.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/gc_weakref.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/recompiler.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/__init__.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/api.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/cparser.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/vengine_gen.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/commontypes.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi
remote:            running build_ext
remote:            building '_cffi_backend' extension
remote:            creating build/temp.linux-x86_64-2.7
remote:            creating build/temp.linux-x86_64-2.7/c
remote:            gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/app/.heroku/python/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
remote:            c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
remote:             #include <ffi.h>
remote:                             ^
remote:            compilation terminated.
remote:            error: command 'gcc' failed with exit status 1
remote:            
remote:            ----------------------------------------
remote:        Command "/app/.heroku/python/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-9VEW36-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-xJda06-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-9VEW36-build/
remote: 
remote:  !     Push rejected, failed to compile Python app
remote: 
remote: Verifying deploy...
remote: 
remote: !       Push rejected to gratipay.
remote: 
To [email protected]:gratipay.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:gratipay.git'
[gratipay] $

What gives?

chadwhitacre · 2016-05-10T11:15:32Z

Hypothesis: the whitespace prefixing cffi is causing the buildpack not to see it.

chadwhitacre · 2016-05-10T11:21:04Z

Oooh, we have ./vendor/ in there, too. Here's pip-grep.

chadwhitacre · 2016-05-10T11:35:03Z

Hmm ... indeed, I don't see "Noticed cffi. Bootstrapping libffi." in the build log. I've cloned the buildpack locally to see about pip-grep. Correcting for https://github.com/whit537/heroku-buildpack-python/pull/2, pip-grep indeed does not find cffi in our requirements file.

$ ../../heroku/heroku-buildpack-python/vendor/pip-pop/pip-grep requirements.txt cffi
Not found.
$

I'm not sure it finds anything:

$ ../../heroku/heroku-buildpack-python/vendor/pip-pop/pip-grep requirements.txt cryptography
Not found.
$

chadwhitacre · 2016-05-10T11:35:44Z

Yeah, it's the local file paths that are confusing it. If I add cffi on a line by itself ...

$ ../../heroku/heroku-buildpack-python/vendor/pip-pop/pip-grep requirements.txt cffi
Package cffi found!
$

chadwhitacre · 2016-05-10T11:36:29Z

So the fix is to teach pip-grep about local file paths, and the workaround is to manually do the libffi bootstrapping.

chadwhitacre · 2016-05-10T11:51:09Z

Alright, fix reticketed as heroku/heroku-buildpack-python#299. Now for the workaround ...

chadwhitacre · 2016-05-10T11:55:03Z

the workaround is to manually do the libffi boostrapping.

... or maybe to include cffi at the end of requirements.txt?

chadwhitacre · 2016-05-10T11:57:33Z

Harumph. Same result with just cffi. 😕

Processing ./vendor/jmespath.py-0.9.0.tar.gz
Processing ./vendor/botocore-1.4.13.tar.gz
Processing ./vendor/boto3-1.3.0.tar.gz
Processing ./vendor/pycparser-2.14.tar.gz
Processing ./vendor/cffi-1.6.0.tar.gz
Processing ./vendor/enum34-1.1.4.tar.gz
Processing ./vendor/idna-2.1.tar.gz
Processing ./vendor/ipaddress-1.0.16.tar.gz
Processing ./vendor/cryptography-1.3.2.tar.gz
Collecting cffi==1.6.0 (from -r requirements.txt (line 67))
  Could not find a version that satisfies the requirement cffi==1.6.0 (from -r requirements.txt (line 67)) (from versions: )
No matching distribution found for cffi==1.6.0 (from -r requirements.txt (line 67))
make: *** [env] Error 1
$

chadwhitacre · 2016-05-10T11:58:51Z

Gah. Do we have to fork the buildpack? 😞

chadwhitacre · 2016-05-10T12:23:07Z

Forked to https://github.com/gratipay/buildpack-python, with a workaround on the gratipay-prod branch.

$ heroku config:set BUILDPACK_URL="https://github.com/gratipay/buildpack-python.git#gratipay-prod"
Setting BUILDPACK_URL and restarting ⬢ gratipay... done, v1595
BUILDPACK_URL: https://github.com/gratipay/buildpack-python.git#gratipay-prod
$

chadwhitacre · 2016-05-10T12:26:48Z

remote: -----> Noticed cffi. Bootstrapping libffi.

👍

chadwhitacre · 2016-05-10T12:27:55Z

Blam. Okay!

chadwhitacre · 2016-05-10T12:29:22Z

Aaaaaand we have crypto! 💃 😳 🎏

chadwhitacre · 2016-05-11T21:15:43Z

[gratipay] $ diff -rub working.log borked.log 
--- working.log 2016-05-11 17:14:46.000000000 -0400
+++ borked.log  2016-05-11 17:15:10.000000000 -0400
@@ -55,5 +55,5 @@
 -----> Compressing...
        Done: 66.2M
 -----> Launching...
-       Released v1597
+       Released v1598
        https://gratipay.herokuapp.com/ deployed to Heroku
[gratipay] $