Cannot decode 502

[shipperizer]

Looks like Locust doesn't handle properly 502

[2016-09-21 15:30:12,835] arbalester/ERROR/stderr: Traceback (most recent call last):
[2016-09-21 15:30:12,835] arbalester/ERROR/stderr: File "/home/shipperizer/code/locustio/env/lib/python2.7/site-packages/gevent/pywsgi.py", line 884, in handle_one_response
[2016-09-21 15:30:12,836] arbalester/ERROR/stderr: self.run_application()
[2016-09-21 15:30:12,836] arbalester/ERROR/stderr: File "/home/shipperizer/code/locustio/env/lib/python2.7/site-packages/gevent/pywsgi.py", line 870, in run_application
[2016-09-21 15:30:12,836] arbalester/ERROR/stderr: self.result = self.application(self.environ, self.start_response)
[2016-09-21 15:30:12,836] arbalester/ERROR/stderr: File "/home/shipperizer/code/locustio/env/lib/python2.7/site-packages/flask/app.py", line 2000, in __call__
[2016-09-21 15:30:12,836] arbalester/ERROR/stderr: return self.wsgi_app(environ, start_response)
[2016-09-21 15:30:12,836] arbalester/ERROR/stderr: File "/home/shipperizer/code/locustio/env/lib/python2.7/site-packages/flask/app.py", line 1991, in wsgi_app
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: response = self.make_response(self.handle_exception(e))
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: File "/home/shipperizer/code/locustio/env/lib/python2.7/site-packages/flask/app.py", line 1567, in handle_exception
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: reraise(exc_type, exc_value, tb)
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: File "/home/shipperizer/code/locustio/env/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: response = self.full_dispatch_request()
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: File "/home/shipperizer/code/locustio/env/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: rv = self.handle_user_exception(e)
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: File "/home/shipperizer/code/locustio/env/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: reraise(exc_type, exc_value, tb)
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: File "/home/shipperizer/code/locustio/env/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: rv = self.dispatch_request()
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: File "/home/shipperizer/code/locustio/env/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: return self.view_functions[rule.endpoint](**req.view_args)
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: File "/home/shipperizer/code/locustio/env/lib/python2.7/site-packages/locust/cache.py", line 16, in wrapper
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: cache["result"] = func(*args, **kwargs)
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: File "/home/shipperizer/code/locustio/env/lib/python2.7/site-packages/locust/web.py", line 182, in request_stats
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: return json.dumps(report)
[2016-09-21 15:30:12,837] arbalester/ERROR/stderr: File "/usr/lib64/python2.7/json/__init__.py", line 244, in dumps
[2016-09-21 15:30:12,838] arbalester/ERROR/stderr: return _default_encoder.encode(obj)
[2016-09-21 15:30:12,838] arbalester/ERROR/stderr: File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
[2016-09-21 15:30:12,838] arbalester/ERROR/stderr: chunks = self.iterencode(o, _one_shot=True)
[2016-09-21 15:30:12,838] arbalester/ERROR/stderr: File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
[2016-09-21 15:30:12,838] arbalester/ERROR/stderr: return _iterencode(o, 0)
[2016-09-21 15:30:12,838] arbalester/ERROR/stderr: File "/usr/lib64/python2.7/json/encoder.py", line 184, in default
[2016-09-21 15:30:12,838] arbalester/ERROR/stderr: raise TypeError(repr(o) + " is not JSON serializable")
[2016-09-21 15:30:12,838] arbalester/ERROR/stderr: TypeError: HTTPError(u'502 Server Error: Bad Gateway for url: http://st-service/api/v1',) is not JSON serializable
[2016-09-21 15:30:12,841] arbalester/ERROR/stderr: {'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT': '*/*',
 'HTTP_ACCEPT_ENCODING': 'gzip, deflate, sdch',
 'HTTP_ACCEPT_LANGUAGE': 'en-GB,en-US;q=0.8,en;q=0.6',
 'HTTP_CONNECTION': 'keep-alive',
 'HTTP_COOKIE': 'cookie-concent=agreed; _ga=GA1.1.1822795932.1465397875; jenkins-timestamper-offset=-3600000; iconSize=16x16; csrf=1v7J3THHOlKkX9RK7b0HLSw71zklIhTm',
 'HTTP_HOST': 'localhost:8089',
 'HTTP_REFERER': 'http://localhost:8089/',
 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.101 Safari/537.36',
 'HTTP_X_REQUESTED_WITH': 'XMLHttpRequest',
 'PATH_INFO': '/stats/requests',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_PORT': '35782',
 'REQUEST_METHOD': 'GET',
 'SCRIPT_NAME': '',
 'SERVER_NAME': 'arbalester',
 'SERVER_PORT': '8089',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SOFTWARE': 'gevent/1.1 Python/2.7',
 'werkzeug.request': <Request 'http://localhost:8089/stats/requests' [GET]>,
 'wsgi.errors': <locust.log.StdErrWrapper object at 0x7fd074e46d10>,
 'wsgi.input': <gevent.pywsgi.Input object at 0x7fd055c89050>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0)} failed with TypeError

I tried to debug it and this is the result of the var report here https://github.com/locustio/locust/blob/master/locust/web.py#L154 is

{'errors': [{'error': HTTPError(u'502 Server Error: Bad Gateway for url: http://st-service/api/v1',), 'method': 'GET', 'occurences': 2, 'name': '/api/v1/complex'}, {'error': HTTPError(u'502 Server Error: Bad Gateway for url: http://st-service/api/v1f4084ce9-2bc9-440b-aae3-9f62ae626531',), 'method': 'GET', 'occurences': 1, 'name': '/api/v1/f4084ce9-2bc9-440b-aae3-9f62ae626531'}], 'stats': [{'median_response_time': 410, 'min_response_time': 48, 'current_rps': 92.88888888888889, 'name': '/api/v1', 'num_failures': 2, 'max_response_time': 4980, 'avg_content_length': 7413, 'avg_response_time': 528.8645655877342, 'method': 'GET', 'num_requests': 1174}, {'median_response_time': 370, 'min_response_time': 45, 'current_rps': 48.333333333333336, 'name': '/api/v1/f4084ce9-2bc9-440b-aae3-9f62ae626531', 'num_failures': 1, 'max_response_time': 4878, 'avg_content_length': 54, 'avg_response_time': 492.1566666666667, 'method': 'GET', 'num_requests': 600}, {'median_response_time': 370, 'min_response_time': 43, 'current_rps': 50.0, 'name': '/api/v1/f4084ce9-2bc9-440b-aae3-9f62ae626534', 'num_failures': 0, 'max_response_time': 4878, 'avg_content_length': 313, 'avg_response_time': 482.47611202635915, 'method': 'GET', 'num_requests': 607}, {'median_response_time': 370, 'min_response_time': 43, 'current_rps': 191.22222222222223, 'name': 'Total', 'num_failures': 3, 'max_response_time': 4980, 'avg_content_length': 3748, 'avg_response_time': 507.7883242335153, 'method': None, 'num_requests': 2381}], 'state': 'hatching', 'total_rps': 191.22222222222223, 'fail_ratio': 0.0012583892617449664, 'user_count': 944}

therefore fails to json encode the report['errors'][0]['error'] as not built-in (and w/o a defined encoder i presume)

[shipperizer]

something as simple as

    for error in report['errors']:
        if error['error'].__class__.__module__ != '__builtin__':
            error['error'] = str(error['error'])

makes it work, so if you are willing to accept a PR let me know

[heyman]

Thanks for te report. Could you please provide more info on how to reproduce this?

[heyman]

I suspect this error might have been introduced in #451 and I might know a simple fix, but I'd love to be able to reproduce it so that I can write a test.

[shipperizer]

well, my configuration is a ecs instance behind an elb...and i suppose the elb can't keep up therefore it throws 502

[heyman]

NVM, I believe managed to reproduce it, and I've pushed a fix. Locust should be able to handle failures and exceptions without crashing.

Could you try with the latest master and confirm that it's been fixed?

[shipperizer]

sure, just tried and looks fine now, thanks a lot

[heyman]

Great! Thanks again for reporting.