mysteriously wedged sds

[rhs]

My ambassador got into a mysteriously wedged state. It was responding just fine to /ambassador/* queries, but it refused to route to my upstream "hello" cluster.

After some head scratching I managed to track this down to the sds service being wedged. I was able to diagnose this because a) the sds service was hanging when I queried it, and b) I got the appended stack trace from the logs.

This raises the obvious question of why it got wedged in this way, but perhaps more importantly this points to the need for some kind of health checking on the various envoy "plugin" services.

[rhs@venture hello]$ kubectl logs ambassador-sds-3085809130-cp47x  
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 699, in inner
    srv.serve_forever()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 536, in serve_forever
    HTTPServer.serve_forever(self)
  File "/usr/lib/python2.7/SocketServer.py", line 238, in serve_forever
    self._handle_request_noblock()
  File "/usr/lib/python2.7/SocketServer.py", line 297, in _handle_request_noblock
    self.handle_error(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__
    self.handle()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 232, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 267, in handle_one_request
    return self.run_wsgi()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 209, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 200, in execute
    write(data)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 168, in write
    self.send_header(key, value)
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 401, in send_header
    self.wfile.write("%s: %s\r\n" % (keyword, value))
IOError: [Errno 32] Broken pipe

[kflynn]

It seems that the ambassador-sds as shipped can end up hanging after awhile. This is probably because it's running out of memory, despite there being no obvious thing that would cause it to hang onto anything.

Planning to switch it to Python 3.6 and do some bulletproofing. Also see #32.

[kflynn]

Ambassador doesn't ship with SDS any more.