Skip to content
This repository was archived by the owner on Dec 4, 2024. It is now read-only.

Refactor #607 #630

Merged
merged 3 commits into from
Apr 4, 2019
Merged

Refactor #607 #630

merged 3 commits into from
Apr 4, 2019

Conversation

jkoelker
Copy link
Contributor

No description provided.

@gpoulin-hopper @jkoelker
Fallback on next Marathon instance on exception
c9e6a8c 
If requests raise a TimeoutException it might be due to the current
instance being down. We should fallback to the next instance
@gpoulin-hopper @jkoelker
Stick to the host with the last successful request
5e8668f 
If we make a requests to a marathon host and it failed it might be
because the host is down. We shouldn't keep trying making request to
this host and instead requesting the last host that answer correctly.
@mesosphere-ci
Copy link

Can one of the admins verify this patch?

@jkoelker jkoelker mentioned this pull request Mar 26, 2019
Closed
@jkoelker
Copy link
Contributor Author

jkoelker commented Mar 26, 2019 •
edited
Loading 

+ Build container image jkoelker/jkoelker-marathon-lb-dev:8abc3c01764c
+ Pushing image to hub
docker push jkoelker/jkoelker-marathon-lb-dev:8abc3c01764c
The push refers to repository [docker.io/jkoelker/jkoelker-marathon-lb-dev]
cb3554e2c744: Pushed
69475446c461: Layer already exists
2c162559b790: Layer already exists
48d89fd8b315: Layer already exists
e8600637163d: Layer already exists
411a98cdce86: Layer already exists
8abc3c01764c: digest: sha256:3095aca2a5f00e91b4105e42ca6f701c69f7da3dba8cd4e089e211b85e5b58fb size: 1582
+ Build devkit image mesosphere/marathon-lb-devkit:latest
+ Discovering Cluster URL
+ Cluster URL: http://172.17.0.3
+ Discovering Public Node IP
+ Public Node IP: 172.17.0.5
+ Integration Testng with image jkoelker/jkoelker-marathon-lb-dev:8abc3c01764c
============================================================================================= test session starts ==============================================================================================
platform linux -- Python 3.7.2rc1, pytest-3.10.1, py-1.8.0, pluggy-0.9.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /marathon-lb/ci, inifile:
plugins: timeout-1.3.3, dcos-test-utils-0.1
collected 20 items

test_marathon_lb_dcos_e2e.py::test_port[backends/docker_ippc.json] PASSED                                                                                                                                [  5%]
test_marathon_lb_dcos_e2e.py::test_response[backends/docker_ippc.json] PASSED                                                                                                                            [ 10%]
test_marathon_lb_dcos_e2e.py::test_port[backends/ucr_host.json] PASSED                                                                                                                                   [ 15%]
test_marathon_lb_dcos_e2e.py::test_response[backends/ucr_host.json] PASSED                                                                                                                               [ 20%]
test_marathon_lb_dcos_e2e.py::test_port[backends/docker_bridge.json] PASSED                                                                                                                              [ 25%]
test_marathon_lb_dcos_e2e.py::test_response[backends/docker_bridge.json] PASSED                                                                                                                          [ 30%]
test_marathon_lb_dcos_e2e.py::test_port[backends/ucr_ippc.json] PASSED                                                                                                                                   [ 35%]
test_marathon_lb_dcos_e2e.py::test_response[backends/ucr_ippc.json] PASSED                                                                                                                               [ 40%]
test_marathon_lb_dcos_e2e.py::test_port[backends/ucr_bridge.json] PASSED                                                                                                                                 [ 45%]
test_marathon_lb_dcos_e2e.py::test_response[backends/ucr_bridge.json] PASSED                                                                                                                             [ 50%]
test_marathon_lb_dcos_e2e.py::test_port[backends/docker_host.json] PASSED                                                                                                                                [ 55%]
test_marathon_lb_dcos_e2e.py::test_response[backends/docker_host.json] PASSED                                                                                                                            [ 60%]
test_marathon_lb_dcos_e2e.py::test_port[backends_1.9/docker_ippc.json] SKIPPED                                                                                                                           [ 65%]
test_marathon_lb_dcos_e2e.py::test_response[backends_1.9/docker_ippc.json] SKIPPED                                                                                                                       [ 70%]
test_marathon_lb_dcos_e2e.py::test_port[backends_1.9/ucr_host.json] SKIPPED                                                                                                                              [ 75%]
test_marathon_lb_dcos_e2e.py::test_response[backends_1.9/ucr_host.json] SKIPPED                                                                                                                          [ 80%]
test_marathon_lb_dcos_e2e.py::test_port[backends_1.9/docker_bridge.json] SKIPPED                                                                                                                         [ 85%]
test_marathon_lb_dcos_e2e.py::test_response[backends_1.9/docker_bridge.json] SKIPPED                                                                                                                     [ 90%]
test_marathon_lb_dcos_e2e.py::test_port[backends_1.9/docker_host.json] SKIPPED                                                                                                                           [ 95%]
test_marathon_lb_dcos_e2e.py::test_response[backends_1.9/docker_host.json] SKIPPED                                                                                                                       [100%]

==================================================================================== 12 passed, 8 skipped in 169.04 seconds ====================================================================================

@jkoelker
Copy link
Contributor Author

Manually tested by iptables'ing off the host:

2019-03-27 18:05:25,351 marathon_lb: Calling GET http://10.0.4.5:8080/v2/apps.
2019-03-27 18:05:25,770 marathon_lb: received event of type status_update_event
2019-03-27 18:05:28,407 marathon_lb: Error calling GET http://10.0.4.5:8080/v2/apps, trying next host.
2019-03-27 18:05:28,407 marathon_lb: GET http://10.0.4.5:8080/v2/apps
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.7/dist-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/usr/local/lib/python3.7/dist-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connection.py", line 181, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connection.py", line 164, in _new_conn
    (self.host, self.timeout))
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7fd963412320>, 'Connection to 10.0.4.5 timed out. (connect timeout=3.05)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.7/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.7/dist-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.0.4.5', port=8080): Max retries exceeded w
ith url: /v2/apps?embed=apps.tasks (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fd963412320>, 'Connection to 10.0.4.5 timed out. (connect timeout=3.05)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/marathon-lb/marathon_lb.py", line 188, in api_req_raw
    **kwargs
  File "/usr/local/lib/python3.7/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/requests/adapters.py", line 504, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='10.0.4.5', port=8080): Max retries exceeded with url: /v2/apps?embed=apps.tasks (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fd963412320>, 'Connection to 10.0.4.5 timed out. (connect timeout=3.05)'))
2019-03-27 18:05:28,408 marathon_lb: Calling GET http://10.0.7.103:8080/v2/apps.
2019-03-27 18:05:28,433 marathon_lb: GET http://10.0.7.103:8080/v2/apps?embed=apps.tasks
2019-03-27 18:05:28,434 marathon_lb: got apps ['/marathon-lb']
2019-03-27 18:05:28,437 marathon_lb: generating config
2019-03-27 18:05:28,437 marathon_lb: HAProxy dir is /marathon-lb
2019-03-27 18:05:28,437 marathon_lb: reading running config from /marathon-lb/haproxy.cfg
2019-03-27 18:05:28,438 marathon_lb: running config/map is different from generated config - reloading
2019-03-27 18:05:28,438 marathon_lb: writing temp file /tmp/tmpqfzk_q7r that will replace /marathon-lb/domain2backend.map
2019-03-27 18:05:28,438 marathon_lb: writing temp file /tmp/tmpfiznhdk6 that will replace /marathon-lb/app2backend.map
2019-03-27 18:05:28,438 marathon_lb: writing temp file /tmp/tmpyw98olnc that will replace /marathon-lb/haproxy.cfg
2019-03-27 18:05:28,438 marathon_lb: checking config with command: ['haproxy', '-f', '/tmp/tmpyw98olnc', '-c']
[WARNING] 085/180528 (60) : Can't read first line of the server state file '/var/state/haproxy/global'
Configuration file is valid

@jkoelker jkoelker changed the title [DO NOT MERGE] Pr607 Refactor #607 Apr 4, 2019
@jkoelker jkoelker merged commit 5ebb622 into d2iq-archive:master Apr 4, 2019
@jkoelker jkoelker deleted the pr607 branch April 4, 2019 20:49
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@GoelDeepak GoelDeepak GoelDeepak approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jkoelker @mesosphere-ci @GoelDeepak @gpoulin-hopper