disable / avoid urllib3 error output[enhancement]

[reteps]

I wrote a script that grabs information about tickers on my computer at specific times. If I don't have wifi, there is a long error script from it, that I would like to disable. I can only catch your error, not any urllib3 ones. Output:

HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x101b64a58>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 743, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 844, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x101b64a58>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x101b64a58>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/poloniex/__init__.py", line 143, in retrying
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/poloniex/__init__.py", line 215, in __call__
    ret = _get(**payload)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 504, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x101b64a58>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x1026dcc50>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 743, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 844, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x1026dcc50>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x1026dcc50>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/poloniex/__init__.py", line 143, in retrying
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/poloniex/__init__.py", line 215, in __call__
    ret = _get(**payload)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 504, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x1026dcc50>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10279d198>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 743, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 844, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x10279d198>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10279d198>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/poloniex/__init__.py", line 143, in retrying
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/poloniex/__init__.py", line 215, in __call__
    ret = _get(**payload)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 504, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10279d198>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10279d7b8>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 743, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 844, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x10279d7b8>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10279d7b8>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/poloniex/__init__.py", line 143, in retrying
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/poloniex/__init__.py", line 215, in __call__
    ret = _get(**payload)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 504, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10279d7b8>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
[ConnectionError(MaxRetryError("HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x101b64a58>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))",),), ConnectionError(MaxRetryError("HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x1026dcc50>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))",),), ConnectionError(MaxRetryError("HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10279d198>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))",),), ConnectionError(MaxRetryError("HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10279d7b8>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))",),), ConnectionError(MaxRetryError("HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10279ddd8>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))",),)]

I finally caught this error after all that was outputted:

HTTPSConnectionPool(host='poloniex.com', port=443): Max retries exceeded with url: /public?command=returnTicker (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10279ddd8>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

How can I avoid / silence this?

[s4w3d0ff]

All requests.exceptions are captured and by the Poloniex.retry method/decorator. What happened is that your script lost connection with poloniex (or the internet) and was raising requests (urllib3) exceptions.

When this happens, the Poloniex.retry decorator will try the api call again (and again) until either a response is returned from poloniex or the poloniex.retryDelay list is exhausted.

In your case the retry list was exhausted, requests was still raising an error so the wrapper shows the captured exceptions (in the logger), then raises (nothing...)...

https://github.com/s4w3d0ff/python-poloniex/blob/master/poloniex/__init__.py#L135-L156

I can create a new Exception (something like RetryException or RetryExhausted) so users can capture the event. I could also adjust the logger output so it is in the 'debug' level instead of always showing.

[reteps]

Either of those would work for me. Would catching the RetryException avoid that long output? Because if it doesn't, then I would prefer the debug option

[s4w3d0ff]

This should help but you are still going to see a huge traceback if you don't capture the RetryException and you will probably want to set the requests logger to INFO or ERROR to reduce the output.

>>> import logging
>>> from poloniex import Poloniex, RetryException
>>> logging.basicConfig(level=logging.ERROR)
>>> polo = Poloniex()
>>> try:
...     polo.returnTicker()
... except RetryException:
...    print('Retry Max Reached')
... 
Retry Max Reached

import logging
from poloniex import Poloniex, RetryException
logging.basicConfig(level=logging.INFO)
polo = Poloniex()
try:
    polo.returnTicker()
except RetryException:
   print('Retry Max Reached')

INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): poloniex.com
INFO:poloniex:-- delaying for 0s
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): poloniex.com
INFO:poloniex:-- delaying for 2s
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): poloniex.com
INFO:poloniex:-- delaying for 5s
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): poloniex.com
INFO:poloniex:-- delaying for 30s
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): poloniex.com
Retry Max Reached