Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry backoff #33

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Retry backoff #33

wants to merge 2 commits into from

Conversation

Selta
Copy link

@Selta Selta commented Oct 2, 2022

Currently, when running with an influx DB that starts in another container, the Speedflux will complete its 3 retries before the influxDB is accepting connections (even with depends_on set in the compose yaml). According to logs, it often gets through these retries in roughly 1 second, as exampled here:

2022-10-02 05:38:56,504 [INFO] Speedtest CLI data logger to InfluxDB started...
2022-10-02 05:38:57,515 [ERROR] Connection to influx host was refused. This likely means that the DB is down or INFLUX_DB_ADDRESS is incorrect. It's currently 'influx-grafana'
2022-10-02 05:38:57,515 [ERROR] Full Error follows

2022-10-02 05:38:57,515 [ERROR] HTTPConnectionPool(host='influx-grafana', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb36ff9c8e0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2022-10-02 05:38:57,515 [ERROR] Retry 1: Initiliazing DB.
2022-10-02 05:38:57,521 [ERROR] Connection to influx host was refused. This likely means that the DB is down or INFLUX_DB_ADDRESS is incorrect. It's currently 'influx-grafana'
2022-10-02 05:38:57,522 [ERROR] Full Error follows

2022-10-02 05:38:57,522 [ERROR] HTTPConnectionPool(host='influx-grafana', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb36ffbba00>: Failed to establish a new connection: [Errno 111] Connection refused'))
2022-10-02 05:38:57,522 [ERROR] Retry 2: Initiliazing DB.
2022-10-02 05:38:57,527 [ERROR] Connection to influx host was refused. This likely means that the DB is down or INFLUX_DB_ADDRESS is incorrect. It's currently 'influx-grafana'
2022-10-02 05:38:57,527 [ERROR] Full Error follows

2022-10-02 05:38:57,527 [ERROR] HTTPConnectionPool(host='influx-grafana', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb36ff12670>: Failed to establish a new connection: [Errno 111] Connection refused'))
2022-10-02 05:38:57,527 [ERROR] Retry 3: Initiliazing DB.
2022-10-02 05:38:57,533 [ERROR] Database Init failed for 3rd time. Exiting

This PR adds a simple exponential backoff to those retries, allowing influxDB a little more time (~28seconds with the default 2s backoff_in_seconds param) to start and accept connections. Comments and criticisms welcome, of course.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant