Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'BackoffFailed' in logging system tests. #5303

Closed
crwilcox opened this issue May 7, 2018 · 6 comments
Closed

'BackoffFailed' in logging system tests. #5303

crwilcox opened this issue May 7, 2018 · 6 comments
Assignees
Labels
api: logging Issues related to the Cloud Logging API. flaky testing type: process A process-related concern. May include testing, release, or the like.

Comments

@crwilcox
Copy link
Contributor

crwilcox commented May 7, 2018

System tests for logging are failing like this currently:

_____________________________________________________________________________________________________ TestLogging.test_log_handler_sync ______________________________________________________________________________________________________
Traceback (most recent call last):
  File "/Users/crwilcox/workspace/google-cloud-python/logging/tests/system/test_system.py", line 264, in test_log_handler_sync
    entries = _list_entries(logger)
  File "/Users/crwilcox/workspace/google-cloud-python/logging/tests/system/test_system.py", line 67, in _list_entries
    return outer(logger)
  File "/Users/crwilcox/workspace/google-cloud-python/logging/.nox/sys-3-6/lib/python3.6/site-packages/test_utils/retry.py", line 95, in wrapped_function
    return to_wrap(*args, **kwargs)
  File "/Users/crwilcox/workspace/google-cloud-python/logging/.nox/sys-3-6/lib/python3.6/site-packages/test_utils/retry.py", line 155, in wrapped_function
    raise BackoffFailed()
test_utils.retry.BackoffFailed
@tseaver tseaver added testing api: logging Issues related to the Cloud Logging API. type: process A process-related concern. May include testing, release, or the like. flaky labels May 8, 2018
@tseaver
Copy link
Contributor

tseaver commented Jun 14, 2018

Saw it again today.

@tseaver
Copy link
Contributor

tseaver commented Jun 14, 2018

Via #5486.

@tseaver
Copy link
Contributor

tseaver commented Jun 26, 2018

I don't believe #5486 actually fixed the test_utils.retry.BackoffFailed problem: it just added retries for 429s w/ RESOURCE_EXHAUSTED gRPC status codes. See today's failure (the second error, inside TestLogging.test_log_text_with_resource).

@tseaver tseaver reopened this Jun 26, 2018
@tseaver
Copy link
Contributor

tseaver commented Jun 27, 2018

Failed again today

@tseaver
Copy link
Contributor

tseaver commented Jun 28, 2018

And again today

@tseaver tseaver changed the title [Logging] System test failing for logging 'BackoffFailed' in logging system tests. Jun 28, 2018
@tseaver
Copy link
Contributor

tseaver commented Aug 6, 2018

A new failures today.

#5551 set max_tries=9 in the retry wrapper. I wish I had a better idea than to just bump the number higher.

tseaver added a commit that referenced this issue Oct 9, 2018
Also, make logger names unique between test runs.

Closes #5303.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: logging Issues related to the Cloud Logging API. flaky testing type: process A process-related concern. May include testing, release, or the like.
Projects
None yet
Development

No branches or pull requests

2 participants