Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail more gracefully #8456

Merged
merged 2 commits into from
Feb 1, 2021
Merged

Fail more gracefully #8456

merged 2 commits into from
Feb 1, 2021

Conversation

hithwen
Copy link
Contributor

@hithwen hithwen commented Jan 26, 2021

Ceph is flaking on CI with following errors:

=================================== FAILURES ===================================
__________________________________ test_check __________________________________
tests/test_integration.py:18: in test_check
    ceph_check.check(copy.deepcopy(BASIC_CONFIG))
datadog_checks/ceph/ceph.py:319: in check
    raw = self._collect_raw(ceph_cmd, ceph_cluster, instance)
datadog_checks/ceph/ceph.py:73: in _collect_raw
    mon_map = raw['status']['monmap']
E   KeyError: 'status'
---------------------------- Captured stderr setup -----------------------------
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "compose_default" with the default driver
Creating dd-test-ceph ... 
Creating dd-test-ceph ... done
Error response from daemon: Container 1a0fce237e6376b91628fcf97c0a251f89842a752199eb1c64a75cef6c8b3fa8 is not running
------------------------------ Captured log call -------------------------------
WARNING  datadog_checks.base.checks.base.ceph:ceph.py:67 Unable to parse data from cmd=mon_status: get_subprocess_output expected output but had none.
WARNING  datadog_checks.base.checks.base.ceph:ceph.py:67 Unable to parse data from cmd=status: get_subprocess_output expected output but had none.
WARNING  datadog_checks.base.checks.base.ceph:ceph.py:67 Unable to parse data from cmd=df detail: get_subprocess_output expected output but had none.
WARNING  datadog_checks.base.checks.base.ceph:ceph.py:67 Unable to parse data from cmd=osd pool stats: get_subprocess_output expected output but had none.
WARNING  datadog_checks.base.checks.base.ceph:ceph.py:67 Unable to parse data from cmd=osd perf: get_subprocess_output expected output but had none.
WARNING  datadog_checks.base.checks.base.ceph:ceph.py:67 Unable to parse data from cmd=health detail: get_subprocess_output expected output but had none.

Created https://github.com/DataDog/integrations-core/pull/8455/files to see if we can better diagnose why there is no output in the subprocess while this PR attempts to make it fail more gracefully

@hithwen hithwen requested a review from a team as a code owner January 26, 2021 11:17
Copy link
Contributor

@ChristineTChen ChristineTChen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor rewording

ceph/datadog_checks/ceph/ceph.py Outdated Show resolved Hide resolved
@hithwen hithwen merged commit cd28353 into master Feb 1, 2021
@hithwen hithwen deleted the julia/ceph-fail-more-gracefully branch February 1, 2021 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants