Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAOS-12287 test: CR Pass 4 - Orphan container #13063

Merged
merged 10 commits into from
Oct 20, 2023
Merged

DAOS-12287 test: CR Pass 4 - Orphan container #13063

merged 10 commits into from
Oct 20, 2023

Conversation

shimizukko
Copy link
Contributor

Test steps:

  1. Create a pool and a container.
  2. Inject fault to cause orphan container. i.e., container is left in the system, but doesn't appear with daos commands.
  3. Check that the container doesn't appear with daos command.
  4. Stop servers.
  5. Use ddb to verify that the container is left in shards.
  6. Enable the checker.
  7. Set policy to --all-interactive.
  8. Start the checker and query the checker until the fault is detected.
  9. Repair by selecting the destroy option.
  10. Query the checker until the fault is repaired.
  11. Disable the checker.
  12. Run the ddb command and verify that the container is removed from shard.

Create recovery_test_base.py to store methods used across the CR tests.

Add daos faults command support in daos_utils_base.py and daos_utils.py

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_orphan_container ddb_cmd
Test-repeat: 4
Required-githooks: true

Before requesting gatekeeper:

  • Two review approvals and any prior change requests have been resolved.
  • Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • Commit messages follows the guidelines outlined here.
  • Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • You are the appropriate gatekeeper to be landing the patch.
  • The PR has 2 reviews by people familiar with the code, including appropriate watchers.
  • Githooks were used. If not, request that user install them and check copyright dates.
  • Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • All builds have passed. Check non-required builds for any new compiler warnings.
  • Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • If applicable, the PR has addressed any potential version compatibility issues.
  • Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • Extra checks if forced landing is requested
    • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • No new NLT or valgrind warnings. Check the classic view.
    • Quick-build or Quick-functional is not used.
  • Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

@shimizukko
DAOS-12287 test: CR Pass 4 - Orphan container
165195a 
Test steps:
1. Create a pool and a container.
2. Inject fault to cause orphan container. i.e., container is left in the system,
but doesn't appear with daos commands.
3. Check that the container doesn't appear with daos command.
4. Stop servers.
5. Use ddb to verify that the container is left in shards.
6. Enable the checker.
7. Set policy to --all-interactive.
8. Start the checker and query the checker until the fault is detected.
9. Repair by selecting the destroy option.
10. Query the checker until the fault is repaired.
11. Disable the checker.
12. Run the ddb command and verify that the container is removed from shard.

Create recovery_test_base.py to store methods used across the CR tests.

Add daos faults command support in daos_utils_base.py and daos_utils.py

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_orphan_container ddb_cmd
Test-repeat: 4
Required-githooks: true

Signed-off-by: Makito Kano <[email protected]>
@github-actions
Copy link

github-actions bot commented Sep 17, 2023 •
edited
Loading

Bug-tracker data:
Ticket title is 'Pass 4 Automate: Destroy container while leaving its data on targets - Orphan container'
Status is 'In Review'
https://daosio.atlassian.net/browse/DAOS-12287

daosbuild1
daosbuild1 reviewed Sep 17, 2023
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

@shimizukko
DAOS-12287 test: Fix pylint
e4f31f7 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_orphan_container ddb_cmd
Test-repeat: 4
Required-githooks: true
Signed-off-by: Makito Kano <[email protected]>
daosbuild1
daosbuild1 reviewed Sep 17, 2023
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

@shimizukko
DAOS-12287 test: Add recovery_test_base.py
abf68e8 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_orphan_container ddb_cmd
Test-repeat: 4
Required-githooks: true
Signed-off-by: Makito Kano <[email protected]>
daosbuild1
daosbuild1 reviewed Sep 17, 2023
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13063/3/execution/node/750/log

@shimizukko
DAOS-12287 test: Add check_repair() in dmg_utils.py
8314d61 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_orphan_container ddb_cmd
Test-repeat: 4
Required-githooks: true
Signed-off-by: Makito Kano <[email protected]>
@shimizukko
Merge branch 'feature/cat_recovery' into makito/DAOS-12287
00bb8a2 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_orphan_container ddb_cmd
Test-repeat: 4
Required-githooks: true
daosbuild1
daosbuild1 reviewed Sep 23, 2023
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

daltonbohning
daltonbohning reviewed Oct 5, 2023
View reviewed changes
src/tests/ftest/util/recovery_test_base.py Outdated
Comment on lines 39 to 42
cmd_out = run_pcmd(hosts=hosts, command=command)

# return vos_file
for file in cmd_out[0]["stdout"]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to use run_remote than run_pcmd. Something like this...

Suggested change
cmd_out = run_pcmd(hosts=hosts, command=command)
# return vos_file
for file in cmd_out[0]["stdout"]:
from run_utils import run_remote
cmd_out = run_remote(self.log, hosts, command)
# return vos_file
for file in cmd_out.output[0].stdout:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Thanks.

src/tests/ftest/recovery/container_list_consolidation.py Outdated
Comment on lines 170 to 173
# Start server to prepare for the cleanup.
dmg_command.system_start()

report_errors(test=self, errors=errors)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just pointing out that if system_start raises an exception and causes the test to fail, then we would miss the report_errors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added except CommandFailure to catch the system start exception.

@shimizukko
Merge branch 'feature/cat_recovery' into makito/DAOS-12287
546ab8e 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_orphan_container ddb_cmd
Test-repeat: 4
Required-githooks: true
@shimizukko
DAOS-12287 test: Use run_remote() and surround system_start() with tr…
1395c21 
…y except

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_orphan_container ddb_cmd
Test-repeat: 4
Required-githooks: true
Signed-off-by: Makito Kano <[email protected]>
daosbuild1
daosbuild1 reviewed Oct 6, 2023
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

@daosbuild1
Copy link
Collaborator

Test stage Python Bandit check completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13063/5/execution/node/147/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13063/5/execution/node/409/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13063/5/execution/node/410/log

@daosbuild1
Copy link
Collaborator

Test stage Build RPM on Leap 15.4 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13063/5/execution/node/394/log

@daosbuild1
Copy link
Collaborator

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13063/5/execution/node/311/log

@shimizukko
DAOS-12287 test: Update comment
2240e2d 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_orphan_container ddb_cmd
Test-repeat: 4
Required-githooks: true
Signed-off-by: Makito Kano <[email protected]>
daosbuild1
daosbuild1 reviewed Oct 7, 2023
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

@daosbuild1
Copy link
Collaborator

Test stage Python Bandit check completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13063/6/execution/node/147/log

@daosbuild1
Copy link
Collaborator

Test stage Functional on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13063/6/execution/node/757/log

@shimizukko
Copy link
Contributor Author

Blocked due to https://daosio.atlassian.net/browse/DAOS-14473 (Engine crash during dmg check query after the repair.)

@shimizukko
Merge branch 'feature/cat_recovery' into makito/DAOS-12287
1bf8fa7 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_orphan_container ddb_cmd
Test-repeat: 4
Required-githooks: true
daosbuild1
daosbuild1 reviewed Oct 11, 2023
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

@daosbuild1
Copy link
Collaborator

Test stage Python Bandit check completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13063/7/execution/node/147/log

@shimizukko
DAOS-12287 test: Update comment
23c9276 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: pr ddb_cmd
Required-githooks: true
Signed-off-by: Makito Kano <[email protected]>
daosbuild1
daosbuild1 reviewed Oct 11, 2023
View reviewed changes
Copy link
Collaborator

@daosbuild1 daosbuild1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. No errors found by checkpatch.

@daosbuild1
Copy link
Collaborator

Test stage Python Bandit check completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13063/8/execution/node/169/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13063/8/execution/node/884/log

@shimizukko shimizukko marked this pull request as ready for review October 12, 2023 21:20
@shimizukko shimizukko requested a review from a team as a code owner October 12, 2023 21:20
@shimizukko
Copy link
Contributor Author

shimizukko commented Oct 15, 2023 •
edited
Loading

The 8 CR test failures (CR20 to CR27) in daos_test/suite.py are not related to this PR. They are unexpected and Fan Yong is looking into them in DAOS-14513.

@shimizukko shimizukko requested a review from a team October 15, 2023 23:24
@daltonbohning daltonbohning merged commit eb246e0 into feature/cat_recovery Oct 20, 2023
@daltonbohning daltonbohning deleted the makito/DAOS-12287 branch October 20, 2023 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daosbuild1 daosbuild1 daosbuild1 left review comments

@dinghwah dinghwah dinghwah approved these changes

@daltonbohning daltonbohning daltonbohning approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@shimizukko @daosbuild1 @daltonbohning @dinghwah