Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAOS-15008 test: VMD Hot Plug Automate - Replace during no activity #13620

Draft
wants to merge 18 commits into
base: master
Choose a base branch
from
Draft

DAOS-15008 test: VMD Hot Plug Automate - Replace during no activity #13620

wants to merge 18 commits into from
+482 −7

Conversation

shimizukko
Copy link
Contributor

Conduct VMD hot-remove and hot-plug during no activity.

  1. Determine the PCI address (TrAddr) of the disk we'll hot-remove and verify that its state is NORMAL and LED is OFF.
  2. Store the total NVMe size.
  3. Create a pool and a container.
  4. Write data with IOR.
  5. Hot remove the disk we selected at step 1.
  6. Repeatedly call "dmg storage query list-devices" until the disk state becomes UNPLUGGED and LED becomes NA.
  7. Verify that the disk space is down compared to before the remove.
  8. Hot-plug.
  9. Repeatedly call "dmg storage query list-devices" until the disk state becomes NORMAL and LED becomes OFF.
  10. Verify that the disk space is back to the original.
  11. Verify that none of the engines have crashed.
  12. Verify that the disks are healthy by checking the container status.
  13. Rung IOR and check that it works.

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true

Before requesting gatekeeper:

  • Two review approvals and any prior change requests have been resolved.
  • Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • Commit messages follows the guidelines outlined here.
  • Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • You are the appropriate gatekeeper to be landing the patch.
  • The PR has 2 reviews by people familiar with the code, including appropriate watchers.
  • Githooks were used. If not, request that user install them and check copyright dates.
  • Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • All builds have passed. Check non-required builds for any new compiler warnings.
  • Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • If applicable, the PR has addressed any potential version compatibility issues.
  • Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • Extra checks if forced landing is requested
    • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • No new NLT or valgrind warnings. Check the classic view.
    • Quick-build or Quick-functional is not used.
  • Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

@shimizukko
DAOS-15008 test: VMD Hot Plug Automate - Replace during no activity
3cd404e 
Conduct VMD hot-remove and hot-plug during no activity.

1. Determine the PCI address (TrAddr) of the disk we'll hot-remove and verify that its state
is NORMAL and LED is OFF.
2. Store the total NVMe size.
3. Create a pool and a container.
4. Write data with IOR.
5. Hot remove the disk we selected at step 1.
6. Repeatedly call "dmg storage query list-devices" until the disk state becomes UNPLUGGED
and LED becomes NA.
7. Verify that the disk space is down compared to before the remove.
8. Hot-plug.
9. Repeatedly call "dmg storage query list-devices" until the disk state becomes NORMAL and
LED becomes OFF.
10. Verify that the disk space is back to the original.
11. Verify that none of the engines have crashed.
12. Verify that the disks are healthy by checking the container status.
13. Rung IOR and check that it works.

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true

Signed-off-by: Makito Kano <[email protected]>
@github-actions GitHub Actions
Copy link

Bug-tracker data:
Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/DAOS-15008

@shimizukko
DAOS-15008 test: Fix pylint
a0bd39d 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true
Signed-off-by: Makito Kano <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13620/2/execution/node/886/log

@shimizukko
Merge branch 'master' into makito/DAOS-15008
88b7297 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true
@shimizukko
DAOS-15008 test: Update test yaml to support two engines
7badac7 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true
Signed-off-by: Makito Kano <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13620/3/execution/node/885/log

@shimizukko
DAOS-15008 test: Support multiple engines per rank when obtaining NVM…
bb9192f 
…e info

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true
Signed-off-by: Makito Kano <[email protected]>
@shimizukko
Merge branch 'master' into makito/DAOS-15008
b24a533 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true
@shimizukko
DAOS-15008 test: Fix pylint
ad7df9c 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true
Signed-off-by: Makito Kano <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13620/5/testReport/

@shimizukko
DAOS-15008 test: Use 10% pool size
00c1e57 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true
Signed-off-by: Makito Kano <[email protected]>
@shimizukko
Merge branch 'master' into makito/DAOS-15008
10ea9c8 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true
@shimizukko
Merge branch 'master' into makito/DAOS-15008
925279d 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true
@shimizukko
Merge branch 'master' into makito/DAOS-15008
8eea650 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true
@shimizukko
DAOS-15008 test: Update test steps and made other changes
6e78208 
Include set nvme-faulty and replace steps.
Update the logic to check total available NVMe size.

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true
Signed-off-by: Makito Kano <[email protected]>
@github-actions GitHub Actions
Copy link

Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/DAOS-15008

@shimizukko
DAOS-15008 test: Fix pylint
88d72f2 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
Required-githooks: true
Signed-off-by: Makito Kano <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13620/9/testReport/

@shimizukko shimizukko mentioned this pull request Apr 23, 2024
Merged
grom72 added 2 commits April 24, 2024 07:50
@grom72
Force Jenkins to use ci_vmd5 for tests instead of ci_nvme5.
99e4d75 
Test-tag: pr,hw,medium,hot_plug

Required-githooks: true
Quick-functional: true

Skip-coverity-test: true
Skip-fault-injection-test: true
Skip-func-test-el8: true
Skip-nlt: true
Skip-python-bandit: true
Skip-test-el-8.6-rpms: true
Skip-unit-test-memcheck: true
Skip-unit-tests: true

Skip-func-hw-test-medium: true
Skip-func-hw-test-medium-verbs-provider: true
Skip-func-hw-test-medium-md-on-ssd: false
Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false
Skip-func-hw-test-medium-ucx-provider: true
Skip-func-hw-test-large: true

Signed-off-by: Gromadzki, Tomasz <[email protected]>
@grom72
Force to use custom SPDK version.
cda2190 
Test-tag: hw,medium,hot_plug

PR-repos: spdk@PR-61

Required-githooks: true
Quick-functional: true

Skip-coverity-test: true
Skip-fault-injection-test: true
Skip-func-test-el8: true
Skip-nlt: true
Skip-python-bandit: true
Skip-test-el-8.6-rpms: true
Skip-unit-test-memcheck: true
Skip-unit-tests: true

Skip-func-hw-test-medium: true
Skip-func-hw-test-medium-verbs-provider: true
Skip-func-hw-test-medium-md-on-ssd: false
Skip-func-hw-test-medium-verbs-provider-md-on-ssd: false
Skip-func-hw-test-medium-ucx-provider: true
Skip-func-hw-test-large: true

Signed-off-by: Gromadzki, Tomasz <[email protected]>
@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13620/12/execution/node/548/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13620/13/execution/node/543/log

@daosbuild1
Copy link
Collaborator

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13620/13/testReport/

@shimizukko
Merge branch 'master' into makito/DAOS-15008
c1a9c05 
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
PR-repos: spdk@PR-61:53
@shimizukko
DAOS-15008 test: Use rd_fac:1 for container and RP_2GX for IOR object…
55b555b 
… type

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
PR-repos: spdk@PR-61:53
Signed-off-by: Makito Kano <[email protected]>
@shimizukko
Merge branch 'makito/DAOS-15008' of https://github.com/daos-stack/daos …
2180a7c 
…into makito/DAOS-15008

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: test_no_activity
PR-repos: spdk@PR-61:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@shimizukko @daosbuild1 @grom72