DAOS-16446 test: HDF5-VOL test - Set object class and container prope… #15004

shimizukko · 2024-08-26T06:11:36Z

…rties env vars

In HDF5, DFS, MPIIO, or POSIX, object class and container properties are defined during the container create. If it’s DFS, object class is also set to the IOR parameter. However, in HDF5-VOL, object class and container properties are defined with the following environment variables of mpirun.

HDF5_DAOS_OBJ_CLASS (Object class)
HDF5_DAOS_FILE_PROP (Container properties)

The infrastructure to set these variables are already there in run_ior_with_pool(). In file_count_test_base.py, pass in the env vars to run_ior_with_pool(env=env) as a dictionary. Object class is the oclass variable. Container properties can be obtained from self.container.properties.value.

This fix is discussed in PR14964.

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-medium-md-on-ssd: false
Test-tag: test_io_sys_admin test_largefilecount test_smallfilecount

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

…rties env vars In HDF5, DFS, MPIIO, or POSIX, object class and container properties are defined during the container create. If it’s DFS, object class is also set to the IOR parameter. However, in HDF5-VOL, object class and container properties are defined with the following environment variables of mpirun. HDF5_DAOS_OBJ_CLASS (Object class) HDF5_DAOS_FILE_PROP (Container properties) The infrastructure to set these variables are already there in run_ior_with_pool(). In file_count_test_base.py, pass in the env vars to run_ior_with_pool(env=env) as a dictionary. Object class is the oclass variable. Container properties can be obtained from container -> properties field in the test yaml. This fix is discussed in PR14964. Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_io_sys_admin test_largefilecount test_smallfilecount Signed-off-by: Makito Kano <[email protected]>

github-actions · 2024-08-26T06:11:59Z

Ticket title is 'HDF5-VOL test - Set object class and container properties env vars'
Status is 'In Progress'
Labels: 'scrubbed_2.8'
https://daosio.atlassian.net/browse/DAOS-16446

daosbuild1 · 2024-08-26T06:55:06Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15004/1/testReport/

daosbuild1 · 2024-08-26T06:57:13Z

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-15004/1/testReport/

…rror Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_io_sys_admin Signed-off-by: Makito Kano <[email protected]>

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_io_sys_admin

src/tests/ftest/util/file_count_test_base.py

Obtain container properties from self.container and format it. Update the env variable instead of creating a new one and using it. Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_io_sys_admin Signed-off-by: Makito Kano <[email protected]>

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_io_sys_admin

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_io_sys_admin test_largefilecount test_smallfilecount Signed-off-by: Makito Kano <[email protected]>

shimizukko · 2024-08-31T00:31:58Z

io_sys_admin.py ended with WARN as expected.
Checked that the env vars are set with the correct format. -genv HDF5_DAOS_OBJ_CLASS=EC_2P1GX -genv HDF5_DAOS_FILE_PROP="cksum:crc16;cksum_size:16384;srv_cksum:on;rd_fac:1"

#15004) In HDF5, DFS, MPIIO, or POSIX, object class and container properties are defined during the container create. If it’s DFS, object class is also set to the IOR parameter. However, in HDF5-VOL, object class and container properties are defined with the following environment variables of mpirun. HDF5_DAOS_OBJ_CLASS (Object class) HDF5_DAOS_FILE_PROP (Container properties) The infrastructure to set these variables are already there in run_ior_with_pool(). In file_count_test_base.py, pass in the env vars to run_ior_with_pool(env=env) as a dictionary. Object class is the oclass variable. Container properties can be obtained from self.container.properties.value. This fix is discussed in PR #14964. Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_io_sys_admin test_largefilecount test_smallfilecount Signed-off-by: Makito Kano <[email protected]>

#15004) #15098 In HDF5, DFS, MPIIO, or POSIX, object class and container properties are defined during the container create. If it’s DFS, object class is also set to the IOR parameter. However, in HDF5-VOL, object class and container properties are defined with the following environment variables of mpirun. HDF5_DAOS_OBJ_CLASS (Object class) HDF5_DAOS_FILE_PROP (Container properties) The infrastructure to set these variables are already there in run_ior_with_pool(). In file_count_test_base.py, pass in the env vars to run_ior_with_pool(env=env) as a dictionary. Object class is the oclass variable. Container properties can be obtained from self.container.properties.value. This fix is discussed in PR #14964. Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_io_sys_admin test_largefilecount test_smallfilecount Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

* DAOS-16484 test: Exclude local host in default interface selection (#15049) When including the local host in the default interface selection a difference in ib0 speeds will cause the logic to select eth0 and then the tcp provider. Signed-off-by: Phil Henderson <[email protected]> * DAOS-15800 client: create cart context on specific interface (#14804) Cart has added the ability to select network interface on context creation. The daos_agent also added a numa-fabric map that can be queried at init time. Update the DAOS client to query from the agent a map of numa to network interface on daos_init(), and on EQ creation, select the best interface for the network context based on the numa of the calling thread. Signed-off-by: Mohamad Chaarawi <[email protected]> * DAOS-16445 client: Add function to cycle OIDs non-sequentially (#14999) We've noticed that with sequential order, object placement is poor. We get 40% fill for 8GiB files with 25 ranks and 16 targets per rank with EC_2P1G8. With this patch, we get a much better distribution. This patch adds the following: 1. A function for cycling oid.hi incrementing by a large prime 2. For DFS, randomize the starting value 3. Modify DFS to cycle OIDs using the new function. Signed-off-by: Jeff Olivier <[email protected]> * DAOS-16251 dtx: Fix dtx_req_send user-after-free (#15035) In dtx_req_send, since the crt_req_send releases the req reference, din may have been freed when dereferenced for the DL_CDEBUG call. Signed-off-by: Li Wei <[email protected]> * DAOS-16304 tools: Add daos health net-test command (#14980) Wrap self_test to provide a simplified network test to detect obvious client/server connectivity and performance problems. Signed-off-by: Michael MacDonald <[email protected]> * DAOS-16272 dfs: fix get_info returning incorrect oclass (#15048) If user creates a container without --file-oclass, the get_info call was returning the default oclass of a directory on daos fs get-attr. Fix that to properly use the enum types for default scenario. Signed-off-by: Mohamad Chaarawi <[email protected]> * DAOS-15863 container: fix a race for container cache (#15038) * DAOS-15863 container: fix a race for container cache while destroying a container, cont_child_destroy_one() releases its own refcount before waiting, if another ULT releases its refcount, which is the last one, wakes up the waiting ULT and frees it ds_cont_child straightaway, because no one else has refcount. When the waiting ULT is waken up, it will try to change the already freed ds_cont_child. This patch changes the LRU eviction logic and fixes this race. Signed-off-by: Liang Zhen <[email protected]> Signed-off-by: Jeff Olivier <[email protected]> Co-authored-by: Jeff Olivier <[email protected]> * DAOS-16471 test: Reduce targets for ioctl_pool_handles.py (#15063) The dfuse/ioctl_pool_handles.py test is overloading the VM so reduce the number of engine targets. Signed-off-by: Phil Henderson <[email protected]> * DAOS-16483 vos: handle empty DTX when vos_tx_end (#15053) It is possible that the DTX modified nothing when stop currnet backend transaction. Under such case, we may not generate persistent DTX entry. Then need to bypass such case before checking on-disk DTX entry status. The patch makes some clean and removed redundant metrics for committed DTX entries. Enhance vos_dtx_deregister_record() to handle GC case. Signed-off-by: Fan Yong <[email protected]> * DAOS-16271 mercury: Add patch to avoid seg fault in key resolve. (#15067) Signed-off-by: Joseph Moore <[email protected]> * DAOS-16484 test: Support mixed speeds when selecting a default interface (#15050) Allow selecting a default interface that is running at a different speed on different hosts. Primarily this is to support selecting the ib0 interface by default when the launch node has a slower ib0 interface than the cluster hosts. Signed-off-by: Phil Henderson <[email protected]> * DAOS-16446 test: HDF5-VOL test - Set object class and container prope… (#15004) In HDF5, DFS, MPIIO, or POSIX, object class and container properties are defined during the container create. If it’s DFS, object class is also set to the IOR parameter. However, in HDF5-VOL, object class and container properties are defined with the following environment variables of mpirun. HDF5_DAOS_OBJ_CLASS (Object class) HDF5_DAOS_FILE_PROP (Container properties) The infrastructure to set these variables are already there in run_ior_with_pool(). In file_count_test_base.py, pass in the env vars to run_ior_with_pool(env=env) as a dictionary. Object class is the oclass variable. Container properties can be obtained from container -> properties field in the test yaml. This fix is discussed in PR #14964. Signed-off-by: Makito Kano <[email protected]> * DAOS-16447 test: set D_IL_REPORT per test (#15012) set D_IL_REPORT per test instead of setting defaults values in utilities. This allows running without it set. Signed-off-by: Dalton Bohning <[email protected]> * DAOS-16450 test: auto run dfs tests when dfs is modified (#15017) Automatically include dfs tests when dfs files are modified in PRs. Signed-off-by: Dalton Bohning <[email protected]> * DAOS-16510 cq: update pylint to 3.2.7 (#15072) update pylint to 3.2.7 Signed-off-by: Dalton Bohning <[email protected]> * DAOS-16509 test: replace IorTestBase.execute_cmd with run_remote (#15070) replace usage of IorTestBase.execute_cmd with run_remote Signed-off-by: Dalton Bohning <[email protected]> * DAOS-16458 object: fix invalid DRAM access in obj_bulk_transfer (#15026) For EC object update via CPD RPC, when calculate the bitmap to skip some iods for current EC data shard, we may input NULL for "*skips" parameter. It may cause the old logic in obj_get_iods_offs_by_oid() to generate some undefined DRAM for "skips" bitmap. Such bitmap may be over-written by others, as to subsequent obj_bulk_transfer() may be misguided. The patch also fixes a bug inside obj_bulk_transfer() that cast any input RPC as UPDATE/FETCH by force. Signed-off-by: Fan Yong <[email protected]> * DAOS-16486 object: return proper error on stale pool map (#15064) Client with stale pool map may try to send RPC to a DOWN target, if the target was brought DOWN due to faulty NVMe device, the ds_pool_child could have been stopped on the NVMe faulty reaction, We'd ensure proper error code is returned for such case. Signed-off-by: Niu Yawei <[email protected]> * DAOS-16514 vos: fix coverity issue (#15083) Fix coverity 2555843 explict null dereferenced. Signed-off-by: Niu Yawei <[email protected]> * DAOS-16467 rebuild: add DAOS_POOL_RF ENV for massive failure case (#15037) * DAOS-16467 rebuild: add DAOS_PW_RF ENV for massive failure case Allow user to set DAOS_PW_RF as pw_rf (pool wise RF). If SWIM detected engine failure is going to break pw_rf, don't change pool map, also don't trigger rebuild. With critical log message to ask administrator to bring back those engines in top priority (just "system start --ranks=xxx", need not to reintegrate those engines). a few functions renamed to avoid confuse - pool_map_find_nodes() -> pool_map_find_ranks() pool_map_find_node_by_rank() -> pool_map_find_dom_by_rank() pool_map_node_nr() -> pool_map_rank_nr() Signed-off-by: Xuezhao Liu <[email protected]> * DAOS-16508 csum: retry a few times on checksum mismatch on update (#15069) Unlike fetch, we return DER_CSUM on update (turned into EIO by dfs) without any retry. We should retry a few times in case it is a transient error. The patch also prints more information about the actual checksum mismatch. Signed-off-by: Johann Lombardi <[email protected]> * DAOS-10877 vos: gang allocation for huge SV (#14790) To avoid allocation failure on a fragmented system, huge SV allocation will be split into multiple smaller allocations, each allocation size is capped to 8MB (the DMA chunk size, that could avoid huge DMA buffer allocation). The address of such scattered SV payload is represented by 'gang address'. Removed io_allocbuf_failure() vos unit test, it's not applicable in gang SV mode now. Signed-off-by: Niu Yawei <[email protected]> * DAOS-16304 tools: Adjust default RPC size for net-test (#15091) The previous default of 1MiB isn't helpful at large scales. Use a default of 1KiB to get faster results and a better balance between raw latency and bandwidth. Also include calculated rpc throughput and bandwidth in JSON output. Signed-off-by: Michael MacDonald <[email protected]> * SRE-2408 ci: Increase timeout (to 15 minutes) for system restore (#14926) Signed-off-by: Tomasz Gromadzki <[email protected]> * DAOS-16251 object: Fix obj_ec_singv_split overflow (#15045) It has been seen that obj_ec_singv_split may read beyond the end of sgl->sg_iovs[0].iov_buf: iod_size=8569 c_bytes=4288 id_shard=0 tgt_off=1 iov_len=8569 iov_buf_len=8569 The memmove read 4288 bytes from offset 4288, whereas the buffer only had 8569 - 4288 = 4281 bytes from offset 4288. This patch fixes the problem by adding the min(...) expression. Signed-off-by: Li Wei <[email protected]> --------- Signed-off-by: Phil Henderson <[email protected]> Signed-off-by: Mohamad Chaarawi <[email protected]> Signed-off-by: Jeff Olivier <[email protected]> Signed-off-by: Li Wei <[email protected]> Signed-off-by: Michael MacDonald <[email protected]> Signed-off-by: Liang Zhen <[email protected]> Signed-off-by: Fan Yong <[email protected]> Signed-off-by: Joseph Moore <[email protected]> Signed-off-by: Makito Kano <[email protected]> Signed-off-by: Dalton Bohning <[email protected]> Signed-off-by: Niu Yawei <[email protected]> Signed-off-by: Xuezhao Liu <[email protected]> Signed-off-by: Johann Lombardi <[email protected]> Signed-off-by: Tomasz Gromadzki <[email protected]> Co-authored-by: Phil Henderson <[email protected]> Co-authored-by: Jeff Olivier <[email protected]> Co-authored-by: Li Wei <[email protected]> Co-authored-by: Michael MacDonald <[email protected]> Co-authored-by: Liang Zhen <[email protected]> Co-authored-by: Nasf-Fan <[email protected]> Co-authored-by: Joseph Moore <[email protected]> Co-authored-by: Makito Kano <[email protected]> Co-authored-by: Dalton Bohning <[email protected]> Co-authored-by: Niu Yawei <[email protected]> Co-authored-by: Liu Xuezhao <[email protected]> Co-authored-by: Johann Lombardi <[email protected]> Co-authored-by: Tomasz Gromadzki <[email protected]>

#15004) #15098 In HDF5, DFS, MPIIO, or POSIX, object class and container properties are defined during the container create. If it’s DFS, object class is also set to the IOR parameter. However, in HDF5-VOL, object class and container properties are defined with the following environment variables of mpirun. HDF5_DAOS_OBJ_CLASS (Object class) HDF5_DAOS_FILE_PROP (Container properties) The infrastructure to set these variables are already there in run_ior_with_pool(). In file_count_test_base.py, pass in the env vars to run_ior_with_pool(env=env) as a dictionary. Object class is the oclass variable. Container properties can be obtained from self.container.properties.value. This fix is discussed in PR #14964. Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_io_sys_admin test_largefilecount test_smallfilecount Required-githooks: true Signed-off-by: Makito Kano <[email protected]>

#15004) (#15098) In HDF5, DFS, MPIIO, or POSIX, object class and container properties are defined during the container create. If it’s DFS, object class is also set to the IOR parameter. However, in HDF5-VOL, object class and container properties are defined with the following environment variables of mpirun. HDF5_DAOS_OBJ_CLASS (Object class) HDF5_DAOS_FILE_PROP (Container properties) The infrastructure to set these variables are already there in run_ior_with_pool(). In file_count_test_base.py, pass in the env vars to run_ior_with_pool(env=env) as a dictionary. Object class is the oclass variable. Container properties can be obtained from self.container.properties.value. This fix is discussed in PR #14964. Signed-off-by: Makito Kano <[email protected]>

shimizukko added 2 commits August 27, 2024 01:08

DAOS-16446 test: Don't set HDF5_DAOS_FILE_PROP because it's causing e…

f3837fd

…rror Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_io_sys_admin Signed-off-by: Makito Kano <[email protected]>

Merge branch 'master' into makito/DAOS-16446

8271d33

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_io_sys_admin

daltonbohning reviewed Aug 27, 2024

View reviewed changes

src/tests/ftest/util/file_count_test_base.py Outdated Show resolved Hide resolved

shimizukko added 3 commits August 28, 2024 01:38

Merge branch 'master' into makito/DAOS-16446

55cff67

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_io_sys_admin

DAOS-16446 test: Cleanup

1ec8f95

Skip-unit-tests: true Skip-fault-injection-test: true Skip-func-hw-test-medium-md-on-ssd: false Test-tag: test_io_sys_admin test_largefilecount test_smallfilecount Signed-off-by: Makito Kano <[email protected]>

shimizukko marked this pull request as ready for review August 31, 2024 00:33

shimizukko requested review from a team as code owners August 31, 2024 00:33

shimizukko requested review from daltonbohning and dinghwah August 31, 2024 00:33

daltonbohning approved these changes Sep 4, 2024

View reviewed changes

dinghwah approved these changes Sep 5, 2024

View reviewed changes

shimizukko requested a review from a team September 5, 2024 20:33

daltonbohning added the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Sep 5, 2024

daltonbohning merged commit 91de313 into master Sep 5, 2024
49 checks passed

daltonbohning deleted the makito/DAOS-16446 branch September 5, 2024 21:59

shimizukko mentioned this pull request Sep 8, 2024

DAOS-16446 test: HDF5-VOL test - Set object class and container prope… #15098

Merged

18 tasks

This was referenced Oct 28, 2024

Merge upstream/release/2.6 into upstream/google/2.6 #15412

Closed

Merge upstream/release/2.6 into upstream/google/2.6 #15414

Closed

Merge upstream/release/2.6 into upstream/google/2.6 #15415

Closed

This was referenced Nov 6, 2024

Merge upstream/release/2.6 into upstream/google/2.6 #15460

Merged

mjmac/DAOS 16787 google 2.6 #15498

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-16446 test: HDF5-VOL test - Set object class and container prope… #15004

DAOS-16446 test: HDF5-VOL test - Set object class and container prope… #15004

shimizukko commented Aug 26, 2024 •

edited

Loading

github-actions bot commented Aug 26, 2024 •

edited

Loading

daosbuild1 commented Aug 26, 2024

daosbuild1 commented Aug 26, 2024

shimizukko commented Aug 31, 2024 •

edited

Loading

DAOS-16446 test: HDF5-VOL test - Set object class and container prope… #15004

DAOS-16446 test: HDF5-VOL test - Set object class and container prope… #15004

Conversation

shimizukko commented Aug 26, 2024 • edited Loading

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Aug 26, 2024 • edited Loading

daosbuild1 commented Aug 26, 2024

daosbuild1 commented Aug 26, 2024

shimizukko commented Aug 31, 2024 • edited Loading

shimizukko commented Aug 26, 2024 •

edited

Loading

github-actions bot commented Aug 26, 2024 •

edited

Loading

shimizukko commented Aug 31, 2024 •

edited

Loading