Mschaara/dfs dcache merge (#15114)

* DAOS-16484 test: Exclude local host in default interface selection (#15049)

When including the local host in the default interface selection a
difference in ib0 speeds will cause the logic to select eth0 and then
the tcp provider.

Signed-off-by: Phil Henderson <[email protected]>

* DAOS-15800 client: create cart context on specific interface (#14804)

Cart has added the ability to select network interface on context creation. The daos_agent also added a numa-fabric map that can be queried at init time. Update the DAOS client to query from the agent a map of numa to network interface on daos_init(), and on EQ creation, select the best interface for the network context based on the numa of the calling thread.

Signed-off-by: Mohamad Chaarawi <[email protected]>

* DAOS-16445 client: Add function to cycle OIDs non-sequentially (#14999)

We've noticed that with sequential order, object placement is poor.

We get 40% fill for 8GiB files with 25 ranks and 16 targets per rank
with EC_2P1G8. With this patch, we get a much better distribution.

This patch adds the following:

1. A function for cycling oid.hi incrementing by a large prime
2. For DFS, randomize the starting value
3. Modify DFS to cycle OIDs using the new function.

Signed-off-by: Jeff Olivier <[email protected]>

* DAOS-16251 dtx: Fix dtx_req_send user-after-free (#15035)

In dtx_req_send, since the crt_req_send releases the req reference, din
may have been freed when dereferenced for the DL_CDEBUG call.

Signed-off-by: Li Wei <[email protected]>

* DAOS-16304 tools: Add daos health net-test command (#14980)

Wrap self_test to provide a simplified network test
to detect obvious client/server connectivity and
performance problems.

Signed-off-by: Michael MacDonald <[email protected]>

* DAOS-16272 dfs: fix get_info returning incorrect oclass (#15048)

If user creates a container without --file-oclass, the get_info call was
returning the default oclass of a directory on daos fs get-attr. Fix
that to properly use the enum types for default scenario.

Signed-off-by: Mohamad Chaarawi <[email protected]>

* DAOS-15863 container: fix a race for container cache (#15038)

* DAOS-15863 container: fix a race for container cache

while destroying a container, cont_child_destroy_one() releases
its own refcount before waiting, if another ULT releases its
refcount, which is the last one, wakes up the waiting ULT and frees
it ds_cont_child straightaway, because no one else has refcount.

When the waiting ULT is waken up, it will try to change the already
freed ds_cont_child.

This patch changes the LRU eviction logic and fixes this race.


Signed-off-by: Liang Zhen <[email protected]>
Signed-off-by: Jeff Olivier <[email protected]>
Co-authored-by: Jeff Olivier <[email protected]>

* DAOS-16471 test: Reduce targets for ioctl_pool_handles.py (#15063)

The dfuse/ioctl_pool_handles.py test is overloading the VM so reduce the number of engine targets.

Signed-off-by: Phil Henderson <[email protected]>

* DAOS-16483 vos: handle empty DTX when vos_tx_end (#15053)

It is possible that the DTX modified nothing when stop currnet backend
transaction. Under such case, we may not generate persistent DTX entry.
Then need to bypass such case before checking on-disk DTX entry status.

The patch makes some clean and removed redundant metrics for committed
DTX entries.

Enhance vos_dtx_deregister_record() to handle GC case.

Signed-off-by: Fan Yong <[email protected]>

* DAOS-16271 mercury: Add patch to avoid seg fault in key resolve. (#15067)

Signed-off-by: Joseph Moore <[email protected]>

* DAOS-16484 test: Support mixed speeds when selecting a default interface (#15050)

Allow selecting a default interface that is running at a different speed
on different hosts.  Primarily this is to support selecting the ib0
interface by default when the launch node has a slower ib0 interface
than the cluster hosts.

Signed-off-by: Phil Henderson <[email protected]>

* DAOS-16446 test: HDF5-VOL test - Set object class and container prope… (#15004)

In HDF5, DFS, MPIIO, or POSIX, object class and container properties are defined
during the container create. If it’s DFS, object class is also set to the IOR
parameter. However, in HDF5-VOL, object class and container properties are
defined with the following environment variables of mpirun.

HDF5_DAOS_OBJ_CLASS (Object class)
HDF5_DAOS_FILE_PROP (Container properties)

The infrastructure to set these variables are already there in run_ior_with_pool().
In file_count_test_base.py, pass in the env vars to run_ior_with_pool(env=env) as a
dictionary. Object class is the oclass variable. Container properties can be
obtained from container -> properties field in the test yaml.

This fix is discussed in PR #14964.

Signed-off-by: Makito Kano <[email protected]>

* DAOS-16447 test: set D_IL_REPORT per test (#15012)

set D_IL_REPORT per test instead of setting defaults values in
utilities. This allows running without it set.

Signed-off-by: Dalton Bohning <[email protected]>

* DAOS-16450 test: auto run dfs tests when dfs is modified (#15017)

Automatically include dfs tests when dfs files are modified in PRs.

Signed-off-by: Dalton Bohning <[email protected]>

* DAOS-16510 cq: update pylint to 3.2.7 (#15072)

update pylint to 3.2.7

Signed-off-by: Dalton Bohning <[email protected]>

* DAOS-16509 test: replace IorTestBase.execute_cmd with run_remote (#15070)

replace usage of IorTestBase.execute_cmd with run_remote

Signed-off-by: Dalton Bohning <[email protected]>

* DAOS-16458 object: fix invalid DRAM access in obj_bulk_transfer (#15026)

For EC object update via CPD RPC, when calculate the bitmap to skip
some iods for current EC data shard, we may input NULL for "*skips"
parameter. It may cause the old logic in obj_get_iods_offs_by_oid()
to generate some undefined DRAM for "skips" bitmap. Such bitmap may
be over-written by others, as to subsequent obj_bulk_transfer() may
be misguided.

The patch also fixes a bug inside obj_bulk_transfer() that cast any
input RPC as UPDATE/FETCH by force.

Signed-off-by: Fan Yong <[email protected]>

* DAOS-16486 object: return proper error on stale pool map (#15064)

Client with stale pool map may try to send RPC to a DOWN target, if the
target was brought DOWN due to faulty NVMe device, the ds_pool_child could
have been stopped on the NVMe faulty reaction, We'd ensure proper error
code is returned for such case.

Signed-off-by: Niu Yawei <[email protected]>

* DAOS-16514 vos: fix coverity issue (#15083)

Fix coverity 2555843 explict null dereferenced.

Signed-off-by: Niu Yawei <[email protected]>

* DAOS-16467 rebuild: add DAOS_POOL_RF ENV for massive failure case (#15037)

* DAOS-16467 rebuild: add DAOS_PW_RF ENV for massive failure case

Allow user to set DAOS_PW_RF as pw_rf (pool wise RF).
If SWIM detected engine failure is going to break pw_rf, don't change
pool map, also don't trigger rebuild.
With critical log message to ask administrator to bring back those
engines in top priority (just "system start --ranks=xxx", need not to
reintegrate those engines).

a few functions renamed to avoid confuse -
pool_map_find_nodes() -> pool_map_find_ranks()
pool_map_find_node_by_rank() -> pool_map_find_dom_by_rank()
pool_map_node_nr() -> pool_map_rank_nr()

Signed-off-by: Xuezhao Liu <[email protected]>

* DAOS-16508 csum: retry a few times on checksum mismatch on update (#15069)

Unlike fetch, we return DER_CSUM on update (turned into EIO by dfs) without any retry.
We should retry a few times in case it is a transient error.

The patch also prints more information about the actual checksum mismatch.

Signed-off-by: Johann Lombardi <[email protected]>

* DAOS-10877 vos: gang allocation for huge SV (#14790)

To avoid allocation failure on a fragmented system, huge SV allocation will
be split into multiple smaller allocations, each allocation size is capped
to 8MB (the DMA chunk size, that could avoid huge DMA buffer allocation).

The address of such scattered SV payload is represented by 'gang address'.

Removed io_allocbuf_failure() vos unit test, it's not applicable in gang
SV mode now.

Signed-off-by: Niu Yawei <[email protected]>

* DAOS-16304 tools: Adjust default RPC size for net-test (#15091)

The previous default of 1MiB isn't helpful at large scales.
Use a default of 1KiB to get faster results and a better
balance between raw latency and bandwidth.

Also include calculated rpc throughput and bandwidth in
JSON output.

Signed-off-by: Michael MacDonald <[email protected]>

* SRE-2408 ci: Increase timeout (to 15 minutes) for system restore (#14926)

Signed-off-by: Tomasz Gromadzki <[email protected]>

* DAOS-16251 object: Fix obj_ec_singv_split overflow (#15045)

It has been seen that obj_ec_singv_split may read beyond the end of
sgl->sg_iovs[0].iov_buf:

    iod_size=8569
    c_bytes=4288
    id_shard=0
    tgt_off=1
    iov_len=8569
    iov_buf_len=8569

The memmove read 4288 bytes from offset 4288, whereas the buffer only
had 8569 - 4288 = 4281 bytes from offset 4288. This patch fixes the
problem by adding the min(...) expression.

Signed-off-by: Li Wei <[email protected]>

---------

Signed-off-by: Phil Henderson <[email protected]>
Signed-off-by: Mohamad Chaarawi <[email protected]>
Signed-off-by: Jeff Olivier <[email protected]>
Signed-off-by: Li Wei <[email protected]>
Signed-off-by: Michael MacDonald <[email protected]>
Signed-off-by: Liang Zhen <[email protected]>
Signed-off-by: Fan Yong <[email protected]>
Signed-off-by: Joseph Moore <[email protected]>
Signed-off-by: Makito Kano <[email protected]>
Signed-off-by: Dalton Bohning <[email protected]>
Signed-off-by: Niu Yawei <[email protected]>
Signed-off-by: Xuezhao Liu <[email protected]>
Signed-off-by: Johann Lombardi <[email protected]>
Signed-off-by: Tomasz Gromadzki <[email protected]>
Co-authored-by: Phil Henderson <[email protected]>
Co-authored-by: Jeff Olivier <[email protected]>
Co-authored-by: Li Wei <[email protected]>
Co-authored-by: Michael MacDonald <[email protected]>
Co-authored-by: Liang Zhen <[email protected]>
Co-authored-by: Nasf-Fan <[email protected]>
Co-authored-by: Joseph Moore <[email protected]>
Co-authored-by: Makito Kano <[email protected]>
Co-authored-by: Dalton Bohning <[email protected]>
Co-authored-by: Niu Yawei <[email protected]>
Co-authored-by: Liu Xuezhao <[email protected]>
Co-authored-by: Johann Lombardi <[email protected]>
Co-authored-by: Tomasz Gromadzki <[email protected]>

ci/gha_functions.sh

@@ -199,7 +199,7 @@ provision_cluster() {
         while [ $((SECONDS-START)) -lt $wait_seconds ]; do
             if clush -B -S -l root -w "$nodestring" '[ -d /var/chef/reports ]'; then
                 # shellcheck disable=SC2016
-                clush -B -S -l root -w "$nodestring" --connect_timeout 30 --command_timeout 600 "if [ -e /root/job_info ]; then
+                clush -B -S -l root -w "$nodestring" --connect_timeout 30 --command_timeout 900 "if [ -e /root/job_info ]; then
                         cat /root/job_info
                     fi
                     echo \"Last provisioning run info:

docs/admin/env_variables.md

@@ -53,6 +53,7 @@ Environment variables in this section only apply to the server side.
 |DAOS\_DTX\_RPC\_HELPER\_THD|DTX RPC helper threshold. The valid range is [18, unlimited). The default value is 513.|
 |DAOS\_DTX\_BATCHED\_ULT\_MAX|The max count of DTX batched commit ULTs. The valid range is [0, unlimited). 0 means to commit DTX synchronously. The default value is 32.|
 |DAOS\_FORWARD\_NEIGHBOR|Set to enable I/O forwarding on neighbor xstream in the absence of helper threads.|
+|DAOS\_POOL\_RF|Redundancy factor for the pool. The valid range is [1, 4]. The default value is 2.|
 
 ## Server and Client environment variables
 

docs/admin/pool_operations.md

@@ -916,6 +916,30 @@ and possibly repair a pmemobj file. As discussed in the previous section, the
 rebuild status can be consulted via the pool query and will be expanded
 with more information.
 
+## Pool Redundancy Factor
+
+If the DAOS system experiences cascading failures, where the number of failed
+fault domains exceeds a pool's redundancy factor, there could be unrecoverable
+errors and applications could suffer from data loss. This can happen in cases
+of power or network outages and would cause node/engine failures. In most cases
+those failures can be recovered and DAOS engines can be restarted and the system
+can function again.
+
+Administrator can set the default pool redundancy factor by environment variable
+"DAOS_POOL_RF" in the server yaml file. If SWIM detects and reports an engine is
+dead and the number of failed fault domain exceeds or is going to exceed the pool
+redundancy factor, it will not change pool map immediately. Instead, it will give
+critical log message:
+intolerable unavailability: engine rank x
+In this case, the system administrator should check and try to recover those
+failed engines and bring them back with:
+dmg system start --ranks=x
+one by one. A reintegrate call is not needed.
+
+For true unrecoverable failures, the administrator can still exclude engines.
+However, data loss is expected as the number of unrecoverable failures exceeds
+the pool redundancy factor.
+
 ## Recovering Container Ownership
 
 Typically users are expected to manage their containers. However, in the event

src/bio/bio_buffer.c

@@ -34,11 +34,6 @@ dma_alloc_chunk(unsigned int cnt)
 
 	D_ASSERT(bytes > 0);
 
-	if (DAOS_FAIL_CHECK(DAOS_NVME_ALLOCBUF_ERR)) {
-		D_ERROR("Injected DMA buffer allocation error.\n");
-		return NULL;
-	}
-
 	D_ALLOC_PTR(chunk);
 	if (chunk == NULL) {
 		return NULL;
@@ -848,6 +843,7 @@ dma_map_one(struct bio_desc *biod, struct bio_iov *biov, void *arg)
 		bio_iov_set_raw_buf(biov, NULL);
 		return 0;
 	}
+	D_ASSERT(!BIO_ADDR_IS_GANG(&biov->bi_addr));
 
 	if (direct_scm_access(biod, biov)) {
 		struct umem_instance *umem = biod->bd_umem;

src/bio/bio_bulk.c

@@ -1,5 +1,5 @@
 /**
- * (C) Copyright 2021-2022 Intel Corporation.
+ * (C) Copyright 2021-2024 Intel Corporation.
  *
  * SPDX-License-Identifier: BSD-2-Clause-Patent
  */
@@ -640,6 +640,7 @@ bulk_map_one(struct bio_desc *biod, struct bio_iov *biov, void *data)
 		goto done;
 	}
 	D_ASSERT(!BIO_ADDR_IS_DEDUP(&biov->bi_addr));
+	D_ASSERT(!BIO_ADDR_IS_GANG(&biov->bi_addr));
 
 	hdl = bulk_get_hdl(biod, biov, roundup_pgs(pg_cnt), pg_off, arg);
 	if (hdl == NULL) {

src/bio/bio_xstream.c

@@ -30,7 +30,6 @@
 /* SPDK blob parameters */
 #define DAOS_BS_CLUSTER_SZ	(1ULL << 25)	/* 32MB */
 /* DMA buffer parameters */
-#define DAOS_DMA_CHUNK_MB	8	/* 8MB DMA chunks */
 #define DAOS_DMA_CHUNK_CNT_INIT	24	/* Per-xstream init chunks, 192MB */
 #define DAOS_DMA_CHUNK_CNT_MAX	128	/* Per-xstream max chunks, 1GB */
 #define DAOS_DMA_CHUNK_CNT_MIN	32	/* Per-xstream min chunks, 256MB */
@@ -207,7 +206,7 @@ bio_nvme_init(const char *nvme_conf, int numa_node, unsigned int mem_size,
 {
 	char		*env;
 	int		 rc, fd;
-	unsigned int	 size_mb = DAOS_DMA_CHUNK_MB;
+	unsigned int	 size_mb = BIO_DMA_CHUNK_MB;
 
 	if (tgt_nr <= 0) {
 		D_ERROR("tgt_nr: %u should be > 0\n", tgt_nr);

src/cart/README.env

@@ -155,7 +155,7 @@ This file lists the environment variables used in CaRT.
 
  . CRT_CTX_NUM
    If set, specifies the limit of number of allowed CaRT contexts to be created.
-   Valid range is [1, 64], with default being 64 if unset.
+   Valid range is [1, 128], with default being 128 if unset.
 
  . D_FI_CONFIG
    Specifies the fault injection configuration file. If this variable is not set

src/cart/crt_internal_types.h

@@ -14,7 +14,7 @@
 #define CRT_CONTEXT_NULL         (NULL)
 
 #ifndef CRT_SRV_CONTEXT_NUM
-#define CRT_SRV_CONTEXT_NUM (64)	/* Maximum number of contexts */
+#define CRT_SRV_CONTEXT_NUM (128) /* Maximum number of contexts */
 #endif
 
 

src/chk/chk_engine.c

@@ -668,7 +668,7 @@ chk_engine_pool_mbs_one(struct chk_pool_rec *cpr, struct pool_map *map, struct c
 	int			 rc = 0;
 	bool			 unknown;
 
-	dom = pool_map_find_node_by_rank(map, mbs->cpm_rank);
+	dom = pool_map_find_dom_by_rank(map, mbs->cpm_rank);
 	if (dom == NULL) {
 		D_ASSERT(mbs->cpm_rank != dss_self_rank());
 
@@ -777,7 +777,7 @@ chk_engine_find_dangling_pm(struct chk_pool_rec *cpr, struct pool_map *map)
 	int			 j;
 	bool			 down;
 
-	rank_nr = pool_map_find_nodes(map, PO_COMP_ID_ALL, &doms);
+	rank_nr = pool_map_find_ranks(map, PO_COMP_ID_ALL, &doms);
 	if (rank_nr <= 0)
 		D_GOTO(out, rc = rank_nr);
 

src/client/api/SConscript

@@ -17,7 +17,7 @@ def scons():
 
     if prereqs.client_requested():
         libdaos = env.d_library('daos', libdaos_tgts, SHLIBVERSION=API_VERSION,
-                                LIBS=['daos_common'])
+                                LIBS=['daos_common', 'numa'])
         if hasattr(env, 'InstallVersionedLib'):
             env.InstallVersionedLib('$PREFIX/lib64/', libdaos, SHLIBVERSION=API_VERSION)
         else:

src/client/api/event.c

@@ -83,11 +83,24 @@ daos_eq_lib_init(crt_init_options_t *crt_info)
 		D_GOTO(unlock, rc);
 	}
 
-	/* use a global shared context for all eq for now */
-	rc = crt_context_create(&daos_eq_ctx);
+	if (d_dynamic_ctx_g) {
+		char iface[DAOS_SYS_INFO_STRING_MAX];
+
+		rc = dc_mgmt_get_iface(&iface[0]);
+		if (rc && rc != -DER_NONEXIST) {
+			D_ERROR("failed to get iface: " DF_RC "\n", DP_RC(rc));
+			D_GOTO(crt, rc);
+		}
+		/** if no interface returned, use the default */
+		if (rc == -DER_NONEXIST)
+			rc = crt_context_create(&daos_eq_ctx);
+		else
+			rc = crt_context_create_on_iface(iface, &daos_eq_ctx);
+	} else {
+		rc = crt_context_create(&daos_eq_ctx);
+	}
 	if (rc != 0) {
-		D_ERROR("failed to create client context: "DF_RC"\n",
-			DP_RC(rc));
+		D_ERROR("failed to create client context: " DF_RC "\n", DP_RC(rc));
 		D_GOTO(crt, rc);
 	}
 
@@ -656,7 +669,23 @@ daos_eq_create(daos_handle_t *eqh)
 
 	eqx = daos_eq2eqx(eq);
 
-	rc = crt_context_create(&eqx->eqx_ctx);
+	if (d_dynamic_ctx_g) {
+		char iface[DAOS_SYS_INFO_STRING_MAX];
+
+		rc = dc_mgmt_get_iface(&iface[0]);
+		if (rc && rc != -DER_NONEXIST) {
+			D_ERROR("failed to get iface: " DF_RC "\n", DP_RC(rc));
+			return rc;
+		}
+
+		/** if no interface returned, use the default */
+		if (rc == -DER_NONEXIST)
+			rc = crt_context_create(&eqx->eqx_ctx);
+		else
+			rc = crt_context_create_on_iface(iface, &eqx->eqx_ctx);
+	} else {
+		rc = crt_context_create(&eqx->eqx_ctx);
+	}
 	if (rc) {
 		D_WARN("Failed to create CART context; using the global one, "DF_RC"\n", DP_RC(rc));
 		eqx->eqx_ctx = daos_eq_ctx;

src/client/dfs/dfs_internal.h

@@ -99,9 +99,6 @@
 /** Max recursion depth for symlinks */
 #define DFS_MAX_RECURSION  40
 
-/** MAX value for the HI OID */
-#define MAX_OID_HI         ((1UL << 32) - 1)
-
 /* Default power2(bits) size of dir-cache */
 #define DCACHE_SIZE_BITS      16
 /** Size of the hash key prefix */
@@ -196,6 +193,8 @@ struct dfs {
 	daos_handle_t        coh;
 	/** refcount on cont handle that through the DFS API */
 	uint32_t             coh_refcount;
+	/** The last oid.hi in the sequence */
+	uint32_t             last_hi;
 	/** Transaction handle epoch. DAOS_EPOCH_MAX for DAOS_TX_NONE */
 	daos_epoch_t         th_epoch;
 	/** Transaction handle */
@@ -377,20 +376,24 @@ oid_gen(dfs_t *dfs, daos_oclass_id_t oclass, bool file, daos_obj_id_t *oid)
 
 	D_MUTEX_LOCK(&dfs->lock);
 	/** If we ran out of local OIDs, alloc one from the container */
-	if (dfs->oid.hi >= MAX_OID_HI) {
+	if (dfs->oid.hi == dfs->last_hi) {
 		/** Allocate an OID for the namespace */
 		rc = daos_cont_alloc_oids(dfs->coh, 1, &dfs->oid.lo, NULL);
 		if (rc) {
 			D_ERROR("daos_cont_alloc_oids() Failed (%d)\n", rc);
 			D_MUTEX_UNLOCK(&dfs->lock);
 			return daos_der2errno(rc);
 		}
-		dfs->oid.hi = 0;
+		/** Start such that dfs->last_hi will be final value */
+		dfs->oid.hi = dfs->last_hi;
 	}
 
 	/** set oid and lo, bump the current hi value */
 	oid->lo = dfs->oid.lo;
-	oid->hi = dfs->oid.hi++;
+	daos_obj_oid_cycle(&dfs->oid);
+	if (unlikely(dfs->oid.lo == RESERVED_LO && dfs->oid.hi <= 1))
+		daos_obj_oid_cycle(&dfs->oid); /* Avoid reserved oids */
+	oid->hi = dfs->oid.hi;
 	D_MUTEX_UNLOCK(&dfs->lock);
 
 	/** if a regular file, use UINT64 typed dkeys for the array object */

src/client/dfs/mnt.c

@@ -685,20 +685,20 @@ dfs_mount(daos_handle_t poh, daos_handle_t coh, int flags, dfs_t **_dfs)
 
 	/** if RW, allocate an OID for the namespace */
 	if (amode == O_RDWR) {
+		dfs->last_hi = (unsigned int)d_rand();
+		/** Avoid potential conflict with SB or ROOT */
+		if (dfs->last_hi <= 1)
+			dfs->last_hi = 2;
+
 		rc = daos_cont_alloc_oids(coh, 1, &dfs->oid.lo, NULL);
 		if (rc) {
 			D_ERROR("daos_cont_alloc_oids() Failed, " DF_RC "\n", DP_RC(rc));
 			D_GOTO(err_root, rc = daos_der2errno(rc));
 		}
 
-		/*
-		 * if this is the first time we allocate on this container,
-		 * account 0 for SB, 1 for root obj.
-		 */
-		if (dfs->oid.lo == RESERVED_LO)
-			dfs->oid.hi = ROOT_HI + 1;
-		else
-			dfs->oid.hi = 0;
+		dfs->oid.hi = dfs->last_hi;
+		/** Increment so that dfs->last_hi is the last value */
+		daos_obj_oid_cycle(&dfs->oid);
 	}
 
 	dfs->mounted = DFS_MOUNT;
@@ -1045,7 +1045,7 @@ dfs_global2local(daos_handle_t poh, daos_handle_t coh, int flags, d_iov_t glob,
 
 	/** allocate a new oid on the next file or dir creation */
 	dfs->oid.lo = 0;
-	dfs->oid.hi = MAX_OID_HI;
+	dfs->oid.hi = dfs->last_hi;
 
 	rc = D_MUTEX_INIT(&dfs->lock, NULL);
 	if (rc != 0) {

src/client/dfs/obj.c

@@ -86,15 +86,14 @@ dfs_obj_get_info(dfs_t *dfs, dfs_obj_t *obj, dfs_obj_info_t *info)
 			if (dfs->attr.da_dir_oclass_id)
 				info->doi_dir_oclass_id = dfs->attr.da_dir_oclass_id;
 			else
-				rc = daos_obj_get_oclass(dfs->coh, 0, 0, 0,
+				rc = daos_obj_get_oclass(dfs->coh, DAOS_OT_MULTI_HASHED, 0, 0,
 							 &info->doi_dir_oclass_id);
 
 			if (dfs->attr.da_file_oclass_id)
 				info->doi_file_oclass_id = dfs->attr.da_file_oclass_id;
 			else
-				rc = daos_obj_get_oclass(dfs->coh, 0, 0, 0,
+				rc = daos_obj_get_oclass(dfs->coh, DAOS_OT_ARRAY_BYTE, 0, 0,
 							 &info->doi_file_oclass_id);
-
 			if (rc) {
 				D_ERROR("daos_obj_get_oclass() failed " DF_RC "\n", DP_RC(rc));
 				return daos_der2errno(rc);

src/common/checksum.c

@@ -278,6 +278,10 @@ daos_csummer_compare_csum_info(struct daos_csummer *obj,
 		match = daos_csummer_csum_compare(obj, ci_idx2csum(a, i),
 						  ci_idx2csum(b, i),
 						  a->cs_len);
+		if (unlikely(!match))
+			D_ERROR("Checksum mismatch at index %d/%d "DF_CI_BUF" != "DF_CI_BUF"\n", i,
+				a->cs_nr, DP_CI_BUF(ci_idx2csum(a, i), a->cs_len),
+				DP_CI_BUF(ci_idx2csum(b, i), b->cs_len));
 	}
 
 	return match;