DAOS-14261 engine: Add dss_chore for I/O forwarding #13372

liw · 2023-11-22T03:04:04Z

As requested by the Jira ticket, add a new I/O forwarding mechanism,
dss_chore, to avoid creating a ULT for every forwarding task.

Forwarding of object I/O and DTX RPCs is converted to chores.
Cancelation is not implemented, because the I/O forwarding tasks
themselves do not support cancelation yet.
In certain engine configurations, some xstreams do not need to
initialize dx_chore_queue. This is left to future work.

Before requesting gatekeeper:

Two review approvals and any prior change requests have been resolved.
Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
Commit messages follows the guidelines outlined here.
Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

github-actions · 2023-11-22T03:04:21Z

Bug-tracker data:
Ticket title is 'limit ULT creation on helper xstream'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-14261

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-11-22T03:09:10Z

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/1/execution/node/381/log

daosbuild1 · 2023-11-22T03:10:31Z

Test stage Build RPM on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/1/execution/node/368/log

daosbuild1 · 2023-11-22T03:12:09Z

Test stage Build RPM on Leap 15.4 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/1/execution/node/356/log

daosbuild1 · 2023-11-22T03:12:48Z

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/1/execution/node/450/log

daosbuild1 · 2023-11-22T03:13:58Z

Test stage Build DEB on Ubuntu 20.04 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/1/execution/node/315/log

daosbuild1 · 2023-11-22T03:17:56Z

Test stage Build on Leap 15.4 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/1/execution/node/543/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-11-22T04:19:45Z

Test stage NLT on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/3/execution/node/839/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-11-22T16:52:36Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/4/execution/node/1377/log

daosbuild1 · 2023-11-22T18:11:19Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/4/execution/node/1331/log

daosbuild1 · 2023-11-22T18:20:06Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/4/execution/node/1519/log

daosbuild1 · 2023-11-22T22:30:44Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/4/execution/node/1473/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-11-29T10:32:10Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/5/execution/node/1331/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-12-01T23:27:27Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/9/execution/node/352/log

daosbuild1 · 2023-12-03T12:36:08Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/9/execution/node/448/log

daosbuild1 · 2023-12-03T20:38:35Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/9/execution/node/564/log

daosbuild1 · 2023-12-18T03:17:01Z

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13372/14/testReport/

NiuYawei · 2023-12-19T02:34:49Z

src/engine/srv.c

@@ -773,6 +787,8 @@ dss_start_one_xstream(hwloc_cpuset_t cpus, int tag, int xs_id)
 	} else {
 		dx->dx_main_xs	= (xs_id >= dss_sys_xs_nr) && (xs_offset == 0);
 	}
+	/* See the DSS_XS_IOFW case in sched_ult2xs. */
+	dx->dx_iofw = xs_id >= dss_sys_xs_nr && (!dx->dx_main_xs || dss_tgt_offload_xs_nr == 0);


minor: Seems you don't regard vos xstream as iofw xstream, so it could be simplified as "dx->dx_iofw = xs_id >= dss_sys_xs_nr && !dx->dx_main_xs".

@NiuYawei, I thought about the same when updating the patch last time. Ended up keeping it this way because if we make such a change to dx_iofw, then a dx_iofw xstream != a DSS_XS_IOFW xstream, potentially making our terminology even more complex (i.e., we currently talk about "I/O forward xstream", "offload xstream", "helper xstream", "DSS_XS_OFFLOAD xstream", etc.---already too confusing).

liw

@Nasf-Fan, thanks for the replies.

liw · 2023-12-19T04:50:08Z

src/engine/ult.c

+			D_ASSERT(chore->cho_status != DSS_CHORE_NEW);
+			if (chore->cho_status == DSS_CHORE_DONE)
+				d_list_del_init(&chore->cho_link);
+			ABT_thread_yield();


No, list is not cleared before the splice. Have I overlooked anything?

liw · 2023-12-19T04:52:44Z

src/dtx/dtx_rpc.c

+		}
+		rc = dss_chore_delegate(&dca->dca_chore, dtx_rpc_helper);
+	} else {
+		dss_chore_diy(&dca->dca_chore, dtx_rpc_helper);


I prefer not to merge diy with delegate, if that's acceptable to you.

daosbuild1 · 2023-12-24T11:22:20Z

Test stage Functional Hardware Medium completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-13372/15/testReport/

Nasf-Fan · 2024-01-08T02:46:49Z

dtx_common.c and dtx_rpc.c have been much changed recently, need some merge work for this patch.

Required-githooks: true

Signed-off-by: Li Wei <[email protected]> Required-githooks: true

daosbuild1 · 2024-01-18T19:42:17Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/16/execution/node/1387/log

Signed-off-by: Li Wei <[email protected]> Required-githooks: true

daosbuild1 · 2024-01-19T12:22:44Z

Test stage Functional Hardware Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-13372/16/execution/node/1574/log

Signed-off-by: Li Wei <[email protected]> Required-githooks: true

Required-githooks: true

Nasf-Fan · 2024-02-02T14:14:21Z

src/dtx/dtx_rpc.c

-	else
-		rc = dtx_rpc_internal(dca);
+	if (dss_has_enough_helper()) {
+		rc = ABT_eventual_create(0, &dca->dca_chore_eventual);


Why non-coll case need the eventual, but coll case does not?

Is it possible to not create the dca_chore_eventual, instead, depends on the dra->dra_future to check whether related operation has been done or not, then non-coll case and coll case can share the same mechanism or logic.

Why non-coll case need the eventual, but coll case does not?

@Nasf-Fan, the dca_chore_eventual is for dtx_rpc_post to know when it's safe to access dra_future.

Is it possible to not create the dca_chore_eventual, instead, depends on the dra->dra_future to check whether related operation has been done or not, then non-coll case and coll case can share the same mechanism or logic.

AFAICS, it's possible, though not easy. The main trouble is that the number of slots in the future is computed by the chore, so we can't create the future until the chore has run to the point where that number is computed. Also, even if we really want to make such a change in this PR, it will take more changes to share the mechanism with dtx_coll_rpc_helper, since the two code paths use different data structures.

So, what do you think?

Why non-coll case need the eventual, but coll case does not?

@Nasf-Fan, the dca_chore_eventual is for dtx_rpc_post to know when it's safe to access dra_future.

Is it possible to not create the dca_chore_eventual, instead, depends on the dra->dra_future to check whether related operation has been done or not, then non-coll case and coll case can share the same mechanism or logic.

AFAICS, it's possible, though not easy. The main trouble is that the number of slots in the future is computed by the chore, so we can't create the future until the chore has run to the point where that number is computed. Also, even if we really want to make such a change in this PR, it will take more changes to share the mechanism with dtx_coll_rpc_helper, since the two code paths use different data structures.

So, what do you think?

Then ignore my former comment and go ahead.

Features: tx Required-githooks: true

Nasf-Fan · 2024-02-19T05:38:36Z

src/dtx/dtx_rpc.c

-	else
-		rc = dtx_rpc_internal(dca);
+	if (dss_has_enough_helper()) {
+		rc = ABT_eventual_create(0, &dca->dca_chore_eventual);


Why non-coll case need the eventual, but coll case does not?

@Nasf-Fan, the dca_chore_eventual is for dtx_rpc_post to know when it's safe to access dra_future.

Is it possible to not create the dca_chore_eventual, instead, depends on the dra->dra_future to check whether related operation has been done or not, then non-coll case and coll case can share the same mechanism or logic.

AFAICS, it's possible, though not easy. The main trouble is that the number of slots in the future is computed by the chore, so we can't create the future until the chore has run to the point where that number is computed. Also, even if we really want to make such a change in this PR, it will take more changes to share the mechanism with dtx_coll_rpc_helper, since the two code paths use different data structures.

So, what do you think?

Then ignore my former comment and go ahead.

github-actions · 2024-02-19T08:18:55Z

Bug-tracker data:
Ticket title is 'limit ULT creation on helper xstream'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-14261

Features: tx Required-githooks: true

github-actions · 2024-03-19T04:19:11Z

Ticket title is 'limit ULT creation on helper xstream'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-14261

As requested by the Jira ticket, add a new I/O forwarding mechanism, dss_chore, to avoid creating a ULT for every forwarding task. - Forwarding of object I/O and DTX RPCs is converted to chores. - Cancelation is not implemented, because the I/O forwarding tasks themselves do not support cancelation yet. - In certain engine configurations, some xstreams do not need to initialize dx_chore_queue. This is left to future work. Required-githooks: true Skipped-githooks: clang Change-Id: I8d6f9889f5562a8bc3683d26cb830672a8aa40f3 Signed-off-by: Li Wei <[email protected]>

As requested by the Jira ticket, add a new I/O forwarding mechanism, dss_chore, to avoid creating a ULT for every forwarding task. - Forwarding of object I/O and DTX RPCs is converted to chores. - Cancelation is not implemented, because the I/O forwarding tasks themselves do not support cancelation yet. - In certain engine configurations, some xstreams do not need to initialize dx_chore_queue. This is left to future work. Signed-off-by: Li Wei <[email protected]>

daosbuild1 reviewed Nov 22, 2023

View reviewed changes

liw force-pushed the liw/helper-ult branch from bb4c73a to 7b37f1c Compare November 22, 2023 03:41

daosbuild1 reviewed Nov 22, 2023

View reviewed changes

liw force-pushed the liw/helper-ult branch from 7b37f1c to 68dc394 Compare November 22, 2023 03:43

daosbuild1 reviewed Nov 22, 2023

View reviewed changes

liw force-pushed the liw/helper-ult branch from 68dc394 to 98cdd04 Compare November 22, 2023 05:51

daosbuild1 reviewed Nov 22, 2023

View reviewed changes

liw force-pushed the liw/helper-ult branch from 98cdd04 to dfda532 Compare November 29, 2023 05:33

daosbuild1 reviewed Nov 29, 2023

View reviewed changes

liw force-pushed the liw/helper-ult branch 2 times, most recently from a4658c2 to 3325b2a Compare November 30, 2023 02:58

daosbuild1 reviewed Nov 30, 2023

View reviewed changes

liw force-pushed the liw/helper-ult branch from 3325b2a to 7c68a86 Compare November 30, 2023 06:26

daosbuild1 reviewed Nov 30, 2023

View reviewed changes

liw requested a review from Nasf-Fan December 18, 2023 01:20

NiuYawei previously approved these changes Dec 19, 2023

View reviewed changes

liw commented Dec 19, 2023

View reviewed changes

liw added 2 commits January 17, 2024 17:08

Merge branch 'master' into liw/helper-ult

0e34213

Required-githooks: true

Convert dtx_coll_rpc_helper to dss_chore

e29ef9c

Signed-off-by: Li Wei <[email protected]> Required-githooks: true

liw dismissed NiuYawei’s stale review via e29ef9c January 18, 2024 05:44

Add debug

1756df3

Signed-off-by: Li Wei <[email protected]> Required-githooks: true

liw added 2 commits January 23, 2024 14:59

Fix performance regression

9a08623

Signed-off-by: Li Wei <[email protected]> Required-githooks: true

Merge branch 'master' into liw/helper-ult

fd39efa

Required-githooks: true

Nasf-Fan reviewed Feb 2, 2024

View reviewed changes

Nasf-Fan self-requested a review February 2, 2024 14:20

Merge branch 'master' into liw/helper-ult

5775432

Features: tx Required-githooks: true

Nasf-Fan approved these changes Feb 19, 2024

View reviewed changes

liw requested review from a team as code owners February 19, 2024 08:18

Merge branch 'master' into liw/helper-ult

e7a80ec

Features: tx Required-githooks: true

liw requested a review from NiuYawei March 19, 2024 04:19

NiuYawei approved these changes Mar 19, 2024

View reviewed changes

liw requested a review from a team March 21, 2024 23:53

NiuYawei merged commit 9351bd2 into master Mar 22, 2024
48 of 49 checks passed

NiuYawei deleted the liw/helper-ult branch March 22, 2024 01:28

DAOS-14261 engine: Add dss_chore for I/O forwarding #13372

DAOS-14261 engine: Add dss_chore for I/O forwarding #13372

Conversation

liw commented Nov 22, 2023 • edited Loading

Before requesting gatekeeper:

Gatekeeper:

github-actions bot commented Nov 22, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 22, 2023

daosbuild1 commented Nov 22, 2023

daosbuild1 commented Nov 22, 2023

daosbuild1 commented Nov 22, 2023

daosbuild1 commented Nov 22, 2023

daosbuild1 commented Nov 22, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 22, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 22, 2023

daosbuild1 commented Nov 22, 2023

daosbuild1 commented Nov 22, 2023

daosbuild1 commented Nov 22, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Nov 29, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Dec 1, 2023

daosbuild1 commented Dec 3, 2023

daosbuild1 commented Dec 3, 2023

daosbuild1 commented Dec 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daosbuild1 commented Dec 24, 2023

Nasf-Fan commented Jan 8, 2024

daosbuild1 commented Jan 18, 2024

daosbuild1 commented Jan 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 19, 2024

github-actions bot commented Mar 19, 2024

liw commented Nov 22, 2023 •

edited

Loading