DAOS-13658 dfuse: Add filesystem query command. #12367

ashleypittman · 2023-06-12T16:27:43Z

Use atomics to track number of pool and containers open, as well
as inodes and file handles.

Required-githooks: true

Signed-off-by: Ashley Pittman [email protected]

Use atomics to track number of pool and containers open, as well as inodes and file handles. Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

github-actions · 2023-06-12T17:48:17Z

Bug-tracker data:
Ticket title is 'Add tracking of memory consumption into dfuse.'
Status is 'In Progress'
Labels: 'triaged'
https://daosio.atlassian.net/browse/DAOS-13658

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

Test-tag: dfuse Required-githooks: true

daosbuild1

LGTM. No errors found by checkpatch.

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-06-18T00:55:36Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12367/8/execution/node/1243/log

daosbuild1 · 2023-06-18T01:11:24Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12367/8/execution/node/1290/log

daosbuild1

LGTM. No errors found by checkpatch.

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1 · 2023-06-19T16:47:29Z

Test stage checkpatch completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12367/12/execution/node/146/log

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1 · 2023-06-19T16:58:27Z

Test stage checkpatch completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12367/13/execution/node/146/log

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-06-20T14:43:33Z

Test stage Unit Test on EL 8 completed with status UNSTABLE. https://build.hpdd.intel.com/job/daos-stack/job/daos//view/change-requests/job/PR-12367/16/testReport/

Required-githooks: true

daosbuild1

LGTM. No errors found by checkpatch.

src/client/dfuse/ops/ioctl.c

daltonbohning · 2023-06-26T16:12:12Z

src/control/cmd/daos/filesystem.go

+type fsDfuseQueryCmd struct {
+	daosCmd
+
+	Path string `long:"path" description:"Path to evict from dfuse" required:"1"`


The command is memquery, but this says Path to evict?

Will update.

daltonbohning · 2023-06-26T16:16:16Z

src/include/dfuse_ioctl.h

+struct dfuse_mem_query {
+	uint64_t inode_count;
+	uint64_t fh_count;
+	uint64_t pool_count;
+	uint64_t container_count;
+	ino_t    ino;
+	bool     found;
+};


Should we future-proof ourselves by adding a version to this struct? Just thinking about what would happen if middlware / third-party compiled with this version of the struct, then recompiled a new version of DAOS without recompiling middleware. @mchaarawi I remember we ran into something similar with DFS before. Thoughts?

It would be possible to but I think we'd just switch to a new cmd number if we did that, ioctls are integer commands so don't have the same issues as c functions which are resolved by name.

daltonbohning · 2023-06-26T16:17:34Z

utils/node_local_test.py

@@ -2563,6 +2563,13 @@ def test_uns_create(self):
        print(stbuf)
        assert stbuf.st_ino < 100
        print(os.listdir(path))
+        rc = run_daos_cmd(self.conf, ['filesystem', 'memquery', '--path', self.dfuse.dir])


Should we support positional?

filesystem memquery <path>

Probably, I'll speak to @tanabarr about how to add that.

ashleypittman · 2023-06-29T08:54:51Z

Here's the help text with the lastest push to this PR.
`
(wolf-63) [08:51] [0] ampittma@wolf-63:~/coral/daos $ daos filesystem memquery --help
Usage:
daos [OPTIONS] filesystem memquery [path]

Application Options:
--debug enable debug output
--verbose enable verbose output (when applicable)
-j, --json enable JSON output

Help Options:
-h, --help Show this help message

[memquery command arguments]
path: DFuse path to query
(wolf-63) [08:53] [0] ampittma@wolf-63:/coral/daos $ daos filesystem memquery
ERROR: daos: the required argument path was not provided
(wolf-63) [08:53] [1] ampittma@wolf-63:~/coral/daos $
`

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

mchaarawi · 2023-06-30T13:15:51Z

utils/node_local_test.py

+        rc = run_daos_cmd(self.conf, ['filesystem', 'memquery', self.dfuse.dir])
+        print(rc)
+        assert rc.returncode == 0
+        rc = run_daos_cmd(self.conf, ['filesystem', 'memquery', self.dfuse.dir], use_json=True)
+        print(rc)
+        assert rc.returncode == 0


i'd like to see a real test here that actually verifies the collected query stats.

mchaarawi · 2023-06-30T13:23:53Z

src/client/dfuse/dfuse.h

+	ATOMIC uint64_t      di_inode_count;
+	ATOMIC uint64_t      di_fh_count;
+	ATOMIC uint64_t      di_pool_count;
+	ATOMIC uint64_t      di_container_count;


i don't think memquery really associate with these counters. a user would think this tool is actually printing memory usage not those counters. so the tool name probably makes more sense to be called stat or probably just query

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

Required-githooks: true

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1

LGTM. No errors found by checkpatch.

daosbuild1 · 2023-06-30T21:12:47Z

Test stage Functional Hardware Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-12367/20/execution/node/1271/log

Multiple cherry-picks to add daos fs query feature for fuse statistics. Helpful for understanding and tuning DAOS performance when dfuse is used. DAOS-13625 dfuse: Merge the info and projection_info structs. (#11881) DAOS-13658 dfuse: Add filesystem query command. (#12367) DAOS-12751 control: Add a daos filesystem evict command. (#12331) DAOS-12751 dfuse: Improve evict command. (#12633) DAOS-13625 dfuse: Remove dfuse_projection_info entirely. (#12796) DAOS-13625 dfuse: Replace fs_handle with dfuse_info. (#12894) DAOS-13625 dfuse: Add core inode_lookup() and inode_decref() functions. (#12573) DAOS-14411 dfuse: Add per-container statistics. (#12819) Signed-off-by: Ashley Pittman <[email protected]>

Multiple cherry-picks to add daos fs query feature for fuse statistics. Helpful for understanding and tuning DAOS performance when dfuse is used. DAOS-13625 dfuse: Merge the info and projection_info structs. (#11881) DAOS-13658 dfuse: Add filesystem query command. (#12367) DAOS-12751 control: Add a daos filesystem evict command. (#12331) DAOS-12751 dfuse: Improve evict command. (#12633) DAOS-13625 dfuse: Remove dfuse_projection_info entirely. (#12796) DAOS-13625 dfuse: Replace fs_handle with dfuse_info. (#12894) DAOS-13625 dfuse: Add core inode_lookup() and inode_decref() functions. (#12573) DAOS-14411 dfuse: Add per-container statistics. (#12819) Features: dfuse Required-githooks: true Change-Id: I8ae3cc743697c2434ae0d54b382ee6c585a3b033 Signed-off-by: Ashley Pittman <[email protected]> Signed-off-by: Jeff Olivier <[email protected]>

Multiple cherry-picks to add daos fs query feature for fuse statistics. Helpful for understanding and tuning DAOS performance when dfuse is used. DAOS-13625 dfuse: Merge the info and projection_info structs. (#11881) DAOS-13658 dfuse: Add filesystem query command. (#12367) DAOS-12751 control: Add a daos filesystem evict command. (#12331) DAOS-12751 dfuse: Improve evict command. (#12633) DAOS-13625 dfuse: Remove dfuse_projection_info entirely. (#12796) DAOS-13625 dfuse: Replace fs_handle with dfuse_info. (#12894) DAOS-13625 dfuse: Add core inode_lookup() and inode_decref() functions. (#12573) DAOS-14411 dfuse: Add per-container statistics. (#12819) DAOS-14411 control: Expose dfuse statistics as yaml. (#13876) Features: dfuse Required-githooks: true Change-Id: I8ae3cc743697c2434ae0d54b382ee6c585a3b033 Signed-off-by: Ashley Pittman <[email protected]> Signed-off-by: Jeff Olivier <[email protected]>

Multiple cherry-picks to add daos fs query feature for fuse statistics. Helpful for understanding and tuning DAOS performance when dfuse is used. DAOS-13625 dfuse: Merge the info and projection_info structs. (#11881) DAOS-13658 dfuse: Add filesystem query command. (#12367) DAOS-12751 control: Add a daos filesystem evict command. (#12331) DAOS-12751 dfuse: Improve evict command. (#12633) DAOS-13625 dfuse: Remove dfuse_projection_info entirely. (#12796) DAOS-13625 dfuse: Replace fs_handle with dfuse_info. (#12894) DAOS-13625 dfuse: Add core inode_lookup() and inode_decref() functions. (#12573) DAOS-14411 dfuse: Add per-container statistics. (#12819) DAOS-14411 control: Expose dfuse statistics as yaml. (#13876) Changed base branch to google/2.4 for daos_build test Change-Id: I8ae3cc743697c2434ae0d54b382ee6c585a3b033 Signed-off-by: Ashley Pittman <[email protected]> Signed-off-by: Jeff Olivier <[email protected]>

DAOS-13658 dfuse: Add counts for number of inodes etc.

6a12f3d

Use atomics to track number of pool and containers open, as well as inodes and file handles. Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1 reviewed Jun 12, 2023

View reviewed changes

Fix counting for ie and fh.

84a641f

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1 reviewed Jun 13, 2023

View reviewed changes

Count down fh.

fc52410

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1 reviewed Jun 13, 2023

View reviewed changes

Avoid a crash on shutdown.

fa4ddcf

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1 reviewed Jun 15, 2023

View reviewed changes

Fix a crash.

1872e92

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1 reviewed Jun 16, 2023

View reviewed changes

Tweak GHA again.

2d0ceb2

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1 reviewed Jun 16, 2023

View reviewed changes

Merge branch 'master' into amd/dfuse-desc-tracing

10c6ef3

Test-tag: dfuse Required-githooks: true

daosbuild1 reviewed Jun 16, 2023

View reviewed changes

ashleypittman added 2 commits June 16, 2023 14:30

Merge branch 'master' into amd/dfuse-desc-tracing

02900f5

Back out GHA change.

bb9d607

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1 reviewed Jun 16, 2023

View reviewed changes

Merge branch 'master' into amd/dfuse-desc-tracing

8a61525

daosbuild1 reviewed Jun 19, 2023

View reviewed changes

Cleanup after merge.

b1844e6

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1 reviewed Jun 19, 2023

View reviewed changes

Update var name.

37d6bda

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1 reviewed Jun 19, 2023

View reviewed changes

Add the go code and the ioctl.

a5781b1

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

Move the ioctl to it's own function.

a5f0220

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1 reviewed Jun 20, 2023

View reviewed changes

Merge branch 'master' into amd/dfuse-desc-tracing

1a17287

Required-githooks: true

daosbuild1 reviewed Jun 26, 2023

View reviewed changes

daltonbohning reviewed Jun 26, 2023

View reviewed changes

Merge branch 'master' into amd/dfuse-desc-tracing

b52dab9

Make agrumnent positional.

a1cdcf5

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

ashleypittman dismissed tanabarr’s stale review via a1cdcf5 June 29, 2023 08:56

daosbuild1 reviewed Jun 29, 2023

View reviewed changes

ashleypittman requested a review from daltonbohning June 30, 2023 08:05

mchaarawi reviewed Jun 30, 2023

View reviewed changes

ashleypittman added 3 commits June 30, 2023 15:10

Rename to query and add an actual test.

f3503dd

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

Merge branch 'master' into amd/dfuse-desc-tracing

de3333b

Required-githooks: true

Remove a variable.

ccabb05

Required-githooks: true Signed-off-by: Ashley Pittman <[email protected]>

daosbuild1 reviewed Jun 30, 2023

View reviewed changes

mchaarawi approved these changes Jun 30, 2023

View reviewed changes

daltonbohning approved these changes Jun 30, 2023

View reviewed changes

ashleypittman changed the title ~~DAOS-13658 dfuse: Add counts for number of inodes etc.~~ DAOS-13658 dfuse: Add filesystem query command. Jul 3, 2023

ashleypittman merged commit 03ef5e9 into master Jul 3, 2023

ashleypittman deleted the amd/dfuse-desc-tracing branch July 3, 2023 06:52

jolivier23 mentioned this pull request Apr 26, 2024

DAOS-14411 dfuse: Add daos fs query #14261

Merged

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-13658 dfuse: Add filesystem query command. #12367

DAOS-13658 dfuse: Add filesystem query command. #12367

ashleypittman commented Jun 12, 2023

daosbuild1 left a comment

github-actions bot commented Jun 12, 2023 •

edited

Loading

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 commented Jun 18, 2023

daosbuild1 commented Jun 18, 2023

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 commented Jun 19, 2023

daosbuild1 commented Jun 19, 2023

daosbuild1 left a comment

daosbuild1 commented Jun 20, 2023

daosbuild1 left a comment

daltonbohning Jun 26, 2023

ashleypittman Jun 26, 2023

daltonbohning Jun 26, 2023

ashleypittman Jun 26, 2023

daltonbohning Jun 26, 2023

ashleypittman Jun 26, 2023

ashleypittman commented Jun 29, 2023 •

edited

Loading

daosbuild1 left a comment

mchaarawi Jun 30, 2023

mchaarawi Jun 30, 2023 •

edited

Loading

daosbuild1 left a comment

daosbuild1 left a comment

daosbuild1 commented Jun 30, 2023

DAOS-13658 dfuse: Add filesystem query command. #12367

DAOS-13658 dfuse: Add filesystem query command. #12367

Conversation

ashleypittman commented Jun 12, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 12, 2023 • edited Loading

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Jun 18, 2023

daosbuild1 commented Jun 18, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Jun 19, 2023

daosbuild1 commented Jun 19, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Jun 20, 2023

daosbuild1 left a comment

Choose a reason for hiding this comment

daltonbohning Jun 26, 2023

Choose a reason for hiding this comment

ashleypittman Jun 26, 2023

Choose a reason for hiding this comment

daltonbohning Jun 26, 2023

Choose a reason for hiding this comment

ashleypittman Jun 26, 2023

Choose a reason for hiding this comment

daltonbohning Jun 26, 2023

Choose a reason for hiding this comment

ashleypittman Jun 26, 2023

Choose a reason for hiding this comment

ashleypittman commented Jun 29, 2023 • edited Loading

daosbuild1 left a comment

Choose a reason for hiding this comment

mchaarawi Jun 30, 2023

Choose a reason for hiding this comment

mchaarawi Jun 30, 2023 • edited Loading

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

daosbuild1 commented Jun 30, 2023

github-actions bot commented Jun 12, 2023 •

edited

Loading

ashleypittman commented Jun 29, 2023 •

edited

Loading

mchaarawi Jun 30, 2023 •

edited

Loading