kernfs_memcg: Add helpers to gather memcgroup related data #96

imran-kn · 2024-08-01T04:50:20Z

This as of now is just a dump of some of my bespoke debug scripts.

imran-kn · 2024-10-21T16:57:46Z

This as of now is just a dump of some of my bespoke debug scripts.

I have added other helpers and modified the earlier ones, so that they work with other UEK versions as well

biger410

Thanks for creating memcg helpers. I added couple comments. And beside that, please create a bug for it and put the number in the git log.

drgn_tools/kernfs_memcg.py

Signed-off-by: Imran Khan <[email protected]>

imran-kn · 2024-11-25T13:17:20Z

Thanks @biger410 for reviewing this. I have addressed your review comments. Could you please have a look and let me know if you have any further feedback

Orabug: 37322867 Signed-off-by: Imran Khan <[email protected]>

biger410 · 2024-11-26T00:57:02Z

The new changes look good to me. One more request is that can we add a corelen module for it? It should run either with -M option or -A option.

imran-kn · 2024-11-26T05:27:41Z

The new changes look good to me. One more request is that can we add a corelen module for it? It should run either with -M option or -A option.

I have added corelens module.

python3 -m drgn_tools.corelens ~/Local-vmcores/Workqueue-study/UEK-6-rds-issue/VMCORE -d ~/Local-vmcores/Workqueue-study/UEK-6-rds-issue/

page: 0xffffc35c881b0000 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b0040 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b0080 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b00c0 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b0100 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b0140 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b0180 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b01c0 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

biger410 · 2024-11-26T22:02:43Z

Is there a paste error? How could this trigger the corelen cmd?

python3 -m drgn_tools.corelens ~/Local-vmcores/Workqueue-study/UEK-6-rds-issue/VMCORE -d ~/Local-vmcores/Workqueue-study/UEK-6-rds-issue/

And with 10000 default, how could user run corelen cmd to dump all pages?

imran-kn · 2024-11-28T05:12:34Z

Is there a paste error? How could this trigger the corelen cmd?
python3 -m drgn_tools.corelens ~/Local-vmcores/Workqueue-study/UEK-6-rds-issue/VMCORE -d ~/Local-vmcores/Workqueue-study/UEK-6-rds-issue/
And with 10000 default, how could user run corelen cmd to dump all pages?

Yes, it was a copy paste error and missed the -M part . The actual command is:
python3 -m drgn_tools.corelens ~/Local-vmcores/Workqueue-study/UEK-6-rds-issue/VMCORE -d ~/Local-vmcores/Workqueue-study/UEK-6-rds-issue/ -M kernfs_memcg

and output is like the one shown below:

`imran@imran-metabox:~/oracle-samples-drgn-tools/drgn-tools$ python3 -m drgn_tools.corelens ~/Local-vmcores/Workqueue-study/UEK-6-rds-issue/VMCORE -d ~/Local-vmcores/Workqueue-study/UEK-6-rds-issue/ -M kernfs_memcg
page: 0xffffc35c881b0000 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b0040 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b0080 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b00c0 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b0100 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b0140 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b0180 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

page: 0xffffc35c881b01c0 cgroup: /user.slice/user-0.slice/session-7128.scope state: CSS_ONLINE|CSS_VISIBLE path: /datastore/oracle_230124_0745EST/diag/asm/cell/scaqat06celadm02/incpkg/pkg_76/seq_1/cell/IPSPKG_20230120060027_COM_1.zip

............................................................

Regarding default page count of 10K, I am using this value to make sure we don't end up spending a lot of time while collecting this data. We have seen that scanning all pages can take hours , so my idea here is that we get information of 10K pages and if that does not indicate anything conclusive , we can later scan all pages. Let me know if its sounds okay or using something other than 10K would be more acceptable
`

biger410 · 2024-12-03T23:44:59Z

The default value is good to me, to dump all pages, just pass option "-m 0". I think we should make the default value and how to dump all page clear in the cmd help doc for user.

imran-kn · 2024-12-04T05:46:53Z

The default value is good to me, to dump all pages, just pass option "-m 0". I think we should make the default value and how to dump all page clear in the cmd help doc for user.

I have modified cmd help to provide this information as per your suggestion. The cmd help looks like below
`python3 -m drgn_tools.corelens -M kernfs_memcg --help
usage: kernfs_memcg [-h] [--max MAX]

Print information related to pages, that are pinning memcgroup(s)

options:
-h, --help show this help message and exit
--max MAX, -m MAX Maximum number of pages to show. By default first 10000 such pages are shown. Use 0 to list all such pages.`

Please let me know if it looks okay.

biger410 · 2024-12-04T16:29:52Z

Looks good to me. @brenns10 do you want to take a look as well?

brenns10

Thanks, these are looking really good. I have a few small comments, the only major thing here is the issue with PG_slab.

drgn_tools/kernfs_memcg.py

Orabug: 37322867 Signed-off-by: Imran Khan <[email protected]>

imran-kn · 2024-12-12T14:46:26Z

@brenns10 I have addressed your review comments . Could you please have a look once more?

biger410

Can you reorganize the patch to address the review feedback while not add new commits for that? Once done reorganize patch, git push --force-with-lease can help overwrite the previous commits.

biger410 · 2024-12-12T17:46:31Z

drgn_tools/kernfs_memcg.py

 class PagesPinningMemcgroups(CorelensModule):
    """Print information related to pages, that are pinning memcgroup(s)"""

-    name = "kernfs_memcg"
+    name = "pages-pinning-memcg"
+    run_when = "never"


It makes senes this only run when user requests it explicitly. Should we make the default behavior to dump all pages pinning zombie cgroup, that looks more common to me when troubleshooting zombie cgroup issues?

Makes sense. I have modified it to dump all pages by default.

biger410 · 2024-12-12T17:49:00Z

drgn_tools/kernfs_memcg.py

+
+    def run(self, prog: Program, args: argparse.Namespace) -> None:
+        get_num_mem_cgroups(prog)
+


Can we make this more generic like the output of cat /proc/cgroups, not only support memory cgroup, but other cgroup as well? It doesn't have to be include with this pull request, you can start a new one for it if you want.

Sure. I can include this change in a different PR. I included memcgroup numbers here because we often run into issue due to zombie memcgroups and hence having their number readily available will help us to decide if we need to run PagesPinningMemcgroups module.

drgn_tools/kernfs_memcg.py

Add kernfs based helpers to extract memcg related information, like number of active and inactive memcgroups, page cache pages pinning memcgroups etc. Orabug: 37322867 Signed-off-by: Imran Khan <[email protected]>

Orabug: 37322867 Signed-off-by: Imran Khan <[email protected]>

imran-kn · 2024-12-13T00:29:08Z

@biger410 , @brenns10, I have addressed your review comments. Could you please have a look ?

biger410

Looks good to me. Thanks @imran-kn

oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Aug 1, 2024

imran-kn force-pushed the kernfs_memcg_work branch from 1909949 to 8c618c9 Compare October 21, 2024 16:54

imran-kn changed the title ~~DRAFT: kernfs_memcg: starting work.~~ kernfs_memcg: Add helpers to gather memcgroup related data Oct 21, 2024

imran-kn force-pushed the kernfs_memcg_work branch 2 times, most recently from 2d9b918 to d84a204 Compare October 22, 2024 09:51

imran-kn assigned pssatapathy-oracle and biger410 Oct 22, 2024

imran-kn requested review from biger410 and pssatapathy-oracle November 19, 2024 19:23

imran-kn assigned imran-kn and unassigned biger410 and pssatapathy-oracle Nov 19, 2024

biger410 reviewed Nov 22, 2024

View reviewed changes

drgn_tools/kernfs_memcg.py Outdated Show resolved Hide resolved

drgn_tools/kernfs_memcg.py Outdated Show resolved Hide resolved

drgn_tools/kernfs_memcg.py Outdated Show resolved Hide resolved

imran-kn added a commit that referenced this pull request Nov 25, 2024

kernfs,memcg: Addreess review comment for PR #96.

bb98c05

Signed-off-by: Imran Khan <[email protected]>

imran-kn added a commit that referenced this pull request Nov 26, 2024

kernfs,memcg: Addreess review comment for PR #96.

e0875fa

Orabug: 37322867 Signed-off-by: Imran Khan <[email protected]>

imran-kn force-pushed the kernfs_memcg_work branch from bb98c05 to e0875fa Compare November 26, 2024 00:19

imran-kn force-pushed the kernfs_memcg_work branch 2 times, most recently from ada1a81 to fe8d6ee Compare December 4, 2024 05:31

brenns10 requested changes Dec 4, 2024

View reviewed changes

imran-kn added a commit that referenced this pull request Dec 12, 2024

kernfs,memcg: Addreess review comment for PR #96.

c712265

Orabug: 37322867 Signed-off-by: Imran Khan <[email protected]>

biger410 reviewed Dec 12, 2024

View reviewed changes

brenns10 reviewed Dec 12, 2024

View reviewed changes

drgn_tools/kernfs_memcg.py Show resolved Hide resolved

drgn_tools/kernfs_memcg.py Outdated Show resolved Hide resolved

imran-kn added 2 commits December 13, 2024 11:07

kernfs,memcg: Add helpers to get memcg info

30e57f6

Add kernfs based helpers to extract memcg related information, like number of active and inactive memcgroups, page cache pages pinning memcgroups etc. Orabug: 37322867 Signed-off-by: Imran Khan <[email protected]>

kernfs,memcg: Add test case for memcg helpers.

7636da0

Orabug: 37322867 Signed-off-by: Imran Khan <[email protected]>

imran-kn force-pushed the kernfs_memcg_work branch from c712265 to 7636da0 Compare December 13, 2024 00:08

brenns10 approved these changes Dec 13, 2024

View reviewed changes

biger410 approved these changes Dec 13, 2024

View reviewed changes

imran-kn merged commit f395a8b into main Dec 13, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kernfs_memcg: Add helpers to gather memcgroup related data #96

kernfs_memcg: Add helpers to gather memcgroup related data #96

imran-kn commented Aug 1, 2024

imran-kn commented Oct 21, 2024

biger410 left a comment •

edited

Loading

imran-kn commented Nov 25, 2024

biger410 commented Nov 26, 2024

imran-kn commented Nov 26, 2024

biger410 commented Nov 26, 2024 •

edited

Loading

imran-kn commented Nov 28, 2024 •

edited

Loading

biger410 commented Dec 3, 2024

imran-kn commented Dec 4, 2024

biger410 commented Dec 4, 2024

brenns10 left a comment

imran-kn commented Dec 12, 2024

biger410 left a comment

biger410 Dec 12, 2024

imran-kn Dec 13, 2024

biger410 Dec 12, 2024

imran-kn Dec 13, 2024

imran-kn commented Dec 13, 2024

biger410 left a comment


		def run(self, prog: Program, args: argparse.Namespace) -> None:
		get_num_mem_cgroups(prog)

kernfs_memcg: Add helpers to gather memcgroup related data #96

kernfs_memcg: Add helpers to gather memcgroup related data #96

Conversation

imran-kn commented Aug 1, 2024

imran-kn commented Oct 21, 2024

biger410 left a comment • edited Loading

Choose a reason for hiding this comment

imran-kn commented Nov 25, 2024

biger410 commented Nov 26, 2024

imran-kn commented Nov 26, 2024

biger410 commented Nov 26, 2024 • edited Loading

imran-kn commented Nov 28, 2024 • edited Loading

biger410 commented Dec 3, 2024

imran-kn commented Dec 4, 2024

biger410 commented Dec 4, 2024

brenns10 left a comment

Choose a reason for hiding this comment

imran-kn commented Dec 12, 2024

biger410 left a comment

Choose a reason for hiding this comment

biger410 Dec 12, 2024

Choose a reason for hiding this comment

imran-kn Dec 13, 2024

Choose a reason for hiding this comment

biger410 Dec 12, 2024

Choose a reason for hiding this comment

imran-kn Dec 13, 2024

Choose a reason for hiding this comment

imran-kn commented Dec 13, 2024

biger410 left a comment

Choose a reason for hiding this comment

biger410 left a comment •

edited

Loading

biger410 commented Nov 26, 2024 •

edited

Loading

imran-kn commented Nov 28, 2024 •

edited

Loading