Display information about container limits in javacore

[ashu-mehra]

It would be useful to include container limits relevant to JVM in javacore.

Following information can be added to javacore:

• Indicate if JVM is in container.
• If not in container but running directly on host in a cgroup, display cgroup name for memory, cpu, and cpuset subsystems.
• Display memory limit imposed by the container.
• Display cpu count, cpu period and quota (and shares if required) as specified by the container.
• Display final cpu count calculated by JVM based on number of cpus and cpu quota

Current javacore includes cpu information at two places:

1. Under GPINFO subcomponent as:

2XHOSLEVEL     OS Level         : Linux 3.10.0-693.17.1.el7.x86_64
2XHCPUS        Processors -
3XHCPUARCH       Architecture   : amd64
3XHNUMCPUS       How Many       : 4
3XHNUMASUP       NUMA is either not supported or has been disabled by user

2. Under ENVINFO subcomponent as:

1CICPUINFO     CPU Information
NULL           ------------------------------------------------------------------------
2CIPHYSCPU     Physical CPUs: 4
2CIONLNCPU     Online CPUs: 4
2CIBOUNDCPU    Bound CPUs: 4
2CITARGETCPU   Target CPUs: 4

Also, there is no information on total physical memory of the system in javacore. NATIVEMEMINFO subcomponent displays only JVM's usage.

Proposal

My proposal is to add a subsection to javacore under ENVINFO subcomponent that covers above mentioned information as:

Container Information/Cgroup Information
----------------------------------------
cgroup names: // this will be shown only if jvm is running directly on host in cgroup
	subsystem		cgroup
	memory			<cgroup name>
	cpuset			<cgroup name>
	cpu			<cgroup name>

Memory limit:
CPU Set:
CPU Period:
CPU Quota:

[ashu-mehra]

@DanHeidinga @pshipton can you please provide your inputs on this.

[pshipton]

The following section is found in the ENVINFO subcomponent. My vote is to put the new information after this section and before the CPU Information

1CISYSINFO     System Information
NULL           ------------------------------------------------------------------------
2CISYSINFO     Hypervisor name = VMWare
2CISYSINFO     /proc/sys/kernel/core_pattern = |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I %h
2CISYSINFO     /proc/sys/kernel/core_uses_pid = 1

@keithc-ca fyi

[pshipton]

Before the System Information section, or after the CPU Information is also fine with me.

[SueChaplain]

Which release is this being targeted for please?

[ashu-mehra]

@SueChaplain If you are looking for java release, I believe this should be part of java 11.
If you are looking for OpenJ9 release, then I am not sure.

[pshipton]

@SueChaplain this is unlikely to make 0.9.0

[pshipton]

@joransiu fyi

[joransiu]

+1. We recently worked on a performance issue that was a result of an unexpected cpu quota being set on the container running the JVM. It would be great if such container constraints were dumped into javacore as suggested here.

[bharathappali]

@joransiu I'm working on to add the cpu quota information in java dump. Can i know in which format it would be good to have. Like cpu quota and cpu period seperately or a ratio of it

[bharathappali]

@keithc-ca I have added changes in OpenJ9 to get the cgroup information in a iterator model similar to approach taken for displaying env and system limits. you can have a look at the changes of OMR and Openj9. I feel this approach is better than the earlier one OMR and Openj9 for displaying Cgroup information. Can you please provide your comment on which approach is preferable

Note : its a work in progress one and its not completely functional and working.

[keithc-ca]

I noticed that in your openj9 branch, all the changed files have altered permissions. Was that intentional, and if so, why?

[bharathappali]

@keithc-ca sorry for that it was not intentional. Its due to my dev environment issue. Will fix it.

[ashu-mehra]

fyi - as per the changes done by @bharathappali javacore is now displaying following subsystem metrics (more than what I have mentioned in my proposal earlier)

memory subsystem:
    memory.usage_in_bytes		 # show current usage for memory
    memory.memsw.usage_in_bytes	 # show current usage for memory+Swap
    memory.limit_in_bytes		 # set/show limit of memory usage
    memory.memsw.limit_in_bytes	 # set/show limit of memory+Swap usage
    memory.failcnt			 # show the number of memory usage hits limits
    memory.memsw.failcnt		 # show the number of memory+Swap hits limits
    memory.max_usage_in_bytes	 # show max memory usage recorded
    memory.memsw.max_usage_in_bytes # show max memory+Swap usage recorded
    memory.oom_control		 # set/show oom controls.

cpuset subsystem:
    cpuset.cpus        # list of CPUs in that cpuset
    cpuset.mems        # list of Memory Nodes in that cpuset
    cpuset.cpu_exclusive flag        # is cpu placement exclusive?
    cpuset.mem_exclusive flag        # is memory placement exclusive?

cpu subsystem:
    cpu.cfs_period_us        # the length of a period (in microseconds)
    cpu.cfs_quota_us        # the total available run-time within a period (in microseconds)
    cpu.shares        # cpu shares
    cpu.stat        # exports throttling statistics

More information about these are available here:
https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt
https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/sec-cpu

[SueChaplain]

@pshipton - now that 0.9.0 is slightly delayed, will this go in please?

[pshipton]

@SueChaplain I don't expect there will be changes for this in 0.9.0

[SueChaplain]

@ashu-mehra Please can you append a full javadump that includes this section. We need a new one for the user docs, so might as well use one that contains this extra information. Thanks

[ashu-mehra]

@SueChaplain, @bharathappali is still working on finalizing the contents of this section.
@bharathappali can you please attach a new javacore here once you are done your changes for this issue.

[pshipton]

Is this going to make the 0.12 release, or should I move it out of plan? i.e. will it be delivered by Monday, seems unlikely.
@DanHeidinga ?

[pshipton]

Since this didn't make the feature complete date, I'm moving it out of the 0.12 milestone as it's not a blocker. If it does happen to make it in, we can fix the milestone afterwards.