ppc64_cpu failed on system with 1280 cores

[davidzengxhsh]

The command "ppc64_cpu --frequency" failed on a Power system with 1280 threads as below:

ppc64_cpu --frequency

sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument
sched_setaffinity: Invalid argument

[davidzengxhsh]

cat /etc/os-release

NAME="Red Hat Enterprise Linux"
VERSION="8.0 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.0"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.0 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.0:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.0
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.0"

ppc64_cpu --cores-present

Number of cores present = 160

ppc64_cpu --smt

SMT=8

[nathanlynch]

Looks like ppc64_cpu needs changes for this to work on systems with 1024+ CPUs:

static void *soak(void *arg)
{
	unsigned int cpu = (long)arg;
	cpu_set_t cpumask;

	CPU_ZERO(&cpumask);
	CPU_SET(cpu, &cpumask);

	if (sched_setaffinity(0, sizeof(cpumask), &cpumask)) {
		perror("sched_setaffinity");
		pthread_exit(NULL);
	}

From sched_setaffinity's man page:

   Handling systems with large CPU affinity masks

       The underlying system calls (which represent CPU masks as bit masks
       of type unsigned long *) impose no restriction on the size of the CPU
       mask.  However, the cpu_set_t data type used by glibc has a fixed
       size of 128 bytes, meaning that the maximum CPU number that can be
       represented is 1023.  If the kernel CPU affinity mask is larger than
       1024, then calls of the form:

           sched_getaffinity(pid, sizeof(cpu_set_t), &mask);

       fail with the error EINVAL, the error produced by the underlying
       system call for the case where the mask size specified in cpusetsize
       is smaller than the size of the affinity mask used by the kernel.
       (Depending on the system CPU topology, the kernel affinity mask can
       be substantially larger than the number of active CPUs in the
       system.)

       When working on systems with large kernel CPU affinity masks, one
       must dynamically allocate the mask argument (see CPU_ALLOC(3)).