-
Notifications
You must be signed in to change notification settings - Fork 184
FAQ
To display detected Intel® RDT capabilities please use -d option or -D for more verbose output including cache information. Those options display Intel® RDT capabilities supported via MSR (legacy) and the OS/Kernel interfaces.
Example pqos -d command output:
Hardware capabilities
Monitoring
Cache Monitoring Technology (CMT) events:
LLC Occupancy (LLC)
Memory Bandwidth Monitoring (MBM) events:
Total Memory Bandwidth (TMEM)
Local Memory Bandwidth (LMEM)
Remote Memory Bandwidth (RMEM) (calculated)
PMU events:
Instructions/Clock (IPC)
LLC misses
Allocation
Cache Allocation Technology (CAT)
L3 CAT
CDP: disabled
Num COS: 16
Memory Bandwidth Allocation (MBA)
Num COS: 8
OS capabilities (Linux kernel 4.13.0-rc5)
Monitoring
Cache Monitoring Technology (CMT) events:
LLC Occupancy (LLC)
Memory Bandwidth Monitoring (MBM) events:
Total Memory Bandwidth (TMEM)
Local Memory Bandwidth (LMEM)
Remote Memory Bandwidth (RMEM) (calculated)
PMU events:
Instructions/Clock (IPC)
LLC misses
Allocation
Cache Allocation Technology (CAT)
L3 CAT
CDP: disabled
Num COS: 8
Memory Bandwidth Allocation (MBA)
Num COS: 8
Note: The pqos -s command combined with the -v or -V option can also be used to display detected Intel® RDT capabilities and cache information. Please see the next FAQ below for more information.
To print the current Intel® RDT configuration, please use pqos -s command.
Example pqos -s command output:
L3CA/MBA COS definitions for Socket 0:
L3CA COS0 => MASK 0x7ff
L3CA COS1 => MASK 0x7ff
…
L3CA COS14 => MASK 0x7ff
L3CA COS15 => MASK 0x7ff
MBA COS0 => 100% available
MBA COS1 => 100% available
…
MBA COS6 => 100% available
MBA COS7 => 100% available
Core information for socket 0:
Core 0, L2ID 0, L3ID 0 => COS0, RMID0
Core 1, L2ID 1, L3ID 0 => COS0, RMID0
…
Core 46, L2ID 14, L3ID 0 => COS0, RMID0
Core 47, L2ID 15, L3ID 0 => COS0, RMID0
The -v option will print additional log information messages containing the detected Intel® RDT capabilities and cache information.
Example pqos –s -v command output:
INFO: CACHE: type 1, level 1, max id sharing this cache 2 (1 bits)
INFO: CACHE: type 2, level 1, max id sharing this cache 2 (1 bits)
INFO: CACHE: type 3, level 2, max id sharing this cache 2 (1 bits)
INFO: CACHE: type 3, level 3, max id sharing this cache 32 (5 bits)
INFO: Monitoring capability detected
INFO: CPUID.0x7.0: L3 CAT supported
INFO: CDP is disabled
INFO: L3 CAT details: CDP support=1, CDP on=0, #COS=16, #ways=11, ways contention bit-mask 0x600
INFO: L3 CAT details: cache size 23068672 bytes, way size 2097152 bytes
INFO: L3CA capability detected
INFO: CPUID 0x10.0: L2 CAT not supported!
INFO: L2CA capability not detected
INFO: MBA details: #COS=8, linear, max=90, step=10
INFO: MBA capability detected
INFO: resctrl detected
INFO: OS support for CMT detected
INFO: OS support for L3 CAT detected
INFO: OS support for MBA detected
INFO: resctrl not mounted
L3CA/MBA COS definitions for Socket 0:
L3CA COS0 => MASK 0x7ff
L3CA COS1 => MASK 0x7ff
…
The -voption can be replaced with -V to include more verbose debug log messages.
WARN: Error opening file '/dev/cpu/0/msr'!
ERROR: CDP detection error!
ERROR: Fatal error encounter in CAT discovery!
ERROR: discover_capabilities() error 1
Error initializing PQoS library!
This is usually seen when the utility is run without root privileges. Another possible cause is when the msr driver has not been auto-loaded.
For some modular kernels the driver may need to be loaded manually by running:
modprobe msr
pqos: error while loading shared libraries: libpqos-0.1.5.so: cannot open shared object file: No such file or directory
This occurs when the library default installation directory is not included in the linker search path.
To add the path, as root (not with sudo), run:
# echo /usr/local/lib > /etc/ld.so.conf.d/<FILENAME>.conf
Then to update the linker cache run:
sudo ldconfig
sudo: pqos: command not found
This occurs when the default installation directory has not been added to the secure path section in the sudoers file.
To fix this, open the sudoers file as root with:
visudo
and append the default installation path to the list of secure paths:
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
ERROR: IPC and/or LLC miss performance counters already in use!
Use -r option to start monitoring anyway.
Monitoring start error on core(s) 0, status 6
This is a warning that another application may be using PMU counters to monitor resources. To force the PQoS utility to start monitoring anyway, add the -r option to the command.
Example: pqos -r -p llc:34534 or pqos -r -m :[0-47]
PID 1 monitoring start error,status 1
This is usually caused when the system is using a kernel that does not support PQoS Perf events such as:
- llc_occupancy - llc (LLC occupancy)
- local_mb - mbl (Local Memory B/W)
- total_mb - mbr (Used with MBL to calculate Remote Memory B/W)
NOTE: Kernel version 4.1 or newer is required for PID monitoring.
Q: For example, if I have a kernel module run on a particular core, will it be limited to the LLC allocation for that particular core or are the limitations just for userspace applications?
A: Currently, the utility and library offer static CAT configuration. This means cores are assigned to classes of service and this configuration doesn't change when tasks are scheduled in and out on these cores. Consequently Linux kernel threads running on this core will be also affected. Depending on the application, this may be a good or a bad thing. There is work in progress to enable CAT configuration on task basis to address this gap.
Yes it can, user applications can modify COS bitmasks at runtime. However, currently CAT can only be managed through the MSR interface. The downside of the MSR interface is that the technologies can only be managed on logical core basis - no task or process tracking capability. There is work in progress to enable CAT configuration on task basis to address this gap.
One bit in a COS bitmask corresponds to one cache way. Cache way size can be calculated manually with (Total LLC size divided by number of ways). This information can also be retrieved from the utility by running pqos -s -v and looking for the following lines:
...
INFO: LLC cache size 36700160 bytes, 20 ways
INFO: LLC cache way size 1835008 bytes
...
Note: These lines will only be logged if CAT support is detected.
Each COS bitmask must have at least 2 contiguous bits set (i.e. 2 cache ways).
For example, for a CPU with 20MB of LLC and 20 cache ways, each way is ~1MB in size. In this case, at least 2MB of cache must be assigned to each COS, which can be incremented by 1MB thereafter.
The number of classes of service is product dependent. To find out how many COS are supported by your product run
pqos -s -v and look for the CAT details message:
...
INFO: CAT details: CDP support=1, CDP on=0, #COS=16, #ways=20, ways contention bit-mask 0xc0000
...
In this example, the number of COS supported by the product is 16.
Note: These lines will only be logged if CAT support is detected.
| Linux Distribution | Release |
|---|---|
| Ubuntu | 16.04 "The Xenial Xerus" |
| 16.10 "The Yakkety Yak" (Devel) | |
| Debian | 9 "Stretch" (Testing) |
| "Sid" (Unstable) | |
| Fedora | 24/25/26 (Devel) |
| EPEL 7 | |
| CentOS | 7 |
| OpenSuse | 13 |
| Leap 42 | |
| Tumbleweed | |
| SLES | 11/12 |
| Clear Linux | 5370 - latest |
To install using the "Advanced Package Tool", or apt do:
$ sudo apt-get install intel-cmt-cat
To install using the "Yellowdog Updater, Modified", or yum do:
$ sudo yum install intel-cmt-cat
To install using the "Dandified Yum", or dnf do:
$ sudo dnf install intel-cmt-cat
Note: RPM name has an OS extension so the final name will be OS dependent.
The below examples are based on Linux CentOS.
Basic steps to build source and binary packages in your home directory:
Create the rpmbuild directory structure:
$ rpmdev-setuptree
Next, download the intel-cmt-cat tar file and copy into ~/rpmbuild/SOURCES directory:
$ wget https://github.com/01org/intel-cmt-cat/archive/v0.1.4.tar.gz
$ cp ./v0.1.4.tar.gz ~/rpmbuild/SOURCES
To build, do:
$ rpmbuild -ba path/to/intel-cmt-cat.spec
To install, do:
$ sudo rpm -i ~/rpmbuild/RPMS/x86_64/intel-cmt-cat-0.1.4-3.el7.centos.x86_64.rpm
To rebuild the package from a SRPM file in your home directory, do:
$ rpmbuild --rebuild path/to/intel-cmt-cat-0.1.4.src.rpm
To install, do:
$ sudo rpm -i ~/rpmbuild/RPMS/x86_64/intel-cmt-cat-0.1.4-3.el7.centos.x86_64.rpm
If you would like to install Python wrapper as a dependency in your application
and you are using Pipfile for specifying dependencies, use the following code:
[packages]
...your dependencies here...
[packages.pqos]
editable = true
path = "path-to-intel-cmt-cat/lib/python"
Alternatively, you may use the following commands:
cd path-to-intel-cmt-cat/lib/python
python -m pip wheel --wheel-dir=/tmp .
python -m pip install --no-index --find-links=/tmp pqos
rm -rf /tmp/pqos*.whl
Note: use python3 instead of python in above commands if your system runs Python 2.x under python command.
You may also use python specific to your Python environment created by virtualenv.
Python wrapper for libpqos requires built version of libpqos.
If libpqos is installed in system, Python wrapper will find it.
Alternatively, if the library is not installed in the system or you would like to use a different version of the library,
Python wrapper can look for the library in a directory specified by LD_LIBRARY_PATH environment variable,
for example:
LD_LIBRARY_PATH=./intel-cmt-cat/lib python my_script.py