-
Notifications
You must be signed in to change notification settings - Fork 184
Usage Examples
NOTE: Must be run as root.
Monitor all events on cores 0 to 11:
pqos -m all:0-11
pqos -m :0-11
Monitor LLC on cores 0, 2 and 6:
pqos -m llc:0,2,6
Monitor local memory B/W on cores 0-2 and remote memory B/W on cores 3, 4 and 5:
pqos -m "mbl:0-2;mbr:3,4,5"
Monitor events on groups of cores (aggregate statistics):
pqos -m "all:[0-11];llc:[12,13,14];mbl:[15-17,20]"
Reset Monitoring: Reclaims in-use RMID's.
pqos -r
Example CMT/MBM usage scenario
A user has a host machine running 3 guest VM's with 3 cores assigned to each guest.
- VM0 - cores 0-2
- VM1 - cores 3-5
- VM2 - cores 6-8
To monitor all events (LLC occupancy, local and remote memory B/W) run:
pqos -m "all:[0-2],[3-5],[6-8];"
Console output:
CORE IPC MISSES LLC[KB] MBL[MB/s] MBR[MB/s]
0-2 0.28 7893k 383.2 901.2 430.8
3-5 0.28 45k 25.3 361282.6 22.4
6-8 0.26 89468k 6778.8 43904.3 4.3
Set COS 1 to the first 4 cache ways and COS 2 to the next 8 cache ways:
pqos -e "llc:1=0x000f;llc:2=0x0ff0;"
Set COS 1 on all sockets, COS 2 on socket 0 and 1 and COS 3 on sockets 2 to 3:
pqos -e "llc:1=0x000f;llc@0,1:2=0x0ff0;llc@2-3:3=0x3c"
Console output for pqos -s to show current configuration:
L3CA COS definitions for Socket 0:
L3CA COS0 => MASK 0xfffff
L3CA COS1 => MASK 0xf
L3CA COS2 => MASK 0xff0
L3CA COS3 => MASK 0xfffff
...
L3CA COS definitions for Socket 1:
L3CA COS0 => MASK 0xfffff
L3CA COS1 => MASK 0xf
L3CA COS2 => MASK 0xff0
L3CA COS3 => MASK 0xfffff
...
L3CA COS definitions for Socket 2:
L3CA COS0 => MASK 0xfffff
L3CA COS1 => MASK 0xf
L3CA COS2 => MASK 0xfffff
L3CA COS3 => MASK 0x3c
...
L3CA COS definitions for Socket 3:
L3CA COS0 => MASK 0xfffff
L3CA COS1 => MASK 0xf
L3CA COS2 => MASK 0xfffff
L3CA COS3 => MASK 0x3c
...
Associate cores 0, 2, and 6 to 10 with COS 1 and core 1 to COS 2:
pqos -a "llc:1=0,2,6-10;llc:2=1;"
Enable, disable L3 CDP (v0.1.5+):
pqos -R l3cdp-on
pqos -R l3cdp-off
Enable, disable L3 CDP (v0.1.4):
pqos -S cdp-on
pqos -S cdp-off
Use current L3 CDP settings and set COS 1 code and data bitmasks (v0.1.5+):
pqos -e "llc:1d=0xfff;llc:1c=0xfff00;"
Use current L3 CDP settings and set COS 1 code and data bitmasks (v0.1.4):
pqos -S cdp-any -e "llc:1d=0xfff;llc:1c=0xfff00;"
or
pqos -e "llc:1d=0xfff;llc:1c=0xfff00;"
Show current CAT settings:
pqos -s
Reset CAT:
Sets all COS to default (fill into all ways) and associates all cores with COS 0.
pqos -R
Example CAT usage scenario
A user has a host machine running 3 guest VM's. Each guest is assigned 3 cores and a priority.
- VM0 - cores 0-2 (P5)
- VM1 - cores 3-5 (P2)
- VM2 - cores 6-8 (P1)
As VM0 has the highest priority it will be assigned 8 exclusive LLC ways. VM1 and VM2 are relatively low priority so VM1 is assigned 6 ways and VM2 is assigned 4 ways, 2 of which will be shared.
-
First, set the 3 COS bitmasks for each VM:
pqos -e "llc:1=0x00ff;llc:2=0x3f00;llc:3=0xf000;" -
Next, associate each COS with the cores where each VM is running:
pqos -a "llc:1=0-2;llc:2=3-5;llc:3=6-8;"
VM 0 now has exclusive access to 8 LLC ways, VM1 has exclusive access to 4 ways and shared access to 2 ways and VM2 has exclusive access to 2 ways and shared access to another 2 ways. All other cores have access to all other ways.
Set COS 1 to 50% available and COS 2 to 70% available:
pqos -e "mba:1=50;mba:2=70;"
Set COS 1 on all sockets, COS 2 on socket 0 and 1 and COS 3 on sockets 2 to 3:
Note: MBA rounds numbers given to it.
pqos -e "mba:1=80;mba@0,1:2=64;mba@2-3:3=85"
Console output for pqos -s to show current configuration:
L3CA/MBA COS definitions for Socket 0:
MBA COS0 => 100% available
MBA COS1 => 80% available
MBA COS2 => 60% available
MBA COS3 => 100% available
...
L3CA/MBA COS definitions for Socket 1:
MBA COS0 => 100% available
MBA COS1 => 80% available
MBA COS2 => 60% available
MBA COS3 => 100% available
...
L3CA/MBA COS definitions for Socket 2:
MBA COS0 => 100% available
MBA COS1 => 80% available
MBA COS2 => 100% available
MBA COS3 => 90% available
...
L3CA/MBA COS definitions for Socket 3:
MBA COS0 => 100% available
MBA COS1 => 80% available
MBA COS2 => 100% available
MBA COS3 => 90% available
...
Show current MBA settings:
pqos -s
Reset MBA: Sets all COS to default and associates all cores with COS 0.
pqos -R
Monitoring by PIDs is available only when pqos uses OS interface ( --iface os ).
Console output for pqos --iface os -s to show current configuration:
PID association information:
COS1 => (none)
COS2 => (none)
COS3 => (none)
COS4 => (none)
COS5 => (none)
COS6 => (none)
COS7 => (none)
COS8 => (none)
COS9 => (none)
COS10 => (none)
COS11 => (none)
COS12 => (none)
COS13 => (none)
COS14 => (none)
COS15 => (none)
pqos --iface os -V -s shows also PID association information
PID association information:
COS0 => 1, 2, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 26, 27, 28, 29,
30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55,
56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81,
82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106,
107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121...
PIDs are associated by default with CAT class 0
Monitor all events for multiple PID's 116 and 119 to 121:
pqos --iface os -p all:116,119-121
Console output:
TIME 2017-08-17 14:35:03
PID CORE IPC MISSES LLC[KB] MBL[MB/s] MBR[MB/s]
116 N/A 1.82 2140k 1344.0 1.9 0.0
119 N/A 1.29 288k 704.0 2.2 0.0
120 N/A 2.16 657k 256.0 9.6 0.0
121 N/A 1.27 718k 192.0 63.4 0.0
Monitor all events for top 10 most active (CPU utilizing) PIDs:
pqos --iface os -p
Console output:
TIME 2017-08-17 14:35:03
PID CORE IPC MISSES LLC[KB] MBL[MB/s] MBR[MB/s]
3013 N/A 1.82 562140k 1344.0 1.9 0.0
1580 N/A 1.29 288k 704.0 2.2 0.0
1668 N/A 2.16 657k 256.0 9.6 0.0
1685 N/A 1.27 718k 192.0 63.4 0.0
590 N/A 1.14 0k 160.0 0.0 0.0
1576 N/A 0.24 3k 32.0 0.1 0.0
8 N/A 0.53 3k 32.0 0.1 0.0
77 N/A 0.47 0k 0.0 0.0 0.0
10 N/A 0.48 0k 0.0 0.0 0.0
1687 N/A 0.44 0k 0.0 0.0 0.0
Allocation by PIDs is available only when pqos uses OS interface ( --iface os ).
Set COS 1 to the first 4 cache ways and COS 2 to the next 8 cache ways:
pqos --iface os -e "llc:1=0x000f;llc:2=0x0ff0;"
Console output:
NOTE: Mixed use of MSR and kernel interfaces to manage
CAT or CMT & MBM may lead to unexpected behavior.
SOCKET 0 L3CA COS1 => MASK 0xf
SOCKET 1 L3CA COS1 => MASK 0xf
SOCKET 0 L3CA COS2 => MASK 0xff0
SOCKET 1 L3CA COS2 => MASK 0xff0
Allocation configuration altered.
Associate cores 0, 2, 3, 4, 5, 6 with CAT class 0 and cores 1, 7 to 10 with class 1:
pqos --iface os -a "core:0=0,2,3,4,5,6;core:1=1,7-10;"
sudo pqos --iface os -s:
Core information for socket 0:
Core 0, L2ID 0, L3ID 0 => COS0
Core 1, L2ID 1, L3ID 0 => COS1
Core 2, L2ID 2, L3ID 0 => COS0
Core 3, L2ID 3, L3ID 0 => COS0
Core 4, L2ID 4, L3ID 0 => COS0
Core 5, L2ID 5, L3ID 0 => COS0
Core 6, L2ID 6, L3ID 0 => COS0
Core 7, L2ID 8, L3ID 0 => COS1
Core 8, L2ID 9, L3ID 0 => COS1
Core 9, L2ID 10, L3ID 0 => COS1
Core 10, L2ID 11, L3ID 0 => COS1
Associate process ID 116 with CAT class 1 and process ID 121with class 2 (only V1.1+):
pqos --iface os -a "pid:1=116;pid:2=119-121;"
sudo pqos --iface os -s:
PID association information:
COS1 => 116
COS2 => 119, 120, 121
COS3 => (none)
COS4 => (none)
COS5 => (none)
COS6 => (none)
COS7 => (none)
COS8 => (none)
COS9 => (none)
COS10 => (none)
COS11 => (none)
COS12 => (none)
COS13 => (none)
COS14 => (none)
COS15 => (none)
Allocation reset: Sets all COS to default and associates all cores with COS 0.
pqos --iface os -R
If you require system wide interface enforcement you can do so by setting the "RDT_IFACE" environment variable.
Set interface for only MSR use:
Write RDT_IFACE=MSR in /etc/environment or sudo RDT_IFACE=MSR pqos -V -s
sudo RDT_IFACE=MSR pqos --iface os -V -s will not work:
NOTE: Mixed use of MSR and kernel interfaces to manage
CAT or CMT & MBM may lead to unexpected behavior.
Interface initialization error!
Your system has been restricted to use the MSR interface only!
Set interface for only OS use:
Write RDT_IFACE=OS in /etc/environment or sudo RDT_IFACE=OS pqos -V -s
sudo RDT_IFACE=OS pqos -V -s will not work:
NOTE: Mixed use of MSR and kernel interfaces to manage
CAT or CMT & MBM may lead to unexpected behavior.
Interface initialization error!
Your system has been restricted to use the OS interface only!
Error initializing PQoS library!
The --iface parameter is used to select interface for program Intel(R) RDT technologies. Three options are allowed:
-
--iface msris used to configure the platform by programming the hardware (MSR's) directly. -
--iface osis used to configure the platform by Linux kernel extensions. -
--iface auto(default options) automatic sets one of both msr or os interfaces.
If automatic detection is selected and "RDT_IFACE" environment variable is set, then:
- If "RDT_IFACE" environment variable is set to "OS", then only
--iface autoor--iface osoptions are allowed. An error will be reported when trying to set up the--iface msr. Pqos will use the os interface if is supported. - If "RDT_IFACE" environment variable is set to "MSR", then only
--iface autoor--iface msroptions are allowed. An error will be reported when trying to set up the--iface os. The pqos will use the msr interface. - If "RDT_IFACE" environment variable is set to different value, then error will be reported.
If "RDT_IFACE" environment variable is no set, then:
- When run pqos with
--iface autoand resctrl is supported, then os interface will be used. - When run pqos with
--iface autoand resctrl is not supported, then msr interface will be used.
In order to detect and use model-specific non-architectural RDT features, set the "RDT_PROBE_MSR" environment variable when using the utilities with the MSR interface selected.
sudo RDT_IFACE=MSR RDT_PROBE_MSR=1 pqos -V -s
or
sudo RDT_PROBE_MSR=1 pqos --iface msr -V -s
The rdtset tool provides support to set up the CAT (Cache Allocation Technology) and MBA (Memory Bandwidth Allocation) capabilities for a task and set its CPU affinity. Current Intel(R) RDT allocation operations of the utility are based on controlling MSR registers (via libpqos library). Class of service 0 (CLOS0) is assumed as default one. In command mode, rdtset forks and one process executes the command. Another process waits for the task to terminate and restores default allocation state by assigning cpu's back to CLOS0. This behavior is not in place in PID mode.
Run "id" command on CPU 1 using four L3 cache-ways (mask 0xf), keeping sudo elevated privileges:
-t 'l3=0xf;cpu=1' -c 1 -k id
CPU 1 uses four L3 cache-ways (mask 0xf)
-t 'l3=0xf;cpu=1'
CPU 1 uses one L2 (mask 0x1) and four L3 (mask 0xf) cache-ways
-t 'l2=0x1;l3=0xf;cpu=1'
CPU 1 uses one L2 (mask 0x1) and four L3 (mask 0xf) cache-ways, CPU 2 uses one L2 (mask 0x1) and default number of L3 cache-ways, L2 cache-ways used by CPU 1 and 2 are overlapping
-t 'l2=0x1;l3=0xf;cpu=1' -t 'l2=0x1;cpu=2'
CPU 2 uses four L3 cache-ways (mask 0xf), CPUs 3-5 share four L3 cache-ways (mask 0xf0), L3 cache-ways used by CPU 2 and 3-4 are non-overlapping
-t 'l3=0xf;cpu=2' -t 'l3=0xf0;cpu=3,4,5'
*CPUs 0-2 share four L3 cache-ways (mask 0xf), CPUs 3-5 share four L3 cache-ways (mask 0xf0), L3 cache-ways used by CPUs 0-2 and 3-5 are non-overlapping
-t 'l3=0xf;cpu=0-2' -t 'l3=0xf0;cpu=3,4,5'
On CDP enabled system, CPU 1 uses four L3 cache-ways for code (mask 0xf) and four L3 cache-ways for data (mask 0xf0), data and code L3 cache-ways are non-overlapping
-t 'l3=0xf,0xf0;cpu=1'
CPU 1 uses four L3 (mask 0xf) cache-ways and can utilize up to 50% of available memory bandwidth
-t 'mba=50;l3=0xf;cpu=1'
Specified processes use four L3 cache-ways (mask 0xf)
--iface os -t 'l3=0xf' -p 23187,567-570
Restrict memory B/W availability to 50% for the memtester application (using PID allocation)
--iface os -t 'mba=50' -k memtester 10M
CPUs 0,1,2,3,4,5
-c 0-3,4,5
reset allocation for CPUs 0,1,2,3,4,5
-r 0-3,4,5
Configure allocation and CPU affinity for BASH process
-t 'l3=0xf;cpu=0-2' -t 'l3=0xf0;cpu=3,4,5' -c 0-5 -p $BASHPID
Change allocation configuration of CPUs used by BASH process
-r 0-5 -t 'l3=0xff;cpu=0-5' -c 0-5 -p $BASHPID
Reset allocation configuration of CPUs used by BASH process
-r 0-5 -p $BASHPID