how to pin prox VM cpu (OS and prox app)?

[pinggit]

say I have a few CPUs in compute, but I only want my gen VM to use only:

OS: CPU1, 7
PROX: 
    TX CPU 3,9
    RX CPU 4,10 

how to implement this?

[kirankn80]

Przemek can comment on this..

[PrzemekGrygiel]

Ping
Please for gen VM allocate cores (pin in nova) 3,9,4,10, then
in rapid/prox configuration allocate
gencores = 3,9
latcores = 4,10

For OS we don't need to do anything, they just need to be excluded from an isolation list. Isolated list (tuned isoloted cpus and kernel isolcpu add all cores except this for OS).
There is my the example for RHOSP with end to end cores allocation etc
https://docs.google.com/document/d/1O8SyGvdDUOcrVEdP_k-rokJnQcwvF8soqwqBXENDHIY/edit#heading=h.iaclu0979282

[pinggit]

this seems does not work. @PrzemekGrygiel

gencores = [3, 9]
latcores = [4, 10]

[root@stack2-jump rapid]# ./runrapid.py --runtime 5
Using 'rapid.env' as name for the environment
Using 'basicrapid.test' for test case definition
Using 'machine.map' for machine mapping
Runtime: 5
Traceback (most recent call last):
  File "./runrapid.py", line 960, in <module>
    clients[-1].fork_cmd(cmd, 'PROX Testing on TestM%d'%(vm+1))
  File "/opt/openstackrapid/samplevnf/VNFs/DPPD-PROX/helper-scripts/rapid/prox_ctrl.py", line 104, in fork_cmd
    % (name, ex.returncode, ex.output.strip()))
RuntimeError: Child process 'PROX Testing on TestM1' failed:
ssh returned exit status 1:
Command line: /root/prox/build/prox -e -t -o cli -f /root/gen.cfg
=== prox 20.05 ===
        Using DPDK 19.11.0
        ncurses version = 5.9 (ncurses 5.9.20130511)
        TERM left unchanged to unknown
        Does not support Intel RDT Monitoring capability
=== Parsing configuration file '/root/gen.cfg' ===
        *** Reading [lua] section ***
        *** Reading [variables] section ***
        *** Reading [eal options] section ***
        *** Reading [cache set #] sections ***
        *** Reading [port #] sections ***
        *** Reading [defaults] section ***
        *** Reading [global] section ***
        *** Reading [core #] sections ***
=== Setting up RTE EAL ===
        Worker threads core mask is 0x619
        With master core index 0, full core mask is 0x619
        EAL command line: /root/prox/build/prox -c0x619 --master-lcore=0 -n4
EAL: Detected 10 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: lcore 10 unavailable
EAL: invalid coremask, please check specified cores are part of 0-9

Usage: /root/prox/build/prox [options]

EAL common options:
  -c COREMASK         Hexadecimal bitmask of cores to run on
  -l CORELIST         List of cores to run on
                      The argument format is <c1>[-c2][,c3[-c4],...]
                      where c1, c2, etc are core indexes between 0 and 128
  --lcores COREMAP    Map lcore set to physical cpu set
                      The argument format is
                            '<lcores[@cpus]>[<,lcores[@cpus]>...]'
                      lcores and cpus list are grouped by '(' and ')'
                      Within the group, '-' is used for range separator,
                      ',' is used for single number separator.
                      '( )' can be omitted for single element group,
                      '@' can be omitted if cpus and lcores have the same value
  -s SERVICE COREMASK Hexadecimal bitmask of cores to be used as service cores
  --master-lcore ID   Core ID that is used as master
  --mbuf-pool-ops-name Pool ops name for mbuf to use
  -n CHANNELS         Number of memory channels
  -m MB               Memory to allocate (see also --socket-mem)
  -r RANKS            Force number of memory ranks (don't detect)
  -b, --pci-blacklist Add a PCI device in black list.
                      Prevent EAL from using this PCI device. The argument
                      format is <domain:bus:devid.func>.
  -w, --pci-whitelist Add a PCI device in white list.
                      Only use the specified PCI devices. The argument format
                      is <[domain:]bus:devid.func>. This option can be present
                      several times (once per device).
                      [NOTE: PCI whitelist cannot be used with -b option]
  --vdev              Add a virtual device.
                      The argument format is <driver><id>[,key=val,...]
                      (ex: --vdev=net_pcap0,iface=eth2).
  --iova-mode   Set IOVA mode. 'pa' for IOVA_PA
                      'va' for IOVA_VA
  -d LIB.so|DIR       Add a driver or driver directory
                      (can be used multiple times)
  --vmware-tsc-map    Use VMware TSC map instead of native RDTSC
  --proc-type         Type of this process (primary|secondary|auto)
  --syslog            Set syslog facility
  --log-level=<int>   Set global log level
  --log-level=<type-match>:<int>
                      Set specific log level
  -v                  Display version information on startup
  -h, --help          This help
  --in-memory   Operate entirely in memory. This will
                      disable secondary process support
  --base-virtaddr     Base virtual address

EAL options for DEBUG use only:
  --huge-unlink       Unlink hugepage files after init
  --no-huge           Use malloc instead of hugetlbfs
  --no-pci            Disable PCI
  --no-hpet           Disable HPET
  --no-shconf         No shared config (mmap'd files)

EAL Linux options:
  --socket-mem        Memory to allocate on sockets (comma separated values)
  --socket-limit      Limit memory allocation on sockets (comma separated values)
  --huge-dir          Directory where hugetlbfs is mounted
  --file-prefix       Prefix for hugepage filenames
  --create-uio-dev    Create /dev/uioX (usually done by hotplug)
  --vfio-intr         Interrupt mode for VFIO (legacy|msi|msix)
  --legacy-mem        Legacy memory mode (no dynamic allocation, contiguous segments)
  --single-file-segments Put all hugepage memory in single files
  --match-allocations Free hugepages exactly as allocated

===== Application Usage =====

error   Error in rte_eal_init()
EAL: FATAL: Invalid 'command line' arguments.
EAL: Invalid 'command line' arguments.
error   Error in rte_eal_init()

[pinggit]

it works with this config in /root/prox/helper-scripts/rapid/basicrapid.test or
/opt/openstackrapid/samplevnf/VNFs/DPPD-PROX/helper-scripts/rapid/basicrapid.test file:

gencores = [5, 4]
latcores = [8, 9]

where 5, 4, 8, 9 is VCPU ID, mapping to CPUID: 3,9,1,7

just a memo, this won't work - need to avoid using vCPU 0:

gencores = [5, 4]
latcores = [1, 0]

it seems like vCPU 0 is reserved for some tasks.

[root@stack2-jump rapid]# ./runrapid.py --runtime 5
Using 'rapid.env' as name for the environment
Using 'basicrapid.test' for test case definition
Using 'machine.map' for machine mapping
Runtime: 5
Traceback (most recent call last):
  File "./runrapid.py", line 960, in <module>
    clients[-1].fork_cmd(cmd, 'PROX Testing on TestM%d'%(vm+1))
  File "/opt/openstackrapid/samplevnf/VNFs/DPPD-PROX/helper-scripts/rapid/prox_ctrl.py", line 104, in fork_cmd
    % (name, ex.returncode, ex.output.strip()))
RuntimeError: Child process 'PROX Testing on TestM1' failed:
ssh returned exit status 1:
Command line: /root/prox/build/prox -e -t -o cli -f /root/gen.cfg
=== prox 20.05 ===
        Using DPDK 19.11.0
        ncurses version = 5.9 (ncurses 5.9.20130511)
        TERM left unchanged to unknown
        Does not support Intel RDT Monitoring capability
=== Parsing configuration file '/root/gen.cfg' ===
        *** Reading [lua] section ***
        *** Reading [variables] section ***
        *** Reading [eal options] section ***
        *** Reading [cache set #] sections ***
        *** Reading [port #] sections ***
        *** Reading [defaults] section ***
        *** Reading [global] section ***
        *** Reading [core #] sections ***
=== Setting up RTE EAL ===
        Worker threads core mask is 0x619
        With master core index 0, full core mask is 0x619
        EAL command line: /root/prox/build/prox -c0x619 --master-lcore=0 -n4
EAL: Detected 10 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: lcore 10 unavailable
EAL: invalid coremask, please check specified cores are part of 0-9
Usage: /root/prox/build/prox [options]
EAL common options:
  -c COREMASK         Hexadecimal bitmask of cores to run on
  -l CORELIST         List of cores to run on
                      The argument format is <c1>[-c2][,c3[-c4],...]
                      where c1, c2, etc are core indexes between 0 and 128

[pinggit]

this is a working rapid config:

[root@stack2-jump rapid]# cat /root/prox/helper-scripts/rapid/basicrapid.test
[DEFAULT]
name = BasicSwapTesting
number_of_tests = 2
total_number_of_test_machines = 2
prox_socket = true
prox_launch_exit = true
lat_percentile = 99
tasks=[0]

[TestM1]
name = Generator
config_file = gen.cfg
dest_vm = 2
gencores = [5, 4]
latcores = [8, 9]
#gencores = [2]
#latcores = [1]

[TestM2]
name = Swap
config_file = swap.cfg
cores = [2]

[BinarySearchParams]
drop_rate_threshold = 0.001
lat_perc_threshold = 300
lat_avg_threshold = 1000
lat_max_threshold = 20000
accuracy = 0.1
#startspeed = 300
startspeed = 3

[test1]
test=warmuptest
flowsize=512
packetsize=64
warmupspeed=1
warmuptime=2
[test2]

test=flowsizetest
packetsizes=[64,256,512,1024,1500]
# the number of flows in the list need to be powers of 2, max 2^20
# Select from following numbers: 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65535, 131072, 262144, 524280, 1048576
flows=[16384]