Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cyclictest benchmark UOS geting high #4258

Closed
paspndevops opened this issue Dec 16, 2019 · 27 comments
Closed

cyclictest benchmark UOS geting high #4258

paspndevops opened this issue Dec 16, 2019 · 27 comments
Assignees
@fuzhongl
Labels
priority: P3-Medium type: bug Something isn't working

Comments

@paspndevops
Copy link

paspndevops commented Dec 16, 2019 •
edited
Loading

ubu_12h_S

Hi,

Could you please guide me to get better cpu latency result on UOS here i attached 12hrs latency with stress for your reference.

  • "cyclictest -mn -Sp90 -i200 -h400 -d720m"

SOS Kernel : 4.19.78-98.iot-lts-lts2018-sos
UOS Kernel : 4.15.0-72-lowlatency

For regular kernel i am getting latency from 100 to 200 but with UOS i am getting more then 5000.

Thank!
@gvancuts
Copy link

Hi @paspndevops , can you let us know how you have configured your system (ACRN version, scenario used, etc.)? Specifically, are you using an RTVM to run the UOS and have you performed any specific configuration to leverage CAT (if available on your platform).

@paspndevops
Copy link
Author

@gvancuts

Hi gvancuts,
acrn-dm -v
DM version is: 1.4-unstable- (daily tag:), build by root@2019-11-25 09:11:52

-scenario - SDC2
-UOS is preempt kernel low letency(Ubuntu)/Fedora 30 (5.2.17-rt9 PREEMPT-RT)

-SOS kernel is 4.19.78-98.iot-lts-lts2018-sos

Q1. Shell i get good result if i install preempt-rt SOS kernel ? How

Thanks !

@Mingyuan18
Copy link

[External_System_ID] ACRN-5861

@Mingyuan18 Mingyuan18 added priority: P3-Medium type: bug Something isn't working labels Dec 17, 2019
@fuzhongl
Copy link
Contributor

@paspndevops Please share your UOS launch script.
Thanks!

@paspndevops
Copy link
Author

@fuzhongl
Here i attached my Ubuntu UOS launcher script for your reference.

launch_Ubuntu.txt

@fuzhongl
Copy link
Contributor

@paspndevops
There are two issues in your script:

  1. No RTVM option(--rtvm \ and --lapic_pt ); please refer to
    https://github.com/projectacrn/acrn-hypervisor/blob/master/devicemodel/samples/nuc/launch_hard_rt_vm.sh

2)You are using virtio_blk for UOS Image; We prefer passthrough SSD to UOS for RTVM
-s 3,virtio-blk,/root/ubuntu_images/uosUBU.img \

So, the UOS latency will be highfor your case.
thanks！

@paspndevops
Copy link
Author

@fuzhongl

Hi ,
I tried with below DM but no improvement on latency also please let me know how to parse passthru.

mem_size=1048M

acrn-dm -A -m $mem_size -s 0:0,hostbridge
--lapic_pt
--rtvm
-s 5,virtio-console,@stdio:stdio_port
-s 6,virtio-hyper_dmabuf
-s 3,virtio-blk,/root/ubuntu_images/uosUBU.img
-s 4,virtio-net,tap1
-s 7,virtio-rnd
--ovmf /usr/share/acrn/bios/OVMF.fd
$pm_channel $pm_by_vuart $pm_vuart_node
$logger_setting
--mac_seed $mac_seed
-U 495ae2e5-2603-4d64-af76-d4bc5a8ec0e5
$vm_name

Thanks !

@fuzhongl
Copy link
Contributor

@paspndevops Which platform are you using?
please run "lshw" in sos and share the info.
Thanks!

@paspndevops
Copy link
Author

@fuzhongl

intel-desktop
description: Desktop Computer
width: 64 bits
capabilities: smbios-3.1 dmi-3.1 vsyscall32
configuration: boot=normal chassis=desktop uuid=A648A33A-633A-A4BA-64A6-94C69110EF7C
*-core
description: Motherboard
product: NUC7i7BNB
vendor: Intel Corporation
physical id: 0
version: J31145-304
serial: GEBN72200F0J
slot: Default string
*-firmware
description: BIOS
vendor: Intel Corp.
physical id: 0
version: BNKBL357.86A.0080.2019.0725.1139
date: 07/25/2019
size: 64KiB
capacity: 8128KiB

Thanks!

@paspndevops
Copy link
Author

@fuzhongl

000399 000000

Total: 002999974

Min Latencies: 00002

Avg Latencies: 00006

Max Latencies: 01737

Histogram Overflows: 00003

Histogram Overflow at cycle number:

Thread 0: 562117 847108 2877099

cyclictest -mn -Sp90 -i200 -h400 -D10m

@fuzhongl
Copy link
Contributor

@paspndevops Please follow the link to set BIOS: chapter Recommended BIOS settings
https://projectacrn.github.io/latest/getting-started/rt_industry.html
Thanks!

@paspndevops
Copy link
Author

@fuzhongl
i followed BIOS settings as per documentation please suggest me on following for passthru:

2)You are using virtio_blk for UOS Image; We prefer passthrough SSD to UOS for RTVM
-s 3,virtio-blk,/root/ubuntu_images/uosUBU.img \

Thanks!

@fuzhongl
Copy link
Contributor

@paspndevops
How many SSD in your device? Please using "lspci -nn"to check bdf info(not found with lshw log) then follow the link to add SSD passthrough to UOS:
https://github.com/projectacrn/acrn-hypervisor/blob/master/devicemodel/samples/nuc/launch_hard_rt_vm.sh
Or send out the log of "lspci -nn", I will update it for you.
Thanks!

@paspndevops
Copy link
Author

@fuzhongl

00:00.0 Host bridge [0600]: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge [8086:5af0] (rev 0b)
00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:5927] (rev 06)
00:08.0 System peripheral [0880]: Intel Corporation Skylake Gaussian Mixture Model [8086:1911]
00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller [8086:9d2f] (rev 21)
00:14.2 Signal processing controller [1180]: Intel Corporation Sunrise Point-LP Thermal subsystem [8086:9d31] (rev 21)
00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-LP CSME HECI #1 [8086:9d3a] (rev 21)
00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-LP SATA Controller [AHCI mode] [8086:9d03] (rev 21)
00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port [8086:9d10] (rev f1)
00:1c.5 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #6 [8086:9d15] (rev f1)
00:1c.7 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #8 [8086:9d17] (rev f1)
00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-LP PCI Express Root Port #9 [8086:9d18] (rev f1)
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:9d4e] (rev 21)
00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-LP PMC [8086:9d21] (rev 21)
00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-LP HD Audio [8086:9d71] (rev 21)
00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-LP SMBus [8086:9d23] (rev 21)
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (4) I219-V [8086:15d8] (rev 21)
3a:00.0 Network controller [0280]: Intel Corporation Wireless 8265 / 8275 [8086:24fd] (rev 78)
3b:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5229 PCI Express Card Reader [10ec:5229] (rev 01)
3c:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:2522]

Thanks!

@fuzhongl
Copy link
Contributor

fuzhongl commented Dec 19, 2019 •
edited
Loading

@paspndevops Nor sure which SSD you installed SOS rootfs in, if you install SOS rootfs in SATA SSD;then you can install UOS in NVMe SSD and passthrough it to UOS; please refer this link:
https://github.com/projectacrn/acrn-hypervisor/blob/master/devicemodel/samples/nuc/launch_hard_rt_vm.sh
The modification: only change vpid and bdf info(marked by bold)
passthru_vpid=(
["eth"]="8086 156f"
["sata"]="8086 9d03"
["nvme"]="8086 2522"
)
passthru_bdf=(
["eth"]="0000:00:1f.6"
["sata"]="0000:00:17.0"
["nvme"]="0000:3c:00.0"
)

-s 2,passthru,3c/0/0 \

@paspndevops
Copy link
Author

@fuzhongl

Thanks for your continues support i tried with below changes and capturing bench marking for 12hrs getting latency more then 1000:

Could you please suggest me on Kernel modification if any

/usr/bin/acrn-dm -A -m $mem_size -s 0:0,hostbridge
--lapic_pt
--rtvm
--virtio_poll 1000000
-U 495ae2e5-2603-4d64-af76-d4bc5a8ec0e5
-s 2,passthru,3c/0/0 \
-s 3,virtio-console,@stdio:stdio_port
-s 8,virtio-net,tap0 \ ** test ***
$pm_channel $pm_by_vuart
--ovmf /usr/share/acrn/bios/OVMF.fd
hard_rtvm

Thanks !

@fuzhongl
Copy link
Contributor

fuzhongl commented Dec 23, 2019 •
edited
Loading

@paspndevops After you copy everything to RTVM, could you please remove following(marked by bold) in script?
-s 8,virtio-net,tap0 \ ** test **
Then, retest cyclictest.
Or you should passthough NIC to RTVM.
Thanks!

@paspndevops
Copy link
Author

@fuzhongl
Hi,
Please find below observations on Ubuntu SOS:

acrn-dm -A -m $mem_size -s 0:0,hostbridge -s 1:0,lpc -l com1,stdio
-s 2,pci-gvt -G "$2"
-s 5,virtio-console,@pty:pty_port
-s 6,virtio-hyper_dmabuf
-s 3,virtio-blk,/root/ubuntu_images/uosUBU.img
$logger_setting
--mac_seed $mac_seed
-U d2795438-25d6-11e8-864e-cb7a18b34643
-k /boot/acrn/org.clearlinux.iot-lts2018-preempt-rt.4.19.72-18
-B "root=/dev/vda2 rw rootwait maxcpus=1 nohpet console=tty0 console=hvc0
console=ttyS0 no_timer_check ignore_loglevel log_buf_len=16M
consoleblank=0 tsc=reliable i915.avail_planes_per_pipe=$4
i915.enable_hangcheck=0 i915.nuclear_pageflip=1 i915.enable_guc_loading=0
i915.enable_guc_submission=0 i915.enable_guc=0" $vm_name

root@ubuntu:~# uname -a
Linux ubuntu 4.19.72-18.iot-lts2018-preempt-rt #1 SMP PREEMPT RT Tue Nov 12 11:59:53 PST 2019 x86_64 x86_64 x86_64 GNU/Linux

Result for 30 Minutes :

Total: 008999996

Min Latencies: 00002

Avg Latencies: 00005

Max Latencies: 00484

Histogram Overflows: 00001

Histogram Overflow at cycle number:

Thread 0: 179201

Thanks!

@fuzhongl
Copy link
Contributor

fuzhongl commented Dec 23, 2019 •
edited
Loading

@paspndevops Do you mean the latency of RTVM is larger than normal VM?
Max Latencies: 1000+ vs 00484

It is unfair RTVM 12hrs Vs normal VM 30 Minutes
Could you please retest them with 12hrs both?
Thanks!

@gvancuts
Copy link

@fuzhongl , is it appropriate to use the SDC2 scenario in this case? That User VM UUID (495ae2e5-2603-4d64-af76-d4bc5a8ec0e5) does not have the GUEST_FLAG_HIGHEST_SEVERITY set, does that get set by acrn-dm when launching the VM? Or should we really need to either use the industrial scenario or update the SDC2 to add this?

@fuzhongl
Copy link
Contributor

@gvancuts Checked with ACRN ARCH, GUEST_FLAG_HIGHEST_SEVERITY should not be set for SDC2 scenario. The industrial scenario is preferred for RTVM.
Thanks!

@paspndevops
Copy link
Author

@fuzhongl
I tried with both VM 's for 12Hrs but no change in latency please give me some steps to achieve good latency for fedora-30 server Guest UOS.Or sample RTVM launcher script for FC30 UOS

Also tried with both SDC2 and Industrial scenarios ..

Thanks!

@fuzhongl
Copy link
Contributor

@paspndevops Please share your UOS launch script for both Guest VM.
Thanks!

@paspndevops
Copy link
Author

paspndevops commented Dec 26, 2019 •
edited
Loading

@fuzhongl

Laun_UOS.txt (UOS 1)

[root@localhost ~]# cyclictest
/dev/cpu_dma_latency set to 0us
policy: other/other: loadavg: 0.00 0.08 0.08 1/95 697

T: 0 ( 697) P: 0 I:1000 C: 2037 Min: 7 Act: 10 Avg: 9 Max: 235

nvme.txt (UOS 2)

root@clr-preempt-rt ~/rt-tests # cyclictest
/dev/cpu_dma_latency set to 0us
policy: other/other: loadavg: 0.36 0.27 0.14 1/115 17646

T: 0 (17646) P: 0 I:1000 C: 7442 Min: 4 Act: 5 Avg: 5 Max: 9

Thanks!

@paspndevops
Copy link
Author

@fuzhongl

Two launcher scripts are attached as link on above thread(Laun_UOS.txt, nvme.txt)

Thanks !

@fuzhongl
Copy link
Contributor

@paspndevops The only difference is virtio_blk Vs NVMe SSD passthrough, correct?
The RT kernel and kernel cmdline is te same.
The only difference is virtio_blk Vs NVMe SSD passthrough, correct?
Thanks!

@paspndevops
Copy link
Author

@fuzhongl

created new thread UOS CPU Latency Fedora Server 30-rt9
thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fuzhongl fuzhongl

Labels
priority: P3-Medium type: bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gvancuts @fuzhongl @Mingyuan18 @paspndevops