Simple Measurement of UPF Performance

[s5uishida]

Hi,

Using your tool, I briefly measured the following UPFs on Virtualbox VMs.

• Open5GS UPF (TUN, TAP)
• free5GC UPF
• UPG-VPP (VPP-DPDK UPF)
• eUPF (eBPF/XDP UPF)

https://github.com/s5uishida/simple_measurement_of_upf_performance

Once again, thank you very much for the excellent tool!

[linouxis9]

Hi @s5uishida!
Super cool, thanks for sharing!! :-) And thanks for showcasing one of the use cases of PacketRusher!
You might want to try to measure the performance of these UPF with multiple UEs doing iperf traffic at the same time as well!
I expect some of these UPF (UPG-VPP and eUPF for instance but also PacketRusher) to be much more performant when used with real network interfaces, but nonetheless, it's a good test :-)
I'm also in the process of replacing PacketRusher's userplane with eUPF's eBPF/XDP code, as the current userplane uses free5gc's gtp5g kernel module which has a lot of issues, but sadly this might not be the best choice according to your results.
Thanks again and cheers!
Valentin

[s5uishida]

Hi @linouxis9

Sorry, I'll add some information.

1. I executed ip vrf exec vrf0000001000 bash and then pinged.
2. I also tried ip vrf exec vrf0000001000 ping 192.168.16.152 directly.

The results were the same.
I built it using your instructions.

Thank you very much for your efforts!

[linouxis9]

Hi @s5uishida,

I was able to reproduce the issue you found and it should be fixed in the latest commit of the eUPF branch.
Could you try and see if it's improved on your side?
On my side, I have compared PacketRusher using eBPF vs gtp5g using proprietary UPF and I'm able to reach 2.25 GB/s with a single UE using eBPF vs 5 GB/s using gtp5g. Of course in generic mode for now. Trading performance for flexibility of eBPF seems worthwhile to me, especially since you can always add more UEs to increase traffic and in the real world, no UE is going to reach individually 2.2 GB/s anyway.
The next step is to add support for concurrent PDU Sessions with multiple UEs at the same time (without the need for multiple gNodeB IPs like with gtp5g), multiple PDU Sessions for a single UE, and PDU Session handover.

Thanks a lot and Merry Christmas!
Valentin

[s5uishida]

Hi @linouxis9

Thank you very much for your response.

I tried the latest ebpf branch and the same occurred.

Also, I think that the flexibility and performance of eBPF is very meaningful, too.

I appreciate your very kind efforts.

Merry Christmas!

[linouxis9]

Hi @s5uishida,

When you say the same occured, do you mean it worked and you again observed no performance changes, or the ping did not work?

Thanks a lot for contributing to PacketRusher through your testing and suggestions!!

Valentin

[s5uishida]

Hi @linouxis9

There was no response to ping.

After getting it working fine on the main branch, I switched to the ebpf branch again without rebooting the VM, and both ping and iperf3 worked.

The startup log on ebpf branch is as follows.

# ./packetrusher ue
INFO[0000] Loaded config at: /root/PacketRusher/config/config.yml 
INFO[0000] PacketRusher version 1.0.1                   
INFO[0000] ---------------------------------------      
INFO[0000] [TESTER] Starting test function: Testing an ue attached with configuration 
INFO[0000] [TESTER][UE] Number of UEs: 1                
INFO[0000] [TESTER][UE] disableTunnel is false          
INFO[0000] [TESTER][GNB] Control interface IP/Port: 192.168.66.131/9487 
INFO[0000] [TESTER][GNB] Data interface IP/Port: 192.168.13.131/2152 
INFO[0000] [TESTER][AMF] AMF IP/Port: 192.168.66.111/38412 
INFO[0000] ---------------------------------------      
INFO[0000] [GNB] SCTP/NGAP service is running           
INFO[0000] [GNB] Initiating NG Setup Request            
INFO[0000] [GNB][SCTP] Receive message in 0 stream      
INFO[0000] [GNB][NGAP] Receive NG Setup Response        
INFO[0000] [GNB][AMF] AMF Name: open5gs-amf0            
INFO[0000] [GNB][AMF] State of AMF: Active              
INFO[0000] [GNB][AMF] Capacity of AMF: 255              
INFO[0000] [GNB][AMF] PLMNs Identities Supported by AMF -- mcc: 001 mnc:01 
INFO[0000] [GNB][AMF] List of AMF slices Supported by AMF -- sst:01 sd:010203 
INFO[0001] [TESTER] TESTING REGISTRATION USING IMSI 0000001000 UE 
INFO[0001] [GNB] Received incoming connection from new UE 
INFO[0001] [UE] Initiating Registration                 
INFO[0001] [UE] Switched from state 0 to state 1        
INFO[0001] [GNB][SCTP] Receive message in 1 stream      
INFO[0001] [GNB][NGAP] Receive Downlink NAS Transport   
INFO[0001] [UE][NAS] Message without security header    
INFO[0001] [UE][NAS] Receive Authentication Request     
INFO[0001] [UE][NAS][MAC] Authenticity of the authentication request message: OK 
INFO[0001] [UE][NAS][SQN] SQN of the authentication request message: VALID 
INFO[0001] [UE][NAS] Send authentication response       
INFO[0001] [UE] Switched from state 1 to state 2        
INFO[0001] [GNB][SCTP] Receive message in 1 stream      
INFO[0001] [GNB][NGAP] Receive Downlink NAS Transport   
INFO[0001] [UE][NAS] Message with security header       
INFO[0001] [UE][NAS] Message with integrity and with NEW 5G NAS SECURITY CONTEXT 
INFO[0001] [UE][NAS] successful NAS MAC verification    
INFO[0001] [UE][NAS] Receive Security Mode Command      
INFO[0001] [UE][NAS] Type of ciphering algorithm is 5G-EA0 
INFO[0001] [UE][NAS] Type of integrity protection algorithm is 128-5G-IA2 
INFO[0001] [GNB][SCTP] Receive message in 1 stream      
INFO[0001] [GNB][NGAP] Receive Initial Context Setup Request 
INFO[0001] [GNB][UE] UE Context was created with successful 
INFO[0001] [GNB][UE] UE RAN ID 1                        
INFO[0001] [GNB][UE] UE AMF ID 1                        
INFO[0001] [GNB][UE] UE Mobility Restrict --Plmn-- Mcc: not informed Mnc: not informed 
INFO[0001] [GNB][UE] UE Masked Imeisv: 1110000000ffff00 
INFO[0001] [GNB][UE] Allowed Nssai-- Sst: [01] Sd: [010203] 
INFO[0001] [GNB][NAS][UE] Send Registration Accept.     
INFO[0001] [GNB][NGAP][AMF] Send Initial Context Setup Response. 
INFO[0001] [GNB] Initiating Initial Context Setup Response 
INFO[0001] [UE][NAS] Message with security header       
INFO[0001] [UE][NAS] Message with integrity and ciphered 
INFO[0001] [UE][NAS] successful NAS MAC verification    
INFO[0001] [UE][NAS] successful NAS CIPHERING           
INFO[0001] [UE][NAS] Receive Registration Accept        
INFO[0001] [UE][NAS] UE 5G GUTI: [192 0 5 222]          
INFO[0001] [UE] Switched from state 2 to state 3        
INFO[0001] [UE] Initiating New PDU Session              
INFO[0001] [UE] Initiating New PDU Session              
INFO[0001] [GNB][SCTP] Receive message in 1 stream      
INFO[0001] [GNB][NGAP] Receive Downlink NAS Transport   
INFO[0001] [UE][NAS] Message with security header       
INFO[0001] [UE][NAS] Message with integrity and ciphered 
INFO[0001] [UE][NAS] successful NAS MAC verification    
INFO[0001] [UE][NAS] successful NAS CIPHERING           
INFO[0001] [UE][NAS] Receive Configuration Update Command 
INFO[0001] [UE] Initiating Configuration Update Complete 
INFO[0001] [GNB][SCTP] Receive message in 1 stream      
INFO[0001] [GNB][NGAP] Receive PDU Session Resource Setup Request 
INFO[0001] [GNB][NGAP][UE] PDU Session was created with successful. 
INFO[0001] [GNB][NGAP][UE] PDU Session Id: 1            
INFO[0001] [GNB][NGAP][UE] NSSAI Selected --- sst: NSSAI was not selected sd: NSSAI was not selected 
INFO[0001] [GNB][NGAP][UE] PDU Session Type: ipv4       
INFO[0001] [GNB][NGAP][UE] QOS Flow Identifier: 1       
INFO[0001] [GNB][NGAP][UE] Uplink Teid: 1               
INFO[0001] [GNB][NGAP][UE] Downlink Teid: 1             
INFO[0001] [GNB][NGAP][UE] Non-Dynamic-5QI: 9           
INFO[0001] [GNB][NGAP][UE] Priority Level ARP: 8        
INFO[0001] [GNB][NGAP][UE] UPF Address: 192.168.13.151 :2152 
INFO[0001] [GNB] Initiating PDU Session Resource Setup Response 
INFO[0001] [UE][NAS] Message with security header       
INFO[0001] [UE][NAS] Message with integrity and ciphered 
INFO[0001] [UE][NAS] successful NAS MAC verification    
INFO[0001] [UE][NAS] successful NAS CIPHERING           
INFO[0001] [UE][NAS] Receive DL NAS Transport           
INFO[0001] [UE][NAS] Receiving PDU Session Establishment Accept 
INFO[0001] [UE][NAS] PDU session QoS RULES: [1 0 6 49 49 1 1 255 1] 
INFO[0001] [UE][NAS] PDU session DNN: internet          
INFO[0001] [UE][NAS] PDU session NSSAI -- sst: 1 sd: 123 
INFO[0001] [UE][NAS] PDU address received: 10.45.0.2    
{"level":"debug","time":"2023-12-25T11:57:05+09:00","message":"EBPF: Put QER: internalId=0, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:5 MaxBitrateUL:1000000 MaxBitrateDL:100000 StartUL:0 StartDL:0}"}
{"level":"debug","time":"2023-12-25T11:57:05+09:00","message":"EBPF: Put FAR: internalId=0, qerInfo={Action:2 OuterHeaderCreation:0 Teid:1 RemoteIP:2534254784 LocalIP:2198710464 IfIndex:2 TransportLevelMarking:0}"}
{"level":"debug","time":"2023-12-25T11:57:05+09:00","message":"EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:0 QerId:0 SdfFilter:<nil>}"}
{"level":"debug","time":"2023-12-25T11:57:05+09:00","message":"EBPF: Put FAR: internalId=1, qerInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:2534254784 LocalIP:2198710464 IfIndex:2 TransportLevelMarking:0}"}
INFO[0001] Attached XDP program to iface "enp0s9" (index 2) 
INFO[0001] Attached XDP program to iface "vrf0000001000" (index 5) 
WARN[0001] [UE][GTP] You are using PacketRusher's eBPF Preview mode: 
INFO[0001] [UE][GTP] Interface vrf0000001000 has successfully been configured for UE 10.45.0.2 
INFO[0001] [UE][GTP] You can do traffic for this UE using VRF vrf0000001000, eg: 
INFO[0001] [UE][GTP] sudo ip vrf exec vrf0000001000 iperf3 -c IPERF_SERVER -p PORT -t 9000 

[infinitydon]

@linouxis9 - Saw the issue you opened at the travelping github page, I think that will be a great way also to have very good performance.
Hopefully the travelping engineers respond on time :). Below are some things that came to my mind concerning this:

• Currently travelping upg-vpp does not support multi-queue (spread GTPU processing across multiple CPU cores) for increased performance, upstream VPP supports it but for some technical reasons, upg-vpp doesn't. So single core performance will be tied to the CPU model in terms of the frequency, the higher the frequency the more performance can be achieved without using multiple cores. While everyone may not be interested in very high throughput rate, I just thought of mentioning it.
• With the above, I don't know if you have also considered the native GTPU plugin for VPP (not sure if it is actively maintained)? Travelping is not using it also but I am not sure why.
• From experience, DPDK seems to be the best NIC driver selection for the upg-vpp, kernel based NIC driver performance can be poor (also in the Telco production networks that I have worked on, GTPU processing is done using DPDK). This is just FYI
• I am not a programmer but I think going this route in my opinion can make software design of PackerRusher simpler since VPP will take care of the underlying routing/performance responsibility. Also since VPP supports a lot of different NIC drivers including XDP, maybe you won't have to spend more effort on the eBPF code?

[linouxis9]

Hi @infinitydon,

Thanks a lot for your messages and advices!
I did look into the native GTP-U VPP plugin, and it could probably work for PacketRusher's use cases.
UPG-VPP probably does not use it because it lacks features required for an UPF (SDF filter...), similar to kernel GTP-U module's limitations.
However, PacketRusher, as a gNodeB simulator, does not need all that, so it's probably doable for us to integrate it rather than the full UPG-VPP!
The native GTP-U VPP plugin is basically a glorified GTP-U encapsulation/decapsulation plugin, and is pretty much what we need.

I initially wanted to avoid VPP/DPDK because one of the goals of PacketRusher is to be easy to use (just compile PacketRusher and you are good to go), while DPDK has some complex requirements which might not be easy to understand for everyone.
However, if we can use VPP with either DPDK and XDP at choice, it would be perfect! By default, PacketRusher would use XDP and users could select DPDK if they have the requirements (supported hardware NIC or virtualized NIC)

I'll try to work on a PoC.

Thanks a lot @infinitydon!
Valentin

[s5uishida]

Hi @linouxis9

I found the following article on free5GC Blogs.

https://free5gc.org/blog/20240110/20240110/

According to that, although detailed configuration information is not shown, such good results are quite unexpected. I also measured it on Virtualbox VMs, but the results were not as good as that.

FYI.

[linouxis9]

Hi @s5uishida,

Yes, I saw the blog post as well! I think the main difference which could explain the speed difference between your environment and theirs would be that they had a better CPU / allocated more cores to the UPF, but yes it's indeed very interesting to observe.

Cheers!
Valentin

[s5uishida]

Hi @linouxis9

My CPU is Core i5 10400 (6 cores).
The CPU of the free5GC blog is Core i5 12400 (6 cores).

Both UPF VMs have two CPU cores. Regarding memory, my VM has 8GB and the VM of the free5GC blog has 2GB.
The CPU generations are two different, and I was surprised that there would be such a difference.