-
Notifications
You must be signed in to change notification settings - Fork 157
Hooking up BESS Ports
If you are walking through the Intro to BESS, you have just written your first BESS configuration script using synthetic traffic generated by a FlowGen module and released out to a Sink module which deletes all of the packets. This document will explain how to set up BESS ports, which will let you read in or write out packets from a network interface card, a virtual machine, an application, etc.
Working with Ports operates in two stages. You create a port in your configuration script, and then you create a module to read or write to that port. The modules that connect to ports are PortInc, PortOut, QueueInc, and QueueOut. We will show you how to use them in the following examples.
BESS utilizes DPDK Poll-Mode Drivers (PMDs) to perform high-performance packet I/O with direct hardware NIC access. To use PMDs, first you need to unbind network interfaces from Linux kernel device drivers, then bind them to a special module (uio_pci_generic, vfio-pci, or igb_uio) to enable user-space I/O.
DPDK provides a script to (un)bind network interfaces. The following command with --status can be used to list all network interfaces in the system.
$ bin/dpdk_devbind.py --status
Network devices using DPDK-compatible driver
============================================
<none>
Network devices using kernel driver
===================================
0000:01:00.0 'MT27500 Family [ConnectX-3]' if=rename6,p2p1 drv=mlx4_core unused=
0000:03:00.0 'Ethernet Connection X552 10 GbE SFP+' if=eth0 drv=ixgbe unused=
0000:03:00.1 'Ethernet Connection X552 10 GbE SFP+' if=eth1 drv=ixgbe unused=
0000:07:00.0 'I210 Gigabit Network Connection' if=p5p1 drv=igb unused= *Active*
0000:08:00.0 'I210 Gigabit Network Connection' if=p6p1 drv=igb unused=
Other network devices
=====================
<none>Suppose we want the Intel X552 dual-port NIC (03:00.0 and 03:00.1) to be used by BESS. These ports are currently used by the kernel ixgbe driver as eth0 and eth1. The command below binds the ports to uio_pci_generic for user-space I/O.
$ sudo modprobe uio_pci_generic
$ sudo bin/dpdk_devbind.py -b uio_pci_generic 03:00.0 03:00.1Then you can see the ports moved to under the "DPDK-compatible driver" section. The ports are ready to be used by BESS.
$ bin/dpdk_devbind.py --status
Network devices using DPDK-compatible driver
============================================
0000:03:00.0 'Ethernet Connection X552 10 GbE SFP+' drv=uio_pci_generic unused=
0000:03:00.1 'Ethernet Connection X552 10 GbE SFP+' drv=uio_pci_generic unused=
Network devices using kernel driver
===================================
0000:01:00.0 'MT27500 Family [ConnectX-3]' if=rename6,p2p1 drv=mlx4_core unused=uio_pci_generic
0000:07:00.0 'I210 Gigabit Network Connection' if=p5p1 drv=igb unused=uio_pci_generic *Active*
0000:08:00.0 'I210 Gigabit Network Connection' if=p6p1 drv=igb unused=uio_pci_generic
Other network devices
=====================
<none>NOTE: If you connect to a server with ssh connection, do not unbind the interface on which the connection is running.
Now that you have configured your network interface to use the DPDK driver, BESS can automatically find it when it starts up. To create a port connecting to that driver, you can add the following to your script:
myport::PMDPort(port_id=0, num_inc_q=2, num_out_q=2)
This tells BESS to create a port connecting to the first of your network interfaces, and create two input and output queues on it. You can read in packets from this port, just like from Source or Flowgen, with
input0::QueueIn(port=myport, qid=0) #This module will read from myport on queue 0
input1::QueueIn(port=myport, qid=1) #This module will read from myport on queue 1
You can write out packets to this port just like Sink() with
output0::QueueOut(port=myport, qid=0) #This module will write to myport on queue 0
output1::QueueOut(port=myport, qid=1) #This module will write to myport on queue 1
BESS also supports DPDK Vhost PMD ports, which can be used forward packets between virtual machines. Creating a Vhost PMD port is very much like creating a physical PMD port, but instead of passing the constructor a PCI address or a DPDK port ID, you pass in a DPDK vdev specifier. The command below will create a new Vhost-backed PMDPort with a single queue and a chardev at /tmp/my_vhost.sock. For a more detailed description of the fields you can set see the DPDK documentation.
my_vhost::PMDPort(name='my_vhost', vdev='eth_vhost0,iface=/tmp/my_vhost.sock,queues=1')
You can now use my_vhost and other Vhost ports you create in combination with any number of bess modules, to connect VMs and process the packets flowing between them.
If you're using Qemu as your VMM, you can connect my_vport to a VM with a command line like this.
qemu-system-x86_64 <other args...>
-chardev socket,id=mychr,path=/tmp/my_vhost.sock \
-netdev vhost-user,id=mydev,chardev=mychr,vhostforce,queues=1 \
-device virtio-net-pci,netdev=mydev
BESS also has support for interfaces called VPorts which are exposed to the kernel. If you want to connect containers to BESS, a VPort is the way to go. VPorts require the BESS kernel module. See here for how to build and load the BESS kernel module.
With the BESS kernel module built and loaded we can now create VPorts and use them to provide networking between containers.
Before doing anything, boot a container.
docker pull ubuntu
docker run -i -d --net=none --name=vport_test ubuntu /usr/bin/tail -f /dev/null
Currently vport_test has no network interfaces. You need to write a bessctl script to create and plumb together the interfaces that will allow the host (or other containers) to talk to vport_test. Let's do that step-by-step (skip to the bottom of this section for the complete script).
First, create a VPort and attach it to vport_test by adding the following to your bessctl script.
container_if = VPort(ifname='eth_vport_test', docker='vport_test', ip_addrs=['10.255.99.2/24'])
If you would rather attach the VPort to your container with its CID you can pass container_pid=<CID> instead of docker=... to the constructor. Further, you can also attach the VPort directly to the container's network namespace by passing netns=<NAMESPACE> instead.
Now, if you run ifconfig inside the container you should see a new interface eth_vport_test.
Next, to allow the host to talk to the container, create another VPort by adding the following to your bessctl script. Note the absence of a docker, container_pid or netns field. This means the interface will be created on the host.
host_if = VPort(ifname='eth_host', ip_addrs=['10.255.99.1/24'])
If you run ifconfig on the host you should now see a new interface eth_host; however, the interfaces are not yet plumbed together, so any attempt to send packets on either will fail. To fix this, connect your interfaces with PortInc and PortOut modules by adding the following to your script.
PortInc(port=host_if) -> PortOut(port=container_if)
PortInc(port=container_if) -> PortOut(port=host_if)
Now, on the host you should be able to ping into the container, and vice versa. For the sake of completeness, your final bessctl script should look like the following.
container_if = VPort(ifname='eth_vport_test', docker='vport_test', ip_addrs=['10.255.99.2/24'])
host_if = VPort(ifname='eth_host', ip_addrs=['10.255.99.1/24'])
PortInc(port=host_if) -> PortOut(port=container_if)
PortInc(port=container_if) -> PortOut(port=host_if)
This example only provides basic connectivity, but you can place an arbitrarily complex pipeline of other BESS modules between the PortInc/PortOut pairs to realize more complex policies/behavior.