Initial integration with Oxide Packet Transformation Engine

[bnaecker]

• Brings in OPTE via the opte-ioctl and opte crates.
• Modifies the instance-ensure request from Nexus to the sled agent, to
  carry the actual information required for setting up the guest OPTE
  port. This includes the actual IP subnet and MAC, rather than things
  like the VPC Subnet UUID.
• Adds a database query and method to extract the above information from
  both the network interface and VPC subnet tables.
• Adds OPTE port for the guests (and currently still a VNIC on top),
  with the right OPTE settings for traffic to flow between two guests in
  the same VPC subnet. That's the virtual-to-physical mapping and a
  router entry for the subnet.
• Adds the VNICs over each OPTE port to the running zone. Note that this
  removes the specification of guest NICs for the zone itself as VNICs.
  They are passed as OPTE ports, and the VNIC is pulled out internally,
  so hopefully little will need to change when the VNIC is removed
  entirely.
• Store the main underlay address for the sled agent, currently its
  dropshot server IP address, in the instance manager, and forward to
  each instance. It's then used as the underlay address when setting up
  the OPTE ports for the guest.
• Fixes bad naming of VNICs used to emulate Chelsio NICs in the current
  setup. Removes the unnecessary loopback address, since the sled agent
  provides its main listening address as the underlay address when
  create OPTE ports.
• Adds better installation of OPTE, xde, and the kernel bits they rely
  on. Don't install opteadm or xde directly, do everything through
  their package repos.

[rcgoodfellow]

This is awesome @bnaecker! Really excited to see this work getting integrated into the control plane 🎉 . Just a few comments on comments above.

[bnaecker]

Though I'm moving this to a draft, this PR is definitely ready for review. There are a few reasons for the draft status. First, any CI is blocked on being able to pull OPTE as a dependency. @rzezeski has started turning the crank on making OPTE public, but it won't be instantaneous. Second, given that fact, I'm going to split out the updates to the scripts in ./tools. Those are really independent.

[rzezeski]

Thanks for putting this all together @bnaecker. I realize that the work and effort required here is not fully appreciated if one were to look at the lines of code alone. Especially given all the testing woes you've hit, plenty of which were my fault. So thank you for your patience and diligence in getting this across the line.

For the most part this seems as I would expect. Some of my comments don't require any action, just stuff I wanted to call out. Though I did not think to explicitly differentiate what needs action and what doesn't. Feel free to reach out for clarification in that regard or if any of my comments are confusing.

[bnaecker]

I've verified that all the tests pass locally, and that I can successfully ping between two Alpine guests. This is no longer blocked on OPTE going public, but the linking failure we've seen before is back. I'm going to mark this open, with the assumption it'll wait until that gets fixed to merge.

[bnaecker]

@rcgoodfellow @rmustacc @rzezeski I'm planning to merge this once CI is green, unless there are objections. Below is some output showing successfully netcat-ing between two guests using this branch.

Guest 1

This is the nc server, with IP 172.30.0.5.

localhost:~# nc -l -s 172.30.0.5 -p 5556
hello
from
the
other
side
^Z[2]+  Stopped                    nc -l -s 172.30.0.5 -p 5556
localhost:~#

Guest 2

This is the nc client, with IP 172.30.0.6

localhost:~# nc 172.30.0.5 5556
hello
from
the
other
side
^Z[2]+  Stopped                    nc 172.30.0.5 5556
localhost:~#

OPTE trace output

Here's the output of opte/dtrace/opte-trace opte-port-process while the above was happening.

AME         DIR FLOW                                        MBLK               RESULT
opte1        out XXX,0.0.0.0:0,0.0.0.0:0                     0xfffffe59539aaa20 Drop { reason: Layer { name: "arp" } }
opte1        out XXX,0.0.0.0:0,0.0.0.0:0                     0xfffffe59539aaa20 Drop { reason: Layer { name: "arp" } }
opte1        out XXX,0.0.0.0:0,0.0.0.0:0                     0xfffffe595399f600 Hairpin(Packet { avail: 80, source: Allocated, segs: [PacketSeg { mp: 0xfffffe595ba91c60, dblk: 0xfffffe595ba8f900, len: 42, avail: 80 }], state: Initialized { len: 42 } })
opte1        out TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe595399f600 Modified
opte0        in  TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe595b877500 Modified
opte0        out TCP,172.30.0.5:5556,172.30.0.6:46459        0xfffffe59535fa4c0 Modified
opte1        in  TCP,172.30.0.5:5556,172.30.0.6:46459        0xfffffe594b3ba7a0 Modified
opte1        out TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe595399f600 Modified
opte0        in  TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe594a66aa40 Modified
opte1        out TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe595399f600 Modified
NAME         DIR FLOW                                        MBLK               RESULT
opte0        in  TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe5963740d40 Modified
opte0        out TCP,172.30.0.5:5556,172.30.0.6:46459        0xfffffe59535fa4c0 Modified
opte1        in  TCP,172.30.0.5:5556,172.30.0.6:46459        0xfffffe596363e140 Modified
opte1        out TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe59537285e0 Modified
opte0        in  TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe5963740a80 Modified
opte0        out TCP,172.30.0.5:5556,172.30.0.6:46459        0xfffffe59535fa4c0 Modified
opte1        in  TCP,172.30.0.5:5556,172.30.0.6:46459        0xfffffe594b3baae0 Modified
opte1        out TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe59537285e0 Modified
opte0        in  TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe594ac9c920 Modified
opte0        out TCP,172.30.0.5:5556,172.30.0.6:46459        0xfffffe595387f9c0 Modified
NAME         DIR FLOW                                        MBLK               RESULT
opte1        in  TCP,172.30.0.5:5556,172.30.0.6:46459        0xfffffe5927dff380 Modified
opte1        out TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe59fb91cd80 Modified
opte0        in  TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe59508b5a80 Modified
opte0        out TCP,172.30.0.5:5556,172.30.0.6:46459        0xfffffe595387f9c0 Modified
opte1        in  TCP,172.30.0.5:5556,172.30.0.6:46459        0xfffffe5a013b20a0 Modified
opte1        out TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe595399f600 Modified
opte0        in  TCP,172.30.0.6:46459,172.30.0.5:5556        0xfffffe594ffbdf40 Modified
opte0        out TCP,172.30.0.5:5556,172.30.0.6:46459        0xfffffe59fe43a860 Modified
opte1        in  TCP,172.30.0.5:5556,172.30.0.6:46459        0xfffffe5953434280 Modified

Caveat

There's a caveat here. The firewall rules are not propagated correctly in this branch. In particular, the guests still can't communicate with one another without adding a rule like:

bnaecker@feldspar : ~/omicron $ pfexec /opt/oxide/opte/bin/opteadm add-fw-rule -p opte0 --action allow --dir in --hosts subnet=172.30.0.0/22 --ports any --priority 10 --protocol tcp

where opte0 is the server guest. I'm planning to do the work to include the current set of firewall rules for the guest in another PR. This branch has lived way too long already.

[rzezeski]

@bnaecker I think that makes perfect sense. There is plenty of major improvement here already that seems ready for integration. I think doing the firewall stuff in a follow up makes it cleaner and easier to focus review anyways. So +1 to your plan.