OpenWRT 18.06 VM as router

HOWTO: OPENWRT Routing

Here's a quick and dirty on getting OpenWRT routing for your kamikazi cluster.

                Inside Kamikazi               |  
|-----------|  |----------------------------| | |------------|  
|  OpenWRT  |==|  External OpenVSwitch br0  |=|=|  internet  |  
|-----------|  |----------------------------| | |------------|  
      ==                                      |  Outside World
|-------------------------------|             |--------------- 
|  Internal OpenVSwitch xenbr0  |  
|-------------------------------|==========================...  
      ==              ==              ==              ==  
|------------|  |------------|  |------------|  |------------|  
|  Domain 0  |  |  Domain 1  |  |  Domain 2  |  |  Domain n  |  
|------------|  |------------|  |------------|  |------------|  

We'll assume you've gotten started with a USB image booting kamikazi 16.04.
I also assume you're either using the local desktop interface or connecting via x2go.
All of this is going to be done from a root terminal, which can be opened from the desktop.
If you're SSHing in, a keystore containing an authorized_keys file will need to be setup on the USB device, because password authentication over SSH is always disallowed from any external access.
The easiest way to get a root filemanager on the desktop, is by clicking the Disks icon on the desktop,
selecting any mounted disk in the left pane, and clicking on the mounted filesystem path in the right pane.
The first icon in the toolbar will be a ! inside of a tab icon, indicating the filemanager is in superuser mode.

First step is having a datastore, I like to use btrfs, so we'll use some functionality of it in this demo.
If you're not using btrfs, just use mkdir instead of btrfs subvolume create.

Create a new btrfs -- I won't prescribe how, short of mentioning you have
your choice of gparted, gnome-disks, or the terminal with parted or cfdisk and mkfs.btrfs.

I made my btrfs on /dev/sdz10.

Once you've created your btrfs, use the 'btrfs' tool to label it something descriptive.

btrfs filesystem label big-storage /dev/sdz10

The next time kamikazi is rebooted, the btrfs partition will be mounted under /mnt/btrfs by label.
If you followed this example, you should get /mnt/btrfs/big-storage/
otherwise you'll have to alter these instructions to suit your mount point.

Once it's back up and running and the kamikazi-boot script has mounted your partition,
it's time to create some subvolumes.

cd /mnt/btrfs/big-storage
btrfs subvolume create vms
btrfs subvolume create templates
cd vms
btrfs subvolume create openwrt

Download the OpenWRT 15.05 xen kernel and rootfs from
https://downloads.openwrt.org/releases/18.06.0/targets/x86/64/
using the built in browser, Midori, or pasting the wget links.

Here are the direct links we'll be using.

wget https://downloads.openwrt.org/releases/18.06.0/targets/x86/64/openwrt-18.06.0-x86-64-vmlinuz  
wget https://downloads.openwrt.org/releases/18.06.0/targets/x86/64/openwrt-18.06.0-x86-64-rootfs-ext4.img.gz  

Now, we'll extract the root filesystem image.

You can use the filemanager to open the .gz and drag the .img file into the right place, or

gunzip openwrt-18.06.0-x86-64-rootfs-ext4.img.gz

This should result in a ~256MB disk image.

Create a configuration file in the same directory with the contents:

#!/usr/sbin/xl create
name = "openwrt"
vcpus = 2
# 256MB+ is overkill, it'll do fine with 64MB, but 128MB is a good middle ground.
memory = 128
maxmem = 128
vif = [ 'mac=02:00:02:00:00:62','mac=02:00:02:00:00:42,bridge=br0' ]
disk = [ '/mnt/btrfs/big-storage/vms/openwrt/openwrt-18.06.0-x86-64-rootfs-ext4.img,,xvda' ]
kernel = "/mnt/btrfs/big-storage/vms/openwrt/openwrt-18.06.0-x86-64-vmlinuz"
extra = "root=PARTUUID=82d11cbf-02 rootfstype=ext4 rootwait xencons=hvc console=tty0 console=hvc0,115200n8 noinitrd"
on_reboot = 'restart'
on_crash = 'restart'

Navigate to the directory and Right click in an empty area of the directory window slightly below any listed files.
Select: Create New -> Empty File
Name the file something like openwrt-18.06.conf.
Right click the new file, and select properties from the context menu.
Click over to the Permissions tab, and change the Execute dropdown from Nobody to Owner.
Right click the file again, and choose "gedit" from the list, or "Open With..." if it does not appear. Paste the contents of the file, and edit the options to suit.
As little as 64MB of ram is 'enough' for a whole subnet, a gigabyte is just massive overkill.

Make sure you get your disk and kernel image paths right.
You may need to remove the second ethernet adapter like this the first time you start it:

vif = [ 'mac=02:00:02:00:00:62' ]

Here's a link to the xen documentation. We already default to openvswitch.
http://wiki.xenproject.org/wiki/Xen_Networking#Open_vSwitch
Also, some valid random MAC addresses in the Xen OUI space probably wouldn't hurt

Starting the vm is as easy as executing the config file by doubleclicking it from a superuser filemanager. Or, you can start it manually from a terminal or ssh...

xl create openwrt-18.06.conf

You may have to navigate to the directory beforehand, or provide the full path to the configuration file.

Some example configuration files have been collected here:
https://github.com/kamilion/kamikazi-core/tree/master/resources/xenial/config/xenguests

Once you have the VM running, we'll have to get to it's console to configure it.

Running VMs can be queried with: xl list and killed by name: xl shutdown openwrt or xl destroy openwrt
Get to the console with xl console openwrt after the VM has finished booting.

Press enter, and you should be greeted with the OpenWRT shell prompt.
First thing you'll need to do is disable the default bridge by replacing it with eth0
and setting the default static IP of the LAN interface to an IP of your choice, other than 192.168.1.1.

uci show network
uci set network.lan.ifname='eth0'
uci set network.lan.ipaddr='10.0.1.254'
uci commit

If you have a static IP from your provider, you can also do that before triggering the reboot.

uci set network.wan.proto='static'
uci set network.wan.ipaddr='192.168.99.5'
uci set network.wan.netmask='255.255.255.248'
uci set network.wan.gateway='192.168.99.1'
uci set network.wan.dns='8.8.8.8 8.8.4.4'
uci commit

This will open up the admin interface to the outside world from a fresh install, post-static IP.

uci add firewall rule
uci set firewall.@rule[-1].src='wan'
uci set firewall.@rule[-1].target='ACCEPT'
uci set firewall.@rule[-1].proto='tcp'
uci set firewall.@rule[-1].dest_port='22'
uci set firewall.@rule[-1].name='sshd'
uci commit firewall

uci add firewall rule
uci set firewall.@rule[-1].src='wan'
uci set firewall.@rule[-1].target='ACCEPT'
uci set firewall.@rule[-1].proto='tcp'
uci set firewall.@rule[-1].dest_port='80'
uci set firewall.@rule[-1].name='http'
uci commit firewall

uci add firewall rule
uci set firewall.@rule[-1].src='wan'
uci set firewall.@rule[-1].target='ACCEPT'
uci set firewall.@rule[-1].proto='tcp'
uci set firewall.@rule[-1].dest_port='443'
uci set firewall.@rule[-1].name='https'
uci commit firewall

It's a good idea to reboot openwrt when you're done, so you know the config stuck.

reboot

Now, it's time to define an external bridge on kamikazi.
Navigate to the USB configuration store:

cd /isodevice/boot/config/

Create a directory/folder named interfaces if one doesn't already exist,
and copy the current configuration inside.

mkdir -p /isodevice/boot/config/interfaces/
cp /etc/network/interfaces.d/xenbr0 interfaces/xenbr0
cp /etc/network/interfaces.d/xenbr0 interfaces/br0

Here, we've copied the same configuration twice -- now we need to edit
them both and choose an network adapter to remove from xenbr0 and
add it to the br0 bridge. Do a search and replace inside the br0 file and
replace all occurrences of xenbr0 with br0, then remove all the ports
except the one you've chosen (I like em2, usually)

cd interfaces
gedit xenbr0 br0

This part is really up to you, depending on the number and type of ethernet, vlan, and bond devices you need to use for your uplink.

Check out our network interface examples directory for some more concrete configurations.
https://github.com/kamilion/kamikazi-core/tree/master/resources/xenial/mods/etc/network/interfaces.examples

These are standard debian/ubuntu network interface definitions.
You'll find more specific information in /usr/share/doc/openvswitch-switch/README.Debian.gz about the configuration stanzas.

It's normally a good idea to reboot (triggering the kamikazi init scripts) after messing around with physical disk configurations or physical network port-to-bridge assignments, but if you know how it works under the hood; feel free to skip the reboot.