Skip to content

Commit

Permalink
For Warewulf, create IPoIB devices for IB partitions, mount Lustre
Browse files Browse the repository at this point in the history
Warewulf doesn't handle IB partitions (or Eth VLAN's), see
warewulf/warewulf3#166

Instead create a script to generate the ifcfg- files for the IB
partitions, bring them up, and mount Lustre.
  • Loading branch information
jabl committed Dec 8, 2018
1 parent c6b5a89 commit 4648dab
Show file tree
Hide file tree
Showing 3 changed files with 72 additions and 0 deletions.
3 changes: 3 additions & 0 deletions tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,6 @@

- include_tasks: redhat.yml
when: lustre_client_enabled and ansible_os_family == "RedHat"

- include_tasks: ohpc.yml
when: lustre_client_enabled and ansible_connection == 'chroot'
20 changes: 20 additions & 0 deletions tasks/ohpc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
# tasks file for ansible-role-lustre_client

- name: Script for creating and bringing up IB partition ipoib devices
template:
src: ifcfg-ibpart-create.sh.j2
dest: /usr/local/sbin/ifcfg-ibpart-create.sh
mode: 0744
owner: root
group: root

- name: Run the script from rc.local
lineinfile:
path: /etc/rc.d/rc.local
line: '/usr/local/sbin/ifcfg-ibpart-create.sh'

- name: Make sure rc.local is executable
file:
path: /etc/rc.d/rc.local
mode: 0744
49 changes: 49 additions & 0 deletions templates/ifcfg-ibpart-create.sh.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#!/bin/bash

# Warewulf doesn't support IB partitions (or eth VLAN's),
# https://github.com/warewulf/warewulf3/issues/166
# So create the ifcfg- files and bring up the Lustre networks here
# instead, and launch from rc.local.

{% for dev in lustre_network_devices %}

parent={{ dev.split('.')[0] }}

parent_ip=$(ip a show dev ${parent} scope global | awk '{if ($1 == "inet") {print $2}}' | awk -F/ '{print $1}')

# Device IP
dip=$(echo $parent_ip | awk 'BEGIN {FS="."; OFS="."} {print $1, $2 + {{ loop.index }}, $3, $4}')
pkey="0x{{ dev.split('.')[1] }}"

cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-{{ dev }}
DEVICE={{ dev }}
NAME={{ dev }}
PHYSDEV=${parent}
BOOTPROTO=none
TYPE=InfiniBand
IPADDR=${dip}
ONBOOT=yes
NETMASK=255.255.0.0
PKEY=yes
PKEY_ID=${pkey}
IPV6INIT=no
MTU=65520
CONNECTED_MODE=yes
NM_CONTROLLED=no
EOF

echo ${pkey} > /sys/class/net/${parent}/create_child
ifup {{ dev }}

{% endfor %}

# Separate loop here to allow parallel initialization of devices
{% for dev in lustre_network_devices %}
while ! dmesg | grep '{{ dev }}: link becomes ready'
do
sleep 1
done
{% endfor %}

modprobe lustre
mount /scratch

0 comments on commit 4648dab

Please sign in to comment.