nomad starts before network is ready

[fredrikhgrelland]

fredrikhgrelland/data-mesh#59

[zhenik]

Options so far:

1. Move away from systemd, provision via ansible
2. Additional checks of daemon services(part of ansible scripts), before provisioning of infrastructure.

[fredrikhgrelland]

Hmm. Third option: restart services in ansible AS prestep

[fredrikhgrelland]

Hmm. Third option: restart services in ansible AS prestep

This^

[fredrikhgrelland]

fredrikhgrelland/data-mesh#59

[zhenik]

I have the same issue with vault #36

[fredrikhgrelland]

I have the same issue with vault #36

Can you elaborate? Same how?

[zhenik]

I have 3 listeners in Vault config

# (docker0)
listener "tcp" {
  address       = "172.17.0.1:8200"
  tls_disable   = 1
}
# (eth0)
listener "tcp" {
  address       = "10.0.2.15:8200"
  tls_disable   = 1
}
# (eth1)
listener "tcp" {
  address       = "10.0.3.10:8200"
  tls_disable   = 1
}

If any of the network is not ready, vault fail with (current example docker0 network is not ready)

Error initializing listener of type tcp: listen tcp 172.17.0.1:8200: bind: cannot assign requested address

• (unfortunately, I did not make listeners work with {{ GetInterfaceIP "docker0" }}:8200 for example )

[fredrikhgrelland]

I guess you must make vault wait for docker?

[zhenik]

I guess you must make vault wait for docker?

One of the solutions.

[fredrikhgrelland]

Findings when getting network exhausted. Nomad starting with systemd. Note Networks are missing Networks: null
running: nomad node status --json -self

{
    "Attributes": {
        "os.signals": "SIGIO,SIGXFSZ,SIGCHLD,SIGCONT,SIGSEGV,SIGURG,SIGPIPE,SIGQUIT,SIGTTIN,SIGWINCH,SIGXCPU,SIGILL,SIGIOT,SIGUSR1,SIGPROF,SIGTSTP,SIGSYS,SIGTRAP,SIGFPE,SIGINT,SIGKILL,SIGSTOP,SIGTTOU,SIGUSR2,SIGABRT,SIGHUP,SIGTERM,SIGALRM,SIGBUS",
        "kernel.version": "4.15.0-58-generic",
        "driver.docker.bridge_ip": "172.17.0.1",
        "cpu.numcores": "2",
        "kernel.name": "linux",
        "driver.exec": "1",
        "nomad.version": "0.11.3",
        "unique.storage.bytesfree": "59849322496",
        "os.version": "18.04",
        "driver.docker": "1",
        "memory.totalbytes": "2090106880",
        "driver.docker.version": "19.03.11",
        "consul.version": "1.8.0",
        "cpu.modelname": "Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz",
        "unique.consul.name": "vagrant",
        "consul.server": "true",
        "os.name": "ubuntu",
        "driver.docker.volumes.enabled": "true",
        "unique.cgroup.mountpoint": "/sys/fs/cgroup",
        "driver.docker.os_type": "linux",
        "unique.storage.bytestotal": "66355564544",
        "cpu.frequency": "2495",
        "unique.storage.volume": "/dev/mapper/vagrant--vg-root",
        "unique.hostname": "vagrant",
        "driver.raw_exec": "1",
        "cpu.totalcompute": "4990",
        "driver.docker.runtimes": "runc",
        "consul.datacenter": "dc1",
        "cpu.arch": "amd64",
        "nomad.revision": "8918fc804a0c6758b6e3e9960e4eb2e605e38552",
        "consul.revision": "44e17c81c",
        "nomad.advertise.address": "10.0.2.15:4646"
    },
    "CSIControllerPlugins": null,
    "CSINodePlugins": null,
    "CreateIndex": 7,
    "Datacenter": "dc1",
    "Drain": false,
    "DrainStrategy": null,
    "Drivers": {
        "docker": {
            "Attributes": {
                "driver.docker.volumes.enabled": "true",
                "driver.docker.bridge_ip": "172.17.0.1",
                "driver.docker.runtimes": "runc",
                "driver.docker.os_type": "linux",
                "driver.docker": "true",
                "driver.docker.version": "19.03.11"
            },
            "Detected": true,
            "HealthDescription": "Healthy",
            "Healthy": true,
            "UpdateTime": "2020-06-18T11:41:36.700438287Z"
        },
        "raw_exec": {
            "Attributes": {
                "driver.raw_exec": "true"
            },
            "Detected": true,
            "HealthDescription": "Healthy",
            "Healthy": true,
            "UpdateTime": "2020-06-18T11:41:36.607249009Z"
        },
        "exec": {
            "Attributes": {
                "driver.exec": "true"
            },
            "Detected": true,
            "HealthDescription": "Healthy",
            "Healthy": true,
            "UpdateTime": "2020-06-18T11:41:36.609163702Z"
        },
        "qemu": {
            "Attributes": null,
            "Detected": false,
            "HealthDescription": "",
            "Healthy": false,
            "UpdateTime": "2020-06-18T11:41:36.612574651Z"
        },
        "java": {
            "Attributes": null,
            "Detected": false,
            "HealthDescription": "",
            "Healthy": false,
            "UpdateTime": "2020-06-18T11:41:36.61565029Z"
        }
    },
    "Events": [
        {
            "CreateIndex": 0,
            "Details": null,
            "Message": "Node registered",
            "Subsystem": "Cluster",
            "Timestamp": "2020-06-18T11:41:36Z"
        }
    ],
    "HTTPAddr": "10.0.2.15:4646",
    "HostVolumes": null,
    "ID": "c16f83e4-dad8-9624-2680-91e258ef5696",
    "Links": {
        "consul": "dc1.vagrant"
    },
    "Meta": {
        "connect.sidecar_image": "envoyproxy/envoy:v1.11.2@sha256:a7769160c9c1a55bb8d07a3b71ce5d64f72b1f665f10d81aa1581bc3cf850d09",
        "connect.log_level": "info"
    },
    "ModifyIndex": 10,
    "Name": "vagrant",
    "NodeClass": "",
    "NodeResources": {
        "Cpu": {
            "CpuShares": 4990
        },
        "Devices": null,
        "Disk": {
            "DiskMB": 57076
        },
        "Memory": {
            "MemoryMB": 1993
        },
        "Networks": null
    },
    "Reserved": {
        "CPU": 0,
        "Devices": null,
        "DiskMB": 0,
        "IOPS": 0,
        "MemoryMB": 0,
        "Networks": null
    },
    "ReservedResources": {
        "Cpu": {
            "CpuShares": 0
        },
        "Disk": {
            "DiskMB": 0
        },
        "Memory": {
            "MemoryMB": 0
        },
        "Networks": {
            "ReservedHostPorts": ""
        }
    },
    "Resources": {
        "CPU": 4990,
        "Devices": null,
        "DiskMB": 57076,
        "IOPS": 0,
        "MemoryMB": 1993,
        "Networks": null
    },
    "SchedulingEligibility": "eligible",
    "Status": "ready",
    "StatusDescription": "",
    "StatusUpdatedAt": 1592480508,
    "TLSEnabled": false
}

[fredrikhgrelland]

hashicorp/nomad#7232