0.95 nomad arm reports 0mb memory available

[markopolo123]

Nomad version

0.95 and 0.94

Operating system and Environment details

Linux Raspbian Buster on Raspberry PI 4

Issue

cannot allocate jobs as nodes are always exhausted of memory - reports 0B available.
setting memory in client.hcl does not help.

Reproduction steps

pi@leader-01:~ $ nomad node status 74f1611a
ID = 74f1611a
Name = client-01
Class =
DC = homecluster
Drain = false
Eligibility = eligible
Status = ready
Uptime = 1h22m55s
Driver Status = docker,exec

Node Events
Time Subsystem Message
2019-08-26T22:59:16+01:00 Cluster Node registered

Allocated Resources
CPU Memory Disk
0/6000 MHz 0 B/0 B 0 B/16 EiB

Host Resource Utilization
CPU Memory Disk
60/6000 MHz 351 MiB/3.8 GiB 1.8 GiB/58 GiB

Allocations
No allocations placed

Job file (if appropriate)

using example.nomad from nomad init.

Nomad Client logs (if appropriate)

If possible please post relevant logs in the issue.

Logs and other artifacts may also be sent to: [email protected]

Please link to your Github issue in the email and reference it in the subject
line:

To: [email protected]

Subject: GH-1234: Errors garbage collecting allocs

Emails sent to that address are readable by all HashiCorp employees but are not publicly visible.

Nomad Server logs (if appropriate)

[endocrimes]

@markopolo123 Could you share your client configuration? - I'm wondering if this could be related to #6134

[markopolo123]

contents of /etc/nomad.d/client.hcl:

client {
    enabled = true

    node_class = ""
    no_host_uuid = false


    max_kill_timeout = "30s"

    network_speed = 0
    cpu_total_compute = 0
    memory_total_mb = 2048 <--- makes no difference
    gc_interval = "1m"
    gc_disk_usage_threshold = 80
    gc_inode_usage_threshold = 70
    gc_parallel_destroys = 2

    reserved {
        cpu = 0
        memory = 0
        disk = 0
    }



    }

nomad agent-info

client
  heartbeat_ttl = 16.792778043s
  known_servers = 192.168.0.100:4647
  last_heartbeat = 5.623203085s
  node_id = 74f1611a-cbc0-8cdb-09fe-886bbba59594
  num_allocations = 0
runtime
  arch = arm
  cpu_count = 4
  goroutines = 80
  kernel.name = linux
  max_procs = 4
  version = go1.11.11

Let me know if you'd like any more information!

[markopolo123]

Update; running a 64 bit OS and using the arm64 nomad package works fine:

ubuntu@leader-01:~$ nomad node status 3fc5e79c
ID            = 3fc5e79c
Name          = leader-01
Class         = <none>
DC            = homecluster
Drain         = false
Eligibility   = eligible
Status        = ready
Uptime        = 2h4m6s
Driver Status = docker,exec

Node Events
Time                  Subsystem  Message
2019-08-30T20:29:08Z  Cluster    Node registered

Allocated Resources
CPU           Memory          Disk
100/6000 MHz  64 MiB/3.7 GiB  300 MiB/54 GiB

Allocation Resource Utilization
CPU         Memory
0/6000 MHz  2.6 MiB/3.7 GiB

Host Resource Utilization
CPU           Memory           Disk
105/6000 MHz  534 MiB/3.7 GiB  2.5 GiB/58 GiB

Allocations
ID        Node ID   Task Group   Version       Desired  Status    Created     Modified
738ca791  3fc5e79c  registry     274879932592  run      running   6m19s ago   5m25s ago

[cthayer]

For anyone else who finds this thread, I had to set the memory_total_mb to 1024 to get it to work with the arm binary. (raspberry pi 4 with 4GB ram running Raspbian Buster Lite 2019-09-26)

client {
  ...
  memory_total_mb = 1024
  ...
}

I tried switching to a 64bit kernel with 32bit userspace (still Raspbian Buster Lite as above) and running the arm64 package, but it couldn't execute so I'm stuck with 1024GB ram that nomad can use on my pi :(

[cthayer]

I notice that the merged commit shown above is in the 0.10 milestone and I've only tried up to version 0.9.7 so perhaps this is fixed in 0.10 for arm?

(I'm trying to hold off from going to 0.10 until all of the features have landed)

[jahentao]

I have the same problem when running nomad of version 0.9.7 on Raspberry pi 4 with 4GB ram.

[endocrimes]

@jahentao You'd need to be running Nomad 0.10 or higher to have this work :)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.