Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nodes won't garbage collect #3658

Closed
apenney opened this issue Dec 13, 2017 · 21 comments
Closed

Nodes won't garbage collect #3658

apenney opened this issue Dec 13, 2017 · 21 comments
Labels
stage/waiting-reply theme/client type/bug

Comments

@apenney
Copy link

apenney commented Dec 13, 2017 •
edited by dadgar
Loading

Nomad version

Nomad v0.7.0

Operating system and Environment details

Ubuntu 16.04
consul 0.9.2/0.9.3

Issue

Nodes won't GC old jobs:

root@ip-10-30-1-43:/etc/nomad.d# nomad job status | wc -l
220

I see submit's back as far as September. I've enabled debug logging, and my client settings are:

client {
    enabled = true
    gc_max_allocs = 1000

    node_class = "uat"
    no_host_uuid = false


    max_kill_timeout = "30s"

    network_speed = 0
    cpu_total_compute = 0

    gc_interval = "1m"
    gc_disk_usage_threshold = 80
    gc_inode_usage_threshold = 70
    gc_parallel_destroys = 2

    reserved {
        cpu = 0
        memory = 0
        disk = 0
        reserved_ports = "22"
    }

    options = {
        "driver.blacklist" = "raw_exec,qemu"
        "docker.auth.config" = "/etc/nomad.d/.dockercfg"
    }

    meta = {
    }
    }

I see zero log lines matching 'garbage', which is what I expect to see based on gc.go.

Reproduction steps

This one is hard, I don't know why the gc gets skipped/ignored so I'm not sure what to say about reproduction.

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

I deleted all the lines matching 'secret', and snipped out some company name stuff. Hopefully this shows the lack of garbage/gc tho.

Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "381e8b66-8190-e2e1-7594-6e6c16c5d804" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "38d77692-9c1a-6b9c-185b-a99902b580b2" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "399c836c-9e5f-6eb5-5e57-346d995eb61c" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "3b983e9d-f6d1-7f0c-8d0e-e1209abd0bea" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "3c1a5d50-9cf1-5070-c616-faa5c878307a" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "3c4c8178-c8a5-a4fc-505e-12439e498d2e" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "3d74e161-f4ad-fb47-6bf4-24308c042ea8" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "3ebb8609-0a0f-22ec-da55-091301002b03" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/local/packetbeat.ctmpl" => "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/new/packetbeat.yml"
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) checking template bdd2cd5c1f0936027fea314561f2e73c
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/local/symbiosis.uat.ctmpl" => "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/new/symbiosis.conf"
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "403baf3c-00a2-6e5d-5f14-88fcd4cc688d" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "41c3c6a5-e52d-308c-3dcc-0b7f45fe8a1d" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "44aae9b7-d17e-79d1-33e0-ed31e5bc7cd0" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "44ae7315-243e-cfdb-3f34-d95b277a9565" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "454db4b9-733a-1211-bb5c-370cb8a4bcc1" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "457d3482-09fc-142a-3e15-6b450bad3583" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: waiting for Vault token for task api in alloc "34d1328e-8da7-25e8-eee0-989aaf1ec8ca"
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: error fetching stats of task api: stats collection hasn't started yet
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: driver.docker: re-attaching to docker process: e72ce139be202e49712c1cc61eef410f7268e645ca9b0a770d45982c4e70bc11
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5c82b5f7-574a-ad91-8517-c9ac7a3a46e8" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 35dd56b3-109c-606b-2ef7-2a5e87720286 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 381e8b66-8190-e2e1-7594-6e6c16c5d804 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 38d77692-9c1a-6b9c-185b-a99902b580b2 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "46b50b1e-042d-f0dc-f4ca-c616d94a3b0e" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 399c836c-9e5f-6eb5-5e57-346d995eb61c for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "478c768c-aa3b-a869-daa8-da41132590b8" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 3b983e9d-f6d1-7f0c-8d0e-e1209abd0bea for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 3c1a5d50-9cf1-5070-c616-faa5c878307a for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 3c4c8178-c8a5-a4fc-505e-12439e498d2e for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "48ae0f5e-53e1-7a09-0ced-9970a2e1c86f" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "4a594eca-8a63-acbe-99e9-11c2673c2519" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "4b8e3fe5-0ffe-8bf9-44b5-679f14a560ae" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "4c48c911-2fee-7f13-175f-3f706efa6ddb" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "4cc8f47d-3f32-61f5-df8a-33ddf439eba3" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "4cd9956c-80be-23d0-1cb5-2ace59bcfd3c" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "4f7cbfbd-f4b1-6644-a561-973d9aae0128" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5050f1e7-c45b-0136-af40-4b9c15f3d41c" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "50d7116e-808b-4ae0-f6f2-5be1a06ad746" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5149620f-f211-cf40-a102-ac51c8c23caa" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "515ac6ae-0076-d6ec-ffc3-c53ef2262725" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5162b0ed-a687-ac1d-4f30-33406acaac63" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "51b7fca8-5941-2b38-7681-75b3bc495cec" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "51bb7b8e-2f7d-b478-aca7-6083fa94fb9b" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "54dd04ba-52fb-aa87-33b9-7825853888e0" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "556ad3d2-0612-5229-261b-e0000e09d67b" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "55d24bd1-93ab-b99b-ffe3-2075b4a7c27d" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "562cbd87-3d30-5d7c-b714-5d5ed92c3dea" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "570da840-3b69-3bba-b3c1-a94af4aa1f63" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "58927b32-e1f9-6628-2bb6-b8192366c032" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5977c5d3-5541-caed-df66-ce0b57a91889" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5a2dc874-8c15-d3a0-12d1-4ec3c3a591c4" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5b3533fa-1a2e-18e8-a54f-4689f41056e9" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5ba17d69-11b6-056b-adb9-e7cca5151640" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 3d74e161-f4ad-fb47-6bf4-24308c042ea8 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5c6aff2d-f11f-f18e-fef6-b8e92a13c167" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 3ebb8609-0a0f-22ec-da55-091301002b03 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 403baf3c-00a2-6e5d-5f14-88fcd4cc688d for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 41c3c6a5-e52d-308c-3dcc-0b7f45fe8a1d for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 44aae9b7-d17e-79d1-33e0-ed31e5bc7cd0 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 44ae7315-243e-cfdb-3f34-d95b277a9565 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 454db4b9-733a-1211-bb5c-370cb8a4bcc1 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 457d3482-09fc-142a-3e15-6b450bad3583 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: driver.docker: re-attached to handle: DOCKER:{"Version":"0.7.0","Image":"tricorder:1.7.0-SNAPSHOT","ImageID":"sha256:1908e504484857f3396bff0997c32a810835b0ceb98b1478e403322c7617e98c","ContainerID":"e72ce139be202e49712c1cc61eef410f7268e645ca9b0a770d45982c4e70bc11","KillTimeout":5000000000,"MaxKillTimeout":30000000000,"PluginConfig":{"Pid":31748,"AddrNet":"unix","AddrName":"/tmp/plugin834450568"}}
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5c82b5f7-574a-ad91-8517-c9ac7a3a46e8 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 46b50b1e-042d-f0dc-f4ca-c616d94a3b0e for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 478c768c-aa3b-a869-daa8-da41132590b8 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 48ae0f5e-53e1-7a09-0ced-9970a2e1c86f for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 4a594eca-8a63-acbe-99e9-11c2673c2519 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 4b8e3fe5-0ffe-8bf9-44b5-679f14a560ae for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 4c48c911-2fee-7f13-175f-3f706efa6ddb for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 4cc8f47d-3f32-61f5-df8a-33ddf439eba3 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 4cd9956c-80be-23d0-1cb5-2ace59bcfd3c for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5050f1e7-c45b-0136-af40-4b9c15f3d41c for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 50d7116e-808b-4ae0-f6f2-5be1a06ad746 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 515ac6ae-0076-d6ec-ffc3-c53ef2262725 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5162b0ed-a687-ac1d-4f30-33406acaac63 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 51b7fca8-5941-2b38-7681-75b3bc495cec for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 51bb7b8e-2f7d-b478-aca7-6083fa94fb9b for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 54dd04ba-52fb-aa87-33b9-7825853888e0 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 556ad3d2-0612-5229-261b-e0000e09d67b for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 55d24bd1-93ab-b99b-ffe3-2075b4a7c27d for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 562cbd87-3d30-5d7c-b714-5d5ed92c3dea for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 570da840-3b69-3bba-b3c1-a94af4aa1f63 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 58927b32-e1f9-6628-2bb6-b8192366c032 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5977c5d3-5541-caed-df66-ce0b57a91889 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5a2dc874-8c15-d3a0-12d1-4ec3c3a591c4 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5b3533fa-1a2e-18e8-a54f-4689f41056e9 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5ba17d69-11b6-056b-adb9-e7cca5151640 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5c6aff2d-f11f-f18e-fef6-b8e92a13c167 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) rendered "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/local/symbiosis.uat.ctmpl" => "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/new/symbiosis.conf"
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) checking template 0d345435336ca507d95eac9d44730182
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/local/secrets.ctmpl" => "/var/nomad/alloc/06540f5a-3a84-7ba7-e2f7-cf403a0fd43e/api/new/secrets.conf"
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies





















Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) watching 24 dependencies
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: (runner) all templates rendered
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: unblocking task api for alloc "06540f5a-3a84-7ba7-e2f7-cf403a0fd43e": Template
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 4f7cbfbd-f4b1-6644-a561-973d9aae0128 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 5149620f-f211-cf40-a102-ac51c8c23caa for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: driver.docker: version of executor: 1.1.0
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: driver.docker: image "tricorder:1.7.0-SNAPSHOT" (sha256:1908e504484857f3396bff0997c32a810835b0ceb98b1478e403322c7617e98c) reference count incremented: 1
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: starting task context for 'api' (alloc '5d3fbe30-29a0-8245-b179-6f317fd6b35e')
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: waiting for Vault token for task api in alloc "5d3fbe30-29a0-8245-b179-6f317fd6b35e"
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: consul.sync: successfully updated services in Consul
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "89a880f4-2f7a-3b44-c1e4-2f8c22b89179" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 89a880f4-2f7a-3b44-c1e4-2f8c22b89179 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "8be4fa7a-ad4f-ae00-47cd-9d5ac0c663cd" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 8be4fa7a-ad4f-ae00-47cd-9d5ac0c663cd for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "8d60c99e-b719-6abb-0a3d-b004b08f069c" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 8d60c99e-b719-6abb-0a3d-b004b08f069c for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: starting task runners for alloc '5d3fbe30-29a0-8245-b179-6f317fd6b35e'
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "5dc5b022-30b2-a5ae-875e-d9ee382db679" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "8e895f9b-d99b-c4ee-a214-9e2f4db5c8d7" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 8e895f9b-d99b-c4ee-a214-9e2f4db5c8d7 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "8ec810f6-e606-9e14-51a6-2de2b1dbf28f" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 8ec810f6-e606-9e14-51a6-2de2b1dbf28f for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "9326e3ee-e253-776f-2340-92cd84021861" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 9326e3ee-e253-776f-2340-92cd84021861 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "94936d21-87ec-68da-ac40-31fc7d8f74fd" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 94936d21-87ec-68da-ac40-31fc7d8f74fd for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "8fb00559-13f0-c137-786a-25c91ddb806e" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 8fb00559-13f0-c137-786a-25c91ddb806e for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "95effff9-5d45-c61a-0f80-58a946c8f17b" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "971b712b-52da-486d-1ad5-878259681530" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 971b712b-52da-486d-1ad5-878259681530 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 95effff9-5d45-c61a-0f80-58a946c8f17b for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "9550675b-f357-9159-065c-9df3c43df388" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation 9550675b-f357-9159-065c-9df3c43df388 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "c6128070-e458-e8a5-29d8-5ceaca237ca2" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation c6128070-e458-e8a5-29d8-5ceaca237ca2 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "c624f808-ce02-85b1-e0fd-6baaa0d44660" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation c624f808-ce02-85b1-e0fd-6baaa0d44660 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "c6d0970f-3e26-14ed-0484-67bfd26f2f63" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation c6d0970f-3e26-14ed-0484-67bfd26f2f63 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "c737bb4f-acfa-7702-3066-3d4da26ea150" in terminal status, waiting for destroy
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: marking allocation c737bb4f-acfa-7702-3066-3d4da26ea150 for GC
Dec 13 16:59:07 ip-10-30-1-43 nomad[29064]: client: alloc "c9fd26c2-e11f-621f-8fdb-8c07b816a4a7" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation c9fd26c2-e11f-621f-8fdb-8c07b816a4a7 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ce236fc1-94d8-f1d2-2516-73ca47a43a57" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation ce236fc1-94d8-f1d2-2516-73ca47a43a57 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ce69ca97-1b9b-d1df-52ce-57b0c1a6d0c0" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation ce69ca97-1b9b-d1df-52ce-57b0c1a6d0c0 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "cf95d8b3-5847-e53d-9908-cef313e624c4" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation cf95d8b3-5847-e53d-9908-cef313e624c4 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d04bca0a-de7a-227b-b8b5-571dc701d68f" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d04bca0a-de7a-227b-b8b5-571dc701d68f for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d16d4240-ee9a-9bea-4eac-23a032b75336" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d16d4240-ee9a-9bea-4eac-23a032b75336 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d173f415-f515-5965-fda3-c591b4a0c136" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d173f415-f515-5965-fda3-c591b4a0c136 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d2b14329-7241-ff9e-6d84-d7c8965a38cc" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d2b14329-7241-ff9e-6d84-d7c8965a38cc for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "f4eb781e-738b-70f3-17e3-797ca0148864" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation f4eb781e-738b-70f3-17e3-797ca0148864 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9e7ef90a-a863-b0aa-a3e7-9a17e1830700" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 9e7ef90a-a863-b0aa-a3e7-9a17e1830700 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c2e3d045-bcaf-69a4-aff9-9e4e2f9b7073" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation c2e3d045-bcaf-69a4-aff9-9e4e2f9b7073 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: retrieved Vault token for task api in alloc "34d1328e-8da7-25e8-eee0-989aaf1ec8ca"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) creating new runner (dry: false, once: false)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) final config: {"Consul":{"Address":"10.30.1.43:8500","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"4dd5818e-8c54-477c-5d93-59e21cb3f989","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000},"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"WARN","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/scribe.conf","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/scribe.uat.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/filebeat.yml","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/filebeat.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/packetbeat.yml","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/packetbeat.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"}],"Vault":{"Address":"http://vault.service.consul:8200","Enabled":true,"Grace":0,"RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0}}
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) creating watcher
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) starting
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) running initial templates
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 682fda72530db76619ac43b9c401faa0
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) was not watching 6 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/scribe/host)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/scribe/port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/auth0/realm)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/auth0/domain)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/auth0/port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/scribe/auth0/client_id)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 915a99f10d848e1e2574aa93400c1e4e
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) was not watching 2 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/elasticsearch/cluster_name)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/uat/logger)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template ad21b17e45fc4983f2cbb42e63caad8a
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 2 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 8 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: driver.docker: re-attaching to docker process: fb33dae2482d130cc42ad298ccda03656277fe3785e3d929046bc73afc628852
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: driver.docker: re-attached to handle: DOCKER:{"Version":"0.7.0","Image":"ingestion:1.7.0-SNAPSHOT","ImageID":"sha256:5bf29961b34956d131dc0707009b66e3b07165d49d92511d0dd8e833125713a3","ContainerID":"fb33dae2482d130cc42ad298ccda03656277fe3785e3d929046bc73afc628852","KillTimeout":5000000000,"MaxKillTimeout":30000000000,"PluginConfig":{"Pid":26992,"AddrNet":"unix","AddrName":"/tmp/plugin936240069"}}
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d62bec89-0c99-4a20-a47d-c4cd275eb355" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d62bec89-0c99-4a20-a47d-c4cd275eb355 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 5dc5b022-30b2-a5ae-875e-d9ee382db679 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "f143817c-b23d-563c-4f60-b84e34cb7ae3" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation f143817c-b23d-563c-4f60-b84e34cb7ae3 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "60f13b2f-17d3-34ff-7ce0-8cb6a0aec7a8" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 60f13b2f-17d3-34ff-7ce0-8cb6a0aec7a8 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "61621de8-e028-5d72-c0b8-298398bc471a" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 61621de8-e028-5d72-c0b8-298398bc471a for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "62360a88-840e-3c45-0648-d63177699216" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 62360a88-840e-3c45-0648-d63177699216 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "6276e411-655d-1738-ff54-5c16472ee7e2" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/elasticsearch/cluster_name)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/scribe/auth0/client_id)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/auth0/domain)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 682fda72530db76619ac43b9c401faa0
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 4 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 915a99f10d848e1e2574aa93400c1e4e
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template ad21b17e45fc4983f2cbb42e63caad8a
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies








Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 8 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/auth0/realm)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/scribe/port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/scribe/host)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 682fda72530db76619ac43b9c401faa0
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 915a99f10d848e1e2574aa93400c1e4e
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template ad21b17e45fc4983f2cbb42e63caad8a
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies








Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 8 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/auth0/port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 682fda72530db76619ac43b9c401faa0
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/scribe.uat.ctmpl" => "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/scribe.conf"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 915a99f10d848e1e2574aa93400c1e4e
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 6276e411-655d-1738-ff54-5c16472ee7e2 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "63a1179d-0ae4-620c-2f27-e117c513f135" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 63a1179d-0ae4-620c-2f27-e117c513f135 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "65f06722-772a-8c48-d2e1-183d68307535" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 65f06722-772a-8c48-d2e1-183d68307535 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template ad21b17e45fc4983f2cbb42e63caad8a
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies








Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 8 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency vault.read(secret/uat/logger)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 682fda72530db76619ac43b9c401faa0
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/scribe.uat.ctmpl" => "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/scribe.conf"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 915a99f10d848e1e2574aa93400c1e4e
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/filebeat.ctmpl" => "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/filebeat.yml"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template ad21b17e45fc4983f2cbb42e63caad8a
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/local/packetbeat.ctmpl" => "/var/nomad/alloc/34d1328e-8da7-25e8-eee0-989aaf1ec8ca/api/new/packetbeat.yml"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies








Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 8 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) all templates rendered
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: retrieved Vault token for task api in alloc "5d3fbe30-29a0-8245-b179-6f317fd6b35e"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) creating new runner (dry: false, once: false)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "6b8c58f9-d5a2-900e-ef6d-ec258fb819d8" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) final config: {"Consul":{"Address":"10.30.1.43:8500","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"4dd5818e-8c54-477c-5d93-59e21cb3f989","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000},"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"WARN","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/tricorder.conf","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/tricorder.uat.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/secrets.conf","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/secrets.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/filebeat.yml","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/filebeat.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"},{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","Destination":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/packetbeat.yml","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Blacklist":[],"Custom":[],"Pristine":false,"Whitelist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":420,"Source":"/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/packetbeat.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}"}],"Vault":{"Address":"http://vault.service.consul:8200","Enabled":true,"Grace":0,"RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0}}
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) creating watcher
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) starting
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) running initial templates
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0af01921e526f7ed9227f49001c30d2e
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "6c9ff7d7-827e-3e91-3ccc-efe1b62590b8" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "6d00d422-5bcc-0015-571a-5b1bff4e2efe" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "6d88842f-13b6-fc9f-667c-1da28934de23" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "6f3568d9-8b8f-fbe1-27f8-759983b61c00" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) was not watching 19 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/tricorder/host)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/tricorder/port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/tricorder/external_host)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/tricorder/external_port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/schema_audit/bucket)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/auth0/domain)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/auth0/port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/tricorder/auth0/client_id)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/influx/postgres/database)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.get(uat/postgres/consul/address)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/influx/postgres/port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(database/creds/tricorder-uat)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/influx/postgres/test)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/elasticsearch/cluster_name)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/uat/tricorder)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.get(uat/elasticsearch/consul/address)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.block(uat/elasticsearch/transport/port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding kv.get(uat/tricorder/deploy/git_sha)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/mrn)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0d345435336ca507d95eac9d44730182
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "710df41f-94f0-b3a9-ae4b-90e0082f3f69" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "717e8e23-478b-5a89-23db-4991edf8fe37" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7180fee2-c804-7484-ad90-5bb6f8727039" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7214744e-86e6-0e98-d7c7-46564eec83d8" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7405bfcb-6679-661c-28d6-1e74ff100c58" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "756baf79-f842-b57b-e541-56b5010c0972" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "768581b0-7a97-c2dc-4c7a-597e097cab56" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "76c8e63c-15d4-945a-c073-1d290bffce92" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "79f5ed48-6e4e-f5b9-878a-70aa206770a8" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7bfa7846-32ed-c73e-096b-7ec0c6de4b4a" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "89514bb8-2395-3e2d-446c-51dec5eeb9d4" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7d7fd2a3-932d-f644-5478-7e0ba59e16e7" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7d9d1c40-4e30-13ad-505e-1e703fe8ae15" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7dea9772-5e2f-0551-2184-ccf0eb4f6003" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7e88bb46-d31f-88aa-86d1-6c71bbd8d407" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7ec8e849-5a76-0c5a-570d-963d3233727d" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "7f8bab5e-e0e7-6d07-26a3-26f0f821e1ab" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "80ff8676-2a91-791b-d217-ea2119647c16" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "83b19aa3-9ec4-72f3-be49-ebd09efe756b" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "84592f6a-a5ea-e44d-e4b9-ea3d95b0e24b" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "861e6f18-d5d4-189e-2974-2dd4308944e9" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "87491d75-e296-15ee-d265-fc2d766a8bc0" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) was not watching 9 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/s3/encryption/production-integration-messages-es)
[vault.read snips]
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/s3/encryption/staging-integration-messages-es)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/s3/encryption/uat-integration-messages-es)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template d74433fe4620cb7647d82eaf4f8e7835
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) was not watching 1 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (watcher) adding vault.read(secret/uat/logger)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 362d96b61c7729f34665b804cc36fefe
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 2 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 29 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "8856ec7d-f8ac-a58f-9a56-b961e3720bc1" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "92df5c45-bbac-1175-d526-eca9f0641fad" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "8dfc4075-ed7b-d3df-7f11-e30f1c189d10" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "8e5c4942-f826-661b-9461-506b1cc48b89" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "95929532-3909-e568-aca4-362079c7b7b5" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9618af56-658e-1e10-a45f-13bf23b81a10" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "60bcb176-7853-af47-0c3b-d382600eb35f" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9bacbac5-f1e9-f19b-6288-269ae56c5b64" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d6f24bdc-b3ad-5aaa-241c-d74bb3191245" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: error fetching stats of task api: stats collection hasn't started yet
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9bf6f578-3e6f-e10f-3589-8515450f3c83" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d751bf37-e051-42fb-61b4-3c279db8f983" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d779c585-addd-a9e2-df7a-2e253f6a9c2d" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9c0f45eb-8196-4b4d-6158-e53cd6935ed0" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a2d72cd6-9876-0d38-8ef6-728f01220902" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a3580421-357c-8b67-fc1a-63f15a508a3a" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a3d8d31c-9104-972d-2dde-05bd2fa27549" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a40bf408-d692-b116-dc5e-979b5511deed" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a46d87cc-0b6a-ed11-7332-ad9cf83e1746" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a4e7aaa8-47e3-2233-acc7-3ecd187d1fa0" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.get(uat/tricorder/deploy/git_sha)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0af01921e526f7ed9227f49001c30d2e
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 18 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0d345435336ca507d95eac9d44730182
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a5777b51-9280-128c-1c72-102e5a5737c9" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a5bc53b9-aa97-f73c-a425-79484e819223" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a65e9103-3651-7a69-d8ee-a94f83c52079" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a8a89a65-1c62-35ea-d5d9-2cf2ec3dc646" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a947549d-2684-b846-3f61-d1df70c710be" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "a9f8f4cc-96dd-6bca-cc7d-d819049b0d98" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "aad858f5-32d7-f4eb-1041-434d5218561a" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ab158044-69a9-9467-d8c9-54b154b194e1" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 9 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template d74433fe4620cb7647d82eaf4f8e7835
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 2 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 362d96b61c7729f34665b804cc36fefe
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 2 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies

[snip still needed stuff]
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 29 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/elasticsearch/transport/port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/tricorder/external_host)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/tricorder/host)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/tricorder/port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/tricorder/auth0/client_id)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/auth0/domain)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.get(uat/postgres/consul/address)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/auth0/port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/schema_audit/bucket)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/influx/postgres/test)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/influx/postgres/port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/elasticsearch/cluster_name)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.get(uat/elasticsearch/consul/address)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/influx/postgres/database)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency vault.read(secret/s3/encryption/production-integration-messages-es)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency vault.read(secret/mrn)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency vault.read(secret/uat/tricorder)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency kv.block(uat/tricorder/external_port)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) receiving dependency vault.read(secret/s3/encryption/staging-docbot-out)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) initiating run
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0af01921e526f7ed9227f49001c30d2e
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ac834e5b-025c-a947-eac0-93323bdf25ea" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "aea79f81-4f59-0a83-7319-6edae590f61f" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 1 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0d345435336ca507d95eac9d44730182
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "af195838-dd14-6612-e6c3-4c9ad03ab455" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "af54da3b-6a54-514b-9cad-94d907301aed" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "b1deefd6-79e4-013b-a863-3a3bcb39b5f2" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "b5651fa2-fa12-f74d-5c5e-be231a5731ce" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "b73bba18-671e-5f9d-4217-1c9b2b22f723" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "b97cd23b-354a-c4f2-2753-9252eea00c61" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ba5f1ce0-2202-245f-5ea3-438651c8ca97" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ba63d222-db04-e7f7-02b3-2479d10ea7c5" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "bad0bc14-7e5e-0728-9ee9-ab9bb49739dd" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "bce49302-0578-ceaf-4288-645552124ed1" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "bf28e0af-7f35-9258-e946-c48bf94819ce" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "bf5ae2d1-60fd-d040-5321-714198eea793" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "bf966954-eb9b-090c-2708-5c8709f3cb50" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c03b42b7-ff48-189b-0e2b-f57e3888bb0d" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c07b875d-d7d5-9b46-6ca2-4687682532eb" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d91ddb19-42f6-ea2b-3d68-c41d094d6efa" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "db913530-1ece-1d26-0e62-893e742ff758" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "dc4add48-2db4-2c53-1287-e845372d7445" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "dd088b39-d286-5e48-9b5b-0abde1631951" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "dd0cfd72-3a94-a9b5-1afc-320b3388499f" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "de08ffbd-1bbf-615b-6a69-56d3c56cc170" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "de2d724a-1ef3-7414-be4e-c9075158995b" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "de72049a-f30d-cd1f-2341-c56dc1afffcf" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "de9cabd7-4b85-152a-da52-13706bb01fd6" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c131ee79-b8d3-31e1-f3b6-a21a1ce4c8d8" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c17c6e42-1445-27b2-aa83-d38a9f0b788b" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c1f47321-c997-e448-dc70-c24a14a16019" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c26ffb7b-2449-32ea-d7e8-5251e6a59a85" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c2871eff-20c1-15dd-03f8-1f75a7f49aca" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "e1c7fe90-38d4-a1c8-f63b-90144c3400c8" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "e2963bd6-2b29-4783-c570-867bd81f4504" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "c2de33c2-31b0-ad88-883e-d8c75a681723" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "e5fe5d98-e783-50f3-2b14-dcc91e04e0bb" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "e71b3486-559e-e304-b6f5-17e74a50b34f" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "e71b4a8e-a7dd-b54a-8789-544c314668e2" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "e8b6ff71-fdad-216e-9e45-03e5c5e5250c" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "921e7989-36c1-3248-e7dc-4f4256295b54" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9aa58df0-9bd1-2015-61c9-7089bf6d3132" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ed22dad2-8a86-2771-b756-86b0d5713ee6" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "989066bb-e16d-54da-1fd6-a9ee4ab6b35f" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "97d494a3-c113-ca17-8be2-2f1fb4438f06" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "5e0496dd-f515-86fc-fd64-5d19807061a0" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9d9e09b9-ce50-b7c7-1777-7dce63b8557c" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "d9149162-0f62-5187-e3d2-eb7361037ce5" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) missing data for 7 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template d74433fe4620cb7647d82eaf4f8e7835
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "9b1f7023-1b14-0604-b03b-e01ad934930e" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "edd4d0f9-9eb8-f740-3495-64feac1104c5" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "ee504eb8-c2a6-e9e0-966b-104adfdefc78" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "f068f3b8-1386-b330-fe8e-1ba9b19a740c" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: alloc "f39327f9-cc03-c013-8d40-5d57d2b609db" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: unblocking task api for alloc "34d1328e-8da7-25e8-eee0-989aaf1ec8ca": Template
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: driver.docker: version of executor: 1.1.0
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 6b8c58f9-d5a2-900e-ef6d-ec258fb819d8 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 6c9ff7d7-827e-3e91-3ccc-efe1b62590b8 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 6d00d422-5bcc-0015-571a-5b1bff4e2efe for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 6d88842f-13b6-fc9f-667c-1da28934de23 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 6f3568d9-8b8f-fbe1-27f8-759983b61c00 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 710df41f-94f0-b3a9-ae4b-90e0082f3f69 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 717e8e23-478b-5a89-23db-4991edf8fe37 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7180fee2-c804-7484-ad90-5bb6f8727039 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7214744e-86e6-0e98-d7c7-46564eec83d8 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7405bfcb-6679-661c-28d6-1e74ff100c58 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 756baf79-f842-b57b-e541-56b5010c0972 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 768581b0-7a97-c2dc-4c7a-597e097cab56 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 76c8e63c-15d4-945a-c073-1d290bffce92 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 79f5ed48-6e4e-f5b9-878a-70aa206770a8 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7bfa7846-32ed-c73e-096b-7ec0c6de4b4a for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 89514bb8-2395-3e2d-446c-51dec5eeb9d4 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7d7fd2a3-932d-f644-5478-7e0ba59e16e7 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7d9d1c40-4e30-13ad-505e-1e703fe8ae15 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7dea9772-5e2f-0551-2184-ccf0eb4f6003 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7e88bb46-d31f-88aa-86d1-6c71bbd8d407 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7ec8e849-5a76-0c5a-570d-963d3233727d for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 7f8bab5e-e0e7-6d07-26a3-26f0f821e1ab for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 80ff8676-2a91-791b-d217-ea2119647c16 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 83b19aa3-9ec4-72f3-be49-ebd09efe756b for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 84592f6a-a5ea-e44d-e4b9-ea3d95b0e24b for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 861e6f18-d5d4-189e-2974-2dd4308944e9 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 87491d75-e296-15ee-d265-fc2d766a8bc0 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 8856ec7d-f8ac-a58f-9a56-b961e3720bc1 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 92df5c45-bbac-1175-d526-eca9f0641fad for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 8dfc4075-ed7b-d3df-7f11-e30f1c189d10 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 8e5c4942-f826-661b-9461-506b1cc48b89 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 95929532-3909-e568-aca4-362079c7b7b5 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 9618af56-658e-1e10-a45f-13bf23b81a10 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 60bcb176-7853-af47-0c3b-d382600eb35f for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 9bacbac5-f1e9-f19b-6288-269ae56c5b64 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d6f24bdc-b3ad-5aaa-241c-d74bb3191245 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 9bf6f578-3e6f-e10f-3589-8515450f3c83 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d751bf37-e051-42fb-61b4-3c279db8f983 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation d779c585-addd-a9e2-df7a-2e253f6a9c2d for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation 9c0f45eb-8196-4b4d-6158-e53cd6935ed0 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a0ff8bf3-3d82-c513-002b-189a82f87f98 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a2d72cd6-9876-0d38-8ef6-728f01220902 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a3580421-357c-8b67-fc1a-63f15a508a3a for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a3d8d31c-9104-972d-2dde-05bd2fa27549 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a40bf408-d692-b116-dc5e-979b5511deed for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a46d87cc-0b6a-ed11-7332-ad9cf83e1746 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a4e7aaa8-47e3-2233-acc7-3ecd187d1fa0 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a5777b51-9280-128c-1c72-102e5a5737c9 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a5bc53b9-aa97-f73c-a425-79484e819223 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a65e9103-3651-7a69-d8ee-a94f83c52079 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a8a89a65-1c62-35ea-d5d9-2cf2ec3dc646 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a947549d-2684-b846-3f61-d1df70c710be for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation a9f8f4cc-96dd-6bca-cc7d-d819049b0d98 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation aad858f5-32d7-f4eb-1041-434d5218561a for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation ab158044-69a9-9467-d8c9-54b154b194e1 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation aea79f81-4f59-0a83-7319-6edae590f61f for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation af54da3b-6a54-514b-9cad-94d907301aed for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation b1deefd6-79e4-013b-a863-3a3bcb39b5f2 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation b73bba18-671e-5f9d-4217-1c9b2b22f723 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation b5651fa2-fa12-f74d-5c5e-be231a5731ce for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation b97cd23b-354a-c4f2-2753-9252eea00c61 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation ba5f1ce0-2202-245f-5ea3-438651c8ca97 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation ba63d222-db04-e7f7-02b3-2479d10ea7c5 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation bad0bc14-7e5e-0728-9ee9-ab9bb49739dd for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation bce49302-0578-ceaf-4288-645552124ed1 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation bf28e0af-7f35-9258-e946-c48bf94819ce for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation bf5ae2d1-60fd-d040-5321-714198eea793 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: ==> Nomad agent configuration:
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:                 Client: true
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:              Log Level: DEBUG
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:                 Region: us-east-1 (DC: us-east-1)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:                 Server: false
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:                Version: 0.7.0
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: ==> Nomad agent started! Log data will stream in below:
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.625309 [INFO] client: using state directory /var/nomad/client
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.625584 [INFO] client: using alloc directory /var/nomad/alloc
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.627017 [DEBUG] client: built-in fingerprints: [arch cgroup consul cpu host memory network nomad signal storage vault env_aws env_gce]
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.627770 [INFO] fingerprint.cgroups: cgroups are available
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.628533 [DEBUG] client: fingerprinting cgroup every 15s
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.630171 [INFO] fingerprint.consul: consul agent is available
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.630591 [DEBUG] fingerprint.cpu: frequency: 2400 MHz
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.630809 [DEBUG] fingerprint.cpu: core count: 2
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.632534 [DEBUG] client: fingerprinting consul every 15s
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.633659 [DEBUG] fingerprint.network: link speed for ens3 set to 10000
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.633943 [DEBUG] fingerprint.network: Detected interface ens3 with IP: 10.30.1.43
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.644098 [INFO] fingerprint.vault: Vault is available
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.644543 [DEBUG] client: fingerprinting vault every 15s
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.647351 [WARN]: fingerprint.env_aws: Could not read value for attribute "public-ipv4"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.650979 [WARN]: fingerprint.env_aws: Could not read value for attribute "public-hostname"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.653953 [DEBUG] fingerprint.env_gce: Could not read value for attribute "machine-type"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.654092 [DEBUG] client: applied fingerprints [arch cgroup consul cpu host memory network nomad signal storage vault env_aws]
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.654139 [DEBUG] driver.docker: using client connection initialized from environment
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation bf966954-eb9b-090c-2708-5c8709f3cb50 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.656375 [DEBUG] driver.exec: exec driver is enabled
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.656475 [DEBUG] client: available drivers [docker exec]
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.656499 [DEBUG] client: drivers skipped due to white/blacklist: [raw_exec qemu]
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.656856 [DEBUG] client: fingerprinting docker every 15s
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.656892 [DEBUG] client: fingerprinting exec every 15s
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.656929 [DEBUG] client: fingerprinting rkt every 15s
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.658521 [DEBUG] client.consul: bootstrap contacting following Consul DCs: ["us-east-1"]
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.659825 [INFO] driver.docker: re-attaching to docker process: 6e9a311e11251f04cf92c9bde5dc460ba22fffaeca0b782e284fcda73a55b818
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.660421 [DEBUG] driver.docker: re-attached to handle: DOCKER:{"Version":"0.7.0","Image":"symbiosis:1.7.0-SNAPSHOT","ImageID":"sha256:d4b632ec32d3b55198fe95af7fc2347b7ef7e7cabfc36f73a778b93d0e619824","ContainerID":"6e9a311e11251f04cf92c9bde5dc460ba22fffaeca0b782e284fcda73a55b818","KillTimeout":5000000000,"MaxKillTimeout":30000000000,"PluginConfig":{"Pid":21963,"AddrNet":"unix","AddrName":"/tmp/plugin612029439"}}
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.661211 [DEBUG] client: alloc "001d90fa-f585-e05a-c9a9-e692dd8be47a" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.661402 [INFO] client: marking allocation 001d90fa-f585-e05a-c9a9-e692dd8be47a for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.661784 [DEBUG] client: alloc "015b6bd1-21a8-3b19-c9ac-8fe6ef444c6d" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.661930 [INFO] client: marking allocation 015b6bd1-21a8-3b19-c9ac-8fe6ef444c6d for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.662617 [DEBUG] client: alloc "0360f079-e7dc-3c0c-e57d-9e9a481929ea" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.662774 [INFO] client: marking allocation 0360f079-e7dc-3c0c-e57d-9e9a481929ea for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: marking allocation c03b42b7-ff48-189b-0e2b-f57e3888bb0d for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.663308 [INFO] client.consul: discovered following Servers: 10.50.1.95:4647,10.50.1.29:4647,10.50.1.101:4647
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.663340 [DEBUG] client: alloc "062e4938-ddad-0fe6-8a34-36c89639bc4d" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.665772 [INFO] client: marking allocation 062e4938-ddad-0fe6-8a34-36c89639bc4d for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.663532 [DEBUG] client: alloc "0440a31d-14d9-3b0d-73e9-1a4f2f274023" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.663785 [DEBUG] client: alloc "05412a3d-6609-646a-6bc2-89947a18f919" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.667155 [INFO] client: marking allocation 05412a3d-6609-646a-6bc2-89947a18f919 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.666176 [DEBUG] client: alloc "03f5f08f-1f8d-66f7-e28a-22f22465694d" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.667739 [INFO] client: marking allocation 03f5f08f-1f8d-66f7-e28a-22f22465694d for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.666728 [INFO] client: marking allocation 0440a31d-14d9-3b0d-73e9-1a4f2f274023 for GC
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.668817 [DEBUG] driver.docker: version of executor: 1.1.0
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.668858 [DEBUG] driver.docker: image "symbiosis:1.7.0-SNAPSHOT" (sha256:d4b632ec32d3b55198fe95af7fc2347b7ef7e7cabfc36f73a778b93d0e619824) reference count incremented: 1
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.670010 [DEBUG] client: starting task context for 'api' (alloc '06540f5a-3a84-7ba7-e2f7-cf403a0fd43e')
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.671949 [DEBUG] client: waiting for Vault token for task api in alloc "06540f5a-3a84-7ba7-e2f7-cf403a0fd43e"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.672792 [WARN] client: error fetching stats of task api: stats collection hasn't started yet
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:07.675821 [DEBUG] client: alloc "093d27b4-3a9e-3307-b1f3-f9e5eae4d878" in terminal status, waiting for destroy
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: error fetching stats of task api: stats collection hasn't started yet
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [INFO] (runner) rendered "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/tricorder.uat.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/tricorder.conf"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) checking template 0d345435336ca507d95eac9d44730182
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendered "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/tricorder.uat.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/tricorder.conf"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 0d345435336ca507d95eac9d44730182
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [INFO] (runner) rendered "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/ingestion.uat.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/ingestion.conf"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) checking template 3f899786697a252f914e48c385b3fac5
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendered "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/ingestion.uat.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/ingestion.conf"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 3f899786697a252f914e48c385b3fac5
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) rendering "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/secrets.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/secrets.conf"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/secrets.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/secrets.conf"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) checking template d74433fe4620cb7647d82eaf4f8e7835
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template d74433fe4620cb7647d82eaf4f8e7835
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) rendering "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/filebeat.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/filebeat.yml"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/filebeat.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/filebeat.yml"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) checking template 362d96b61c7729f34665b804cc36fefe
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 362d96b61c7729f34665b804cc36fefe
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) rendering "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/packetbeat.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/packetbeat.yml"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/local/packetbeat.ctmpl" => "/var/nomad/alloc/5d3fbe30-29a0-8245-b179-6f317fd6b35e/api/new/packetbeat.yml"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) diffing and updating dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies


[snip]












[snip]























Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) watching 29 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) all templates rendered
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08.589761 [DEBUG] client: unblocking task api for alloc "5d3fbe30-29a0-8245-b179-6f317fd6b35e": Template

Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) rendering "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/secrets.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/secrets.conf"


Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) checking template de2638ff63506a18feeccea5c5b770a0







Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) rendering "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/filebeat.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/filebeat.yml"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) checking template 20d663d8d756b7418c40e5e6c095cba2


Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) rendering "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/packetbeat.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/packetbeat.yml"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) diffing and updating dependencies






































Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) watching 36 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08 [DEBUG] (runner) all templates rendered
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08.603922 [DEBUG] client: unblocking task api for alloc "fa37f5c3-e340-0a1c-81ab-ed90d142de75": Template
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08.605626 [DEBUG] consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks



Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 29 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) all templates rendered
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: unblocking task api for alloc "5d3fbe30-29a0-8245-b179-6f317fd6b35e": Template
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/secrets.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/secrets.conf"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08.613266 [DEBUG] http: Request /v1/status/leader (29.942089ms)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template de2638ff63506a18feeccea5c5b770a0
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/filebeat.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/filebeat.yml"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) checking template 20d663d8d756b7418c40e5e6c095cba2
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) rendering "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/local/packetbeat.ctmpl" => "/var/nomad/alloc/fa37f5c3-e340-0a1c-81ab-ed90d142de75/api/new/packetbeat.yml"
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) diffing and updating dependencies




































Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) watching 36 dependencies
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: (runner) all templates rendered
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: unblocking task api for alloc "fa37f5c3-e340-0a1c-81ab-ed90d142de75": Template
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: http: Request /v1/status/leader (29.942089ms)
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08.631187 [DEBUG] consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08.640709 [DEBUG] client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039'
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08.640805 [DEBUG] client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039'
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039'
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039'
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08.649242 [DEBUG] consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08.665178 [DEBUG] consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: consul.sync: registered 1 services, 1 checks; deregistered 0 services, 0 checks
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:08.820070 [DEBUG] consul.sync: registered 0 services, 0 checks; deregistered 0 services, 0 checks
Dec 13 16:59:08 ip-10-30-1-43 nomad[29064]: consul.sync: registered 0 services, 0 checks; deregistered 0 services, 0 checks
Dec 13 16:59:16 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:16.744385 [DEBUG] http: Request /v1/agent/health?type=client (458.142µs)
Dec 13 16:59:16 ip-10-30-1-43 nomad[29064]: http: Request /v1/agent/health?type=client (458.142µs)
Dec 13 16:59:16 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:16.978960 [DEBUG] client: 34 evaluations triggered by node update
Dec 13 16:59:16 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:16.979022 [DEBUG] client: state updated to ready
Dec 13 16:59:16 ip-10-30-1-43 nomad[29064]: client: 34 evaluations triggered by node update
Dec 13 16:59:16 ip-10-30-1-43 nomad[29064]: client: state updated to ready
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:17.129672 [DEBUG] client: updated allocations at index 605006 (total 253) (pulled 3) (filtered 250)
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:17.129982 [DEBUG] client: allocs: (added 0) (removed 0) (updated 3) (ignore 250)
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:17.130055 [DEBUG] client: dropping update to terminal alloc '0b1e29f2-976e-1c85-2ab5-0689915a6244'
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:17.130081 [DEBUG] client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039'
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:17.130102 [DEBUG] client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565'
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: updated allocations at index 605006 (total 253) (pulled 3) (filtered 250)
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: allocs: (added 0) (removed 0) (updated 3) (ignore 250)
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '0b1e29f2-976e-1c85-2ab5-0689915a6244'
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039'
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565'
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:17.285999 [DEBUG] client: updated allocations at index 605038 (total 253) (pulled 3) (filtered 250)
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:17.286431 [DEBUG] client: allocs: (added 0) (removed 0) (updated 3) (ignore 250)
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: updated allocations at index 605038 (total 253) (pulled 3) (filtered 250)
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:17.286569 [DEBUG] client: dropping update to terminal alloc '0b1e29f2-976e-1c85-2ab5-0689915a6244'
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:17.286602 [DEBUG] client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039'
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:17.286629 [DEBUG] client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565'
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: allocs: (added 0) (removed 0) (updated 3) (ignore 250)
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '0b1e29f2-976e-1c85-2ab5-0689915a6244'
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039'
Dec 13 16:59:17 ip-10-30-1-43 nomad[29064]: client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565'
Dec 13 16:59:23 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:23.615107 [DEBUG] http: Request /v1/status/leader (914.296µs)
Dec 13 16:59:23 ip-10-30-1-43 nomad[29064]: http: Request /v1/status/leader (914.296µs)
Dec 13 16:59:26 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:26.745767 [DEBUG] http: Request /v1/agent/health?type=client (257.378µs)
Dec 13 16:59:26 ip-10-30-1-43 nomad[29064]: http: Request /v1/agent/health?type=client (257.378µs)
Dec 13 16:59:36 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:36.747088 [DEBUG] http: Request /v1/agent/health?type=client (242.798µs)
Dec 13 16:59:36 ip-10-30-1-43 nomad[29064]: http: Request /v1/agent/health?type=client (242.798µs)
Dec 13 16:59:38 ip-10-30-1-43 nomad[29064]:     2017/12/13 16:59:38.616545 [DEBUG] http: Request /v1/status/leader (508.628µs)
Dec 13 16:59:38 ip-10-30-1-43 nomad[29064]: http: Request /v1/status/leader (508.628µs)

Job file (if appropriate)
@schmichael
Copy link
Member

Would you be able to test Nomad 0.7.1-rc1 that was just released Monday?
Link to Linux amd64 binary: https://releases.hashicorp.com/nomad/0.7.1-rc1/nomad_0.7.1-rc1_linux_amd64.zip
Release notes: https://groups.google.com/d/msg/nomad-tool/-aUbGM2_ou0/FMawwLq1AgAJ

I believe #3445 fixes your issue.

@dadgar
Copy link
Contributor

dadgar commented Dec 13, 2017

@apenney

  1. Of those jobs how many are dead?
  2. Can you do a curl -XPUT http://127.0.0.1:4646/v1/system/gc and collect the logs from all the servers.
  3. For a job that is dead and you believe should be garbage collected can you collect: curl http://127.0.0.1:4646/v1/job/<job>/allocations

@apenney
Copy link
Author

apenney commented Dec 13, 2017 •
edited by dadgar
Loading

0.7.1 doesn't seem to have helped @schmichael, I'm still in the same situation.

1: Most of them are dead:

root@ip-10-30-1-43:/var/nomad/alloc# nomad job status | grep dead | wc -l
167

2:

Here's the logs from the 0.7.1-rc1 box:

Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.423133 [DEBUG] client: alloc "8d60c99e-b719-6abb-0a3d-b004b08f069c" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "8d60c99e-b719-6abb-0a3d-b004b08f069c" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.436503 [DEBUG] client: alloc "97d494a3-c113-ca17-8be2-2f1fb4438f06" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "97d494a3-c113-ca17-8be2-2f1fb4438f06" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 8d60c99e-b719-6abb-0a3d-b004b08f069c for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.436824 [INFO] client.gc: marking allocation 8d60c99e-b719-6abb-0a3d-b004b08f069c for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.449583 [INFO] client.gc: marking allocation 97d494a3-c113-ca17-8be2-2f1fb4438f06 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 97d494a3-c113-ca17-8be2-2f1fb4438f06 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.450270 [DEBUG] client: alloc "4f7cbfbd-f4b1-6644-a561-973d9aae0128" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "4f7cbfbd-f4b1-6644-a561-973d9aae0128" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.473093 [INFO] client.gc: marking allocation 4f7cbfbd-f4b1-6644-a561-973d9aae0128 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 4f7cbfbd-f4b1-6644-a561-973d9aae0128 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.474111 [DEBUG] client: alloc "aad858f5-32d7-f4eb-1041-434d5218561a" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "aad858f5-32d7-f4eb-1041-434d5218561a" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.486222 [INFO] client.gc: marking allocation aad858f5-32d7-f4eb-1041-434d5218561a for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation aad858f5-32d7-f4eb-1041-434d5218561a for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.487075 [DEBUG] client: alloc "454db4b9-733a-1211-bb5c-370cb8a4bcc1" waiting for remote previous alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" to terminate
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "454db4b9-733a-1211-bb5c-370cb8a4bcc1" waiting for remote previous alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" to terminate
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.488876 [DEBUG] client: alloc "454db4b9-733a-1211-bb5c-370cb8a4bcc1" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "454db4b9-733a-1211-bb5c-370cb8a4bcc1" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.523354 [INFO] client.gc: marking allocation 454db4b9-733a-1211-bb5c-370cb8a4bcc1 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 454db4b9-733a-1211-bb5c-370cb8a4bcc1 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.524139 [DEBUG] client: alloc "60bcb176-7853-af47-0c3b-d382600eb35f" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "60bcb176-7853-af47-0c3b-d382600eb35f" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.537929 [DEBUG] client: alloc "76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca" waiting for remote previous alloc "403baf3c-00a2-6e5d-5f14-88fcd4cc688d" to terminate
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca" waiting for remote previous alloc "403baf3c-00a2-6e5d-5f14-88fcd4cc688d" to terminate
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.538940 [DEBUG] client: alloc "76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.543500 [DEBUG] client: updated allocations at index 607150 (total 260) (pulled 149) (filtered 111)
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: updated allocations at index 607150 (total 260) (pulled 149) (filtered 111)
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.549697 [INFO] client.gc: marking allocation 76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.549881 [INFO] client.gc: marking allocation 60bcb176-7853-af47-0c3b-d382600eb35f for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 76764b5b-cdec-5cc0-bcc6-3bb52f8e14ca for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 60bcb176-7853-af47-0c3b-d382600eb35f for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.550959 [DEBUG] client: alloc "c03b42b7-ff48-189b-0e2b-f57e3888bb0d" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "c03b42b7-ff48-189b-0e2b-f57e3888bb0d" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.563213 [INFO] client.gc: marking allocation c03b42b7-ff48-189b-0e2b-f57e3888bb0d for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation c03b42b7-ff48-189b-0e2b-f57e3888bb0d for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.563757 [DEBUG] client: alloc "65f06722-772a-8c48-d2e1-183d68307535" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "65f06722-772a-8c48-d2e1-183d68307535" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.576996 [INFO] client.gc: marking allocation 65f06722-772a-8c48-d2e1-183d68307535 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 65f06722-772a-8c48-d2e1-183d68307535 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.577865 [DEBUG] client: alloc "7bfa7846-32ed-c73e-096b-7ec0c6de4b4a" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "7bfa7846-32ed-c73e-096b-7ec0c6de4b4a" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.591752 [INFO] client.gc: marking allocation 7bfa7846-32ed-c73e-096b-7ec0c6de4b4a for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 7bfa7846-32ed-c73e-096b-7ec0c6de4b4a for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.592901 [DEBUG] client: alloc "c17c6e42-1445-27b2-aa83-d38a9f0b788b" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "c17c6e42-1445-27b2-aa83-d38a9f0b788b" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.605541 [INFO] client.gc: marking allocation c17c6e42-1445-27b2-aa83-d38a9f0b788b for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation c17c6e42-1445-27b2-aa83-d38a9f0b788b for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.605839 [DEBUG] client: alloc "cf95d8b3-5847-e53d-9908-cef313e624c4" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "cf95d8b3-5847-e53d-9908-cef313e624c4" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.618239 [INFO] client.gc: marking allocation cf95d8b3-5847-e53d-9908-cef313e624c4 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation cf95d8b3-5847-e53d-9908-cef313e624c4 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.619272 [DEBUG] client: alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" waiting for remote previous alloc "756baf79-f842-b57b-e541-56b5010c0972" to terminate
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" waiting for remote previous alloc "756baf79-f842-b57b-e541-56b5010c0972" to terminate
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.620149 [DEBUG] client: alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "a0ff8bf3-3d82-c513-002b-189a82f87f98" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.633089 [INFO] client.gc: marking allocation a0ff8bf3-3d82-c513-002b-189a82f87f98 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation a0ff8bf3-3d82-c513-002b-189a82f87f98 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.633913 [DEBUG] client: alloc "89a880f4-2f7a-3b44-c1e4-2f8c22b89179" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "89a880f4-2f7a-3b44-c1e4-2f8c22b89179" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.647838 [INFO] client.gc: marking allocation 89a880f4-2f7a-3b44-c1e4-2f8c22b89179 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.648249 [DEBUG] client: alloc "74b37c99-5f20-e968-b05a-3642ade44ae9" waiting for remote previous alloc "fa37f5c3-e340-0a1c-81ab-ed90d142de75" to terminate
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 89a880f4-2f7a-3b44-c1e4-2f8c22b89179 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "74b37c99-5f20-e968-b05a-3642ade44ae9" waiting for remote previous alloc "fa37f5c3-e340-0a1c-81ab-ed90d142de75" to terminate
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.649543 [DEBUG] client: alloc "74b37c99-5f20-e968-b05a-3642ade44ae9" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "74b37c99-5f20-e968-b05a-3642ade44ae9" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.661565 [INFO] client.gc: marking allocation 74b37c99-5f20-e968-b05a-3642ade44ae9 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 74b37c99-5f20-e968-b05a-3642ade44ae9 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.662386 [DEBUG] client: alloc "5162b0ed-a687-ac1d-4f30-33406acaac63" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "5162b0ed-a687-ac1d-4f30-33406acaac63" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.674501 [INFO] client.gc: marking allocation 5162b0ed-a687-ac1d-4f30-33406acaac63 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 5162b0ed-a687-ac1d-4f30-33406acaac63 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.674834 [DEBUG] client: alloc "7dea9772-5e2f-0551-2184-ccf0eb4f6003" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "7dea9772-5e2f-0551-2184-ccf0eb4f6003" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.686990 [INFO] client.gc: marking allocation 7dea9772-5e2f-0551-2184-ccf0eb4f6003 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 7dea9772-5e2f-0551-2184-ccf0eb4f6003 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.687898 [DEBUG] client: alloc "8ec810f6-e606-9e14-51a6-2de2b1dbf28f" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "8ec810f6-e606-9e14-51a6-2de2b1dbf28f" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.699946 [INFO] client.gc: marking allocation 8ec810f6-e606-9e14-51a6-2de2b1dbf28f for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 8ec810f6-e606-9e14-51a6-2de2b1dbf28f for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.700405 [DEBUG] client: alloc "2f531a29-431e-4d31-451b-d08c84cc2176" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "2f531a29-431e-4d31-451b-d08c84cc2176" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.713843 [INFO] client.gc: marking allocation 2f531a29-431e-4d31-451b-d08c84cc2176 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 2f531a29-431e-4d31-451b-d08c84cc2176 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.714922 [DEBUG] client: alloc "d62bec89-0c99-4a20-a47d-c4cd275eb355" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "d62bec89-0c99-4a20-a47d-c4cd275eb355" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.727722 [INFO] client.gc: marking allocation d62bec89-0c99-4a20-a47d-c4cd275eb355 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation d62bec89-0c99-4a20-a47d-c4cd275eb355 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.728291 [DEBUG] client: alloc "5c6aff2d-f11f-f18e-fef6-b8e92a13c167" waiting for remote previous alloc "dd0cfd72-3a94-a9b5-1afc-320b3388499f" to terminate
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "5c6aff2d-f11f-f18e-fef6-b8e92a13c167" waiting for remote previous alloc "dd0cfd72-3a94-a9b5-1afc-320b3388499f" to terminate
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.729906 [DEBUG] client: alloc "5c6aff2d-f11f-f18e-fef6-b8e92a13c167" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "5c6aff2d-f11f-f18e-fef6-b8e92a13c167" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.747433 [INFO] client.gc: marking allocation 5c6aff2d-f11f-f18e-fef6-b8e92a13c167 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 5c6aff2d-f11f-f18e-fef6-b8e92a13c167 for GC
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.747842 [DEBUG] client: alloc "8e895f9b-d99b-c4ee-a214-9e2f4db5c8d7" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "8e895f9b-d99b-c4ee-a214-9e2f4db5c8d7" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.759880 [DEBUG] client: updated allocations at index 607151 (total 260) (pulled 133) (filtered 127)
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: updated allocations at index 607151 (total 260) (pulled 133) (filtered 127)
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:19.761176 [DEBUG] client: alloc "58927b32-e1f9-6628-2bb6-b8192366c032" in terminal status, waiting for destroy
Dec 13 19:35:19 ip-10-30-1-43 nomad[15209]: client: alloc "58927b32-e1f9-6628-2bb6-b8192366c032" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.456301 [DEBUG] client: 34 evaluations triggered by node update
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.456421 [DEBUG] client: state updated to ready
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: 34 evaluations triggered by node update
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: state updated to ready
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.733452 [INFO] client.gc: marking allocation 8e895f9b-d99b-c4ee-a214-9e2f4db5c8d7 for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 8e895f9b-d99b-c4ee-a214-9e2f4db5c8d7 for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.734360 [INFO] client.gc: marking allocation 58927b32-e1f9-6628-2bb6-b8192366c032 for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 58927b32-e1f9-6628-2bb6-b8192366c032 for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.735073 [DEBUG] client: alloc "3ebb8609-0a0f-22ec-da55-091301002b03" waiting for remote previous alloc "03f5f08f-1f8d-66f7-e28a-22f22465694d" to terminate
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "3ebb8609-0a0f-22ec-da55-091301002b03" waiting for remote previous alloc "03f5f08f-1f8d-66f7-e28a-22f22465694d" to terminate
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.736350 [DEBUG] client: alloc "3ebb8609-0a0f-22ec-da55-091301002b03" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "3ebb8609-0a0f-22ec-da55-091301002b03" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.751440 [INFO] client.gc: marking allocation 3ebb8609-0a0f-22ec-da55-091301002b03 for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 3ebb8609-0a0f-22ec-da55-091301002b03 for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.752284 [DEBUG] client: alloc "c2871eff-20c1-15dd-03f8-1f75a7f49aca" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "c2871eff-20c1-15dd-03f8-1f75a7f49aca" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.794053 [INFO] client.gc: marking allocation c2871eff-20c1-15dd-03f8-1f75a7f49aca for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation c2871eff-20c1-15dd-03f8-1f75a7f49aca for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.794479 [DEBUG] client: alloc "3b983e9d-f6d1-7f0c-8d0e-e1209abd0bea" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "3b983e9d-f6d1-7f0c-8d0e-e1209abd0bea" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.795495 [DEBUG] client: updated allocations at index 607152 (total 260) (pulled 130) (filtered 130)
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: updated allocations at index 607152 (total 260) (pulled 130) (filtered 130)
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.830706 [DEBUG] client: alloc "af54da3b-6a54-514b-9cad-94d907301aed" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "af54da3b-6a54-514b-9cad-94d907301aed" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.843986 [INFO] client.gc: marking allocation af54da3b-6a54-514b-9cad-94d907301aed for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation af54da3b-6a54-514b-9cad-94d907301aed for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.845132 [DEBUG] client: alloc "23251e63-074b-59d9-80ef-310fce801d42" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "23251e63-074b-59d9-80ef-310fce801d42" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.857386 [INFO] client.gc: marking allocation 23251e63-074b-59d9-80ef-310fce801d42 for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 23251e63-074b-59d9-80ef-310fce801d42 for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.858984 [DEBUG] client: alloc "ba63d222-db04-e7f7-02b3-2479d10ea7c5" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "ba63d222-db04-e7f7-02b3-2479d10ea7c5" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.864374 [DEBUG] client: updated allocations at index 607204 (total 260) (pulled 127) (filtered 133)
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: updated allocations at index 607204 (total 260) (pulled 127) (filtered 133)
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.871166 [INFO] client.gc: marking allocation 3b983e9d-f6d1-7f0c-8d0e-e1209abd0bea for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 3b983e9d-f6d1-7f0c-8d0e-e1209abd0bea for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.871939 [INFO] client.gc: marking allocation ba63d222-db04-e7f7-02b3-2479d10ea7c5 for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.871855 [DEBUG] client: alloc "dd0cfd72-3a94-a9b5-1afc-320b3388499f" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation ba63d222-db04-e7f7-02b3-2479d10ea7c5 for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "dd0cfd72-3a94-a9b5-1afc-320b3388499f" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.885526 [INFO] client.gc: marking allocation dd0cfd72-3a94-a9b5-1afc-320b3388499f for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation dd0cfd72-3a94-a9b5-1afc-320b3388499f for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.885815 [DEBUG] client: alloc "50d7116e-808b-4ae0-f6f2-5be1a06ad746" waiting for remote previous alloc "4a594eca-8a63-acbe-99e9-11c2673c2519" to terminate
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "50d7116e-808b-4ae0-f6f2-5be1a06ad746" waiting for remote previous alloc "4a594eca-8a63-acbe-99e9-11c2673c2519" to terminate
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.887516 [DEBUG] client: alloc "50d7116e-808b-4ae0-f6f2-5be1a06ad746" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "50d7116e-808b-4ae0-f6f2-5be1a06ad746" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.898990 [INFO] client.gc: marking allocation 50d7116e-808b-4ae0-f6f2-5be1a06ad746 for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client.gc: marking allocation 50d7116e-808b-4ae0-f6f2-5be1a06ad746 for GC
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.900023 [DEBUG] client: alloc "de9cabd7-4b85-152a-da52-13706bb01fd6" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: alloc "de9cabd7-4b85-152a-da52-13706bb01fd6" in terminal status, waiting for destroy
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:20.905435 [DEBUG] client: updated allocations at index 607216 (total 260) (pulled 124) (filtered 136)
Dec 13 19:35:20 ip-10-30-1-43 nomad[15209]: client: updated allocations at index 607216 (total 260) (pulled 124) (filtered 136)
Dec 13 19:35:34 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:34.169026 [DEBUG] http: Request /v1/agent/health?type=client (267.633µs)
Dec 13 19:35:34 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (267.633µs)
Dec 13 19:35:40 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:40.257261 [DEBUG] http: Request /v1/status/leader (8.952714ms)
Dec 13 19:35:40 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (8.952714ms)
Dec 13 19:35:44 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:44.174416 [DEBUG] http: Request /v1/agent/health?type=client (4.448616ms)
Dec 13 19:35:44 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (4.448616ms)
Dec 13 19:35:47 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:47.858337 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (3.829894ms)
Dec 13 19:35:47 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (3.829894ms)
Dec 13 19:35:54 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:54.176002 [DEBUG] http: Request /v1/agent/health?type=client (241.173µs)
Dec 13 19:35:54 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (241.173µs)
Dec 13 19:35:55 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:55.259426 [DEBUG] http: Request /v1/status/leader (547.139µs)
Dec 13 19:35:55 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (547.139µs)
Dec 13 19:35:58 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:58.018504 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (245.288µs)
Dec 13 19:35:58 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (245.288µs)
Dec 13 19:35:58 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:35:58.037559 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (349.099µs)
Dec 13 19:35:58 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (349.099µs)
Dec 13 19:36:04 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:36:04.177455 [DEBUG] http: Request /v1/agent/health?type=client (256.83µs)
Dec 13 19:36:04 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (256.83µs)
Dec 13 19:36:10 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:36:10.261140 [DEBUG] http: Request /v1/status/leader (564.829µs)
Dec 13 19:36:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (564.829µs)
Dec 13 19:36:14 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:36:14.178673 [DEBUG] http: Request /v1/agent/health?type=client (216.296µs)
Dec 13 19:36:14 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (216.296µs)
Dec 13 19:36:24 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:36:24.189967 [DEBUG] http: Request /v1/agent/health?type=client (10.144846ms)
Dec 13 19:36:24 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (10.144846ms)
Dec 13 19:36:25 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:36:25.262840 [DEBUG] http: Request /v1/status/leader (551.843µs)
Dec 13 19:36:25 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (551.843µs)
Dec 13 19:36:34 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:36:34.191218 [DEBUG] http: Request /v1/agent/health?type=client (242.978µs)
Dec 13 19:36:34 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (242.978µs)
Dec 13 19:36:40 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:36:40.264611 [DEBUG] http: Request /v1/status/leader (474.955µs)
Dec 13 19:36:40 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (474.955µs)
Dec 13 19:36:44 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:36:44.192342 [DEBUG] http: Request /v1/agent/health?type=client (217.45µs)
Dec 13 19:36:44 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (217.45µs)
Dec 13 19:36:54 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:36:54.193885 [DEBUG] http: Request /v1/agent/health?type=client (267.08µs)
Dec 13 19:36:54 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (267.08µs)
Dec 13 19:36:55 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:36:55.266656 [DEBUG] http: Request /v1/status/leader (689.019µs)
Dec 13 19:36:55 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (689.019µs)
Dec 13 19:37:04 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:37:04.195508 [DEBUG] http: Request /v1/agent/health?type=client (283.838µs)
Dec 13 19:37:04 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (283.838µs)
Dec 13 19:37:10 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:37:10.268937 [DEBUG] http: Request /v1/status/leader (965.641µs)
Dec 13 19:37:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (965.641µs)
Dec 13 19:37:14 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:37:14.197216 [DEBUG] http: Request /v1/agent/health?type=client (316.67µs)
Dec 13 19:37:14 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (316.67µs)
Dec 13 19:37:18 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:37:18.463994 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (216.193µs)
Dec 13 19:37:18 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (216.193µs)
Dec 13 19:37:24 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:37:24.203260 [DEBUG] http: Request /v1/agent/health?type=client (4.790514ms)
Dec 13 19:37:24 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (4.790514ms)
Dec 13 19:37:25 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:37:25.270538 [DEBUG] http: Request /v1/status/leader (503.546µs)
Dec 13 19:37:25 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (503.546µs)
Dec 13 19:37:29 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:37:29.099502 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (307.012µs)
Dec 13 19:37:29 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (307.012µs)
Dec 13 19:37:29 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:37:29.128447 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (174.287µs)
Dec 13 19:37:29 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (174.287µs)
Dec 13 19:37:34 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:37:34.208357 [DEBUG] http: Request /v1/agent/health?type=client (228.085µs)
Dec 13 19:37:34 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (228.085µs)
Dec 13 19:37:40 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:37:40.272654 [DEBUG] http: Request /v1/status/leader (955.632µs)
Dec 13 19:37:40 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (955.632µs)
Dec 13 19:37:44 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:37:44.209568 [DEBUG] http: Request /v1/agent/health?type=client (234.668µs)
Dec 13 19:37:44 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (234.668µs)
Dec 13 19:37:54 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:37:54.210789 [DEBUG] http: Request /v1/agent/health?type=client (247.035µs)
Dec 13 19:37:54 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (247.035µs)
Dec 13 19:37:55 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:37:55.274748 [DEBUG] http: Request /v1/status/leader (904.08µs)
Dec 13 19:37:55 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (904.08µs)
Dec 13 19:38:04 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:04.212259 [DEBUG] http: Request /v1/agent/health?type=client (201.362µs)
Dec 13 19:38:04 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (201.362µs)
Dec 13 19:38:10 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:10.277078 [DEBUG] http: Request /v1/status/leader (975.083µs)
Dec 13 19:38:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (975.083µs)
Dec 13 19:38:14 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:14.213465 [DEBUG] http: Request /v1/agent/health?type=client (220.513µs)
Dec 13 19:38:14 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (220.513µs)
Dec 13 19:38:16 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:16.307316 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (229.282µs)
Dec 13 19:38:16 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (229.282µs)
Dec 13 19:38:23 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:23.631207 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (219.83µs)
Dec 13 19:38:23 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (219.83µs)
Dec 13 19:38:24 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:24.214647 [DEBUG] http: Request /v1/agent/health?type=client (198.392µs)
Dec 13 19:38:24 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (198.392µs)
Dec 13 19:38:25 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:25.278952 [DEBUG] http: Request /v1/status/leader (611.382µs)
Dec 13 19:38:25 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (611.382µs)
Dec 13 19:38:26 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:26.609274 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (309.496µs)
Dec 13 19:38:26 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (309.496µs)
Dec 13 19:38:26 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:26.626161 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (210.562µs)
Dec 13 19:38:26 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (210.562µs)
Dec 13 19:38:33 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:33.698901 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (247.474µs)
Dec 13 19:38:33 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (247.474µs)
Dec 13 19:38:33 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:33.722253 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (89.375µs)
Dec 13 19:38:33 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (89.375µs)
Dec 13 19:38:34 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:34.215804 [DEBUG] http: Request /v1/agent/health?type=client (157.479µs)
Dec 13 19:38:34 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (157.479µs)
Dec 13 19:38:40 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:40.280577 [DEBUG] http: Request /v1/status/leader (514.259µs)
Dec 13 19:38:40 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (514.259µs)
Dec 13 19:38:44 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:44.216929 [DEBUG] http: Request /v1/agent/health?type=client (234.622µs)
Dec 13 19:38:44 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (234.622µs)
Dec 13 19:38:49 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:49.561948 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (262.264µs)
Dec 13 19:38:49 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (262.264µs)
Dec 13 19:38:54 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:54.218095 [DEBUG] http: Request /v1/agent/health?type=client (213.49µs)
Dec 13 19:38:54 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (213.49µs)
Dec 13 19:38:55 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:55.282517 [DEBUG] http: Request /v1/status/leader (612.803µs)
Dec 13 19:38:55 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (612.803µs)
Dec 13 19:38:59 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:59.926153 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (276.397µs)
Dec 13 19:38:59 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (276.397µs)
Dec 13 19:38:59 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:38:59.952674 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (249.274µs)
Dec 13 19:38:59 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (249.274µs)
Dec 13 19:39:04 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:04.219420 [DEBUG] http: Request /v1/agent/health?type=client (230.915µs)
Dec 13 19:39:04 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (230.915µs)
Dec 13 19:39:10 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:10.106300 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (380.693µs)
Dec 13 19:39:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (380.693µs)
Dec 13 19:39:10 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:10.124885 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (106.49µs)
Dec 13 19:39:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (106.49µs)
Dec 13 19:39:10 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:10.289128 [DEBUG] http: Request /v1/status/leader (5.290217ms)
Dec 13 19:39:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (5.290217ms)
Dec 13 19:39:14 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:14.220878 [DEBUG] http: Request /v1/agent/health?type=client (319.849µs)
Dec 13 19:39:14 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (319.849µs)
Dec 13 19:39:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:20.581199 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (296.461µs)
Dec 13 19:39:20 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (296.461µs)
Dec 13 19:39:20 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:20.598649 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (132.847µs)
Dec 13 19:39:20 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (132.847µs)
Dec 13 19:39:24 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:24.222382 [DEBUG] http: Request /v1/agent/health?type=client (228.187µs)
Dec 13 19:39:24 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (228.187µs)
Dec 13 19:39:25 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:25.291681 [DEBUG] http: Request /v1/status/leader (1.511535ms)
Dec 13 19:39:25 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (1.511535ms)
Dec 13 19:39:25 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:25.668224 [DEBUG] http: Request /v1/system/gc (693.903µs)
Dec 13 19:39:25 ip-10-30-1-43 nomad[15209]: http: Request /v1/system/gc (693.903µs)
Dec 13 19:39:30 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:30.836928 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (231.916µs)
Dec 13 19:39:30 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (231.916µs)
Dec 13 19:39:30 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:30.858899 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (231.254µs)
Dec 13 19:39:30 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (231.254µs)
Dec 13 19:39:34 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:34.223638 [DEBUG] http: Request /v1/agent/health?type=client (214.667µs)
Dec 13 19:39:34 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (214.667µs)
Dec 13 19:39:40 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:40.296374 [DEBUG] http: Request /v1/status/leader (535.247µs)
Dec 13 19:39:40 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (535.247µs)
Dec 13 19:39:41 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:41.341266 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (248.522µs)
Dec 13 19:39:41 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (248.522µs)
Dec 13 19:39:41 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:41.356719 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (287.126µs)
Dec 13 19:39:41 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (287.126µs)
Dec 13 19:39:44 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:44.224862 [DEBUG] http: Request /v1/agent/health?type=client (219.14µs)
Dec 13 19:39:44 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (219.14µs)
Dec 13 19:39:51 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:51.568582 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (4.993536ms)
Dec 13 19:39:51 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (4.993536ms)
Dec 13 19:39:51 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:51.593021 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (128.688µs)
Dec 13 19:39:51 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (128.688µs)
Dec 13 19:39:54 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:54.226374 [DEBUG] http: Request /v1/agent/health?type=client (223.71µs)
Dec 13 19:39:54 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (223.71µs)
Dec 13 19:39:55 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:39:55.304475 [DEBUG] http: Request /v1/status/leader (560.34µs)
Dec 13 19:39:55 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (560.34µs)
Dec 13 19:40:00 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:00.790799 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (227.27µs)
Dec 13 19:40:00 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (227.27µs)
Dec 13 19:40:02 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:02.175335 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (229.492µs)
Dec 13 19:40:02 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (229.492µs)
Dec 13 19:40:02 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:02.215937 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (173.094µs)
Dec 13 19:40:02 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (173.094µs)
Dec 13 19:40:02 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:02.897214 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (242.247µs)
Dec 13 19:40:02 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (242.247µs)
Dec 13 19:40:04 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:04.227936 [DEBUG] http: Request /v1/agent/health?type=client (217.322µs)
Dec 13 19:40:04 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (217.322µs)
Dec 13 19:40:10 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:10.311347 [DEBUG] http: Request /v1/status/leader (5.536999ms)
Dec 13 19:40:10 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (5.536999ms)
Dec 13 19:40:11 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:11.397282 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (175.538µs)
Dec 13 19:40:11 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (175.538µs)
Dec 13 19:40:11 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:11.421125 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (207.155µs)
Dec 13 19:40:11 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (207.155µs)
Dec 13 19:40:13 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:13.535315 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (257.324µs)
Dec 13 19:40:13 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (257.324µs)
Dec 13 19:40:13 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:13.562052 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (227.65µs)
Dec 13 19:40:13 ip-10-30-1-43 nomad[15209]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (227.65µs)
Dec 13 19:40:14 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:14.230066 [DEBUG] http: Request /v1/agent/health?type=client (274.836µs)
Dec 13 19:40:14 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (274.836µs)
Dec 13 19:40:18 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:18.259910 [DEBUG] http: Request /v1/jobs (10.676546ms)
Dec 13 19:40:18 ip-10-30-1-43 nomad[15209]: http: Request /v1/jobs (10.676546ms)
Dec 13 19:40:19 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:19.979375 [DEBUG] http: Request /v1/jobs (10.359401ms)
Dec 13 19:40:19 ip-10-30-1-43 nomad[15209]: http: Request /v1/jobs (10.359401ms)
Dec 13 19:40:23 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:23.321174 [DEBUG] http: Request /v1/jobs (7.162803ms)
Dec 13 19:40:23 ip-10-30-1-43 nomad[15209]: http: Request /v1/jobs (7.162803ms)
Dec 13 19:40:24 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:24.231753 [DEBUG] http: Request /v1/agent/health?type=client (162.462µs)
Dec 13 19:40:24 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (162.462µs)
Dec 13 19:40:25 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:25.313501 [DEBUG] http: Request /v1/status/leader (652.374µs)
Dec 13 19:40:25 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (652.374µs)
Dec 13 19:40:34 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:34.233104 [DEBUG] http: Request /v1/agent/health?type=client (218.8µs)
Dec 13 19:40:34 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (218.8µs)
Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:36.173973 [DEBUG] client: updated allocations at index 607227 (total 260) (pulled 3) (filtered 257)
Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:36.174315 [DEBUG] client: allocs: (added 0) (removed 0) (updated 3) (ignore 257)
Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]: client: updated allocations at index 607227 (total 260) (pulled 3) (filtered 257)
Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:36.174398 [DEBUG] client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039'
Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:36.174435 [DEBUG] client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565'
Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]: client: allocs: (added 0) (removed 0) (updated 3) (ignore 257)
Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]: client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039'
Dec 13 19:40:36 ip-10-30-1-43 nomad[15209]: client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565'
^[[B^[[BDec 13 19:40:40 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:40.315631 [DEBUG] http: Request /v1/status/leader (1.02672ms)
Dec 13 19:40:40 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (1.02672ms)
Dec 13 19:40:44 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:44.234169 [DEBUG] http: Request /v1/agent/health?type=client (203.076µs)
Dec 13 19:40:44 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (203.076µs)
Dec 13 19:40:54 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:54.235522 [DEBUG] http: Request /v1/agent/health?type=client (234.772µs)
Dec 13 19:40:54 ip-10-30-1-43 nomad[15209]: http: Request /v1/agent/health?type=client (234.772µs)
Dec 13 19:40:55 ip-10-30-1-43 nomad[15209]:     2017/12/13 19:40:55.317594 [DEBUG] http: Request /v1/status/leader (996.679µs)
Dec 13 19:40:55 ip-10-30-1-43 nomad[15209]: http: Request /v1/status/leader (996.679µs)

3:

curl http://10.30.1.43:4646/v1/job/uniporter-uat/periodic-1510743600/allocations
[{"ID":"5e0496dd-f515-86fc-fd64-5d19807061a0","EvalID":"95537cd5-6166-f41c-ef7c-c90edd8d87b6","Name":"uniporter-uat/periodic-1510743600.ibm-wfo[0]","NodeID":"7cd5a435-ae09-ab0a-1f90-c4228be55cce","JobID":"uniporter-uat/periodic-1510743600","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"failed","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":true,"Restarts":0,"LastRestart":"0001-01-01T00:00:00Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Received","Time":1511049208686408058,"Message":"","DisplayMessage":"Task received by client","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Task Setup","Time":1511049208686482136,"Message":"Building Task Directory","DisplayMessage":"Building Task Directory","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killing","Time":1511049239383850300,"Message":"","DisplayMessage":"Killing task: vault: failed to derive token: failed to unwrap the token for task \"jtcc\"","Details":null,"FailsTask":true,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"vault: failed to derive token: failed to unwrap the token for task \"jtcc\"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":425697,"ModifyIndex":607221,"CreateTime":1511049208218050041,"ModifyTime":0},{"ID":"fc6d66ba-7719-062a-dc32-889a60709eb1","EvalID":"1500803d-ef4d-6856-1b58-43e11ad812f9","Name":"uniporter-uat/periodic-1510743600.ibm-wfo[0]","NodeID":"7cd5a435-ae09-ab0a-1f90-c4228be55cce","JobID":"uniporter-uat/periodic-1510743600","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"stop","DesiredDescription":"alloc is lost since its node is down","ClientStatus":"complete","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":false,"Restarts":16,"LastRestart":"2017-11-15T11:04:13.390821067Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Driver","Time":1510743820393178908,"Message":"","DisplayMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","GenericSource":""},{"Type":"Driver Failure","Time":1510743820478967357,"Message":"","DisplayMessage":"failed to initialize task \"jtcc\" for alloc \"fc6d66ba-7719-062a-dc32-889a60709eb1\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"failed to initialize task \"jtcc\" for alloc \"fc6d66ba-7719-062a-dc32-889a60709eb1\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Restarting","Time":1510743820479004253,"Message":"","DisplayMessage":"Task restarting in 17.09760235s","Details":null,"FailsTask":false,"RestartReason":"Restart within policy","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":17097602350,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Driver","Time":1510743837577786703,"Message":"","DisplayMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","GenericSource":""},{"Type":"Driver Failure","Time":1510743837652731353,"Message":"","DisplayMessage":"failed to initialize task \"jtcc\" for alloc \"fc6d66ba-7719-062a-dc32-889a60709eb1\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"failed to initialize task \"jtcc\" for alloc \"fc6d66ba-7719-062a-dc32-889a60709eb1\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Restarting","Time":1510743837652766788,"Message":"","DisplayMessage":"Task restarting in 15.611816545s","Details":null,"FailsTask":false,"RestartReason":"Restart within policy","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":15611816545,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Driver","Time":1510743853265843002,"Message":"","DisplayMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","GenericSource":""},{"Type":"Driver Failure","Time":1510743853390778326,"Message":"","DisplayMessage":"failed to initialize task \"jtcc\" for alloc \"fc6d66ba-7719-062a-dc32-889a60709eb1\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"failed to initialize task \"jtcc\" for alloc \"fc6d66ba-7719-062a-dc32-889a60709eb1\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Restarting","Time":1510743853390821067,"Message":"","DisplayMessage":"Exceeded allowed attempts, applying a delay - Task restarting in 167h55m46.723984944s","Details":null,"FailsTask":false,"RestartReason":"Exceeded allowed attempts, applying a delay","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":604546723984944,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killed","Time":1511049197330244197,"Message":"","DisplayMessage":"Task successfully killed","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":418809,"ModifyIndex":607146,"CreateTime":1510743600083464036,"ModifyTime":0}]

@apenney
Copy link
Author

apenney commented Dec 13, 2017

That job:

uniporter-uat/periodic-1511175600         batch           50        dead (stopped)  11/20/17 11:00:00 UTC

@apenney
Copy link
Author

apenney commented Dec 13, 2017 •
edited
Loading

I managed to run nomad stop -purge $job on all these jobs, and they no longer exist:

curl http://10.30.1.43:4646/v1/job/uniporter-uat/periodic-1511953200
job not found

root@ip-10-30-1-43:/var/nomad# curl http://10.30.1.43:4646/v1/job/uniporter-uat/periodic-1511953200/allocations
[{"ID":"3b81f235-c6c2-0a04-9dca-28e9652fa43d","EvalID":"cafede1e-575e-b099-22ce-e6cb837acdb8","Name":"uniporter-uat/periodic-1511953200.ibm-wfo[0]","NodeID":"7cd5a435-ae09-ab0a-1f90-c4228be55cce","JobID":"uniporter-uat/periodic-1511953200","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"stop","DesiredDescription":"alloc not needed due to job update","ClientStatus":"complete","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":false,"Restarts":23,"LastRestart":"2017-12-13T19:52:37.06046043Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Driver","Time":1513194722835388017,"Message":"","DisplayMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","GenericSource":""},{"Type":"Driver Failure","Time":1513194722918026138,"Message":"","DisplayMessage":"failed to initialize task \"jtcc\" for alloc \"3b81f235-c6c2-0a04-9dca-28e9652fa43d\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"failed to initialize task \"jtcc\" for alloc \"3b81f235-c6c2-0a04-9dca-28e9652fa43d\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Restarting","Time":1513194722918074121,"Message":"","DisplayMessage":"Task restarting in 15.960969333s","Details":null,"FailsTask":false,"RestartReason":"Restart within policy","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":15960969333,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Driver","Time":1513194738881431161,"Message":"","DisplayMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","GenericSource":""},{"Type":"Driver Failure","Time":1513194738962467969,"Message":"","DisplayMessage":"failed to initialize task \"jtcc\" for alloc \"3b81f235-c6c2-0a04-9dca-28e9652fa43d\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"failed to initialize task \"jtcc\" for alloc \"3b81f235-c6c2-0a04-9dca-28e9652fa43d\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Restarting","Time":1513194738962563682,"Message":"","DisplayMessage":"Task restarting in 17.911645299s","Details":null,"FailsTask":false,"RestartReason":"Restart within policy","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":17911645299,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Driver","Time":1513194756879242992,"Message":"","DisplayMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"Downloading image cotalabs/uniporter:0.1.1-SNAPSHOT","GenericSource":""},{"Type":"Driver Failure","Time":1513194757059836499,"Message":"","DisplayMessage":"failed to initialize task \"jtcc\" for alloc \"3b81f235-c6c2-0a04-9dca-28e9652fa43d\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"failed to initialize task \"jtcc\" for alloc \"3b81f235-c6c2-0a04-9dca-28e9652fa43d\": Failed to pull `cotalabs/uniporter:0.1.1-SNAPSHOT`: API error (404): {\"message\":\"manifest for cotalabs/uniporter:0.1.1-SNAPSHOT not found\"}\n","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Restarting","Time":1513194757060460430,"Message":"","DisplayMessage":"Task restarting in 15.453403262s","Details":null,"FailsTask":false,"RestartReason":"Restart within policy","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":15453403262,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killed","Time":1513194765024101601,"Message":"","DisplayMessage":"Task successfully killed","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":607390,"ModifyIndex":607806,"CreateTime":1513194279915270907,"ModifyTime":0},{"ID":"8c83f1cb-8589-7773-b88a-82f63ad1caf9","EvalID":"d2258389-43f3-8ab7-e738-0e5b57e1b2c6","Name":"uniporter-uat/periodic-1511953200.ibm-wfo[0]","NodeID":"7cdec421-166e-04da-07e1-2b37858357a3","JobID":"uniporter-uat/periodic-1511953200","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"failed","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":true,"Restarts":0,"LastRestart":"0001-01-01T00:00:00Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Received","Time":1513115825621384244,"Message":"","DisplayMessage":"Task received by client","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Task Setup","Time":1513115825621496358,"Message":"Building Task Directory","DisplayMessage":"Building Task Directory","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killing","Time":1513115825636789224,"Message":"","DisplayMessage":"Killing task: vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"8c83f1cb-8589-7773-b88a-82f63ad1caf9\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","Details":null,"FailsTask":true,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"8c83f1cb-8589-7773-b88a-82f63ad1caf9\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":595105,"ModifyIndex":595302,"CreateTime":1513115818001542460,"ModifyTime":0},{"ID":"94936d21-87ec-68da-ac40-31fc7d8f74fd","EvalID":"5039797e-5886-cef2-b7b0-3c85efec7108","Name":"uniporter-uat/periodic-1511953200.ibm-wfo[0]","NodeID":"7cd5a435-ae09-ab0a-1f90-c4228be55cce","JobID":"uniporter-uat/periodic-1511953200","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"failed","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":true,"Restarts":0,"LastRestart":"0001-01-01T00:00:00Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Received","Time":1513112912870960502,"Message":"","DisplayMessage":"Task received by client","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Task Setup","Time":1513112912871048881,"Message":"Building Task Directory","DisplayMessage":"Building Task Directory","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killing","Time":1513112912901171716,"Message":"","DisplayMessage":"Killing task: vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"94936d21-87ec-68da-ac40-31fc7d8f74fd\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","Details":null,"FailsTask":true,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"94936d21-87ec-68da-ac40-31fc7d8f74fd\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":592027,"ModifyIndex":607786,"CreateTime":1513112903271182797,"ModifyTime":0},{"ID":"b73bba18-671e-5f9d-4217-1c9b2b22f723","EvalID":"9d143393-2a24-5f75-7517-7146997074e2","Name":"uniporter-uat/periodic-1511953200.ibm-wfo[0]","NodeID":"7cd5a435-ae09-ab0a-1f90-c4228be55cce","JobID":"uniporter-uat/periodic-1511953200","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"failed","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":true,"Restarts":0,"LastRestart":"0001-01-01T00:00:00Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Received","Time":1513115803904081416,"Message":"","DisplayMessage":"Task received by client","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Task Setup","Time":1513115803904151831,"Message":"Building Task Directory","DisplayMessage":"Building Task Directory","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killing","Time":1513115803931710036,"Message":"","DisplayMessage":"Killing task: vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"b73bba18-671e-5f9d-4217-1c9b2b22f723\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","Details":null,"FailsTask":true,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"b73bba18-671e-5f9d-4217-1c9b2b22f723\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":594855,"ModifyIndex":607786,"CreateTime":1513115803876082994,"ModifyTime":0},{"ID":"c7d1093e-ca1b-f385-0502-34d67cf7b9d3","EvalID":"1b9981c7-fcbf-ea3a-1bf9-ded95897aadc","Name":"uniporter-uat/periodic-1511953200.ibm-wfo[0]","NodeID":"7cdec421-166e-04da-07e1-2b37858357a3","JobID":"uniporter-uat/periodic-1511953200","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"failed","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":true,"Restarts":0,"LastRestart":"0001-01-01T00:00:00Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Received","Time":1513115398637824126,"Message":"","DisplayMessage":"Task received by client","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Task Setup","Time":1513115398637986165,"Message":"Building Task Directory","DisplayMessage":"Building Task Directory","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killing","Time":1513115398653332452,"Message":"","DisplayMessage":"Killing task: vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"c7d1093e-ca1b-f385-0502-34d67cf7b9d3\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","Details":null,"FailsTask":true,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"c7d1093e-ca1b-f385-0502-34d67cf7b9d3\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":594082,"ModifyIndex":594373,"CreateTime":1513115383337841929,"ModifyTime":0},{"ID":"d16d4240-ee9a-9bea-4eac-23a032b75336","EvalID":"7fbe95da-90eb-f40d-cdbb-7ab7ca716a4d","Name":"uniporter-uat/periodic-1511953200.ibm-wfo[0]","NodeID":"7cd5a435-ae09-ab0a-1f90-c4228be55cce","JobID":"uniporter-uat/periodic-1511953200","JobVersion":0,"TaskGroup":"ibm-wfo","DesiredStatus":"run","DesiredDescription":"","ClientStatus":"failed","ClientDescription":"","TaskStates":{"jtcc":{"State":"dead","Failed":true,"Restarts":0,"LastRestart":"0001-01-01T00:00:00Z","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Events":[{"Type":"Received","Time":1513112735189272618,"Message":"","DisplayMessage":"Task received by client","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Task Setup","Time":1513112735189349266,"Message":"Building Task Directory","DisplayMessage":"Building Task Directory","Details":null,"FailsTask":false,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""},{"Type":"Killing","Time":1513112735204538511,"Message":"","DisplayMessage":"Killing task: vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"d16d4240-ee9a-9bea-4eac-23a032b75336\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","Details":null,"FailsTask":true,"RestartReason":"","SetupError":"","DriverError":"","ExitCode":0,"Signal":0,"KillTimeout":0,"KillError":"","KillReason":"vault: failed to derive token: failed to create token for task \"jtcc\" on alloc \"d16d4240-ee9a-9bea-4eac-23a032b75336\": Connection to Vault failed: failed to lookup Vault periodic token: Error making API request.\n\nURL: POST http://vault.service.consul:8200/v1/auth/token/lookup\nCode: 403. Errors:\n\n* permission denied","StartDelay":0,"DownloadError":"","ValidationError":"","DiskLimit":0,"FailedSibling":"","VaultError":"","TaskSignalReason":"","TaskSignal":"","DriverMessage":"","GenericSource":""}]}},"DeploymentStatus":null,"CreateIndex":591541,"ModifyIndex":607786,"CreateTime":1513112730585011087,"ModifyTime":0}]

Then the allocs still can't gc:

Dec 13 19:59:34 ip-10-30-1-43 nomad[28460]: client: allocs: (added 0) (removed 0) (updated 4) (ignore 265)
Dec 13 19:59:34 ip-10-30-1-43 nomad[28460]: client: dropping update to terminal alloc '3b81f235-c6c2-0a04-9dca-28e9652fa43d'
Dec 13 19:59:34 ip-10-30-1-43 nomad[28460]: client: dropping update to terminal alloc '0b1e29f2-976e-1c85-2ab5-0689915a6244'
Dec 13 19:59:34 ip-10-30-1-43 nomad[28460]: client: dropping update to terminal alloc '7180fee2-c804-7484-ad90-5bb6f8727039'
Dec 13 19:59:34 ip-10-30-1-43 nomad[28460]: client: dropping update to terminal alloc '44ae7315-243e-cfdb-3f34-d95b277a9565'

@apenney
Copy link
Author

apenney commented Dec 13, 2017

We ended up bringing down all masters, wiping all raft/client.db, and starting from scratch. All gc'ing was broken across all nodes, and it looks like it's been broken since 0.5->0.6 upgrade.

@dadgar
Copy link
Contributor

dadgar commented Dec 13, 2017

@apenney Unfortunately the logs you gave are client logs not the server logs. So there really isn't enough information to try to track this down.

I am going to close this until we do get more information or you can provide repro steps. If it does happen again, following the steps I outlined in my response should provide the information we need.

@dadgar dadgar closed this as completed Dec 13, 2017
@apenney
Copy link
Author

apenney commented Jan 12, 2018 •
edited by schmichael
Loading

Not sure if I can reopen this but here are the server logs during a gc, as this issue is now reoccurring. Server3 is the master in the below paste.

Symptoms, just to recap: Nodes fill up with allocs, and can't gc. I have over ~3000 jobs when I should have about 20, most of those are "dead" batch jobs that never go away. On the servers I can see it constantly trying to alloc and failing to get vault tokens for jobs that are dead and shouldn't be doing anything.

Server1:

Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:04.740810 [DEBUG] http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (8.153384ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (8.153384ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:04.741363 [DEBUG] http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (6.86882ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (6.86882ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:04.741910 [DEBUG] http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (7.240519ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (7.240519ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:04.742494 [DEBUG] http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (6.516327ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (6.516327ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:04.743035 [DEBUG] http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (7.563235ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (7.563235ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:04.743606 [DEBUG] http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (8.413474ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (8.413474ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:04.744147 [DEBUG] http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (8.832525ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (8.832525ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (8.941974ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:04.744684 [DEBUG] http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (8.941974ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:04.745211 [DEBUG] http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (9.584528ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (9.584528ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (4.438317ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:04.739275 [DEBUG] http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (4.438317ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:04.746330 [DEBUG] http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (11.370118ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (11.370118ms)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:04.747287 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (181.824µs)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (181.824µs)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:04.773791 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (116.087µs)
Jan 12 15:52:04 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (116.087µs)
Jan 12 15:52:05 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:05 [DEBUG] memberlist: TCP connection from=10.50.2.47:35878
Jan 12 15:52:05 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.2.47:35878
Jan 12 15:52:06 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:06.528954 [DEBUG] http: Request /v1/agent/health?type=server (1.063618ms)
Jan 12 15:52:06 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (1.063618ms)
Jan 12 15:52:06 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:06.904706 [DEBUG] http: Request /v1/agent/health?type=client (279.685µs)
Jan 12 15:52:06 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (279.685µs)
Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:07.097761 [DEBUG] http: Request /v1/agent/health?type=client (649.065µs)
Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (649.065µs)
Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:07.184990 [DEBUG] http: Request /v1/agent/health?type=client (192.565µs)
Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (192.565µs)
Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:07.187744 [DEBUG] http: Request /v1/agent/health?type=server (507.651µs)
Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (507.651µs)
Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:07.201052 [DEBUG] http: Request /v1/agent/health?type=server (468.691µs)
Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (468.691µs)
Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:07 [DEBUG] memberlist: TCP connection from=10.50.5.8:52664
Jan 12 15:52:07 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52664
Jan 12 15:52:08 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52666
Jan 12 15:52:08 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:08 [DEBUG] memberlist: TCP connection from=10.50.5.8:52666
Jan 12 15:52:09 ip-10-50-1-29 nomad[1436]: http: Request /v1/status/leader?region=us-east-1&wait=60000ms (540.14µs)
Jan 12 15:52:09 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:09.965589 [DEBUG] http: Request /v1/status/leader?region=us-east-1&wait=60000ms (540.14µs)
Jan 12 15:52:10 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:10.526439 [DEBUG] http: Request /v1/agent/health?type=client (344.193µs)
Jan 12 15:52:10 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (344.193µs)
Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:11.245061 [DEBUG] http: Request /v1/agent/health?type=server (1.566163ms)
Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (1.566163ms)
Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:11.816870 [DEBUG] http: Request /v1/agent/health?type=server (663.73µs)
Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (663.73µs)
Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:11 [DEBUG] memberlist: TCP connection from=10.50.4.247:35548
Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35548
Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:11 [DEBUG] memberlist: TCP connection from=10.50.4.247:35550
Jan 12 15:52:11 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35550
Jan 12 15:52:12 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (217.803µs)
Jan 12 15:52:12 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:12.597037 [DEBUG] http: Request /v1/agent/health?type=client (217.803µs)
Jan 12 15:52:13 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:13.553745 [DEBUG] http: Request /v1/allocations?index=696204&region=us-east-1&stale=&wait=120000ms (511.983994ms)
Jan 12 15:52:13 ip-10-50-1-29 nomad[1436]: http: Request /v1/allocations?index=696204&region=us-east-1&stale=&wait=120000ms (511.983994ms)
Jan 12 15:52:13 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:13.608652 [DEBUG] http: Request /v1/agent/health?type=server (605.496µs)
Jan 12 15:52:13 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (605.496µs)
Jan 12 15:52:14 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (186.597µs)
Jan 12 15:52:14 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:14.362698 [DEBUG] http: Request /v1/agent/health?type=client (186.597µs)
Jan 12 15:52:14 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:14 [DEBUG] memberlist: TCP connection from=10.50.1.12:33592
Jan 12 15:52:14 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.1.12:33592
Jan 12 15:52:15 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:15.242575 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (195.512µs)
Jan 12 15:52:15 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (195.512µs)
Jan 12 15:52:15 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:15.283239 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (212.225µs)
Jan 12 15:52:15 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (212.225µs)
Jan 12 15:52:15 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:15 [DEBUG] memberlist: TCP connection from=10.50.2.47:35958
Jan 12 15:52:15 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.2.47:35958
Jan 12 15:52:16 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:16.532754 [DEBUG] http: Request /v1/agent/health?type=server (1.184321ms)
Jan 12 15:52:16 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (1.184321ms)
Jan 12 15:52:16 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:16.908213 [DEBUG] http: Request /v1/agent/health?type=client (227.075µs)
Jan 12 15:52:16 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (227.075µs)
Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:17.086415 [DEBUG] http: Request /v1/status/leader (1.422652ms)
Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: http: Request /v1/status/leader (1.422652ms)
Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:17.099573 [DEBUG] http: Request /v1/agent/health?type=client (116.697µs)
Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (116.697µs)
Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (218.146µs)
Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:17.187443 [DEBUG] http: Request /v1/agent/health?type=client (218.146µs)
Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:17.191839 [DEBUG] http: Request /v1/agent/health?type=server (2.148459ms)
Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (2.148459ms)
Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:17.212093 [DEBUG] http: Request /v1/agent/health?type=server (8.229039ms)
Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (8.229039ms)
Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52738
Jan 12 15:52:17 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:17 [DEBUG] memberlist: TCP connection from=10.50.5.8:52738
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:18 [DEBUG] memberlist: TCP connection from=10.50.5.8:52742
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52742
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/vault-ui/allocations?all=false&region=us-east-1&stale=&wait=120000ms (637.426µs)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:18.672077 [DEBUG] http: Request /v1/job/vault-ui/allocations?all=false&region=us-east-1&stale=&wait=120000ms (637.426µs)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:18.674354 [DEBUG] http: Request /v1/job/kibana-production/allocations?all=false&region=us-east-1&stale=&wait=120000ms (410.793µs)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/kibana-production/allocations?all=false&region=us-east-1&stale=&wait=120000ms (410.793µs)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:18.684499 [DEBUG] http: Request /v1/job/mitochondrion-staging/allocations?all=false&region=us-east-1&stale=&wait=120000ms (10.800316ms)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/mitochondrion-staging/allocations?all=false&region=us-east-1&stale=&wait=120000ms (10.800316ms)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:18.690559 [DEBUG] http: Request /v1/job/consul-snapshot/allocations?all=false&region=us-east-1&stale=&wait=120000ms (245.419µs)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/consul-snapshot/allocations?all=false&region=us-east-1&stale=&wait=120000ms (245.419µs)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:18.762328 [DEBUG] http: Request /v1/job/symbiosis-uat/allocations?all=false&region=us-east-1&stale=&wait=120000ms (6.315185ms)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/symbiosis-uat/allocations?all=false&region=us-east-1&stale=&wait=120000ms (6.315185ms)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:18.772827 [DEBUG] http: Request /v1/job/symbiosis-production/allocations?all=false&region=us-east-1&stale=&wait=120000ms (2.709907ms)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/symbiosis-production/allocations?all=false&region=us-east-1&stale=&wait=120000ms (2.709907ms)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:18.776125 [DEBUG] http: Request /v1/job/symbiosis-staging/allocations?all=false&region=us-east-1&stale=&wait=120000ms (19.587042ms)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/symbiosis-staging/allocations?all=false&region=us-east-1&stale=&wait=120000ms (19.587042ms)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:18.776803 [DEBUG] http: Request /v1/job/tricorder-production/allocations?all=false&region=us-east-1&stale=&wait=120000ms (2.357293ms)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/tricorder-production/allocations?all=false&region=us-east-1&stale=&wait=120000ms (2.357293ms)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:18.777889 [DEBUG] http: Request /v1/job/kibana-uat/allocations?all=false&region=us-east-1&stale=&wait=120000ms (413.159µs)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/kibana-uat/allocations?all=false&region=us-east-1&stale=&wait=120000ms (413.159µs)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:18.778473 [DEBUG] http: Request /v1/job/scribe-production/allocations?all=false&region=us-east-1&stale=&wait=120000ms (664.869µs)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/scribe-production/allocations?all=false&region=us-east-1&stale=&wait=120000ms (664.869µs)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:18.781360 [DEBUG] http: Request /v1/job/scribe-uat/allocations?all=false&region=us-east-1&stale=&wait=120000ms (2.509439ms)
Jan 12 15:52:18 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/scribe-uat/allocations?all=false&region=us-east-1&stale=&wait=120000ms (2.509439ms)
Jan 12 15:52:19 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:19.958151 [DEBUG] http: Request /v1/regions?region=us-east-1&wait=60000ms (634.289µs)
Jan 12 15:52:19 ip-10-50-1-29 nomad[1436]: http: Request /v1/regions?region=us-east-1&wait=60000ms (634.289µs)
Jan 12 15:52:19 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:19.964389 [DEBUG] http: Request /v1/status/leader?region=us-east-1&wait=60000ms (1.085318ms)
Jan 12 15:52:19 ip-10-50-1-29 nomad[1436]: http: Request /v1/status/leader?region=us-east-1&wait=60000ms (1.085318ms)
Jan 12 15:52:20 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:20.529149 [DEBUG] http: Request /v1/agent/health?type=client (851.152µs)
Jan 12 15:52:20 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (851.152µs)
Jan 12 15:52:20 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:20.725620 [DEBUG] http: Request /v1/allocations?index=696209&region=us-east-1&stale=&wait=120000ms (2.578614755s)
Jan 12 15:52:20 ip-10-50-1-29 nomad[1436]: http: Request /v1/allocations?index=696209&region=us-east-1&stale=&wait=120000ms (2.578614755s)
Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:21.247595 [DEBUG] http: Request /v1/agent/health?type=server (1.077007ms)
Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (1.077007ms)
Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:21.820220 [DEBUG] http: Request /v1/agent/health?type=server (567.847µs)
Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (567.847µs)
Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:21 [DEBUG] memberlist: TCP connection from=10.50.4.247:35628
Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35628
Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:21 [DEBUG] memberlist: TCP connection from=10.50.4.247:35630
Jan 12 15:52:21 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35630
Jan 12 15:52:22 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:22.599732 [DEBUG] http: Request /v1/agent/health?type=client (237.127µs)
Jan 12 15:52:22 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (237.127µs)
Jan 12 15:52:23 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:23.612269 [DEBUG] http: Request /v1/agent/health?type=server (586.58µs)
Jan 12 15:52:23 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (586.58µs)
Jan 12 15:52:23 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:23.626731 [DEBUG] http: Request /v1/job/sftp-s3/versions?diffs=false&index=695801&region=us-east-1&stale=&wait=120000ms (2m6.976314067s)
Jan 12 15:52:23 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/sftp-s3/versions?diffs=false&index=695801&region=us-east-1&stale=&wait=120000ms (2m6.976314067s)
Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:24.171698 [DEBUG] http: Request /v1/system/gc (509.72µs)
Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]: http: Request /v1/system/gc (509.72µs)
Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:24.365619 [DEBUG] http: Request /v1/agent/health?type=client (238.222µs)
Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (238.222µs)
Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:24 [DEBUG] memberlist: TCP connection from=10.50.1.12:33660
Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.1.12:33660
Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:24.955752 [DEBUG] http: Request /v1/regions?region=us-east-1&wait=60000ms (182.903µs)
Jan 12 15:52:24 ip-10-50-1-29 nomad[1436]: http: Request /v1/regions?region=us-east-1&wait=60000ms (182.903µs)
Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:25.253542 [DEBUG] http: Request /v1/allocations?index=696212&region=us-east-1&stale=&wait=120000ms (2.876001562s)
Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: http: Request /v1/allocations?index=696212&region=us-east-1&stale=&wait=120000ms (2.876001562s)
Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:25.537420 [DEBUG] http: Request /v1/nodes?index=696031&region=us-east-1&stale=&wait=10000ms (10.247298219s)
Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: http: Request /v1/nodes?index=696031&region=us-east-1&stale=&wait=10000ms (10.247298219s)
Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:25.570706 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (183.179µs)
Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (183.179µs)
Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:25.611524 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (122.086µs)
Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (122.086µs)
Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:25 [DEBUG] memberlist: TCP connection from=10.50.2.47:36032
Jan 12 15:52:25 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.2.47:36032
Jan 12 15:52:26 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:26.535868 [DEBUG] http: Request /v1/agent/health?type=server (622.076µs)
Jan 12 15:52:26 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (622.076µs)
Jan 12 15:52:26 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:26.911992 [DEBUG] http: Request /v1/agent/health?type=client (1.223915ms)
Jan 12 15:52:26 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (1.223915ms)
Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:27.101542 [DEBUG] http: Request /v1/agent/health?type=client (291.829µs)
Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (291.829µs)
Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (337.031µs)
Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:27.189988 [DEBUG] http: Request /v1/agent/health?type=client (337.031µs)
Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:27.194208 [DEBUG] http: Request /v1/agent/health?type=server (628.072µs)
Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (628.072µs)
Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:27.214834 [DEBUG] http: Request /v1/agent/health?type=server (475.46µs)
Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (475.46µs)
Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52814
Jan 12 15:52:27 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:27 [DEBUG] memberlist: TCP connection from=10.50.5.8:52814
Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.1.95:36852
Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:28 [DEBUG] memberlist: TCP connection from=10.50.1.95:36852
Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/tricorder-production/versions?diffs=false&region=us-east-1&stale=&wait=120000ms (883.538µs)
Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:28.337701 [DEBUG] http: Request /v1/job/tricorder-production/versions?diffs=false&region=us-east-1&stale=&wait=120000ms (883.538µs)
Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:28.339358 [DEBUG] http: Request /v1/job/tricorder-production?region=us-east-1&stale=&wait=120000ms (316.702µs)
Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/tricorder-production?region=us-east-1&stale=&wait=120000ms (316.702µs)
Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52816
Jan 12 15:52:28 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:28 [DEBUG] memberlist: TCP connection from=10.50.5.8:52816
Jan 12 15:52:29 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:29.992566 [DEBUG] http: Request /v1/regions?region=us-east-1&wait=60000ms (626.969µs)
Jan 12 15:52:29 ip-10-50-1-29 nomad[1436]: http: Request /v1/regions?region=us-east-1&wait=60000ms (626.969µs)
Jan 12 15:52:30 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:30.531714 [DEBUG] http: Request /v1/agent/health?type=client (701.375µs)
Jan 12 15:52:30 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (701.375µs)
Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:31.250444 [DEBUG] http: Request /v1/agent/health?type=server (952.828µs)
Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (952.828µs)
Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:31.827289 [DEBUG] http: Request /v1/agent/health?type=server (4.21674ms)
Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (4.21674ms)
Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:31 [DEBUG] memberlist: TCP connection from=10.50.4.247:35700
Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35700
Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:31.884365 [DEBUG] http: Request /v1/allocations?index=696214&region=us-east-1&stale=&wait=120000ms (924.854623ms)
Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: http: Request /v1/allocations?index=696214&region=us-east-1&stale=&wait=120000ms (924.854623ms)
Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35702
Jan 12 15:52:31 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:31 [DEBUG] memberlist: TCP connection from=10.50.4.247:35702
Jan 12 15:52:32 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:32.088790 [DEBUG] http: Request /v1/status/leader (516.314µs)
Jan 12 15:52:32 ip-10-50-1-29 nomad[1436]: http: Request /v1/status/leader (516.314µs)
Jan 12 15:52:32 ip-10-50-1-29 nomad[1436]: memberlist: Initiating push/pull sync with: 10.50.1.95:4648
Jan 12 15:52:32 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:32 [DEBUG] memberlist: Initiating push/pull sync with: 10.50.1.95:4648
Jan 12 15:52:32 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (259.764µs)
Jan 12 15:52:32 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:32.602329 [DEBUG] http: Request /v1/agent/health?type=client (259.764µs)
Jan 12 15:52:33 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:33.616261 [DEBUG] http: Request /v1/agent/health?type=server (1.036229ms)
Jan 12 15:52:33 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (1.036229ms)
Jan 12 15:52:34 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:34.368295 [DEBUG] http: Request /v1/agent/health?type=client (188.306µs)
Jan 12 15:52:34 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (188.306µs)
Jan 12 15:52:34 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:34 [DEBUG] memberlist: TCP connection from=10.50.1.12:33714
Jan 12 15:52:34 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.1.12:33714
Jan 12 15:52:35 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:35 [DEBUG] memberlist: TCP connection from=10.50.2.47:36096
Jan 12 15:52:35 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.2.47:36096
Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:36.025499 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (322.979µs)
Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (322.979µs)
Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:36.050205 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (200.549µs)
Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (200.549µs)
Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:36.538979 [DEBUG] http: Request /v1/agent/health?type=server (563.906µs)
Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (563.906µs)
Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:36.914877 [DEBUG] http: Request /v1/agent/health?type=client (214.858µs)
Jan 12 15:52:36 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (214.858µs)
Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:37.103962 [DEBUG] http: Request /v1/agent/health?type=client (198.812µs)
Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (198.812µs)
Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (323.954µs)
Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:37.192553 [DEBUG] http: Request /v1/agent/health?type=client (323.954µs)
Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:37.198934 [DEBUG] http: Request /v1/agent/health?type=server (2.856772ms)
Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (2.856772ms)
Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:37.234092 [DEBUG] http: Request /v1/agent/health?type=server (16.242204ms)
Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (16.242204ms)
Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:37.414136 [DEBUG] http: Request /v1/job/tricorder-uat/allocations?all=false&index=695870&region=us-east-1&stale=&wait=120000ms (2m5.217383171s)
Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/tricorder-uat/allocations?all=false&index=695870&region=us-east-1&stale=&wait=120000ms (2m5.217383171s)
Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:37 [DEBUG] memberlist: TCP connection from=10.50.5.8:52886
Jan 12 15:52:37 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52886
Jan 12 15:52:38 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:38.045488 [DEBUG] http: Request /v1/job/docbot-production/allocations?all=false&index=695870&region=us-east-1&stale=&wait=120000ms (2m6.455937814s)
Jan 12 15:52:38 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/docbot-production/allocations?all=false&index=695870&region=us-east-1&stale=&wait=120000ms (2m6.455937814s)
Jan 12 15:52:38 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:38 [DEBUG] memberlist: TCP connection from=10.50.5.8:52888
Jan 12 15:52:38 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.5.8:52888
Jan 12 15:52:38 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:38.469131 [DEBUG] http: Request /v1/job/sftp-s3-test?index=695811&region=us-east-1&stale=&wait=120000ms (2m7.087101371s)
Jan 12 15:52:38 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/sftp-s3-test?index=695811&region=us-east-1&stale=&wait=120000ms (2m7.087101371s)
Jan 12 15:52:39 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:39.959955 [DEBUG] http: Request /v1/regions?region=us-east-1&wait=60000ms (620.782µs)
Jan 12 15:52:39 ip-10-50-1-29 nomad[1436]: http: Request /v1/regions?region=us-east-1&wait=60000ms (620.782µs)
Jan 12 15:52:39 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:39.972180 [DEBUG] http: Request /v1/status/leader?region=us-east-1&wait=60000ms (955.891µs)
Jan 12 15:52:39 ip-10-50-1-29 nomad[1436]: http: Request /v1/status/leader?region=us-east-1&wait=60000ms (955.891µs)
Jan 12 15:52:40 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:40.173322 [DEBUG] http: Request /v1/job/scribe-staging/allocations?all=false&index=695870&region=us-east-1&stale=&wait=120000ms (2m3.673234376s)
Jan 12 15:52:40 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/scribe-staging/allocations?all=false&index=695870&region=us-east-1&stale=&wait=120000ms (2m3.673234376s)
Jan 12 15:52:40 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:40.367549 [DEBUG] http: Request /v1/job/scribe-production/allocations?all=false&index=695870&region=us-east-1&stale=&wait=120000ms (2m5.160696839s)
Jan 12 15:52:40 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/scribe-production/allocations?all=false&index=695870&region=us-east-1&stale=&wait=120000ms (2m5.160696839s)
Jan 12 15:52:40 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (211.394µs)
Jan 12 15:52:40 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:40.533214 [DEBUG] http: Request /v1/agent/health?type=client (211.394µs)
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:41.257815 [DEBUG] http: Request /v1/agent/health?type=server (5.086311ms)
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (5.086311ms)
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:41.542672 [DEBUG] http: Request /v1/job/docbot-production?index=694504&region=us-east-1&stale=&wait=120000ms (2m6.159973379s)
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/docbot-production?index=694504&region=us-east-1&stale=&wait=120000ms (2m6.159973379s)
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:41.579486 [DEBUG] http: Request /v1/job/symbiosis-staging/allocations?all=false&index=695873&region=us-east-1&stale=&wait=120000ms (2m0.429229659s)
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/symbiosis-staging/allocations?all=false&index=695873&region=us-east-1&stale=&wait=120000ms (2m0.429229659s)
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:41.592937 [DEBUG] http: Request /v1/job/docbot-production/allocations?all=false&index=696128&region=us-east-1&stale=&wait=120000ms (12.495412ms)
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/docbot-production/allocations?all=false&index=696128&region=us-east-1&stale=&wait=120000ms (12.495412ms)
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:41.758829 [DEBUG] http: Request /v1/allocations?index=696222&region=us-east-1&stale=&wait=120000ms (562.7799ms)
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: http: Request /v1/allocations?index=696222&region=us-east-1&stale=&wait=120000ms (562.7799ms)
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:41.830815 [DEBUG] http: Request /v1/agent/health?type=server (1.076224ms)
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (1.076224ms)
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:41 [DEBUG] memberlist: TCP connection from=10.50.4.247:35788
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35788
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:41 [DEBUG] memberlist: TCP connection from=10.50.4.247:35790
Jan 12 15:52:41 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.4.247:35790
Jan 12 15:52:42 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:42.605002 [DEBUG] http: Request /v1/agent/health?type=client (301.485µs)
Jan 12 15:52:42 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=client (301.485µs)
Jan 12 15:52:43 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:43.445880 [DEBUG] http: Request /v1/job/kibana-staging/allocations?all=false&index=695870&region=us-east-1&stale=&wait=120000ms (2m6.532032772s)
Jan 12 15:52:43 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/kibana-staging/allocations?all=false&index=695870&region=us-east-1&stale=&wait=120000ms (2m6.532032772s)
Jan 12 15:52:43 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:43.593212 [DEBUG] http: Request /v1/job/hashi-ui/allocations?all=false&index=695870&region=us-east-1&stale=&wait=120000ms (2m7.471101758s)
Jan 12 15:52:43 ip-10-50-1-29 nomad[1436]: http: Request /v1/job/hashi-ui/allocations?all=false&index=695870&region=us-east-1&stale=&wait=120000ms (2m7.471101758s)
Jan 12 15:52:43 ip-10-50-1-29 nomad[1436]:     2018/01/12 15:52:43.619118 [DEBUG] http: Request /v1/agent/health?type=server (542.983µs)
Jan 12 15:52:43 ip-10-50-1-29 nomad[1436]: http: Request /v1/agent/health?type=server (542.983µs)
Jan 12 15:52:44 ip-10-50-1-29 nomad[1436]: memberlist: TCP connection from=10.50.1.12:33792

Server2:

3ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.538494 [DEBUG] http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (1.373583ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.539073 [DEBUG] http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (2.160888ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.539625 [DEBUG] http: Request /v1/node/ec20657b-ac9a-c80f-9a87-4098c5d9d665?region=us-east-1&wait=60000ms (1.600722ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (2.160888ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.540221 [DEBUG] http: Request /v1/node/5daf562c-9ccb-f55e-3fda-37fd0bfdd74c?region=us-east-1&wait=60000ms (2.318732ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.541177 [DEBUG] http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (595.869µs)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec20657b-ac9a-c80f-9a87-4098c5d9d665?region=us-east-1&wait=60000ms (1.600722ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.541717 [DEBUG] http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (879.934µs)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/5daf562c-9ccb-f55e-3fda-37fd0bfdd74c?region=us-east-1&wait=60000ms (2.318732ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.542231 [DEBUG] http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (1.517322ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (595.869µs)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.544732 [DEBUG] http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (706.517µs)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.545276 [DEBUG] http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (2.729114ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.545784 [DEBUG] http: Request /v1/node/cfc3db5b-563e-9c80-4e1b-557ecfb1a977?region=us-east-1&wait=60000ms (2.781095ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.546292 [DEBUG] http: Request /v1/node/2bb7c609-f7f5-de03-9609-51ed67e76470?region=us-east-1&wait=60000ms (2.636082ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (879.934µs)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.546814 [DEBUG] http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (3.0296ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.547341 [DEBUG] http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (3.429119ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (1.517322ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.547879 [DEBUG] http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (3.609681ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (706.517µs)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (2.729114ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.548396 [DEBUG] http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (4.242371ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/cfc3db5b-563e-9c80-4e1b-557ecfb1a977?region=us-east-1&wait=60000ms (2.781095ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/2bb7c609-f7f5-de03-9609-51ed67e76470?region=us-east-1&wait=60000ms (2.636082ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (3.0296ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (3.429119ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (3.609681ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (4.242371ms)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.557410 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (121.993µs)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (121.993µs)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.595895 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (152.568µs)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (152.568µs)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25.676433 [DEBUG] http: Request /v1/agent/health?type=server (562.73µs)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (562.73µs)
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25 [DEBUG] memberlist: TCP connection from=10.50.5.8:59084
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59084
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.2.47:36366
Jan 12 15:52:25 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:25 [DEBUG] memberlist: TCP connection from=10.50.2.47:36366
Jan 12 15:52:26 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:26.973049 [DEBUG] http: Request /v1/agent/health?type=server (567.303µs)
Jan 12 15:52:26 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (567.303µs)
Jan 12 15:52:29 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (209.751µs)
Jan 12 15:52:29 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:29.574723 [DEBUG] http: Request /v1/agent/health?type=client (209.751µs)
Jan 12 15:52:29 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:29.685405 [DEBUG] http: Request /v1/agent/health?type=server (590.67µs)
Jan 12 15:52:29 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (590.67µs)
Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:30.217809 [DEBUG] http: Request /v1/agent/health?type=client (646.895µs)
Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (646.895µs)
Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:30.242089 [DEBUG] http: Request /v1/agent/health?type=client (487.638µs)
Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (487.638µs)
Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:30 [DEBUG] memberlist: TCP connection from=10.50.4.247:39998
Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:39998
Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:30 [DEBUG] memberlist: TCP connection from=10.50.4.247:40000
Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40000
Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (272.493µs)
Jan 12 15:52:30 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:30.767758 [DEBUG] http: Request /v1/agent/health?type=client (272.493µs)
Jan 12 15:52:32 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (234.9µs)
Jan 12 15:52:32 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:32.119432 [DEBUG] http: Request /v1/agent/health?type=client (234.9µs)
Jan 12 15:52:32 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (497.767µs)
Jan 12 15:52:32 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:32.241450 [DEBUG] http: Request /v1/agent/health?type=server (497.767µs)
Jan 12 15:52:33 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:33.146783 [DEBUG] http: Request /v1/allocations?index=696215&region=us-east-1&stale=&wait=120000ms (705.424314ms)
Jan 12 15:52:33 ip-10-50-1-101 nomad[25850]: http: Request /v1/allocations?index=696215&region=us-east-1&stale=&wait=120000ms (705.424314ms)
Jan 12 15:52:33 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:33.487380 [DEBUG] http: Request /v1/agent/health?type=server (1.343953ms)
Jan 12 15:52:33 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (1.343953ms)
Jan 12 15:52:33 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:33.853861 [DEBUG] http: Request /v1/agent/health?type=server (912.993µs)
Jan 12 15:52:33 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (912.993µs)
Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:34.231842 [DEBUG] http: Request /v1/allocations?index=696216&region=us-east-1&stale=&wait=120000ms (495.16528ms)
Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]: http: Request /v1/allocations?index=696216&region=us-east-1&stale=&wait=120000ms (495.16528ms)
Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (323.833µs)
Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:34.543238 [DEBUG] http: Request /v1/agent/health?type=client (323.833µs)
Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:34.937950 [DEBUG] http: Request /v1/agent/members?region=us-east-1&wait=60000ms (423.906µs)
Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/members?region=us-east-1&wait=60000ms (423.906µs)
Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:34.953501 [DEBUG] http: Request /v1/status/leader?region=us-east-1&wait=60000ms (437.547µs)
Jan 12 15:52:34 ip-10-50-1-101 nomad[25850]: http: Request /v1/status/leader?region=us-east-1&wait=60000ms (437.547µs)
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:35 [DEBUG] memberlist: TCP connection from=10.50.5.8:59152
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59152
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:35.166523 [DEBUG] http: Request /v1/allocations?index=696217&region=us-east-1&stale=&wait=120000ms (349.708925ms)
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: http: Request /v1/allocations?index=696217&region=us-east-1&stale=&wait=120000ms (349.708925ms)
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:35.170989 [DEBUG] http: Request /v1/status/leader (984.958µs)
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: http: Request /v1/status/leader (984.958µs)
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.1.12:39768
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:35 [DEBUG] memberlist: TCP connection from=10.50.1.12:39768
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:35.680598 [DEBUG] http: Request /v1/agent/health?type=server (921.487µs)
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (921.487µs)
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:35 [DEBUG] memberlist: TCP connection from=10.50.5.8:59154
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59154
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:35 [DEBUG] memberlist: TCP connection from=10.50.2.47:36430
Jan 12 15:52:35 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.2.47:36430
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.000353 [DEBUG] http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (1.553256ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.001122 [DEBUG] http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (1.248687ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.001662 [DEBUG] http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (904.883µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (1.553256ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (1.248687ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.002575 [DEBUG] http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (600.338µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (904.883µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (600.338µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/5daf562c-9ccb-f55e-3fda-37fd0bfdd74c?region=us-east-1&wait=60000ms (911.034µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.003140 [DEBUG] http: Request /v1/node/5daf562c-9ccb-f55e-3fda-37fd0bfdd74c?region=us-east-1&wait=60000ms (911.034µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.003914 [DEBUG] http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (1.808776ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (1.808776ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.007128 [DEBUG] http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (1.015616ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (1.015616ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.008281 [DEBUG] http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (3.830074ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (3.830074ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (4.207059ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.009001 [DEBUG] http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (4.207059ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.009643 [DEBUG] http: Request /v1/node/ec20657b-ac9a-c80f-9a87-4098c5d9d665?region=us-east-1&wait=60000ms (4.669875ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.010135 [DEBUG] http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (2.251914ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.010242 [DEBUG] http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (4.565396ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.010799 [DEBUG] http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (4.95177ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.010819 [DEBUG] http: Request /v1/node/2bb7c609-f7f5-de03-9609-51ed67e76470?region=us-east-1&wait=60000ms (3.275272ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.011289 [DEBUG] http: Request /v1/node/cfc3db5b-563e-9c80-4e1b-557ecfb1a977?region=us-east-1&wait=60000ms (1.768325ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec20657b-ac9a-c80f-9a87-4098c5d9d665?region=us-east-1&wait=60000ms (4.669875ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (2.251914ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (4.565396ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (4.95177ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/2bb7c609-f7f5-de03-9609-51ed67e76470?region=us-east-1&wait=60000ms (3.275272ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/cfc3db5b-563e-9c80-4e1b-557ecfb1a977?region=us-east-1&wait=60000ms (1.768325ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.013597 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (174.675µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (174.675µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (604.553µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.022745 [DEBUG] http: Request /v1/node/ec28c2f0-22c7-c267-f19e-530856c11d80?region=us-east-1&wait=60000ms (604.553µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.023300 [DEBUG] http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (1.019073ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.023813 [DEBUG] http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (1.40664ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.025023 [DEBUG] http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (701.607µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.026284 [DEBUG] http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (716.549µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.027415 [DEBUG] http: Request /v1/node/5daf562c-9ccb-f55e-3fda-37fd0bfdd74c?region=us-east-1&wait=60000ms (2.02512ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.029125 [DEBUG] http: Request /v1/node/cfc3db5b-563e-9c80-4e1b-557ecfb1a977?region=us-east-1&wait=60000ms (1.536367ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.030307 [DEBUG] http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (4.556565ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.030771 [DEBUG] http: Request /v1/node/2bb7c609-f7f5-de03-9609-51ed67e76470?region=us-east-1&wait=60000ms (798.779µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.030929 [DEBUG] http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (2.05353ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.031354 [DEBUG] http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (2.143559ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.031486 [DEBUG] http: Request /v1/node/ec20657b-ac9a-c80f-9a87-4098c5d9d665?region=us-east-1&wait=60000ms (3.568656ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.031982 [DEBUG] http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (2.456053ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.032065 [DEBUG] http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (3.688143ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/6edd0055-769c-bb58-65b9-7c14fb303e8f?region=us-east-1&wait=60000ms (1.019073ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.032696 [DEBUG] http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (3.039851ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/f6bbc417-6087-b19f-5dbf-2d4a1d55a317?region=us-east-1&wait=60000ms (1.40664ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec2bbab7-5122-adf6-bbab-c73ca578ee2b?region=us-east-1&wait=60000ms (701.607µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/7cd5a435-ae09-ab0a-1f90-c4228be55cce?region=us-east-1&wait=60000ms (716.549µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/5daf562c-9ccb-f55e-3fda-37fd0bfdd74c?region=us-east-1&wait=60000ms (2.02512ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/cfc3db5b-563e-9c80-4e1b-557ecfb1a977?region=us-east-1&wait=60000ms (1.536367ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/7cdec421-166e-04da-07e1-2b37858357a3?region=us-east-1&wait=60000ms (4.556565ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/2bb7c609-f7f5-de03-9609-51ed67e76470?region=us-east-1&wait=60000ms (798.779µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/14c862b7-bbf0-d897-e7ca-94932decf275?region=us-east-1&wait=60000ms (2.05353ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec2f2f35-2602-3988-708e-ebab55216529?region=us-east-1&wait=60000ms (2.143559ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec20657b-ac9a-c80f-9a87-4098c5d9d665?region=us-east-1&wait=60000ms (3.568656ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec21d55b-22e7-e1ec-8224-87809b718469?region=us-east-1&wait=60000ms (2.456053ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/169d6030-120a-3b64-fef4-0d9e6aa095f8?region=us-east-1&wait=60000ms (3.688143ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/node/ec22785b-8b37-74cb-8437-5e49f32ef686?region=us-east-1&wait=60000ms (3.039851ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.037591 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (228.116µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (228.116µs)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:36.976002 [DEBUG] http: Request /v1/agent/health?type=server (1.133916ms)
Jan 12 15:52:36 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (1.133916ms)
Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:39.313165 [DEBUG] http: Request /v1/job/helix2-emitter-production/versions?diffs=false&region=us-east-1&stale=&wait=120000ms (4.643344ms)
Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/helix2-emitter-production/versions?diffs=false&region=us-east-1&stale=&wait=120000ms (4.643344ms)
Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:39.327666 [DEBUG] http: Request /v1/job/helix2-emitter-production?region=us-east-1&stale=&wait=120000ms (321.544µs)
Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/helix2-emitter-production?region=us-east-1&stale=&wait=120000ms (321.544µs)
Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (234.403µs)
Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:39.580685 [DEBUG] http: Request /v1/agent/health?type=client (234.403µs)
Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:39.675826 [DEBUG] http: Request /v1/allocations?index=696220&region=us-east-1&stale=&wait=120000ms (661.649061ms)
Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: http: Request /v1/allocations?index=696220&region=us-east-1&stale=&wait=120000ms (661.649061ms)
Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:39.688672 [DEBUG] http: Request /v1/agent/health?type=server (459.85µs)
Jan 12 15:52:39 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (459.85µs)
Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:40.219282 [DEBUG] http: Request /v1/agent/health?type=client (220.826µs)
Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (220.826µs)
Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:40.244322 [DEBUG] http: Request /v1/agent/health?type=client (208.558µs)
Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (208.558µs)
Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:40 [DEBUG] memberlist: TCP connection from=10.50.4.247:40080
Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40080
Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:40 [DEBUG] memberlist: TCP connection from=10.50.4.247:40082
Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40082
Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:40.610226 [DEBUG] http: Request /v1/allocations?index=696221&region=us-east-1&stale=&wait=120000ms (352.745161ms)
Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: http: Request /v1/allocations?index=696221&region=us-east-1&stale=&wait=120000ms (352.745161ms)
Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:40.770709 [DEBUG] http: Request /v1/agent/health?type=client (195.904µs)
Jan 12 15:52:40 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (195.904µs)
Jan 12 15:52:42 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:42.122468 [DEBUG] http: Request /v1/agent/health?type=client (592.037µs)
Jan 12 15:52:42 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (592.037µs)
Jan 12 15:52:42 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:42.244712 [DEBUG] http: Request /v1/agent/health?type=server (524.794µs)
Jan 12 15:52:42 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (524.794µs)
Jan 12 15:52:43 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:43.491105 [DEBUG] http: Request /v1/agent/health?type=server (909.135µs)
Jan 12 15:52:43 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (909.135µs)
Jan 12 15:52:43 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:43.858387 [DEBUG] http: Request /v1/agent/health?type=server (949.047µs)
Jan 12 15:52:43 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (949.047µs)
Jan 12 15:52:44 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:44.545823 [DEBUG] http: Request /v1/agent/health?type=client (189.834µs)
Jan 12 15:52:44 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (189.834µs)
Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:45 [DEBUG] memberlist: TCP connection from=10.50.5.8:59244
Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59244
Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:45 [DEBUG] memberlist: TCP connection from=10.50.1.12:39848
Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.1.12:39848
Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:45.684107 [DEBUG] http: Request /v1/agent/health?type=server (773.292µs)
Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (773.292µs)
Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59246
Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:45 [DEBUG] memberlist: TCP connection from=10.50.5.8:59246
Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:45.774979 [DEBUG] http: Request /v1/job/wfoc-staging/allocations?all=false&index=696129&region=us-east-1&stale=&wait=120000ms (20.477897ms)
Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/wfoc-staging/allocations?all=false&index=696129&region=us-east-1&stale=&wait=120000ms (20.477897ms)
Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:45 [DEBUG] memberlist: TCP connection from=10.50.2.47:36522
Jan 12 15:52:45 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.2.47:36522
Jan 12 15:52:46 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:46.286834 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (224.692µs)
Jan 12 15:52:46 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (224.692µs)
Jan 12 15:52:46 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:46.329205 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (200.015µs)
Jan 12 15:52:46 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (200.015µs)
Jan 12 15:52:46 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:46.978314 [DEBUG] http: Request /v1/agent/health?type=server (509.54µs)
Jan 12 15:52:46 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (509.54µs)
Jan 12 15:52:47 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:47.515690 [DEBUG] http: Request /v1/job/mitochondrion-staging?index=663818&region=us-east-1&stale=&wait=120000ms (2m1.194231162s)
Jan 12 15:52:47 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/mitochondrion-staging?index=663818&region=us-east-1&stale=&wait=120000ms (2m1.194231162s)
Jan 12 15:52:47 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:47.535438 [DEBUG] http: Request /v1/job/mitochondrion-staging/allocations?all=false&index=696134&region=us-east-1&stale=&wait=120000ms (10.323071ms)
Jan 12 15:52:47 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/mitochondrion-staging/allocations?all=false&index=696134&region=us-east-1&stale=&wait=120000ms (10.323071ms)
Jan 12 15:52:49 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:49.584270 [DEBUG] http: Request /v1/agent/health?type=client (667.487µs)
Jan 12 15:52:49 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (667.487µs)
Jan 12 15:52:49 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (559.558µs)
Jan 12 15:52:49 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:49.692211 [DEBUG] http: Request /v1/agent/health?type=server (559.558µs)
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:50.174508 [DEBUG] http: Request /v1/status/leader (535.344µs)
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: http: Request /v1/status/leader (535.344µs)
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:50.223139 [DEBUG] http: Request /v1/agent/health?type=client (192.179µs)
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (192.179µs)
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:50.247697 [DEBUG] http: Request /v1/agent/health?type=client (177.957µs)
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (177.957µs)
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:50.337689 [DEBUG] http: Request /v1/evaluations?region=us-east-1&stale=&wait=120000ms (192.138891ms)
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: http: Request /v1/evaluations?region=us-east-1&stale=&wait=120000ms (192.138891ms)
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40176
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:50 [DEBUG] memberlist: TCP connection from=10.50.4.247:40176
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:50 [DEBUG] memberlist: TCP connection from=10.50.4.247:40178
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40178
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:50.773755 [DEBUG] http: Request /v1/agent/health?type=client (217.207µs)
Jan 12 15:52:50 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (217.207µs)
Jan 12 15:52:52 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:52.125688 [DEBUG] http: Request /v1/agent/health?type=client (607.93µs)
Jan 12 15:52:52 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (607.93µs)
Jan 12 15:52:52 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (550.386µs)
Jan 12 15:52:52 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:52.246526 [DEBUG] http: Request /v1/agent/health?type=server (550.386µs)
Jan 12 15:52:53 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (454.884µs)
Jan 12 15:52:53 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:53.493752 [DEBUG] http: Request /v1/agent/health?type=server (454.884µs)
Jan 12 15:52:53 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:53.861673 [DEBUG] http: Request /v1/agent/health?type=server (572.724µs)
Jan 12 15:52:53 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (572.724µs)
Jan 12 15:52:54 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:54.548297 [DEBUG] http: Request /v1/agent/health?type=client (171.695µs)
Jan 12 15:52:54 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (171.695µs)
Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:55 [DEBUG] memberlist: TCP connection from=10.50.5.8:59324
Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59324
Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:55 [DEBUG] memberlist: TCP connection from=10.50.1.12:39906
Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.1.12:39906
Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:55.690939 [DEBUG] http: Request /v1/agent/health?type=server (3.959848ms)
Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (3.959848ms)
Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:55 [DEBUG] memberlist: TCP connection from=10.50.5.8:59326
Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.5.8:59326
Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:55 [DEBUG] memberlist: TCP connection from=10.50.2.47:36592
Jan 12 15:52:55 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.2.47:36592
Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:56.330196 [DEBUG] http: Request /v1/job/tricorder-production/allocations?all=false&index=695876&region=us-east-1&stale=&wait=120000ms (2.238855ms)
Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/tricorder-production/allocations?all=false&index=695876&region=us-east-1&stale=&wait=120000ms (2.238855ms)
Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:56.878232 [DEBUG] http: Request /v1/nodes?index=696031&region=us-east-1&stale=&wait=10000ms (10.497649584s)
Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: http: Request /v1/nodes?index=696031&region=us-east-1&stale=&wait=10000ms (10.497649584s)
Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:56.904322 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (272.713µs)
Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (272.713µs)
Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:56.925012 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (186.196µs)
Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: http: Request /v1/client/stats?region=us-east-1&wait=60000ms (186.196µs)
Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:56.981661 [DEBUG] http: Request /v1/agent/health?type=server (1.194177ms)
Jan 12 15:52:56 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (1.194177ms)
Jan 12 15:52:57 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:57 [DEBUG] memberlist: Initiating push/pull sync with: 10.50.1.95:4648
Jan 12 15:52:57 ip-10-50-1-101 nomad[25850]: memberlist: Initiating push/pull sync with: 10.50.1.95:4648
Jan 12 15:52:59 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:59.587301 [DEBUG] http: Request /v1/agent/health?type=client (226.539µs)
Jan 12 15:52:59 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (226.539µs)
Jan 12 15:52:59 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:59.696220 [DEBUG] http: Request /v1/agent/health?type=server (752.867µs)
Jan 12 15:52:59 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=server (752.867µs)
Jan 12 15:52:59 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/members?region=us-east-1&wait=60000ms (480.612µs)
Jan 12 15:52:59 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:52:59.938879 [DEBUG] http: Request /v1/agent/members?region=us-east-1&wait=60000ms (480.612µs)
Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (382.354µs)
Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:53:00.225118 [DEBUG] http: Request /v1/agent/health?type=client (382.354µs)
Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:53:00.249966 [DEBUG] http: Request /v1/agent/health?type=client (174.588µs)
Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (174.588µs)
Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:53:00 [DEBUG] memberlist: TCP connection from=10.50.4.247:40258
Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:53:00 [DEBUG] memberlist: TCP connection from=10.50.4.247:40256
Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40258
Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: memberlist: TCP connection from=10.50.4.247:40256
Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:53:00.777635 [DEBUG] http: Request /v1/agent/health?type=client (590.016µs)
Jan 12 15:53:00 ip-10-50-1-101 nomad[25850]: http: Request /v1/agent/health?type=client (590.016µs)
Jan 12 15:53:01 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:53:01.340877 [DEBUG] http: Request /v1/job/ingestion-uat?index=693233&region=us-east-1&stale=&wait=120000ms (2m4.841526643s)
Jan 12 15:53:01 ip-10-50-1-101 nomad[25850]: http: Request /v1/job/ingestion-uat?index=693233&region=us-east-1&stale=&wait=120000ms (2m4.841526643s)
Jan 12 15:53:01 ip-10-50-1-101 nomad[25850]:     2018/01/12 15:53:01.542596 [DEBUG] http: Request /v1/client/stats?region=us-east-1&wait=60000ms (190.684µs)

Server3:

Jan 12 15:50:06 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:09 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:09.906778 [INFO] client.gc: garbage collecting allocation fb7ccb1a-3ca1-79dd-b4a0-68c458ee656e due to new allocations and over max (50)
Jan 12 15:50:09 ip-10-50-1-95 nomad[21952]: client.gc: garbage collecting allocation fb7ccb1a-3ca1-79dd-b4a0-68c458ee656e due to new allocations and over max (50)
Jan 12 15:50:09 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:09.952000 [INFO] client.gc: garbage collecting allocation 4384639f-51b9-7204-e2d7-81c2b504bd97 due to new allocations and over max (50)
Jan 12 15:50:09 ip-10-50-1-95 nomad[21952]: client.gc: garbage collecting allocation 4384639f-51b9-7204-e2d7-81c2b504bd97 due to new allocations and over max (50)
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:10 [INFO] (runner) creating new runner (dry: false, once: false)
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: (runner) creating new runner (dry: false, once: false)
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:10 [INFO] (runner) creating watcher
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: (runner) creating watcher
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:10.447404 [ERR] client: alloc "2c650187-be72-ec50-1197-5fef0a26fe44", task "mmx" failed to build task's template manager: failed to read template: open /var/nomad/alloc/2c650187-be72-ec50-1197-5fef0a26fe44/mmx/local/config.ctmpl: no such file or directory
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: client: alloc "2c650187-be72-ec50-1197-5fef0a26fe44", task "mmx" failed to build task's template manager: failed to read template: open /var/nomad/alloc/2c650187-be72-ec50-1197-5fef0a26fe44/mmx/local/config.ctmpl: no such file or directory
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:10.447705 [INFO] client.gc: marking allocation 2c650187-be72-ec50-1197-5fef0a26fe44 for GC
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: client.gc: marking allocation 2c650187-be72-ec50-1197-5fef0a26fe44 for GC
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:10.635318 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:10.672615 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:10.718613 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:10 [ERR] raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40094->10.50.1.29:4647: i/o timeout
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40094->10.50.1.29:4647: i/o timeout
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:10.774199 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:10.815519 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:10 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:11 [INFO] (runner) creating new runner (dry: false, once: false)
Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:11 [INFO] (runner) creating watcher
Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]: (runner) creating new runner (dry: false, once: false)
Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]: (runner) creating watcher
Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:11.248048 [ERR] client: alloc "d47252ba-d489-069c-063a-74ca1e16380e", task "mmx" failed to build task's template manager: failed to read template: open /var/nomad/alloc/d47252ba-d489-069c-063a-74ca1e16380e/mmx/local/config.ctmpl: no such file or directory
Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]: client: alloc "d47252ba-d489-069c-063a-74ca1e16380e", task "mmx" failed to build task's template manager: failed to read template: open /var/nomad/alloc/d47252ba-d489-069c-063a-74ca1e16380e/mmx/local/config.ctmpl: no such file or directory
Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:11.248443 [INFO] client.gc: marking allocation d47252ba-d489-069c-063a-74ca1e16380e for GC
Jan 12 15:50:11 ip-10-50-1-95 nomad[21952]: client.gc: marking allocation d47252ba-d489-069c-063a-74ca1e16380e for GC
Jan 12 15:50:19 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:19.781007 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:19 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:19 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:19.835517 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:19 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:20 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:20 [ERR] raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40096->10.50.1.29:4647: i/o timeout
Jan 12 15:50:20 ip-10-50-1-95 nomad[21952]: raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40096->10.50.1.29:4647: i/o timeout
Jan 12 15:50:30 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:30 [ERR] raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40130->10.50.1.29:4647: i/o timeout
Jan 12 15:50:30 ip-10-50-1-95 nomad[21952]: raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40130->10.50.1.29:4647: i/o timeout
Jan 12 15:50:40 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:40 [ERR] raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40164->10.50.1.29:4647: i/o timeout
Jan 12 15:50:40 ip-10-50-1-95 nomad[21952]: raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40164->10.50.1.29:4647: i/o timeout
Jan 12 15:50:40 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:40.982217 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:40 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:41 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:41.017821 [ERR] nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:41 ip-10-50-1-95 nomad[21952]: nomad.client: DeriveVaultToken failed (recoverable false): Can't request Vault token for terminal allocation
Jan 12 15:50:47 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:47.538499 [WARN] nomad: eval <Eval "102cafb7-54c8-79c0-65d2-c375985919f6" JobID: "sftp-s3/periodic-1515616860" Namespace: "default"> reached delivery limit, marking as failed
Jan 12 15:50:47 ip-10-50-1-95 nomad[21952]: nomad: eval <Eval "102cafb7-54c8-79c0-65d2-c375985919f6" JobID: "sftp-s3/periodic-1515616860" Namespace: "default"> reached delivery limit, marking as failed
Jan 12 15:50:50 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:50 [ERR] raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40172->10.50.1.29:4647: i/o timeout
Jan 12 15:50:50 ip-10-50-1-95 nomad[21952]: raft: Failed to AppendEntries to {Voter 10.50.1.29:4647 10.50.1.29:4647}: read tcp 10.50.1.95:40172->10.50.1.29:4647: i/o timeout
Jan 12 15:50:56 ip-10-50-1-95 nomad[21952]:     2018/01/12 15:50:56 [INFO] raft: pipelining replication to peer {Voter 10.50.1.29:4647 10.50.1.29:4647}
Jan 12 15:50:56 ip-10-50-1-95 nomad[21952]: raft: pipelining replication to peer {Voter 10.50.1.29:4647 10.50.1.29:4647}

I only have debug enabled on the first two, sorry. Probably makes it easier to read at least.

@apenney
Copy link
Author

apenney commented Jan 12, 2018

@dadgar direct ping as you closed it, you may be able to open it :)

@apenney
Copy link
Author

apenney commented Jan 12, 2018

welp.txt

Added this as a txt file because it's too big, but this is the allocs on a single master node, all unable to GC. You asked for this on #3604 where two other people are experiencing this.

@shantanugadgil
Copy link
Contributor

shantanugadgil commented Jan 12, 2018 •
edited
Loading

I have a similar (though not identical) issue about nodes not collecting garbage correctly.

I am using Nomad 0.7.1/Consul 1.0.2/CentOS 7.4+/Docker CE.

My issue is that "/var/lib/docker" bloats up over time (I have image cleanup purposely set to 'false').

This fills up the disk and nomad reports the machine "ready" but no allocation go the machine as the disk capacity is exhausted.

I don't have the logs with me right now, but it keeps emitting a message along the following line:
disk usage of 98 more than the threshold of 80 but not triggering a gc as no allocations are terminal.

So I have a node which is basically unusable but not being indicated as such by Nomad.

I know that '/var/lib/docker' bloating up is not really Nomad's problem, but just wanted to put this out there.

Regards,
Shantanu

@schmichael
Copy link
Member

@apenney Thanks for the additional logs! In the future please attach them as text files or link a gist to make browsing this issue easier. We'll almost always want to download them to search/grep/etc.

@shantanugadgil As far as we can tell right now @apenney's issue is related to the servers GCing dead jobs. Your issue is specific to clients. Ensure you have not set docker.cleanup.image=true in your configs and reopen #3560 if you think it's the same issue. There may be something we're missing and closed too soon!

@dadgar
Copy link
Contributor

dadgar commented Jan 16, 2018

@apenney Hey I am sorry but I just can't reproduce this. I will reopen when there are clear reproduction steps. The reproduction steps will have to include the configuration and how to get to this state from a fresh cluster.

I have even tested by creating a periodic job that fails in the same fashion as your attached allocations but as soon as I hit the GC endpoint, the jobs and allocations get removed.

@dadgar
Copy link
Contributor

dadgar commented Jan 16, 2018

@apenney And if you post logs can you please post the Nomad Server logs and not the client's logs. After you do the curl -XPUT .../v1/system/gc you should see logs like:

    2018/01/16 23:10:11.956340 [DEBUG] http: Request /v1/system/gc (71.546µs)
    2018/01/16 23:10:11.956464 [DEBUG] worker: dequeued evaluation fd8b7e4c-ca04-0781-91f9-b3b847638f8d
    2018/01/16 23:10:11.956494 [DEBUG] sched.core: forced job GC
    2018/01/16 23:10:11.956653 [DEBUG] sched.core: job GC: 13 jobs, 13 evaluations, 13 allocs eligible
    2018/01/16 23:10:11.962373 [DEBUG] sched.core: forced eval GC
    2018/01/16 23:10:11.962441 [DEBUG] sched.core: forced deployment GC
    2018/01/16 23:10:11.962450 [DEBUG] sched.core: forced node GC

This should be logged if you are in DEBUG level logs regardless of if anything gets garbage collected. So if you don't see that something is odd with your setup.

@apenney
Copy link
Author

apenney commented Feb 13, 2018

@dadgar thanks to the wonderful @jippi we've fixed this!

On our server nodes we had:

    enabled_schedulers = ["service","batch","system"]
    num_schedulers = 2

Removing these two lines immediately fixed the issue, with it starting to delete thousands of piled up batch jobs. We don't really understand -why-, unless it was starved by a low num_schedulers stopping it ever getting to gc?

I just wanted to let you know, and let the world know, in case someone else stumbles over this bug report.

@preetapan
Copy link
Contributor

@apenney glad this was fixed for you.

And you guessed right - see the docs for num_schedulers. By setting it to 2, the ability of nomad servers to be able to run scheduler workers in parallel was severely reduced. Garbage collection is done as an internal scheduled job as well so that explains why.

@groggemans
Copy link
Contributor

groggemans commented Mar 8, 2018 •
edited
Loading

Had the exact same problem, running only 1 scheduler, on 1vCPU test nodes. After removing both lines mentioned above, gc started immediately. After putting both config lines back, GC still works.

On 1vCPU, the default is still 1 scheduler, so it doesn't seem to be the num_schedulers that's creating the issue. (or not alone)

My test setup was updated in-place/with rolling redeploys from 5.6 and did almost all versions until 7.1.

Haven't restarted all my nodes yet, but I'll report back here if I'm able to reproduce the issue after restarting all nodes again.

@groggemans
Copy link
Contributor

groggemans commented Mar 9, 2018 •
edited
Loading

@dadgar @preetapan I was able to reproduce the problem again in my setup by performing a rolling re-deploy.

Removing the enabled_schedulers setting on one of the nodes and restarting nomad fixed the issue again.

When the issue is active, server logs (in debug mode) show the ../system/gc call coming in, but there are no other log entries about GC.

@groggemans
Copy link
Contributor

groggemans commented Mar 9, 2018 •
edited
Loading

I think I found the issue, the default value for enabled_schedulers adds the _core scheduler type.

nomad/nomad/config.go

Line 363 in b3ec943

c.EnabledSchedulers = append(c.EnabledSchedulers, structs.JobTypeCore)

But if a custom value is provided, the _core scheduler type is never added to this list by the config parser.

@dadgar
Copy link
Contributor

dadgar commented Mar 9, 2018

@groggemans Great find. This was definitely not intended and we will get this fixed up for 0.8

This was referenced Mar 12, 2018
Closed
Merged
@github-actions
Copy link

github-actions bot commented Dec 3, 2022

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 3, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
stage/waiting-reply theme/client type/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@apenney @schmichael @preetapan @dadgar @shantanugadgil @groggemans