Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing consul keys: no hint in alloc-status #2578

Closed
stefreak opened this issue Apr 20, 2017 · 4 comments
Closed

Missing consul keys: no hint in alloc-status #2578

stefreak opened this issue Apr 20, 2017 · 4 comments

Comments

@stefreak
Copy link

If you reference consul kv keys from the template stanza, nomad will notice if they don't exist yet and the task is pending until the key is there.

This is fine, but alloc-status should inform the user about this fact.

Nomad version

Output from nomad version
Nomad v0.5.6

Operating system and Environment details

Ubuntu 16.04

Issue

nomad alloc-status should give a hint if nomad is waiting for missing information in consul

Reproduction steps

template stanza with the following template:

 {{ key "this/key/is/missing" }}

Nomad Server logs (if appropriate)

Apr 20 11:47:27 nomad-node0 nomad[1209]:     2017/04/20 11:47:27 [DEBUG] (runner) missing data for 1 dependencies
Apr 20 11:47:27 nomad-node0 nomad[1209]:     2017/04/20 11:47:27 [DEBUG] (runner) diffing and updating dependencies
Apr 20 11:47:27 nomad-node0 nomad[1209]:     2017/04/20 11:47:27 [DEBUG] (runner) kv.block(scorecard/lastIncident) is still needed
Apr 20 11:47:27 nomad-node0 nomad[1209]:     2017/04/20 11:47:27 [DEBUG] (runner) kv.block(scorecard/midonetCountdown) is still needed
Apr 20 11:47:27 nomad-node0 nomad[1209]:     2017/04/20 11:47:27 [DEBUG] (runner) kv.block(scorecard/toilworkPercent) is still needed
Apr 20 11:47:27 nomad-node0 nomad[1209]:     2017/04/20 11:47:27 [DEBUG] (runner) kv.block(scorecard/errorbudget) is still needed
Apr 20 11:47:27 nomad-node0 nomad[1209]:     2017/04/20 11:47:27 [DEBUG] (runner) watching 4 dependencies
@dadgar
Copy link
Contributor

dadgar commented Apr 20, 2017

Yep definitely on the roadmap!

@csawyerYumaed
Copy link
Contributor

I just got hit by this today. I was like UGH why is my job stuck in pending.... This is the relevant logs from the nomad agent node running the job... so it's not even out on the agent to say anything.


    2017/07/19 09:51:50 [INFO] (runner) creating new runner (dry: false, once: false)
    2017/07/19 09:51:50 [INFO] (runner) creating watcher
    2017/07/19 09:51:50 [INFO] (runner) starting
    2017/07/19 09:51:50 [INFO] (runner) initiating run

@stefreak
Copy link
Author

Yeah it seems that it's only logged with debug loglevel.

dadgar added a commit that referenced this issue Aug 10, 2017
This PR does the following:
* Adds a mechanism to emit events in the TaskRunner
* Vendors a new version of Consul-Template that allows extraction of
missing dependencies
* Adds logic to our consul_template.go to determine missing events and
emit them in a batched fashion.
* Refactors the consul_template code to split the run method and take in
a config struct rather than many parameters.

Fixes #2578
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 10, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants