HTTP endpoint to tail job logs, stdout, and stderr

[jippi]

Would be a nice addition for easy debugging

I realise non-nomad version of this would be to pipe everything to syslog or similar, but it requires a few more moving parts

[cbednarski]

@jippi Thanks for the suggestion. We would like to support this because it is a super convenient way to debug things on a small cluster. In a large scale environment we recommend using centralized logging.

[ghost]

+1 On a side note, having the ability to shove this output somewhere in the job definition (like logspout with docker) for 12 factor apps would be nice.

[fernandezvara]

I would point to heka, since the docker log input allows to get all the logs with a simple configuration setting.

http://hekad.readthedocs.org/en/v0.10.0b1/config/inputs/docker_log.html

If you don't like to get the container name as log source, you can add any information you need from the environment variables. (Adding a name_from_env_var, like 'TASK_NAME' will return the value if found instead of the container name).

That way you have a centralized log easily without over configuration.

Hope this helps.

[cbednarski]

From #475

Currently it seems there is no way to view the out put of a job other than using docker logs (if you are scheduling docker) on a individual container. it would be awesome if the logs were made available via a stream or something, this would also make for a useful ui feature in that you can see your jobs get allocated and see there output, perhaps even see the logs of all of the instances of a task in the same stream just tag for each instance.

[cbednarski]

+1 On a side note, having the ability to shove this output somewhere in the job definition (like logspout with docker) for 12 factor apps would be nice.

@rvm2015 Our expectation is that you will have a log shipping agent setup on the host, or co-scheduled in your task group. Can you give us some specific examples of configuration you need to specify that is currently not available in Nomad?

[supernomad]

@cbednarski I cannot speak for @rvm2015 however a simple use case for this would be getting the logs for said shipping agent without having to go to a node that is somewhere on your cluster to get them.

Also there are a lot of times where logs coming out of stdout/stderr or much different than the ones coming out via log files. This can be invaluable when trying to debug why an application seemingly inexplicably died.

[diptanu]

@supernomad Our logging feature, coming with Nomad 0.3 will allow you to stream logs and view the file system of an allocation with Nomad cli. You won't need to go to a Node to stream logs out of the tasks running on allocations in that Node.

The UX is going to be something like this -

nomad logs <allod-id> <task-name> --stdout --follow
nomad alloc-fs-list <alloc-id> /path/to/dir/
nomad alloc-fs-cat <alloc-id> /path/to/file
nomad alloc-fs-stat <alloc-id> /path/to/file

Hope this works for you?

[supernomad]

@diptanu That will work perfectly, but I have two questions. First would there be a way to get the logs from all active allocations? Also, is there any kind of timeline on 0.3?

[diptanu]

@supernomad Yes, you can get logs from all allocations which are still present on the nomad clients.

And regarding 0.3, very soon! We are getting into the crunch mode to get the release out.

[sheerun]

@diptanu I'd prefer --follow flag to be added to fs cat instead to stream logs in custom locations as well.

[jrg72]

Would there be any possibility this could be added to nomad? I'm trying to debug a particularly difficult issue with the Rkt driver and it would be very helpful to have this output while trying to determine if the problem is the ACI or the driver. Thank you.

[dadgar]

Still on the roadmap! But won't be in 0.4.0

[type0lang]

+1. here i have a failed task with 0 stderr or stdout

nomad logs -stderr e3e3d705

[dadgar]

Ah this is actually done! /v1/client/fs/logs.

@type0lang You may have that if it never produce logs because it failed to start (see nomad alloc-status) or the allocation may have been garbage collected

[sheerun]

@dadgar Ideally nomad logs would allow streaming any file, not only stderr / stdout. Sometimes logs are written in separate files, and sometimes we'd like to stream results / progress.

[dadgar]

You can do that using nomad fs command. If you do -f it will follow the
file.
On Tue, Sep 13, 2016 at 12:27 PM Adam Stankiewicz [email protected]
wrote:

@dadgar https://github.com/dadgar Ideally nomad logs would allow
streaming any file, not only stderr / stdout. Sometimes logs are written in
separate files, and sometimes we'd like to stream results / progress.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#277 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA_9ams8PFUussE8epiXDaUA-e6wlNfxks5qpvkNgaJpZM4GOCJj
.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.