Add event logging support #679
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yuvipanda
force-pushed
the
event-logging
branch
from
cdd3b1b to
56479a2
Compare
|
@minrk if I set handler to logging.StreamHandler(), I get this error: Looks like traitlets is trying to do a deep copy maybe? I'm not entirely sure why it's doing that! Any thoughts? |
|
traitlets does a deep copy of the config object. What are you passing to the config? |
- Define & emit events in a unified way with EventLog class - Events can be sent to a sink via logging handlers - Events are JSON formatted, with a 'capsule' that contains useful metadata - Defaults to not doing anything at all unless a handler is configured - Adds 'launch' events that are emitted when a launch is successful. - Schemas are used to make sure we are only capturing the information we really need, and nothing more.
traitlets does a deep copy of config, so you can not pass unpicklable Instance objects through it. Most handlers are unpicklable
This installs binderhub and its dependencies into environment. Also runs npm install and webpack, so we don't have to run it manually. Doing this since travis can't find the jsonlogger package otherwise.
Trying to fix the following error from CI: > traitlets.traitlets.TraitError: No default value found for handlers_maker trait
yuvipanda
force-pushed
the
event-logging
branch
from
4683890 to
046e3a7
Compare
|
The test failures now seem unrelated? |
|
Tests and docs added! |
betatim
reviewed
Oct 4, 2018
minrk
reviewed
Oct 4, 2018
minrk
reviewed
Oct 4, 2018
|
As a design question: will each event type be added as a method on EventLogger, or will there be direct calls to |
Avoids evaluation overhead on object instantiation
This ensures we have a timezone marker
yuvipanda
force-pushed
the
event-logging
branch
from
2649a4d to
03831ba
Compare
|
@minrk I think to begin with we'll add methods to EventLogger, but I want to add schema validation + registration to this soon. Once we do that, we can just have emit. The methods are there just to enforce a schema until then. This will also allow us to make this more generic later on. |
pip install . doesn't seem to run these commands from setup.py
minrk
reviewed
Oct 5, 2018
This was referenced Oct 20, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR aims to address parts of jupyterhub/mybinder.org-deploy#97
Our goal is to have an analytics pipeline that:
developers can see easily & exactly what is being
collected.
the event streams & developers who are adding the
event emitting code.
This PR introduces an EventLog class. We'll iterate on
the design here, but eventually it should be spun out into
its own little library that can be integrated into other
parts of Jupyter.
For mybinder.org, we can sink these events into
Stackdriver with https://cloud.google.com/logging/docs/setup/python.
This should give us an easy sink that we don't have to maintain,
and an easy way to export to files / real-time query. This is
much easier than maintaining something like logstash or fluent-bit.
A lot of this work is inspired by the (IMO very well designed)
Wikimedia Event Logging
system. I think it provides a good mix of analytics, performance,
privacy, transparency, and ease of use for analysis