-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve IF logging protocols #600
Comments
@jmcook1186 and @narekhovhannisyan, so my overall feedback as a user of IF is that there are several issues we have with not just logging but generally diagnosing issues. Unless your manifest is trivial, it's impossible to determine which part of the manifest file is causing the error. Typically, the flow is that you see a seemingly random error appear in the console, start commenting out bits of the manifest file, and re-run until you narrow down the part of the manifest file that is causing the error.
I have the advantage of the last point, and I can rely on you, but clearly, most people do not. So we need a few solutions. Research and document the current state of affairs
After we've done that analysis, what is missing? A first step to all of this, I would say, is documenting what exceptions our plugins and platform throw right now under what conditions and figuring out what exceptions are missing (we've casually discussed several during weekly IF calls but it's unclear to me which we've implemented). This issue, I believe, covers both of the above (#337). Giving context to the end user The name of the exception and the plugin that raised it is not enough information to be able to diagnose the problem. As an end user, I have to do much more work to be able to figure out the root cause. A little bit more context would save a huge amount of effort and make the UX much more pleasant.
NOTE: Ideally I'd like the context to be printed alongside the critical exception, but if that is very challenging one approach might be to just provide context with DEBUG level logging. So the above context is always printed to the DEBUG level. If you get an exception, you re-run with DEBUG level on, it prints an enormous amount into the console and you can scroll up from the exception to see the context like where in the tree or the name of the plugin. We have that as a fallback and it seems reasonable. |
Realised this is a blocker for #592 and #638, and consequently #615 -
|
Hi @narekhovhannisyan - here are some of the key items we need to think, experiment and have async chats about in relation to logging, in order to get this ticket moving, in my opinion.
|
@jmcook1186 another item I realised I forgot to mention is giving the plugin developers the ability to log also, they can provide context about what's going on internally that is really useful for diagnosis. The simplest solution as I see it doesn't require us to build any complex functionality, we just intentionally print certain things to a debug log. So at the point you know what node your working on, print to the debug log "Processing node x.y.z" At the point you know what the state of the component is (i.e. what you are passing to the plugin) print to the debug log "Component state: ....." If it's all printed to the log then, on an error , to get more context you can simply rerun your manifest with a -debug flag and it prints everything, you simply scroll up from the error to find the additional context like current state, node path etc... The pressure from experience is that developers want to print everything to "info" and the console becomes very busy, so some guidance for what gets printed to critical, warn, info, debug is also required (this guidance should be public and what we recommend for plugin developers also). A more advanced solution might be to store some state internally so when a critical exception is thrown, logging at critical the exception plus the path and the component state. If we can do this then the user might not need to rerun the manifest file with the debug flag, but that really is a nice to have as it just saves the user from rerunning to see context which isn't such a big issue. |
Noting that I added this page to the documentation where all the current error classes are listed: https://if.greensoftware.foundation/reference/errors |
Sketching out some thoughts on this topic before working on the ticket description (@jawache for info). Here's three aims for this task: Add context to error reports:Extend our error builder so that it also surfaces:
The new error format could look similar to:
Standardize and document error classesWe want to define a finite list of error classes that can be emitted and definitions for when these errors should be raised, to reduce the ambiguity of error messages from plugins. Here's the list of error classes we currently use across our IF features and plugins
We should check this list and identify any missing classes or classes we can remove. Then we need to determine a strategy for how this list of error classes should be distributed to plugin builders (is documenting on IF site enough?) Add detailed logs (behind
|
@jmcook1186 We can extend the current implementation of the plugin interface to support flags, however, it won't force users to rely on that flag. Maybe we can do some trick to tune that behavior from IF. F.ex. temporary override console with another stub function, and put the original one back after plugin execution. I need to dig in to see what workarounds can be there. |
@jmcook1186 it won't help in case if users will create custom plugins without our template. We can implement condition in our IF error handler, to support certain errors, and If error is not from the supported list, then error out or just log that error without full support. |
expecting to raise a PR tomorrow morning 🙏 |
@narekhovhannisyan please review the PR changes 🙏 |
Sub of #655
What
A set of updates to our logging and error reporting that will make it easier to diagnose and fix errors in IF runs.
Why
Diagnosing errors is one of the main elements of developer experience that we think needs to be improved to encourage IF adoption.
As a developer I want to be able to access rich debugging logs to help me to build with IF.
As a developer, I want to know exactly where int he stack my IF run failed so i can go in and debug.
As a user I want to have fine grained control over the type of logs I see in the console, and to be able to interpret them easily.
As a user, if my IF run fails, I want to know why and I want to be able to fix it quickly.
Context
To help users and developers identify where an error occurred, we should optionally print logs to the console during normal execution. Then, users can see how their IF run is progressing and use the logs to track when an error occurred. This should be configured using a
--debug
flag in the CLI.If the debug flag is provided on the CLI, then logs should be emitted with the following structure:
[LOG LEVEL] [DATE] [MESSAGE]
e.g.
INFO: 2024-03-05T00:00:00.000Z: Starting Impact Framework
Supported logs
The following log messages should be added to IF execution:
load
functionvalidatemanifest
injectEnvironment
Parameterize
getAggregationMethod
initialize()
initPlugin
handModule
computeNode
mergeDefaults
aggregate
aggregateNode
exhaust
ExportYaml
ExportCSV
ExporCSVRaw
For plugins, we want to be able to integrate logs from plugins into the IF logger, but we also want to be able to develop and execute plugins independently of IF. It's important for the developer experience to be able to develop plugins quickly with minimal environment setup and dependencies. To balance these concerns, IF can easily be tweaked to reroute
console.log()
(orconsole.warn
,console.error
,console.info
,console.debug
) calls to the IF logger. This means we can prepend the name of the plugin emitting the message to the log and run everything through the framework's logger, but plugin developers can also just use console.X methods as usual in their development environment with no changes required to integrate into IF.So, to improve plujgin logging we can:
A plugin (
demo-plugin
) that includesconsole.debug("my custom message")
run through IF will be emitted asSoW
--debug
to CLI--debug
flag is upAcceptance Criteria
ie
run with the--debug
flag emits logs showing the execution detailsGIVEN the
--debug
flag is added to the CLI and logs are added to IFWHEN the following command is run:
ie -m manifest.yml --stdout --debug
THEN the following logs should be seen in the console:
console.x
calls are rerouted to the IF loggerGIVEN the
console.x
calls from plugins are rerouted to the IF loggerWHEN a plugin includes a call to
console.debug("message")
and IF is run with the--debug
flag and a manifest that executes the plugin, e.g. `ie -m example.yml --stdout --debugTHEN IF includes the plugin logs in its execution logs, reformatting it as follows:
For example, a plugin called "my-plugin` that includes the following calls:
full set of IF logs could looks as follows:
The text was updated successfully, but these errors were encountered: