Consider kedro run -v verbose mode

[noklam]

Description

The current logging configuration is too granular. We should use root logger only. The exception would be info_logger or error_logger because they are mean to capture that specific log level.

The most common case is that I want to see all the logs during debugging. In production case (packaged project), how do I even change it to see the log if I want to? Editing the configuration file doesn't seem to be a good idea.

Inspired by pytest -v pytest -vv
Proposal:

1. kedro run -v verbose mode - The Project logging level will be set to DEBUG because this should be the most common case, I want to see the log message that I wrote in the project
2. kedro run -vv more verbose mode - Everything should be shown now, including any plugin logging or kedro logs

Context

It's hard to get logging right, in most case I either expect seeing just the default level or I want to see everything during debugging. a -v flag comes in handy and is a common convention among many CLI tools. i.e. pytest.

Possible Implementation

Add new argument in the CLI, and override the level of logging in LOGGING.

Possible Alternatives

Edited 2024-03-07
I have splited this ticket to two tickets

• Change the default_logging.yml and logging.yml to have more sensible default #3687

[noklam]

Putting a note here to remind myself to do a demo when we groom this.

[astrojuanlu]

I've said it in #3446 (comment), but for the record, +1 to this. From https://12factor.net/logs:

A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout.

[astrojuanlu]

Another point in favour of having kedro -vv: sometimes you just want to know what kedro is doing without having to configure the whole stdlib logging. This is how most other tools work.

Here is an example of a user that's puzzled because kedro just hangs https://linen-slack.kedro.org/t/16630828/i-m-encountering-issues-when-running-the-load-ext-kedro-ipyt#3758284a-34fb-435c-b054-dd3713cd78c9

[noklam]

It was moved to discussion due to #3755 (comment)

[astrojuanlu]

To be continued

[astrojuanlu]

Another problem of our approach to logging:

I'm trying to get Kedro to work on Prefect by creating a very simple flow that launches a KedroSession:

from kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project
from prefect import flow


@flow
def main():
    bootstrap_project(Path.cwd())
    with KedroSession.create() as session:
        session.run()


if __name__ == "__main__":
    main()

However,

1. from prefect import flow takes care of configuring my loggers to send the output to the Prefect UI,
2. and bootstrap_project(Path.cwd()) reverts that configuration

And since this is impossible to avoid, because

• Removing logging.yml will lead to Kedro using default_logging.yml anyway,
• Making logging.yml empty will result in a crash,

Then I have no other choice but to take care of this manually:

@@ -9,6 +9,9 @@
     format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
 
 handlers:
+  prefect:
+    class: prefect.logging.handlers.APILogHandler
+
   console:
     class: logging.StreamHandler
     level: INFO
@@ -35,9 +38,11 @@
 loggers:
   kedro:
     level: INFO
+    handlers: [prefect]
 
   spaceflights_pandas_viz2:
     level: INFO
+    handlers: [prefect]
 
 root:
   handlers: [rich, info_file_handler]