Custom Logging Class not working

[bouleq]

Hi !
I am currently trying to use a custom EVENT_LOGGER in superset, i added the code snippets from the official doc but it isn't working:
I think the EVENT_LOGGER has changed and isn't the default DBEventLogger() anymore because we have some new logs on stdout, and there are no more "action logs" in the superset UI.
However, my JSONStdOutEventLogger() isn't used either, i added some prints in the log method to see if it was even called, and it doesn't seem so.

Here are the code snippets i added:

event_logger.py: |-
    from superset.utils.log import AbstractEventLogger
    import json
    class JSONStdOutEventLogger(AbstractEventLogger):
        def log(self, user_id, action, *args, **kwargs):
          records = kwargs.get('records', list())
          dashboard_id = kwargs.get('dashboard_id')
          slice_id = kwargs.get('slice_id')
          duration_ms = kwargs.get('duration_ms')
          referrer = kwargs.get('referrer')
          print("bip boup 1")
          for record in records:
              log = dict(
                  action=action,
                  json=record,
                  dashboard_id=dashboard_id,
                  slice_id=slice_id,
                  duration_ms=duration_ms,
                  referrer=referrer,
                  user_id=user_id
              )
              print(json.dumps(log))
              print("bip boup 2")

configFile: |-
...
  from event_logger import JSONStdOutEventLogger
  EVENT_LOGGER = JSONStdOutEventLogger()

Expected results

I expect the custom log method to be used (ie to see "bip boup" and "bip boup 2" on stdout)

Actual results

Some debug logs and logs on the requests done are on stdout (cf screenshot)

Screenshots

Environment

• superset version: 1.0.1
• python version: 3.8.8
• node.js version: 12

I hope i'm not missing something, thank in advance for your help !

[wiktor2200]

Hello! Have you solved this issue somehow or found some workaround for that?

[bouleq]

Hello, yes we managed to fix it. It turned out that logs were not written at the right place by the logging class. We added this to the code :

with open("/var/lib/superset/action.log","w") as f:
                f.write("\n [LOG]: " + json.dumps(log) + "\n") 

The full code snippet is now:

event_logger.py: |-
    from superset.utils.log import AbstractEventLogger
    import json
    class JSONStdOutEventLogger(AbstractEventLogger):
      def log(self, user_id, action, *args, **kwargs):
          records = kwargs.get('records', list())
          dashboard_id = kwargs.get('dashboard_id')
          slice_id = kwargs.get('slice_id')
          duration_ms = kwargs.get('duration_ms')
          referrer = kwargs.get('referrer')
          for record in records:
              log = dict(
                  action=action,
                  json=record,
                  dashboard_id=dashboard_id,
                  slice_id=slice_id,
                  duration_ms=duration_ms,
                  referrer=referrer,
                  user_id=user_id
              )
              with open("/var/lib/superset/action.log","w") as f:
                f.write("\n [LOG]: " + json.dumps(log) + "\n")

[wiktor2200]

Thanks a lot for your answer.
I think it's worth considering editing documentation here: https://github.com/apache/superset/blob/master/docs/docs/installation/event-logging.mdx and then it would be possible to close this issue.
May I do it and link this issue to PR or do you want to open your own PR?

[bouleq]

Sure you can do it, hope it can help people.

[wiktor2200]

I've managed to do both JSONStdOutEventLogger and DBEventLogger at the same time. Now events are saved to logs in JSON format and they are still visible in Superset's dashboard in activity tab.

event_logger.py: |
from superset.utils.log import DBEventLogger
import json
class JSONStdOutEventLogger(DBEventLogger):
    def log(self, user_id, action, *args, **kwargs):
        records = kwargs.get('records', list())
        dashboard_id = kwargs.get('dashboard_id')
        slice_id = kwargs.get('slice_id')
        duration_ms = kwargs.get('duration_ms')
        referrer = kwargs.get('referrer')
        super().log(user_id, action, *args, **kwargs)

        for record in records:
            if 'rison' in record:
                del record["rison"]
            log = dict(
                action=action,
                json=record,
                dashboard_id=dashboard_id,
                slice_id=slice_id,
                duration_ms=duration_ms,
                referrer=referrer,
                user_id=user_id
            )
            print(json.dumps(log))

logger_settings: |
  from event_logger import JSONStdOutEventLogger
  EVENT_LOGGER = JSONStdOutEventLogger()

[wiktor2200]

Hi! My last PR is merged, finally I've just updated docs. I think this issue could be closed now?
@bouleq could you close it? Or is it anything more still needed for this issue?

[gugacavalieri]

If anyone is coming here in 2024. This is what worked for us with the Superset Helm Chart 0.12.9 version: (We are running Superset in Kubernetes)

# values.yaml file:

configOverrides:
  # We use a custom JSON logger here to output everything to stdout -> FluentBit
  # Also, we prevent superset from using the default logger and flooding the DB with Logs
  # More info: https://superset.apache.org/docs/configuration/event-logging/
  logger_settings: |
    # log everything as JSON
    from superset.utils.log import AbstractEventLogger
    import json
    class JSONStdOutEventLogger(AbstractEventLogger):
        def log(self, user_id, action, *args, **kwargs):
          records = kwargs.get('records', list())
          dashboard_id = kwargs.get('dashboard_id')
          slice_id = kwargs.get('slice_id')
          duration_ms = kwargs.get('duration_ms')
          referrer = kwargs.get('referrer')
          for record in records:
              log = dict(
                  action=action,
                  json=record,
                  dashboard_id=dashboard_id,
                  slice_id=slice_id,
                  duration_ms=duration_ms,
                  referrer=referrer,
                  user_id=user_id
              )
              print(json.dumps(log))

    # use JSON logger and disable logging to the DB Table
    EVENT_LOGGER = JSONStdOutEventLogger()

This code creates a new logger that will log events as JSON in stdout and then prevent the DB Table from overflowing from time to time. The logs can then be flushed out from stdout to a search tool using a logger daemon.

If desired, I can help to update the Kubernetes Docs with this 🧡