Use mashumaro for serializing logging events

[gshank]

resolves #4504

Description

Use mashumaro to serialize json logging.

Checklist

• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change

[gshank]

Mashumaro required that the Event class be a dataclass. Changing the Event class to a dataclass with attributes with defaults means that the attributes of subclasses all have to have defaults too (because non-default attributes can only come before attributes with defaults). In some cases Event subclasses were being constructed without keyword names, which caused the passed in value to be used for the first attribute (i.e. log_version) which didn't work, so all constructors needed to use keyword arguments.

Mashumaro also expects to serialize everything in the class, so it didn't work to have various node objects in the event classes. I switched to putting the node_info construction code in the nodes themselves and passing the node_info the event classes instead of passing in the various node objects. I would have liked to make _event_status and node_info into a separate class, but mypy really didn't like that. The stubs.py file was removed.

I created a separate core/dbt/events/serialization.py file for handling the mashumaro serialization, since these classes didn't need some of the complications in core/dbt/dataclass_schema.py. It's also a good place to put other special serialization code if we need it.

[nathaniel-may]

Integration tests with json serialization on are hanging in the interop tests. I suspect there's an exception being raised in threads again.

[nathaniel-may]

I get a few failures too when I run DBT_LOG_FORMAT=json make integration locally. But it's definitely hanging sometime after the 73% mark.

[nathaniel-may]

I have a couple of concerns with this approach:

• mypy can no longer catch malformed events such as events that have no code field.
• mypy can no longer catch when we fail to provide event attributes even when mypy runs everywhere
• it looks like when an event takes a type that isn't serializable we are forced to convert it to a serializable type outside the event module which leaks a core abstraction of the event module back out to the rest of the code base

Since these warrant a larger discussion, I think we may way to make a new PR with the bug fixes you included here for the patch release, and consider the larger serialization change for 1.1.0.

[gshank]

"it looks like when an event takes a type that isn't serializable we are forced to convert it to a serializable type outside the event module which leaks a core abstraction of the event module back out to the rest of the code base"

I'm not sure that I agree that it is the event module's job to enable serialization for outside classes. In any case, serializing classes that are not needed for the event message feels unnecessary, since it gets thrown away anyway. I think that we're better off providing only the information that is needed for the event/message.

[gshank]

"mypy can no longer catch when we fail to provide event attributes even when mypy runs everywhere"

Providing defaults for attributes of subclasses is something that the Python dataclasses module forces you to do. I think it's probably more important to consistently make the classes dataclasses than make a couple of methods abstractmethods, when executing those methods without an override will throw an error just like the abstract base class.

[nathaniel-may]

The way I see it, the event module is tasked with translating dbt's internal representations into a structured output that is versioned as an API so that downstream applications can rely on it. Serialization and schema conformity are both major parts of creating this API to an enterprise quality. I want this code to be written in such a way that if you are not modifying the event module you do not need to know anything about how it was built or what it needs to continue to conform to the logging API version.

If using dataclasses forces us to weaken our development-time guarantees for this API, we should switch tactics and do something else.

[nathaniel-may]

These recent commits address all 3 of my major concerns from my last review. Mypy is still able to be helpful in the ways it was planned to and there's a way to override serialization in serialization.py. The following remarks are much smaller issues likely with much simpler solutions. Thanks for all the hard work here!

[nathaniel-may]

Added some additional comments. There are still 5 unaddressed comments from my previous review.

[nathaniel-may]

Just finished a quick sync with @gshank to go over the remaining comments. Here are the outcomes from that conversation:

• The node_info field is going to move underneath data. This will require an API version increase.
• The serialization strategy for datetime objects in serialization.py seems to be currently unused so it's going to be removed with a comment for now to avoid confusion between other datetime serializations. Cleaning up these across dbt-core is recorded in [CT-82] Standardize serialization of datetimes #4608
• The "node_info" value is actually not being constructed here, it's just being passed through as a dictionary, so let's skip turning it into a nominal type here for now.
• RunResult is a little tougher to serialize with mashumaro. Let's comment what makes that difficult and leave it as a dictionary value.

[nathaniel-may]

There's just that one value test for the log version that still needs to be updated, but this looks good to me!