Feat: Add pydantic plugin with BaseModel type transformer

[fg91]

TL;DR

The plugin adds type support for pydantic BaseModel.

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed: The transformer does not yet support all Flyte types in a pydantic model (e.g. data frames)
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

The plugin allows passing pydantic models to and between tasks:

Type Example

from pydantic import BaseModel
import flytekitplugins.pydantic
class Config(BaseModel):
    lr: float = 1e-3
    batch_size: int = 32
@task
def train(cfg: Config):
    ...

Tracking Issue

flyteorg/flyte#2686

Follow-up issue

NA

[codecov]

Codecov Report

Merging #1620 (1f20565) into master (baf0b89) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1620      +/-   ##
==========================================
- Coverage   70.10%   70.10%   -0.01%     
==========================================
  Files         328      328              
  Lines       30156    30162       +6     
  Branches     5452     5454       +2     
==========================================
+ Hits        21141    21144       +3     
+ Misses       8485     8476       -9     
- Partials      530      542      +12     

see 10 files with indirect coverage changes

[ArthurBook]

Hi

super useful. thanks for sharing.
Trying this on my end but couldnt get it to play with the flytefiles/directories because they require arbitrary_types_allowed which disables the .schema() method. Here is an example :

class Rax(pydantic.BaseModel, arbitrary_types_allowed=True):
    hax: directory.FlyteDirectory

    @classmethod
    def from_json(cls, json: str) -> "Rax":
        self = cls.parse_raw(json)
        return self

Any thoughts on how to best solve it? I am looking at the built in DataclassTransformer for inspiration

[fg91]

Closing in favor of #1660 where @ArthurBook improved upon this implementation of the pydantic base model type transformer.