[Core feature] Support Pydantic Models for input/output task/workflow types

[cosmicBboy]

Motivation: Why do you think this is important?

pydantic is a widely-used python data validation library. Similar to data classes, it offers a flexible way for users to create custom types that map field names to a variety of value types.

As a pydantic user, I can use my Model definitions in my Flyte workflows so that I don't have to re-implement my custom-defined types in a format that Flyte understands, like data classes.

Goal: What should the final outcome look like, ideally?

I should be able to use pydantic.BaseModel subclasses in my flyte tasks and workflows

from datetime import datetime
from typing import List, Optional
from pydantic import BaseModel


class User(BaseModel):
    id: int
    name = 'John Doe'
    signup_ts: Optional[datetime] = None
    friends: List[int] = []


from flytekit import task


@task
def my_task(user: User) -> User:
    ...

Describe alternatives you've considered

Create some sort of utility function that converts BaseModels to dataclasses.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[ggydush]

We have an implementation for this at Freenome and can potentially help build this into Flytekit!

Our solution involved:

• Serialize Pydantic model to content-aware file (save this in a private field of model, like Flytefile)
• Create metaclass for automatic type registration so inherited models are automatically registered in Flytekit's type system

[cosmicBboy]

Thanks @ggydush! We may want to implement this slightly differently but would love to discuss @eapolinario @wild-endeavor

[cosmicBboy]

Notes from the pydantic brainstorm: https://docs.google.com/document/d/1W6usjjm8d9NEwE0NLxF1Dt9rkTzLuJxw5kGiVHaPCdg/edit#

@eapolinario ^^ if you're interested in reviewing

[cosmicBboy]

Hey @fg91 just a friendly ping on this issue: I think a good milestone for this would be a work-in-progress PR attached to this issue (whatever you may have so far).

[cosmicBboy]

@wild-endeavor

[github-actions]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable.
Thank you for your contribution and understanding! 🙏