Add FlyteNonRecoverableSystemException

[pingsutw]

Tracking issue

NA

Why are the changes needed?

Flyte returns the task when it fails to convert the Python output to a literal or vice versa.

What changes were proposed in this pull request?

Raise FlyteNonRecoverableSystemException if flytekit fails to serialize the output. (e.g. failed to convert python.int to Literal.simple.string).
However, if the error comes from data persistence, we won't change the error type to FlyteNonRecoverableSystemException

How was this patch tested?

remote cluster

pyflyte run --remote flyte-example/improve_error/mismatching_output.py wf

Setup process

from flytekit import task, workflow, ImageSpec

new_flytekit = "git+https://github.com/flyteorg/flytekit@3ef9b11cba76a237bb010c8a764ecdc2c2093089"

image_spec = ImageSpec(
    registry="pingsutw",
    apt_packages=["git"],
    packages=[new_flytekit],
    builder="envd",
)


@task(container_image=image_spec)
def t1() -> int:
    return "hello"


@workflow()
def wf():
    t1()

Screenshots

Before:

After:

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

NA

Docs link

NA

[wild-endeavor]

tried to merge in master, didn't do it correctly though, definitely missing some things, like the double Exception catching that's now in entrypoint.

Regardless I think we should think more about this - I'm not sure it really addresses the issue raised in slack (and commented in this issue here). This issue is on the propeller side.

[wild-endeavor]

forgot to hit submit on that comment. that was from before kevin fixed the merge