Orchestration - Flakiness in small samples when using PNS executor

[Bobgy]

In #5273, I switched to PNS executor by default. After that, it seems lightweight component sample fail more frequently than before.

(It failed 3 times consecutively in an example I found, but it seems to me that the last two failures should be fixed by #5284, we can observe the actual flakiness rate after the change)

Symptom, pipeline components that run too fast fail with:

failed to save outputs: could not chroot into main for artifact collection: container may have exited too quickly

https://oss-prow.knative.dev/view/gs/oss-prow/pr-logs/pull/kubeflow_pipelines/4147/kubeflow-pipeline-sample-test/1369916972707352576#1:build-log.txt%3A5681

Root cause seems to be: argoproj/argo-workflows#1256 (comment)

And workarounds can be:

• (hacky) let the main container sleep for a while
• (stable fix) mount the artifacts on an emptyDir

[Bobgy]

/assign @chensun @neuromage
/cc @Ark-kun @capri-xiyue

I think we need to discuss this problem, from KFP side, we could automatically mount emptyDir for users.

[Bobgy]

/cc @jessesuen @alexec
Do you have any suggestions? Is my above understanding of the workarounds accurate?

[Bobgy]

Hmm, strange, I'm not seeing flakiness anymore

I'll keep observing this problem

[Ark-kun]

I see postsubmit flakyness. https://oss-prow.knative.dev/view/gs/oss-prow/logs/kubeflow-pipeline-postsubmit-integration-test/1371547567694811136

[Bobgy]

I'll revert back to docker as default for current release, but we should evaluate possibility of mounting emptyDir volumes for artifact paths for users.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Bobgy]

We now recommend emissary executor instead: #1654 (comment)