Warning message is confusing when pod logs cannot be retrieved

[jiezhang]

What steps did you take:

After the pod finishes successfully and is later reclaimed, the following warning is displayed. The message often confuses first-time users, and suggests them to check out the troubleshooting guide.

Warning: failed to retrieve pod logs. Possible reasons include cluster autoscaling or pod preemption

What happened:

In fact, the logs can be viewed in Stackdriver Kubernetes Monitoring.

What did you expect to happen:

Remove the warning message.

/kind bug
/area frontend

[Bobgy]

Thanks for the suggestion!
Sounds reasonable to me.

We can only show troubleshooting guide when there is an error, but not a warning.

[jonasdebeukelaer]

happy to fix this
/assign @jonasdebeukelaer

[jonasdebeukelaer]

oh wait is this already done? i.e. just removing 'troubleshooting guide'?

[Bobgy]

@jonasdebeukelaer Thanks for offering help!
This still needs to be done.

Some helpful information for contribution:

1. frontend contribution guide: https://github.com/kubeflow/pipelines/tree/master/frontend
2. Banner component (that shows the troubleshooting link): https://github.com/kubeflow/pipelines/blob/master/frontend/src/components/Banner.tsx
3. Run Details Page's log viewer tab's banner:

   pipelines/frontend/src/pages/RunDetails.tsx

   Line 472 in e52481a

   <Banner

My suggested UX would be to hide the troubleshooting link when the given banner is a warning (it still shows it when the banner is error), but you can take a look and decide if that feels reasonable to you. It already supports hiding the link on ad-hoc usage: https://github.com/kubeflow/pipelines/blob/master/frontend/src/components/Banner.tsx#L72, so we can also dynamically configure it on usages.

[jiezhang]

@Bobgy @jonasdebeukelaer I wonder if it is okay to remove the message completely, or at least lower the level to informational (without the exclamation mark and "Warning" prefix).

[Bobgy]

/reopen
I just tested on 1.0.0 and the problem still exists.

Now it shows fail to retrieve pod logs with error message:

Error response: Could not get main container logs: Error: Unable to retrieve workflow status: [object Object].

We didn't accommodate for the case when workflow was also missing.

[k8s-ci-robot]

@Bobgy: Reopened this issue.

In response to this:

/reopen
I just tested on 1.0.0 and the problem still exists.

Now it shows fail to retrieve pod logs with error message:

Error response: Could not get main container logs: Error: Unable to retrieve workflow status: [object Object].

We didn't accommodate for the case when workflow was also missing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Bobgy]

@jonasdebeukelaer do you want to revisit this?
Or I can follow up too

[jonasdebeukelaer]

hey @Bobgy should be a quick one to fix so happy to do it. In what situations can a workflow be missing?

[Bobgy]

When user configures a TTL to GC workflows. (We have a default TTL of 1day)

In fact, workflow status should be persisted into run details DB rows. So UI shouldn't need to fetch the workflow

[jonasdebeukelaer]

hmm makes sense 👍

[Ark-kun]

@Bobgy The logs are already persisted to the storage the same way as other artifacts. AFAIK, @eterna2 added support to show these logs in the UX when the pod is not available, but this option is turned off by default. Maybe you can enable this option?

[Bobgy]

@Ark-kun this bug: #3711 (comment) must be fixed before logs can be reused from archive.

[haydnkeung]

@Ark-kun How do you enable the option?

[ConverJens]

@Ark-kun @Bobgy Is there any update to this? How do you enable log persistence?

[Bobgy]

No update yet, we need someone from community to fix this problem.

For us, we are on GCP, so GCP stackdriver auto persists all Kubernetes pod logs.

[ConverJens]

@haydnkeung I managed to enable logs persistence.

Check the configmap workflow-controller-configmap and see if archiveLogs: true is set. For me it wasn't, even though I'm on KF 1.1, and I had to set it in the config-map.yaml found in your manifest dir under argo/base.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ra312]

@haydnkeung I managed to enable logs persistence.

Check the configmap workflow-controller-configmap and see if archiveLogs: true is set. For me it wasn't, even though I'm on KF 1.1, and I had to set it in the config-map.yaml found in your manifest dir under argo/base.

Dear @ConverJens , did you restart ml-pipeline-ui (kubeflow UX) after editing configmap?

[ConverJens]

@ra312 I actually redeployed all ml-pipieline components apart from Minio and Mysql so I don't know which is required. However, I don't think the UI has anythinh to do with this but rather it's the api server that needs restarting. The UI will simply pick up the logs as any other artifact.

[ra312]

Thanks, @ConverJens! I will try to do the same.

[rohitgujral]

@ConverJens @ra312 I'm also trying to persist the pipeline pod logs so that if pod gets deleted then logs should be available to the pipeline runs.
I added archiveLogs: true to the argo/base config-map and restarted pipelines deployment but still after deleting the pod, im not seeing logs.

Is there any other step which needs to be done ?
kubeflow version - 1.0.2 and argo version - 2.3.0

[ConverJens]

@rohitgujral The logs tab will only be populated until the pod is removed. The complete logs will be available as a tar.gz under artifacts instead, called main-logs.tar.gz I think.

Note that while the logs tab will always hold the full log, the artifact can lose the final part in the event of your component crashing.I believe that this has to do with the logs not being fully flushed in some instances and in that case, only the logs up the point of error is available.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ra312]

closing since the issue has been reported as fixed in #3848

[ra312]

/close

[google-oss-prow]

@ra312: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.