Error while receiving logs

[pavellety]

Hi,

First of all thank you very much for woking on tart/cirrus-cli. This is wonderful project.

Anyway I have one question regarding the error we started to see quite often on our workers: 17:15:58 Error while receiving logs: rpc error: code = Canceled desc = context canceled.

I have tracked the implementation is here:

logEntry, err := stream.Recv()
                if err == io.EOF {
                        break
                }
                if err != nil {    
                        streamLogger.Warnf("Error while receiving logs: %v", err)
                        return err 
                }

Usually when this error occurs the job will stuck and fail after the configured timeout is reached (resources are blocked).

Does it mean the client is dead?
Should the task immediately fail when this error occurs?
Would it make sense to implement retry mechanism (like fail after certain number of tries)?

We saw some network glitches in our monitoring on that workers so we suspect it can be related to this.

Thank you very much for reply.
Pavel

[edigaryev]

Hello Pavel!

Do I understand correctly that you use cirrus run to run your task on worker machines (and not Persistent Workers)?

The error you're seeing should not normally happen and likely means that the agent running the task has terminated/hung prematurely.

Which environment this happens on (Docker container, Tart VM, etc.)?

[pavellety]

Any update here? Thanks

[edigaryev]

Hello Pavel,

Sorry for late response.

If I understood you right, your primary issue is that the cirrus run terminates because of the task's timeout, but not because of the networking error you're seeing, which could happen much earlier?

We could add a logic to fail the task if the agent misses more than 2 heartbeats (which happen every minute), but that won't fix the networking-related root cause you're experiencing.

[pavellety]

Hi,

thanks for the reply,

If there is not any way how tart could recover from this type of issue I would definitely prefer task to fail as you suggested.

Thanks,
Pavel

[edigaryev]

I've created #652 to address this.

[edigaryev]

Please check out the new 1.104.0 release that include theses changes.

[edigaryev]

Hello @pavellety, are you still experiencing network glitches?

We've tweaked things a bit in the latest 1.106.0 release, and now you'll need to explicitly enable heartbeat monitoring with --heartbeat-timeout flag.

The old behavior can be replicated by passing a --heartbeat-timeout=2m flag to tart run.

[pavellety]

Hello @edigaryev, let me try to bump to the new version and try. Thanks