Unsufficient timeout for opening K8S logstream

[Cl0udius]

Hi.

this is a followup to issue #683.
We currently recognize that the 1 second timeout in between the retries is sometimes not sufficient and leads to job loss when using AWX.

receptor/pkg/workceptor/kubernetes.go

Line 155 in 63ee265

time.Sleep(time.Second)

Would it be possible to make this value configurable from our side and not having this 1 second fixed.
Maybe a Environment variable which we can configure on the POD could do the job.

There could also be a default value which has the one second value.

[fosterseth]

I'd be in favor of being able to override with env variable.

[chadmf]

i endorse this!

[JoelKle]

We have the same problem as @Cl0udius. Our k3s environment is from time to time under load and a timeout of one second with 5 retries leads to job loss.

@fosterseth
This issue is almost a year old. Can I move foward and open a PR for this issue?
We would implement this feature in the same was as the env var RECEPTOR_KUBE_CLIENTSET_RATE_LIMITER https://github.com/ansible/receptor/blob/devel/pkg/workceptor/kubernetes.go#L1275

[aeter]

Together with @JoelKle we agreed that I send a PR with a possible fix for this. Thanks! (the PR is already referenced)
@JoelKle - please confirm when you have the time - to make sure all is ok. Thanks

[JoelKle]

yes, we work together with @aeter and agreed that he submits a pull request