container logs lose line(s) (the last one)

[tnozicka]

At least the last line of logs get lost sometimes causing flakes in various tests. Test use oc logs command.

I've found some old hack with sleep 1 at the end of the container command which appear to mitigate the issue mostly, but flake sometimes as well.

Here are some flakes caused by this:

• [Conformance] deploymentconfigs with custom deployments should run the custom deployment steps flakes #17631 (comment)
• https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17606/test_pull_request_origin_extended_conformance_gce/12465/
• https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17576/test_pull_request_origin_end_to_end/6690/
• more ...

cc: @mfojtik @soltysh @deads2k

[tnozicka]

this is likely related:
#12143 (comment)
moby/moby#29863

[soltysh]

So I was talking about #9285, but it looks like the bits mentioned in projectatomic/docker#218 should be in already.

[tnozicka]

@soltysh I don't think we have projectatomic/docker#218 that pr was closed

https://github.com/projectatomic/docker/blob/master/daemon/logger/journald/read.go#L255

[soltysh]

We'd need to check the current docker or get @runcom look into it

[runcom]

We'd need to check the current docker or get @runcom look into it

I guess we can re-open projectatomic/docker#218 for the latest docker(s) we support and have @nalind ack on it to get it in

[tnozicka]

I've resurrected it already projectatomic/docker#291

[stevekuznetsov]

In our test AMIs we are using

$ rpm -q docker
docker-1.12.6-68.gitec8512b.el7.x86_64

[tnozicka]

that version of docker already has that particular fix (https://github.com/projectatomic/docker/blob/ec8512ba42f349cbe5a63140c4d9c810f7afa527/daemon/logger/journald/read.go#L262-L269)

It must be something else then

[tnozicka]

• https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17479/test_pull_request_origin_end_to_end/6606/
• https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17479/test_pull_request_origin_end_to_end/6613/
• https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17553/test_pull_request_origin_end_to_end/6603/

[stevekuznetsov]

We need to revert the journald limit disable PR.

[soltysh]

We've discussed this with Steve yesterday, I'll open a PR which will enable the journald for all our tests, for now the revert in #17758 should stabilize the queue, slightly.

[tnozicka]

Is this really just jurnald rate limit? I find it suspicious that rate limiting would strip only the last line in the same specific test.

[stevekuznetsov]

I think it ends up being how reproducible our tests are -- always run the same N tests in the same order and generate the same log lines, so if we trigger a rate-limit it will be fairly often in the same place.

[tnozicka]

Last time it was on the last line of different test and we had to add sleep 1 there. Seems like too much of a coincidence.

[stevekuznetsov]

There was also some weirdness with the way that the logs were gathered in docker that Andy was working around by asking for them twice, but I don't think that ever made it in.

[tnozicka]

@stevekuznetsov it is in the docker version you've pasted #17747 (comment)

[stevekuznetsov]

No, @soltysh knows more about that patch from Andy

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close