Build Cop: possible missing report

[tmatsuo]

In the build, one of the failures is for directory container_registry/container_analysis.

The log says:

Command pytest --junitxml=sponge_log.xml failed with exit code 1
Session py-3.8 failed.
[Buildcop] Sending logs to Build Cop Bot...
[Buildcop] See https://github.com/googleapis/repo-automation-bots/tree/master/packages/buildcop.
[Buildcop] Processing sponge_log.xml
[Buildcop] Success! ID=1172713771765038
[Buildcop] Done!

However, we don't see an issue on the repo.
I think the previous issue is:
GoogleCloudPlatform/python-docs-samples#2894

but it was closed 15 days ago.

[tbpg]

Thanks for the report. I'm looking into the logs now.

[tbpg]

I think I found the logs for this particular build. I don't know why the issue was not filed -- I don't see any errors.

I'm going to add some additional logging to the bot to hopefully help next time this happens.

[tbpg]

cc @codyoss, who has also seen something like this.

Please update this thread if you spot this again.

[tmatsuo]

Another instance:

Build:
https://source.cloud.google.com/results/invocations/6bbc9137-ac61-466b-9929-9162b22b1a18

Log:

Session py-3.7 failed.
[Buildcop] Sending logs to Build Cop Bot...
[Buildcop] See https://github.com/googleapis/repo-automation-bots/tree/master/packages/buildcop.
[Buildcop] Processing sponge_log.xml
[Buildcop] Success! ID=1173967683679910
[Buildcop] Done!

My suspicion is that this test failed with somewhat long log output. The xml file is 485 KB. You can download it from the resultstore UI.

[tbpg]

The extra logging was helpful -- the failure was successfully parsed:

[GoogleCloudPlatform/python-docs-samples] example failure: datalabeling.manage_dataset_test: test_list_dataset

Unfortunately, there are no other logs indicating why the issue wasn't filed.

[tbpg]

Another one: d4152082-608d-425a-a5a7-0e6890562a19

[tbpg]

Another one: fdf704c7-bc2b-40c4-8ff6-abc657cc6dec

I see some log entries with that ID. But, I don't see an invocation with a failed test. The previous instances of this bug had logs that showed the failure was successfully parsed. That particular scenario should be fixed with the await PR above.

However, with this build, I don't see the failure at all.

[tbpg]

I checked the logs around that time and don't see any relevant errors. I also downloaded the XML file and confirmed it is successfully parsed.

@orthros is there a way we could get the PubSub publish ID from the bot? When the logs are sent to PubSub, we have logs like this:

[Buildcop] Processing sponge_log.xml
[Buildcop] Success! ID=1186690235465177

I'd like to log that ID from the bot so we can confirm the bot is actually seeing all of the PubSub messages.

[tbpg]

986af32d-f777-40b5-8fbc-bd7c467b912c is the exact same. That's another build for a different Go version from the same overall build.

Notably, the test failures are from the largest sponge_log.xml files for both builds. Perhaps @tmatsuo's idea about the message being too large is correct?

[orthros]

I'd have to do a bit more digging into the PubSub mechanism of this (I'm not terribly familiar with BuildCop) but it sounds like something that is possible.

[bcoe]

@orthros @tbpg, two thoughts:

1. this could be something we improve with the intern project, by moving the test output information into a database, rather than kicking it around in a PubSub message.
2. if we can't wait this long, what if we put build information into a storage bucket with a short-lived retention policy, and just kicked around a reference in the pubsub message?

[tmatsuo]

In an assumption that adding instances helps. Let me know if you don't need this any more.

(build)[https://source.cloud.google.com/results/invocations/fe90a7a5-508a-46db-bc85-4f102c14bb96/targets/github%2Fpython-docs-samples%2Fvision%2Fautoml/tests]

Log:

[Buildcop] Sending logs to Build Cop Bot...
[Buildcop] See https://github.com/googleapis/repo-automation-bots/tree/master/packages/buildcop.
[Buildcop] Processing sponge_log.xml
[Buildcop] Success! ID=1203126179949932
[Buildcop] Done!