CI jobs hanging

[phil-davis]

These CI jobs are hanging:
https://drone.owncloud.com/owncloud/files_primary_s3/1305/20/18

  Scenario: restoring file to old version changes the content of the file                                   # /var/www/owncloud/testrunner/tests/acceptance/features/webUIFiles/versions.feature:25
    Given user "user0" has uploaded file with content "lorem content" to "/randomfile.txt"                  # FeatureContext::userHasUploadedAFileWithContentTo()
    And user "user0" has logged in using the webUI                                                          # WebUILoginContext::theUserLogsInUsingTheWebUI()
    And the user has browsed to the files page                                                              # WebUIFilesContext::theUserBrowsesToTheFilesPage()
    And user "user0" has uploaded file with content "new lorem content" to "/randomfile.txt"                # FeatureContext::userHasUploadedAFileWithContentTo()
    When the user browses directly to display the "versions" details of file "randomfile.txt" in folder "/" # WebUIFilesContext::theUserBrowsesDirectlyToDetailsTabOfFileInFolder()
    And the user restores the file to last version using the webUI                                          # WebUIFilesContext::theUserRestoresTheFileToLastVersionUsingTheWebui()

The output hangs at that point.

https://drone.owncloud.com/owncloud/files_primary_s3/1305/60/16

  Scenario: Restore a file and check, if the content is now in the current file               # /var/www/owncloud/testrunner/tests/acceptance/features/apiVersions/fileVersions.feature:58
    Given user "user0" has uploaded file with content "123" to "/davtest.txt"                 # FeatureContext::userHasUploadedAFileWithContentTo()
    And user "user0" has uploaded file with content "12345" to "/davtest.txt"                 # FeatureContext::userHasUploadedAFileWithContentTo()
    And the version folder of file "/davtest.txt" for user "user0" should contain "1" element # FilesVersionsContext::theVersionFolderOfFileShouldContainElements()
    When user "user0" restores version index "1" of file "/davtest.txt" using the WebDAV API  # FilesVersionsContext::userRestoresVersionIndexOfFile()

The output hangs at that point.

[phil-davis]

https://drone.owncloud.com/owncloud/files_primary_s3/1210
CI passed the nightly on 2019-10-10 05:55am
https://drone.owncloud.com/owncloud/files_primary_s3/1210/17/20

  Background:                                                                               # /drone/server/tests/acceptance/features/webUIFiles/versions.feature:9
    Given these users have been created with default attributes and without skeleton files: # FeatureContext::theseUsersHaveBeenCreatedWithDefaultAttributesAndWithoutSkeletonFiles()
      | username |
      | user0    |

  Scenario: restoring file to old version changes the content of the file                                   # /drone/server/tests/acceptance/features/webUIFiles/versions.feature:25
    Given user "user0" has uploaded file with content "lorem content" to "/randomfile.txt"                  # FeatureContext::userHasUploadedAFileWithContentTo()
    And user "user0" has logged in using the webUI                                                          # WebUILoginContext::theUserLogsInUsingTheWebUI()
    And the user has browsed to the files page                                                              # WebUIFilesContext::theUserBrowsesToTheFilesPage()
    And user "user0" has uploaded file with content "new lorem content" to "/randomfile.txt"                # FeatureContext::userHasUploadedAFileWithContentTo()
    When the user browses directly to display the "versions" details of file "randomfile.txt" in folder "/" # WebUIFilesContext::theUserBrowsesDirectlyToDetailsTabOfFileInFolder()
    And the user restores the file to last version using the webUI                                          # WebUIFilesContext::theUserRestoresTheFileToLastVersionUsingTheWebui()
    Then the content of file "randomfile.txt" for user "user0" should be "lorem content"                    # FeatureContext::contentOfFileForUserShouldBe()
SCENARIO RESULT: (pass)

https://drone.owncloud.com/owncloud/files_primary_s3/1210/45/18

  Scenario: Restore a file and check, if the content is now in the current file               # /drone/server/tests/acceptance/features/apiVersions/fileVersions.feature:58
    Given user "user0" has uploaded file with content "123" to "/davtest.txt"                 # FeatureContext::userHasUploadedAFileWithContentTo()
    And user "user0" has uploaded file with content "12345" to "/davtest.txt"                 # FeatureContext::userHasUploadedAFileWithContentTo()
    And the version folder of file "/davtest.txt" for user "user0" should contain "1" element # FilesVersionsContext::theVersionFolderOfFileShouldContainElements()
    When user "user0" restores version index "1" of file "/davtest.txt" using the WebDAV API  # FilesVersionsContext::userRestoresVersionIndexOfFile()
    Then the content of file "/davtest.txt" for user "user0" should be "123"                  # FeatureContext::contentOfFileForUserShouldBe()

[phil-davis]

2019-10-10 09:57 https://drone.owncloud.com/owncloud/files_primary_s3/1211 also passed
That was PR #251 "Initial drone starlark" with core webUI and API acceptance tests split into 20 parts each.

The merge CI for that has all the pipelines green, but the overall result is red???
https://drone.owncloud.com/owncloud/files_primary_s3/1212

2019-10-10 08:42 https://drone.owncloud.com/owncloud/files_primary_s3/1218 passed
e.g. https://drone.owncloud.com/owncloud/files_primary_s3/1218/60/16 /var/www/owncloud/testrunner/tests/acceptance/features/apiVersions/fileVersions.feature:58 ran fine.
That was PR #253 "Update drone starlark". core webUI and API acceptance tests are split into 25 parts each. It was not merged that day.

2019-10-11 05:56 https://drone.owncloud.com/owncloud/files_primary_s3/1219 was the nightly CI
That just had the scality S3 multi-bucket unit test fails that were a separate issue. (core webUI and API tests in 20 parts each)
https://drone.owncloud.com/owncloud/files_primary_s3/1219/50/16 apiVersions/fileVersions.feature:58 passed (all acceptance tests passed)

2019-10-11 07:32 https://drone.owncloud.com/owncloud/files_primary_s3/1220 was the merge CI of PR #253 (core webUI and API tests in 25 parts each)
All acceptance tests passed (it just had the scality S3 multi-bucket unit test fails)

2019-10-12 05:56 https://drone.owncloud.com/owncloud/files_primary_s3/1222/60/16 was the next nightly. (core webUI and API tests in 20 parts each)
That is when the 2 acceptance test pipelines started hanging.

So "something happened" between the CI running for the merge at 2019-10-11 07:32 and the next nightly at 2019-10-12 05:56

[phil-davis]

The things merged to core master on 2019-10-11 were only:
a) owncloud/core#36263 Updating sabre/http (5.0.2 => 5.0.4) 2019-10-11
b) owncloud/core#36222 only collect shares inside the given path in transfer-ownership command 2019-10-10

I have tried reversing (a) and that does not fix the hangs.

(b) could be an issue? It was merged the day before, but if the daily-master-qa tarball build failed on 2019-10-10 then a problem would not have been noticed then.

[phil-davis]

"fixed" by owncloud/core#36470 - downgrading sabre/http from 5.0.4 to 5.0.2

See #274 for the follow-on issue to really investigate the hanging.