Packaging test fails on fedora 27, centos 7 in verify that the documents are there after restart

[alpar-t]

Running
./gradlew :qa:vagrant:packagingTest -Pvagrant.boxes=fedora-27 Fails with:

ok 236 [UPGRADE] check elasticsearch version is version under test
not ok 237 [UPGRADE] verify that the documents are there after restart
# (in test file /project/build/packaging/bats/tests/80_upgrade.bats, line 111)
#   `curl -s localhost:9200/library/book/1?pretty | grep Elasticsearch' failed
ok 238 [UPGRADE] stop version under test
ok 239 [REINSTALL] install
ok 240 [REINSTALL] purge elasticsearch
ok 241 [REINSTALL] chown directories
ok 242 [REINSTALL] reinstall elasticsearch
ok 243 [REINSTALL] check ownership
REPRODUCE WITH: ./gradlew :qa:vagrant:vagrantFedora27#batsPackagingTest -Dtests.seed=858B8891E728527A

I got the same on centos-7 as well as fedora-28 ( from #30579 ).

[elasticmachine]

Pinging @elastic/es-core-infra

[jasontedor]

Since our packaging tests have been green for almost 10 straight days, I think this has to be an environmental issue. As such, I am going to close this. We can help you sort out what the issue might be via another channel.

[alpar-t]

On closer inspection, the real cause was this:

fedora-28:/project/build/packaging/archives$ sudo bats /project/build/packaging/bats/tests/80_upgrade.bats
 ✗ [UPGRADE] install old version
   (from function `install_package' in file /project/build/packaging/bats/utils/packages.bash, line 77,
    in test file /project/build/packaging/bats/tests/80_upgrade.bats, line 60)
     `install_package -v $(cat upgrade_from_version)' failed
   error: open of elasticsearch-oss-6.2.5-SNAPSHOT.rpm failed: No such file or directory

Indeed that RPM is not on the box ( only 6.3.0 is )

[alpar-t]

There seems to be a mismatch between the tests and the Gradle setup, as VagrantTestPlugin has this:

if (upgradeFromVersion.onOrAfter('6.3.0')) {
                project.dependencies.add(PACKAGING_CONFIGURATION,
                        "org.elasticsearch.distribution.${it}:elasticsearch-oss:${upgradeFromVersion}@${it}")
            }

so the elasticsearch**-oss** distribution is only added from 6.3.0 yet the test tried to install it for 6.2.5.

[alpar-t]

There's this logic that switches the package name in qa/vagrant/src/test/resources/packaging/tests/80_upgrade.bats:

    # TODO: this needs to conditionally change based on version > 6.3.0
    if [ -f upgrade_is_oss ]; then
      export PACKAGE_NAME="elasticsearch-oss"
    else
      skip "upgrade cannot happen from pre 6.3.0 to elasticsearch-oss"
    fi

The test pass on CI, but this test is skipped:

04:23:23 ok 228 # skip (upgrade cannot happen from pre 6.3.0 to elasticsearch-oss) [UPGRADE] install old version
04:23:23 ok 228 # skip (upgrade cannot happen from pre 6.3.0 to elasticsearch-oss) [UPGRADE] install old version

I get the same result from CI if I get the testing seed as above, or if I specify and older version:
../../gradlew clean packagingTest -Pvagrant.boxes=fedora-27 -Dtests.seed=FD628E5E9BAEE17
or
../../gradlew clean packagingTest -Pvagrant.boxes=fedora-28 -Dtests.packaging.upgradeVersion=6.2.5-SNAPSHOT
( the seed is used only if upgradeVersion is not passed in )

A difference between CI and how I ran it locally is that CI calls clean, whereas I did not when I ran into the issue first reported here. This plays an important role because createUpgradeIsOssFile uses onlyIf, so the file is not cleaned up if it shouldn't be there, it's left around if it was there previously.

For completeness this is how to reproduce the first issue I ran into:

../../gradlew clean createUpgradeIsOssFile -Pvagrant.boxes=fedora-27 -Dtests.packaging.upgradeVersion=6.3.0-SNAPSHOT
../../gradlew packagingTest -Pvagrant.boxes=fedora-27 -Dtests.packaging.upgradeVersion=6.2.5-SNAPSHOT

I will reopen the ticket to make some changes to createUpgradeIsOssFile so it isn't as sensitive to clean and hopefully save others some trouble.

Update: removed sections that described a failure of the tests as it turned out that was due to a local change.