[META][Windows CI] Getting windows gradle check to green

[Poojita-Raj]

Coming from #1448, here is the meta issue tracking all failing tasks/tests on windows gradle check that need to be resolved in order to get gradle check to green.

• Windows gradle check failure - checkExtraction tasks failing #4925
• checkLicense tasks failing #4939
• checkNotice tasks failing #4940
• ResourceAwareTasksTests - these just hang for me in intellij.
• HierarchyCircuitBreakerServiceTests.testFallbackG1RegionSize
• PluginsServiceTests.testPluginLoadFailure & PluginsServiceTests.testPluginFromParentClassLoader
• ShardIndexingPressureConcurrentExecutionTests
• :build-tools:integTest - symlink tests failing for non-admin user

Cannot run opensearch as root - Grouping failures caused by running as admin:
Should be fixed on reverting #4656

[mch2]

Running as Administrator is blocked on windows with #4656. Enforced here. There are a few tests that create symlinks that fail example
without admin privileges. I bypassed this check and get the following failures in :server:test. (re-ran a few times to fliter out flaky tests)

HierarchyCircuitBreakerServiceTests.testFallbackG1RegionSize
PluginsServiceTests.testPluginLoadFailure
PluginsServiceTests.testPluginFromParentClassLoader

I ran everything in gradlew internalClusterTest - 2,815 passed, 30 ignored.

[vibrantvarun]

@mch2 @Poojita-Raj Docker tests and running gradle check with admin

[mch2]

Listing project wise stats here, ignored means it is already ignored, not a new ignore.

WITH admin.

rest-api-spec - 1,152 passed, 27 ignored.
server:internalClusterTest - 2,606 passed 47 ignored.
server:test 11,249 passed, 33 ignored, 3 Failed (listed above)

[stephen-crawford]

@mch2 what was your process for bypassing the admin check if you don't mind me asking? Did you incorporate it into a script or did you manually run the tests and just not the check file?

[mch2]

Ran check from intellij with admin:
passed: 15,861
ignored: 113
Failed: 9

The 9 that fail are in ShardIndexingPressureConcurrentExecutionTests. These also fail from command line, added to list.

[mch2]

from command line this test passes:

./gradlew :server:test --tests "org.opensearch.indices.breaker.HierarchyCircuitBreakerServiceTests.testFallbackG1RegionSize"

It consistently fails from intellij.

[mch2]

@scrawfor99 for now i've simply bypassed the check in initializeNatives with Constants.WINDOWS == false. We need a better solution here but it helped identify tests that still don't pass even with admin.

Working on getting a list of failures without admin.

[owaiskazi19]

Tested PluginsServiceTests.testPluginLoadFailure & PluginsServiceTests.testPluginFromParentClassLoader with admin/non-admin users with @vibrantvarun and they passed.

[vibrantvarun]

C:\Users\Varun\project1\OpenSearch>gradlew ":server:test" --tests "org.opensearch.plugins.PluginsServiceTests.testPluginFromParentClassLoader"
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true

> Configure project :plugins:repository-hdfs
hdfsFixture unsupported, please set HADOOP_HOME and put HADOOP_HOME\bin in PATH

> Configure project :qa:os
Cannot add task 'destructiveDistroTest.docker' as a task with that name already exists.
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.5.1
  OS Info               : Windows Server 2022 10.0 (amd64)
  JDK Version           : 11 (Amazon Corretto JDK)
  JAVA_HOME             : C:\Program Files\Amazon Corretto\jdk11.0.16_9
  Random Testing Seed   : 28955231E4B19E1A
  In FIPS 140 mode      : false
=======================================

> Task :server:test
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=warn; support was removed in 17.0
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/C:/Users/Varun/project1/OpenSearch/test/framework/build/distributions/framework-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/C:/Users/Varun/.gradle/wrapper/dists/gradle-7.5.1-all/1ehga6e77gqps5uk2kc5kf1vc/gradle-7.5.1/lib/plugins/gradle-testing-base-7.5.1.jar)
WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

BUILD SUCCESSFUL in 14s
42 actionable tasks: 1 executed, 41 up-to-date 

gradlew ":server:test" --tests "org.opensearch.plugins.PluginsServiceTests.testPluginLoadFailure"
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
Starting a Gradle Daemon, 1 stopped Daemon could not be reused, use --status for details
cache fileHashes.bin (C:\Users\Varun\project1\OpenSearch\.gradle\7.5.1\fileHashes\fileHashes.bin) is corrupt. Discarding.

> Configure project :plugins:repository-hdfs
hdfsFixture unsupported, please set HADOOP_HOME and put HADOOP_HOME\bin in PATH

> Configure project :qa:os
Cannot add task 'destructiveDistroTest.docker' as a task with that name already exists.
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.5.1
  OS Info               : Windows Server 2022 10.0 (amd64)
  JDK Version           : 11 (Amazon Corretto JDK)
  JAVA_HOME             : C:\Program Files\Amazon Corretto\jdk11.0.16_9
  Random Testing Seed   : AA86F954220D0020
  In FIPS 140 mode      : false
=======================================

> Task :server:test
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=warn; support was removed in 17.0
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/C:/Users/Varun/project1/OpenSearch/test/framework/build/distributions/framework-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/C:/Users/Varun/.gradle/wrapper/dists/gradle-7.5.1-all/1ehga6e77gqps5uk2kc5kf1vc/gradle-7.5.1/lib/plugins/gradle-testing-base-7.5.1.jar)
WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

BUILD SUCCESSFUL in 40s
42 actionable tasks: 2 executed, 40 up-to-date

[Rishikesh1159]

Tested ShardIndexingPressureConcurrentExecutionTests in isolation. Looks like a flaky test as it was not failing. Marking it as done.
./gradlew :server:test --tests "org.opensearch.index.ShardIndexingPressureConcurrentExecutionTests.*"

[Poojita-Raj]

Failing for non - Administrator users:

Task :build-tools:integTest

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testTarDoNotPreserveFileTimestamps FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testTar FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testBZip2Tar FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testGZipTar FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testGZipTarDoNotPreserveFileTimestamps FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testBZip2TarDoNotPreserveFileTimestamps FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testBZip2Tar FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testBZip2TarDoNotPreserveFileTimestamps FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testTar FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testTarDoNotPreserveFileTimestamps FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testGZipTar FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testGZipTarDoNotPreserveFileTimestamps FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testBZip2Tar FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testTar FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testTarDoNotPreserveFileTimestamps FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testGZipTarDoNotPreserveFileTimestamps FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testBZip2TarDoNotPreserveFileTimestamps FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testGZipTar FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testGZipTarDoNotPreserveFileTimestamps FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testBZip2Tar FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testTarDoNotPreserveFileTimestamps FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testBZip2TarDoNotPreserveFileTimestamps FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testGZipTar FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

org.opensearch.gradle.tar.SymbolicLinkPreservingTarIT > testTar FAILED
java.nio.file.FileSystemException at SymbolicLinkPreservingTarIT.java:66

45 tests completed, 24 failed

Task :build-tools:integTest FAILED

[saratvemulapalli]

For Symlink tests here is the summary so far:
Symlink creation in windows needs elevated permissions: https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystemException.html
So we can solve the problem in 3 ways:

• Run as non-admin, instead of creating a symlink, copy the directory/content to the place we care about.
• Run as admin and enable symlink permissions(very likely through Java Options).
• Skip symlinks on windows.

I would prefer the first approach as we would still continue to do the tests with regular permissions. You can check for file permissions using @Before annotation for each test and if the permissions dont exist for creating a symlink you could copy the directory.
Here are few solutions which other repositories took:
golang/dep#773
srikanth-lingala/zip4j#278
pypa/flit#599
admb-project/admb#272

[dblock]

@saratvemulapalli The question is, do we support symlinks on windows? that's pretty unusual AFAIK, I would be fine with (3) as well

[mch2]

Dumping all Files.createSymbolicLink locations:

Src:

• SymbolicLinkPreservingUntarTransform.java - looks to be bypassed with [BUG] SymbolicLinkPreservingUntarTransform fails on Windows #1433

Test code:

• PidFileTests.java - testPidFile randomly creats a symlink - the test already catches and swallows the security exception.
• SymbolicLinkPreservingTarIT.java
• EvilSecurityTests.java - all symlink related tests here are skipped by assuming we are not on windows.
• PluginCliTests.java - test20SymlinkPluginsDir. This test passes from CLI because it is not getting executed. With ./gradlew :qa:os:test --tests=org.opensearch.packaging.test.PluginCliTests. I don't see this test run even after adding an Assert.fail(). I think it is because of this line. I also don't see this package listed in a recent passing CI check.

[mch2]

@dblock @saratvemulapalli I think we can safely go the skip route for SymbolicLinkPreservingTarIT because with gradle assemble we skip other distros on windows and only build a zip - have opened a PR here.

All other instances of createSymbolicLink are already skipped.

[mch2]

BwcTest is the only task within check consistently failing for me now - the following passes for me on main. (had to retry a few times to rule out flakiness).

./gradlew check -x bwcTest

BwcTest fails because. do not have windows builds for prior versions.

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':qa:full-cluster-restart:v2.0.0#upgradedClusterTest'.
> Could not resolve all task dependencies for configuration ':qa:full-cluster-restart:opensearch_distro_extracted_testclusters-qa-full-cluster-restart-v2.0.0-1-2.0.0-'.
   > Could not resolve opensearch-distribution:opensearch:2.0.0.
     Required by:
         project :qa:full-cluster-restart
      > Could not resolve opensearch-distribution:opensearch:2.0.0.
         > Could not get resource 'https://artifacts.opensearch.org/releases/core/opensearch/2.0.0/opensearch-min-2.0.0-windows-x64.zip'.
            > Could not HEAD 'https://artifacts.opensearch.org/releases/core/opensearch/2.0.0/opensearch-min-2.0.0-windows-x64.zip'. Received status code 403 from server: Forbidden
      > Could not resolve opensearch-distribution:opensearch:2.0.0.
         > Could not get resource 'https://artifacts.opensearch.org/release-candidates/core/opensearch/2.0.0/opensearch-min-2.0.0-windows-x64.zip'.
            > Could not HEAD 'https://artifacts.opensearch.org/release-candidates/core/opensearch/2.0.0/opensearch-min-2.0.0-windows-x64.zip'. Received status code 403 from server: Forbidden

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.

* Get more help at https://help.gradle.org

BUILD FAILED in 38s

[dblock]

@mch2 Is it worth trying to produce a 1-time windows reference build manually with an older version to ensure we exercise the upgrade path in tests before we ship anything?

[mch2]

I didn't realize bwcTest isn't actually fully executed within check, only a subset, from our docs:

When running ./gradlew check, minimal bwc checks are also run against compatible versions that are not yet released.

I was able to just get a full passing check on 2.4 so looking to see if that subset is sufficient. If not then yeah we should do a manual build and run all of bwcTest.

[mch2]

Looking at a successful check run on a 2.4 backport. The bwc tests are running against 1.3.7 and 2.3.1. The tests clone & compile the previous major & minor where running bwcTest alone does not for the additional versions it checks.

check runs bwcTestSnapshots (config).

./gradlew bwcTestSnapshot passes for me with 2.4.