testListAllIsSorted failures for SMB with a particular seed

[droberts195]

The following tests fail with a specific seed:

org.elasticsearch.index.store.SmbMMapDirectoryTests testListAllIsSorted
org.elasticsearch.index.store.SmbSimpleFSDirectoryTests testListAllIsSorted

The full log is here: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-windows-compatibility/219/console

The repro commands are:

gradle :plugins:store-smb:test -Dtests.seed=CD2A79A33FDBCDD8 -Dtests.class=org.elasticsearch.index.store.SmbSimpleFSDirectoryTests -Dtests.method="testListAllIsSorted" -Dtests.security.manager=true -Dtests.locale=ar-MA -Dtests.timezone=Asia/Ulaanbaatar
gradle :plugins:store-smb:test -Dtests.seed=CD2A79A33FDBCDD8 -Dtests.class=org.elasticsearch.index.store.SmbMMapDirectoryTests -Dtests.method="testListAllIsSorted" -Dtests.security.manager=true -Dtests.locale=ar-QA -Dtests.timezone=NST

The CI build ran on either Windows 10 or Windows Server 2016. I managed to reproduce both failures on Windows Server 2012r2.

The problem is that the Java security manager denies access to a file:

ERROR   0.05s J0 | SmbMMapDirectoryTests.testListAllIsSorted <<< FAILURES!
   > Throwable #1: java.security.AccessControlException: access denied ("java.io.FilePermission" "C:\Users\jenkins\workspace\elastic+elasticsearch+master+multijob-windows-compatibility\plugins\store-smb\build\testrun\test\J0\temp\org.elasticsearch.index.store.SmbMMapDirectoryTests_CD2A79A33FDBCDD8-001\tempDir-006\con" "read")
   > 	at __randomizedtesting.SeedInfo.seed([CD2A79A33FDBCDD8:5706DED1FFE44251]:0)
   > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
   > 	at java.security.AccessController.checkPermission(AccessController.java:884)
   > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   > 	at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
   > 	at sun.nio.fs.WindowsChannelFactory.open(WindowsChannelFactory.java:293)
   > 	at sun.nio.fs.WindowsChannelFactory.newFileChannel(WindowsChannelFactory.java:162)

However, similar commands without the seed work fine, for example:

gradle :plugins:store-smb:test -Dtests.class=org.elasticsearch.index.store.SmbSimpleFSDirectoryTests -Dtests.method="testListAllIsSorted" -Dtests.security.manager=true

So it's a seed-specific failure.

[droberts195]

@tlrx I have been told you know about this plugin, so please could you take a look or reassign to someone else if I've been misinformed.

[dadoonet]

Actually @tlrx :)

[droberts195]

Oops, sorry. I corrected the comment above.

[cbuescher]

I'm able to reproduce this on a virtual Windows 10 box, however the error looks a bit different (probably because using java 9.0.4). For

gradle :plugins:store-smb:test -Dtests.seed=CD2A79A33FDBCDD8 -Dtests.class=org.elasticsearch.index.store.SmbMMapDirectoryTests -Dtests.method="testListAllIsSorted" -Dtests.security.manager=true -Dtests.locale=ar-QA -Dtests.timezone=NST

I get

ERROR   0.50s | SmbMMapDirectoryTests.testListAllIsSorted <<< FAILURES!
   > Throwable #1: java.nio.file.FileSystemException: C:\Users\vagrant\elasticsearch\plugins\store-smb\build\testrun\test\J0\temp\org.elasticsearch.index.store.SmbMMapDirectoryTests_CD2A79A33FDBCDD8-001\tempDir-001\con: The parameter is incorrect.
   >    at __randomizedtesting.SeedInfo.seed([CD2A79A33FDBCDD8:5706DED1FFE44251]:0)
   >    at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
   >    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
   >    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
   >    at java.base/sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:231)
   >    at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)
   >    at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)
   >    at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)
   >    at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:240)
   >    at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)
   >    at org.apache.lucene.mockfile.HandleTrackingFS.newByteChannel(HandleTrackingFS.java:240)
   >    at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)
   >    at java.base/java.nio.file.Files.newByteChannel(Files.java:369)
   >    at java.base/java.nio.file.Files.newByteChannel(Files.java:415)
   >    at org.elasticsearch.index.store.SmbDirectoryWrapper$SmbFSIndexOutput.<init>(SmbDirectoryWrapper.java:63)
   >    at org.elasticsearch.index.store.SmbDirectoryWrapper.createOutput(SmbDirectoryWrapper.java:52)
   >    at org.elasticsearch.index.store.SmbMMapDirectoryTests.testListAllIsSorted(SmbMMapDirectoryTests.java:58)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   >    at java.base/java.lang.Thread.run(Thread.java:844)
  2> NOTE: leaving temporary files on disk at: C:\Users\vagrant\elasticsearch\plugins\store-smb\build\testrun\test\J0\temp\org.elasticsearch.index.store.SmbMMapDirectoryTests_CD2A79A33FDBCDD8-001
  2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1629, maxMBSortInHeap=6.035814031390048, sim=RandomSimilarity(queryNorm=false): {}, locale=ar-QA, timezone=NST
  2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 9.0.4 (64-bit)/cpus=2,threads=1,free=518003192,total=536870912
  2> NOTE: All tests run in this JVM: [SmbMMapDirectoryTests]

And similarly for

gradle :plugins:store-smb:test -Dtests.seed=CD2A79A33FDBCDD8 -Dtests.class=org.elasticsearch.index.store.SmbSimpleFSDirectoryTests -Dtests.method="testListAllIsSorted" -Dtests.security.manager=true -Dtests.locale=ar-MA -Dtests.timezone=Asia/Ulaanbaatar

 > Throwable #1: java.nio.file.FileSystemException: C:\Users\vagrant\elasticsearch\plugins\store-smb\build\testrun\test\J0\temp\org.elasticsearch.index.store.SmbSimpleFSDirectoryTests_CD2A79A33FDBCDD8-001\tempDir-001\con: The parameter is incorrect.
   >    at __randomizedtesting.SeedInfo.seed([CD2A79A33FDBCDD8:5706DED1FFE44251]:0)
   >    at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
   >    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
   >    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
   >    at java.base/sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:231)
   >    at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)
   >    at org.apache.lucene.mockfile.FilterFileSystemProvider.newByteChannel(FilterFileSystemProvider.java:212)
   >    at java.base/java.nio.file.Files.newByteChannel(Files.java:369)
   >    at java.base/java.nio.file.Files.newByteChannel(Files.java:415)
   >    at org.elasticsearch.index.store.SmbDirectoryWrapper$SmbFSIndexOutput.<init>(SmbDirectoryWrapper.java:63)
   >    at org.elasticsearch.index.store.SmbDirectoryWrapper.createOutput(SmbDirectoryWrapper.java:52)
   >    at org.apache.lucene.store.BaseDirectoryTestCase.testListAllIsSorted(BaseDirectoryTestCase.java:1198)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   >    at java.base/java.lang.Thread.run(Thread.java:844)
  2> NOTE: leaving temporary files on disk at: C:\Users\vagrant\elasticsearch\plugins\store-smb\build\testrun\test\J0\temp\org.elasticsearch.index.store.SmbSimpleFSDirectoryTests_CD2A79A33FDBCDD8-001
  2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=348, maxMBSortInHeap=6.647405339044713, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-MA, timezone=Asia/Ulaanbaatar
  2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 9.0.4 (64-bit)/cpus=2,threads=1,free=518777344,total=536870912
  2> NOTE: All tests run in this JVM: [SmbSimpleFSDirectoryTests]

So no direct mentioning of a java.security.AccessControlException here, although this could get wrapped somewhere. Since this fails only with particular seeds, I looked at what gets randomized in the underlying Lucene BaseDirectoryTestCase#testListAllIsSorted. Its mostly a list of names that looks unsuspicios when printed with the above seeds:

1> tweoxp
1> grmrcmuqb
1> kmcn
1> x
1> npravrbm
1> t
1> eao
1> ad
1> rp
1> szjhatc
1> vmuyrh
1> icdmuhs
1> zakohug
1> zknwfn
1> crew
1> kk
1> xuuhorbl
1> m
1> neui
1> con

The test creates either tmp-files (with the name as prefix and a certain suffix) or simply files with the given name and checks that they exist afterwards. The last name seems to cause trouble when creating the output, but its unclear to me so far why none of the earlier cases triggers a similar problem.

[cbuescher]

@romseygeek pointed me to this list of forbidden Windows 10 file names which con is on. So this issue seems to be a Windows only test issue that should only occur rarely. Since the randomization happens in Lucenes BaseDirectoryTestCase, this is probably also the place to fix this since it seems to be a more general problem with directory testing.

[cbuescher]

The underlying issue is tracked in https://issues.apache.org/jira/browse/LUCENE-8163

[jasontedor]

Closing as this is tracked in the upstream Lucene issue.

[cbuescher]

Looks like https://issues.apache.org/jira/browse/LUCENE-8163 made it into Lucene 7.3 which was just merged to master and 6.3 (#29387). Just mentioning it here for future reference.