Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TEST] SecurityRealmSettingsTests failing on Windows #44942

Closed
ywelsch opened this issue Jul 29, 2019 · 8 comments · Fixed by #48666
Closed

[TEST] SecurityRealmSettingsTests failing on Windows #44942

ywelsch opened this issue Jul 29, 2019 · 8 comments · Fixed by #48666
Assignees
@jkakavas
Labels
:Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) >test-failure Triaged test failures from CI

Comments

@ywelsch
Copy link
Contributor

ywelsch commented Jul 29, 2019

Build scan: https://scans.gradle.com/s/o3smubtjyn4gu

Relevant output:

org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests > classMethod FAILED
    java.io.IOException: Could not remove the following files (in the order of attempts):
       C:\Users\jenkins\workspace\elastic+elasticsearch+7.2+multijob-windows-compatibility\os\windows-2012-r2\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_E46537D453124597-001\jwkset-005json: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.2+multijob-windows-compatibility\os\windows-2012-r2\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_E46537D453124597-001\jwkset-005json: The process cannot access the file because it is being used by another process.

       C:\Users\jenkins\workspace\elastic+elasticsearch+7.2+multijob-windows-compatibility\os\windows-2012-r2\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_E46537D453124597-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.2+multijob-windows-compatibility\os\windows-2012-r2\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_E46537D453124597-001
@ywelsch ywelsch added >test-failure Triaged test failures from CI :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) labels Jul 29, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security

@jkakavas jkakavas self-assigned this Jul 29, 2019
@jkakavas
Copy link
Member

Seems similar to #44006 and #44552

@droberts195
Copy link
Contributor

This happened again in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.3+multijob-windows-compatibility/os=windows-2016/69/console:

2> java.io.IOException: Could not remove the following files (in the order of attempts):
     C:\Users\jenkins\workspace\elastic+elasticsearch+7.3+multijob-windows-compatibility\os\windows-2016\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_C3CCAFF8D4417098-001\jwkset-001json: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.3+multijob-windows-compatibility\os\windows-2016\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_C3CCAFF8D4417098-001\jwkset-001json: The process cannot access the file because it is being used by another process.

Seems similar to #44006 and #44552

Maybe #44006 is a bit different, because that's "directory not empty" and probably caused by async code creating a file in a directory during the recursive deletion of the temporary directory hierarchy after the loop thinks it's deleted all the files in the directory.

I agree this is similar to #44552 in that both are breaches of the Windows rule that you can't delete an open file (unless it was specifically opened in a mode that allows it to be deleted while open). However, unfortunately every place where we're deleting open files could require a separate code change to fix - it's not like there's a single magic line of code we can change to make all these Windows problems go away.

@tvernum
Copy link
Contributor

tvernum commented Sep 4, 2019

Another failure
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-windows-compatibility/os=windows-2019/110/console

java.io.IOException
: 
Could not remove the following files (in the order of attempts):
   C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2019\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_4CD7B58A56A26CBC-001\jwkset-001json: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2019\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_4CD7B58A56A26CBC-001\jwkset-001json: The process cannot access the file because it is being used by another process.


   C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2019\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_4CD7B58A56A26CBC-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\elastic+elasticsearch+7.x+multijob-windows-compatibility\os\windows-2019\x-pack\plugin\security\build\testrun\test\temp\org.elasticsearch.xpack.security.authc.SecurityRealmSettingsTests_4CD7B58A56A26CBC-001
at __randomizedtesting.SeedInfo.seed([4CD7B58A56A26CBC]:0)
at org.apache.lucene.util.IOUtils.rm(IOUtils.java:319)
at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)

@alpar-t
Copy link
Contributor

alpar-t commented Sep 10, 2019

Another one: https://gradle-enterprise.elastic.co/s/b25nyp3zemcxc/tests/g5u3xltc4r2du-tn72fxt3goumw

alpar-t added a commit that referenced this issue Sep 10, 2019
@alpar-t
mute on master fo windows
db23b1c 
Tracking #44942
alpar-t added a commit that referenced this issue Sep 10, 2019
@alpar-t
mute on 7.4 fo windows
6bb963b 
Tracking #44942
alpar-t added a commit that referenced this issue Sep 10, 2019
@alpar-t
mute on 7.x fo windows
b40ac6d 
Tracking #44942
@romseygeek
Copy link
Contributor

This has failed again on the 7.5 branch on Windows, which is deeply weird because it ought to be muted - the 7.5 branch has an assumeFalse check for Constants.WINDOWS:

elasticsearch/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authc/SecurityRealmSettingsTests.java

Line 131 in c39c1f4

assumeFalse("https://github.com/elastic/elasticsearch/issues/44942", Constants.WINDOWS);

@romseygeek
Copy link
Contributor

Relevant build scan: https://gradle-enterprise.elastic.co/s/y56ncmvtmm46q/

@alpar-t
Copy link
Contributor

alpar-t commented Oct 21, 2019 

at __randomizedtesting.SeedInfo.seed([171B268676DCC25C]:0)
at org.apache.lucene.util.IOUtils.rm(IOUtils.java:319)
at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)

Looks like the failure is coming from a cleanup rule that is executed despite of the test method being skipped. We could AFAIK skip the suite by moving the assumeFalse in a @BeforeClass

bizybot pushed a commit to bizybot/elasticsearch that referenced this issue Oct 30, 2019
Fix to release system resource after reading JKWSet file
0c8d8f3 
When we load a JSON Web Key (`JWKSet`) from the specified
file using `JWKSet.load` it internally uses `IOUtils.readFileToString`
but the opened `FileInputStream` is never closed after usage.
https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/342

This commit reads the file and parses the `JWKSet` from the string.
This also fixes an issue wherein if the underlying file changed,
for every change event it would add another file watcher. The
change is to only add the file watcher at the start.

Closes elastic#44942
@bizybot bizybot mentioned this issue Oct 30, 2019
Merged
bizybot added a commit that referenced this issue Oct 30, 2019
@bizybot
Fix to release system resource after reading JKWSet file (#48666)
678492d 
When we load a JSON Web Key (JWKSet) from the specified
file using JWKSet.load it internally uses IOUtils.readFileToString
but the opened FileInputStream is never closed after usage.
https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/342

This commit reads the file and parses the JWKSet from the string.

This also fixes an issue wherein if the underlying file changed,
for every change event it would add another file watcher. The
change is to only add the file watcher at the start.

Closes #44942
bizybot added a commit to bizybot/elasticsearch that referenced this issue Oct 30, 2019
@bizybot
Fix to release system resource after reading JKWSet file (elastic#48666)
ce8a3f5 
When we load a JSON Web Key (JWKSet) from the specified
file using JWKSet.load it internally uses IOUtils.readFileToString
but the opened FileInputStream is never closed after usage.
https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/342

This commit reads the file and parses the JWKSet from the string.

This also fixes an issue wherein if the underlying file changed,
for every change event it would add another file watcher. The
change is to only add the file watcher at the start.

Closes elastic#44942
@bizybot bizybot mentioned this issue Oct 30, 2019
Merged
bizybot added a commit to bizybot/elasticsearch that referenced this issue Oct 30, 2019
@bizybot
Fix to release system resource after reading JKWSet file (elastic#48666)
752e2e5 
When we load a JSON Web Key (JWKSet) from the specified
file using JWKSet.load it internally uses IOUtils.readFileToString
but the opened FileInputStream is never closed after usage.
https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/342

This commit reads the file and parses the JWKSet from the string.

This also fixes an issue wherein if the underlying file changed,
for every change event it would add another file watcher. The
change is to only add the file watcher at the start.

Closes elastic#44942
@bizybot bizybot mentioned this issue Oct 30, 2019
Merged
bizybot added a commit to bizybot/elasticsearch that referenced this issue Oct 30, 2019
@bizybot
Fix to release system resource after reading JKWSet file (elastic#48666)
e155e8a 
When we load a JSON Web Key (JWKSet) from the specified
file using JWKSet.load it internally uses IOUtils.readFileToString
but the opened FileInputStream is never closed after usage.
https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/342

This commit reads the file and parses the JWKSet from the string.

This also fixes an issue wherein if the underlying file changed,
for every change event it would add another file watcher. The
change is to only add the file watcher at the start.

Closes elastic#44942
@bizybot bizybot mentioned this issue Oct 30, 2019
Merged
bizybot added a commit that referenced this issue Oct 30, 2019
@bizybot
Fix to release system resource after reading JKWSet file (#48666) (#4…
016b204 
…8679)

When we load a JSON Web Key (JWKSet) from the specified
file using JWKSet.load it internally uses IOUtils.readFileToString
but the opened FileInputStream is never closed after usage.
https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/342

This commit reads the file and parses the JWKSet from the string.

This also fixes an issue wherein if the underlying file changed,
for every change event it would add another file watcher. The
change is to only add the file watcher at the start.

Closes #44942
bizybot added a commit that referenced this issue Oct 30, 2019
@bizybot
Fix to release system resource after reading JKWSet file (#48666) (#4…
547a3cf 
…8678)

When we load a JSON Web Key (JWKSet) from the specified
file using JWKSet.load it internally uses IOUtils.readFileToString
but the opened FileInputStream is never closed after usage.
https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/342

This commit reads the file and parses the JWKSet from the string.

This also fixes an issue wherein if the underlying file changed,
for every change event it would add another file watcher. The
change is to only add the file watcher at the start.

Closes #44942
bizybot added a commit that referenced this issue Oct 30, 2019
@bizybot
Fix to release system resource after reading JKWSet file (#48666) (#4…
c7342dd 
…8677)

When we load a JSON Web Key (JWKSet) from the specified
file using JWKSet.load it internally uses IOUtils.readFileToString
but the opened FileInputStream is never closed after usage.
https://bitbucket.org/connect2id/nimbus-jose-jwt/issues/342

This commit reads the file and parses the JWKSet from the string.

This also fixes an issue wherein if the underlying file changed,
for every change event it would add another file watcher. The
change is to only add the file watcher at the start.

Closes #44942
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jkakavas jkakavas

Labels
:Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix to release system resource after reading JKWSet file bizybot/elasticsearch
7 participants
@romseygeek @tvernum @alpar-t @ywelsch @droberts195 @jkakavas @elasticmachine