Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AzureBlobStoreRepositoryTests.testMultipleSnapshotAndRollback failing #47120

Closed
ywelsch opened this issue Sep 25, 2019 · 4 comments · Fixed by #47176 or #48520
Closed

AzureBlobStoreRepositoryTests.testMultipleSnapshotAndRollback failing #47120

ywelsch opened this issue Sep 25, 2019 · 4 comments · Fixed by #47176 or #48520
Assignees
@tlrx
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI

Comments

@ywelsch
Copy link
Contributor

ywelsch commented Sep 25, 2019

A NullpointerException in the SDK

1> Caused by: java.io.IOException: Can not write blob __sp4a6K8pRL6XeEOszzlrNw
  1> 	at org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:104) ~[main/:?]
  1> 	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:1343) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$2.doRun(BlobStoreRepository.java:1105) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	... 5 more
  1> Caused by: com.microsoft.azure.storage.StorageException: Internal Server Error
  1> 	at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87) ~[azure-storage-8.4.0.jar:?]
  1> 	at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:220) ~[azure-storage-8.4.0.jar:?]
  1> 	at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadFullBlob(CloudBlockBlob.java:1035) ~[azure-storage-8.4.0.jar:?]
  1> 	at com.microsoft.azure.storage.blob.CloudBlockBlob.upload(CloudBlockBlob.java:864) ~[azure-storage-8.4.0.jar:?]
  1> 	at com.microsoft.azure.storage.blob.CloudBlockBlob.upload(CloudBlockBlob.java:743) ~[azure-storage-8.4.0.jar:?]
  1> 	at org.elasticsearch.repositories.azure.AzureStorageService.lambda$writeBlob$15(AzureStorageService.java:332) ~[main/:?]
  1> 	at org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:64) ~[main/:?]
  1> 	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
  1> 	at org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:63) ~[main/:?]
  1> 	at org.elasticsearch.repositories.azure.AzureStorageService.writeBlob(AzureStorageService.java:331) ~[main/:?]
  1> 	at org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:120) ~[main/:?]
  1> 	at org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:102) ~[main/:?]
  1> 	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:1343) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$2.doRun(BlobStoreRepository.java:1105) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	... 5 more
  1> Caused by: java.lang.NullPointerException
  1> 	at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:202) ~[azure-storage-8.4.0.jar:?]
  1> 	at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadFullBlob(CloudBlockBlob.java:1035) ~[azure-storage-8.4.0.jar:?]
  1> 	at com.microsoft.azure.storage.blob.CloudBlockBlob.upload(CloudBlockBlob.java:864) ~[azure-storage-8.4.0.jar:?]
  1> 	at com.microsoft.azure.storage.blob.CloudBlockBlob.upload(CloudBlockBlob.java:743) ~[azure-storage-8.4.0.jar:?]
  1> 	at org.elasticsearch.repositories.azure.AzureStorageService.lambda$writeBlob$15(AzureStorageService.java:332) ~[main/:?]
  1> 	at org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:64) ~[main/:?]
  1> 	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
  1> 	at org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:63) ~[main/:?]
  1> 	at org.elasticsearch.repositories.azure.AzureStorageService.writeBlob(AzureStorageService.java:331) ~[main/:?]
  1> 	at org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:120) ~[main/:?]
  1> 	at org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:102) ~[main/:?]
  1> 	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.snapshotFile(BlobStoreRepository.java:1343) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.repositories.blobstore.BlobStoreRep

Build scan: https://gradle-enterprise.elastic.co/s/43fhrshn37tsc
@ywelsch ywelsch added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Sep 25, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@tlrx tlrx mentioned this issue Sep 26, 2019
Merged
tlrx added a commit that referenced this issue Sep 27, 2019
@tlrx
Injected response errors in Azure repository tests should have a body (…
c5b9cda 
…#47176)

The Azure SDK client expects server errors to have a body, 
something that looks like:

<?xml version="1.0" encoding="utf-8"?>  
<Error>  
  <Code>string-value</Code>  
  <Message>string-value</Message>  
</Error> 

I've forgot to add such errors in Azure tests and that triggers 
some NPE in the client like the one reported in #47120.

Closes #47120
tlrx added a commit that referenced this issue Sep 27, 2019
@tlrx
Injected response errors in Azure repository tests should have a body (…
42ae76a 
…#47176)

The Azure SDK client expects server errors to have a body, 
something that looks like:

<?xml version="1.0" encoding="utf-8"?>  
<Error>  
  <Code>string-value</Code>  
  <Message>string-value</Message>  
</Error> 

I've forgot to add such errors in Azure tests and that triggers 
some NPE in the client like the one reported in #47120.

Closes #47120
@tlrx
Copy link
Member

tlrx commented Oct 25, 2019

This failed again on 7.5:
https://gradle-enterprise.elastic.co/s/guxpi3sxg6sm2
https://gradle-enterprise.elastic.co/s/jw7r4xc6mmwbe/tests/lgycqkw2bfw46-clvf2uhrzpiw4

@tlrx tlrx reopened this Oct 25, 2019
This was referenced Oct 25, 2019
Closed
Merged
tlrx added a commit that referenced this issue Oct 25, 2019
@tlrx
Add missing azure error code (#48520)
1998a10 
In #47176 we changed the internal HTTP server that emulates 
the Azure Storage service so that it includes a response body 
for injected errors. This fixed most of the issues reported in 
#47120 but sadly I missed to map one error to its Azure 
equivalent, and it triggered some CI failures today.

Closes #47120
tlrx added a commit that referenced this issue Oct 25, 2019
@tlrx
Add missing azure error code (#48520)
06d2cc5 
In #47176 we changed the internal HTTP server that emulates 
the Azure Storage service so that it includes a response body 
for injected errors. This fixed most of the issues reported in 
#47120 but sadly I missed to map one error to its Azure 
equivalent, and it triggered some CI failures today.

Closes #47120
tlrx added a commit that referenced this issue Oct 25, 2019
@tlrx
Add missing azure error code (#48520)
7dc72ea 
In #47176 we changed the internal HTTP server that emulates 
the Azure Storage service so that it includes a response body 
for injected errors. This fixed most of the issues reported in 
#47120 but sadly I missed to map one error to its Azure 
equivalent, and it triggered some CI failures today.

Closes #47120
@mark-vieira
Copy link
Contributor

I encountered another instance of this on 7.x:

https://gradle-enterprise.elastic.co/s/42cmniylhr7g4/tests/lgycqkw2bfw46-clvf2uhrzpiw4

@mark-vieira mark-vieira reopened this Nov 11, 2019
@costin costin mentioned this issue Nov 12, 2019
Closed
@tlrx
Copy link
Member

tlrx commented Nov 12, 2019

Thanks @mark-vieira. The failure you mentioned is a bit different from the original one, so I'm closing this in favor of #48978

@tlrx tlrx closed this as completed Nov 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tlrx tlrx

Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
4 participants
@tlrx @ywelsch @mark-vieira @elasticmachine