Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for HTTP Proxies for the GCS repository #82737

Merged
merged 3 commits into from
Jan 20, 2022

Conversation

arteam
Copy link
Contributor

@arteam arteam commented Jan 18, 2022

The change adds 3 new client properties for the GCS repository:

  • gcs.client.default.proxy.type
  • gcs.client.default.proxy.host
  • gcs.client.default.proxy.port

They allow to configure a java.net.Proxy for the GCS SDK to use when communicating with the GCS API.

Resolves #82444

@arteam arteam added >enhancement :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.0.0 auto-backport Automatically create backport pull requests when merged labels Jan 18, 2022
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Jan 18, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

The change adds 3 new client properties for the GCS repository:

* gcs.client.default.proxy.type
* gcs.client.default.proxy.host
* gcs.client.default.proxy.port

They allow to configure a [java.net.Proxy](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/Proxy.html)
for the GCS SDK to use when communicating with the GCS API.

Resolves elastic#82444
@arteam
Copy link
Contributor Author

arteam commented Jan 20, 2022

@elasticmachine update branch

@arteam arteam requested a review from tlrx January 20, 2022 11:36
Copy link
Member

@tlrx tlrx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@arteam arteam merged commit 1ddaf25 into elastic:master Jan 20, 2022
@arteam
Copy link
Contributor Author

arteam commented Jan 20, 2022

Thanks Tanguy!

arteam added a commit to arteam/elasticsearch that referenced this pull request Jan 20, 2022
* Add support for HTTP Proxies for the GCS repository

The change adds 3 new client properties for the GCS repository:

* gcs.client.default.proxy.type
* gcs.client.default.proxy.host
* gcs.client.default.proxy.port

They allow to configure a [java.net.Proxy](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/Proxy.html)
for the GCS SDK to use when communicating with the GCS API.

Resolves elastic#82444
@elasticsearchmachine
Copy link
Collaborator

💚 Backport successful

Status Branch Result
8.0

@arteam arteam deleted the gcs-proxy branch January 20, 2022 14:53
arteam added a commit that referenced this pull request Jan 20, 2022
The change adds 3 new client properties for the GCS repository:

* gcs.client.default.proxy.type
* gcs.client.default.proxy.host
* gcs.client.default.proxy.port

They allow to configure a [java.net.Proxy](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/Proxy.html)
for the GCS SDK to use when communicating with the GCS API.

Resolves #82444
@sangeethdba
Copy link

does these settings work with elastic 6.8.2 or 7.x

@sangeethdba
Copy link

path: /_snapshot/backup, params: {pretty=, repository=backup}
org.elasticsearch.repositories.RepositoryException: [backup] cannot create blob store
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:328) ~[elasticsearch-6.8.2.jar:6.8.2]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:625) ~[elasticsearch-6.8.2.jar:6.8.2]
at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$2(RepositoriesService.java:230) ~[elasticsearch-6.8.2.jar:6.8.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.8.2.jar:6.8.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
Caused by: org.elasticsearch.common.blobstore.BlobStoreException: Unable to check if bucket [com-dbre-backup-np] exists
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:109) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.(GoogleCloudStorageBlobStore.java:75) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:137) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:47) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:324) ~[elasticsearch-6.8.2.jar:6.8.2]
... 6 more
Caused by: com.google.cloud.storage.StorageException: Error getting access token for service account:
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:226) ~[?:?]
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:403) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:196) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:193) ~[?:?]
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89) ~[?:?]
at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:192) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$doesBucketExist$0(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_322]
at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.(GoogleCloudStorageBlobStore.java:75) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:137) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:47) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:324) ~[elasticsearch-6.8.2.jar:6.8.2]
... 6 more
Caused by: java.io.IOException: Error getting access token for service account:
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:395) ~[?:?]
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:179) ~[?:?]
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:165) ~[?:?]
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96) ~[?:?]
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159) ~[?:?]
at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109) ~[?:?]
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469) ~[?:?]
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:400) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:196) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:193) ~[?:?]
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89) ~[?:?]
at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:192) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$doesBucketExist$0(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_322]
at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.(GoogleCloudStorageBlobStore.java:75) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:137) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:47) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:324) ~[elasticsearch-6.8.2.jar:6.8.2]
... 6 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_322]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_322]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_322]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_322]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_322]
at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_322]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:288) ~[?:?]
at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:203) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:189) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1342) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1317) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264) ~[?:?]
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77) ~[?:?]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981) ~[?:?]
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:393) ~[?:?]
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:179) ~[?:?]
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:165) ~[?:?]
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96) ~[?:?]
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159) ~[?:?]
at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109) ~[?:?]
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469) ~[?:?]
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:400) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:196) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:193) ~[?:?]
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89) ~[?:?]
at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:192) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$doesBucketExist$0(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_322]
at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.(GoogleCloudStorageBlobStore.java:75) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:137) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:47) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:324) ~[elasticsearch-6.8.2.jar:6.8.2]

@sangeethdba
Copy link

@tlrx @arteam

elasticsearchmachine pushed a commit that referenced this pull request Feb 15, 2022
Re-adds HTTP proxy settings for the GCS repository type. These settings
were added with #82737.

The docs were accidentally removed as part of
#82996.
elasticsearchmachine pushed a commit that referenced this pull request Feb 15, 2022
Re-adds HTTP proxy settings for the GCS repository type. These settings
were added with #82737.

The docs were accidentally removed as part of
#82996.

(cherry picked from commit e8b34c7)
elasticsearchmachine pushed a commit that referenced this pull request Feb 15, 2022
Re-adds HTTP proxy settings for the GCS repository type. These settings
were added with #82737.

The docs were accidentally removed as part of
#82996.

(cherry picked from commit e8b34c7)
jrodewig added a commit that referenced this pull request Feb 16, 2022
We've had a few users ask about support for HTTP proxy connections to GCS snapshot repositories in 7.x and 6.x.
However, client settings for those connections are only supported in 8.0+.

This adds a note to the 7.17 and 6.8 docs to nudge users to upgrade.

Relates to #82737

Closes #83959
elasticsearchmachine pushed a commit that referenced this pull request Feb 16, 2022
)

We've had a few users ask about support for HTTP proxy connections to GCS snapshot repositories in 7.x and 6.x.
However, client settings for those connections are only supported in 8.0+.

This adds a note to the 7.17 and 6.8 docs to nudge users to upgrade.

Relates to #82737

Closes #83959

(cherry picked from commit 99de635)
weizijun added a commit to weizijun/elasticsearch that referenced this pull request Feb 16, 2022
…ijun/elasticsearch into fix-none-tsdb-index-dimension-tests

* 'fix-none-tsdb-index-dimension-tests' of github.com:weizijun/elasticsearch: (37 commits)
  [docs] Mention JDK 17 in the Contributing docs (elastic#84018)
  Fix GeoIpDownloader startup during rolling upgrade (elastic#84000)
  Script: Fields API for Dense Vector (elastic#83550)
  Move InferenceConfigUpdate under VersionedNamedWriteable (elastic#84022)
  [ML] Fix license feature test cleanup (elastic#84020)
  Replace deprecated api in artifact transforms (elastic#84015)
  QL: Add leniency option to SQL CLI (elastic#83795)
  [Stack Monitoring] add kibana_stats version alias to -mb template (elastic#83930)
  Optimize spliterator for ImmutableOpenMap (elastic#83899)
  Feature usage actions for archive (elastic#83931)
  Use latch to speedup multi feature migration test (elastic#84007)
  Make action names available in NodeClient (elastic#83919)
  [DOCS] Re-add HTTP proxy setings from elastic#82737 (elastic#84001)
  Add CI matrix configuration for snapshot BWC versions (elastic#83990)
  Update YAML Rest tests to check for product header on all responses (elastic#83290)
  TSDB: Add time series aggs cancellation (elastic#83492)
  [DOCS] Fix percolate query headings (elastic#83988)
  [DOCS] Move tip for percolate query example (elastic#83972)
  Simplify LocalExporter cleaner function to fix failing tests (elastic#83812)
  [GCE Discovery] Correcly handle large zones with 500 or more instances (elastic#83785)
  ...
arteam added a commit to arteam/elasticsearch that referenced this pull request Apr 11, 2022
They were added in elastic#82737, but not exposed via the GCSPlugin to end users.
arteam added a commit that referenced this pull request Apr 12, 2022
They were added in #82737, but not exposed via the GCSPlugin to end users.
arteam added a commit to arteam/elasticsearch that referenced this pull request Apr 12, 2022
They were added in elastic#82737, but not exposed via the GCSPlugin to end users.
arteam added a commit to arteam/elasticsearch that referenced this pull request Apr 12, 2022
They were added in elastic#82737, but not exposed via the GCSPlugin to end users.
arteam added a commit to arteam/elasticsearch that referenced this pull request Apr 12, 2022
They were added in elastic#82737, but not exposed via the GCSPlugin to end users.
arteam added a commit that referenced this pull request Apr 12, 2022
They were added in #82737, but not exposed via the GCSPlugin to end users.
arteam added a commit that referenced this pull request Apr 12, 2022
They were added in #82737, but not exposed via the GCSPlugin to end users.
arteam added a commit that referenced this pull request Apr 12, 2022
They were added in #82737, but not exposed via the GCSPlugin to end users.
2lambda123 pushed a commit to 2lambda123/elastic-elasticsearch that referenced this pull request May 3, 2024
We've had a few users ask about support for HTTP proxy connections to GCS snapshot repositories in 7.x and 6.x.
However, client settings for those connections are only supported in 8.0+.

This adds a note to the 7.17 and 6.8 docs to nudge users to upgrade.

Relates to elastic/elasticsearch#82737

Closes elastic/elasticsearch#83959

(cherry picked from commit 99de635)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Automatically create backport pull requests when merged :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >enhancement Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. v8.0.0 v8.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for HTTP Proxies for the Google Cloud Storage repository plugin
5 participants