Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[6.4.0] Seeing some DEADLINE_EXCEEDED fatals in 6.4.0 #19599

Closed
jerrymarino opened this issue Sep 22, 2023 · 2 comments
Closed

[6.4.0] Seeing some DEADLINE_EXCEEDED fatals in 6.4.0 #19599

jerrymarino opened this issue Sep 22, 2023 · 2 comments
Assignees
@sgowroji @Pavank1992 @iancha1992
Labels
type: bug untriaged

Comments

@jerrymarino
Copy link
Contributor

jerrymarino commented Sep 22, 2023 •
edited
Loading

We're seeing a surge of DEADLINE_EXCEEDED after updating to 6.4.0:

I'm just seeing this after a few runs in CI this morning - it's not clear if retries are still working and needs more investigation on my end but I wanted to flag this ASAP
#19035

[2023-09-22T18:41:58.536Z] com.google.devtools.build.lib.remote.common.BulkTransferException: Error while uploading artifact with digest '77d4482ede1a679150c9667caee361df0aa1cb42b5380bfb9cd42bfaf205a1c0/57826528'
[2023-09-22T18:41:58.536Z]      at com.google.devtools.build.lib.remote.util.RxUtils$BulkTransferExceptionCollector.onResult(RxUtils.java:91)
[2023-09-22T18:41:58.536Z]      at io.reactivex.rxjava3.internal.operators.flowable.FlowableCollectSingle$CollectSubscriber.onNext(FlowableCollectSingle.java:94)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapPublisher$SingleFlatMapPublisherObserver.onNext(SingleFlatMapPublisher.java:107)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.flowable.FlowableUsing$UsingSubscriber.onNext(FlowableUsing.java:104)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapPublisher$SingleFlatMapPublisherObserver.onNext(SingleFlatMapPublisher.java:107)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.flowable.FlowableUsing$UsingSubscriber.onNext(FlowableUsing.java:104)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber.innerSuccess(FlowableFlatMapSingle.java:173)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber$InnerObserver.onSuccess(FlowableFlatMapSingle.java:342)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.observers.ResumeSingleObserver.onSuccess(ResumeSingleObserver.java:46)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.single.SingleJust.subscribeActual(SingleJust.java:30)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.single.SingleResumeNext$ResumeMainSingleObserver.onError(SingleResumeNext.java:80)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle$ToSingle.onError(CompletableToSingle.java:73)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.completable.CompletableFromObservable$CompletableFromObservableObserver.onError(CompletableFromObservable.java:51)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.subjects.AsyncSubject$AsyncDisposable.onError(AsyncSubject.java:354)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.subjects.AsyncSubject.onError(AsyncSubject.java:171)
[2023-09-22T18:41:58.537Z]      at com.google.devtools.build.lib.remote.RemoteExecutionCache$1.onError(RemoteExecutionCache.java:232)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.completable.CompletableFromSingle$CompletableFromSingleObserver.onError(CompletableFromSingle.java:41)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.tryOnError(SingleCreate.java:95)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onError(SingleCreate.java:81)
[2023-09-22T18:41:58.537Z]      at com.google.devtools.build.lib.remote.util.AsyncTaskCache$1.onError(AsyncTaskCache.java:340)
[2023-09-22T18:41:58.537Z]      at com.google.devtools.build.lib.remote.util.AsyncTaskCache$Execution.onError(AsyncTaskCache.java:206)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle$ToSingle.onError(CompletableToSingle.java:73)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapCompletable$FlatMapCompletableObserver.onError(SingleFlatMapCompletable.java:97)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.tryOnError(CompletableCreate.java:91)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.onError(CompletableCreate.java:77)
[2023-09-22T18:41:58.537Z]      at com.google.devtools.build.lib.remote.util.RxFutures$OnceCompletableOnSubscribe$1.onFailure(RxFutures.java:102)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1124)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:851)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:214)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:190)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:142)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:851)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:124)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:104)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:851)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:214)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:190)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:142)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:851)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:214)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:190)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:142)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:104)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:55)
[2023-09-22T18:41:58.537Z]      at com.google.devtools.build.lib.remote.util.RxFutures$2.onError(RxFutures.java:257)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:117)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.single.SingleUsing$UsingSingleObserver.onError(SingleUsing.java:180)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.tryOnError(SingleCreate.java:95)
[2023-09-22T18:41:58.537Z]      at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onError(SingleCreate.java:81)
[2023-09-22T18:41:58.537Z]      at com.google.devtools.build.lib.remote.util.RxFutures$OnceSingleOnSubscribe$1.onFailure(RxFutures.java:172)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1124)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
[2023-09-22T18:41:58.537Z]      at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:55)
[2023-09-22T18:41:58.537Z]      at com.google.devtools.build.lib.remote.ByteStreamUploader$Writer.onError(ByteStreamUploader.java:537)
[2023-09-22T18:41:58.537Z]      at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
[2023-09-22T18:41:58.537Z]      at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
[2023-09-22T18:41:58.537Z]      at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
[2023-09-22T18:41:58.537Z]      at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
[2023-09-22T18:41:58.537Z]      at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
[2023-09-22T18:41:58.537Z]      at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[2023-09-22T18:41:58.537Z]      at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
[2023-09-22T18:41:58.537Z]      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[2023-09-22T18:41:58.537Z]      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[2023-09-22T18:41:58.537Z]      at java.base/java.lang.Thread.run(Unknown Source)
[2023-09-22T18:41:58.537Z]      Suppressed: java.io.IOException: Error while uploading artifact with digest '77d4482ede1a679150c9667caee361df0aa1cb42b5380bfb9cd42bfaf205a1c0/57826528'
[2023-09-22T18:41:58.537Z]              at com.google.devtools.build.lib.remote.ByteStreamUploader.lambda$uploadBlobAsync$0(ByteStreamUploader.java:176)
[2023-09-22T18:41:58.537Z]              at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:203)
[2023-09-22T18:41:58.537Z]              at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:190)
[2023-09-22T18:41:58.537Z]              at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133)
[2023-09-22T18:41:58.537Z]              ... 57 more
[2023-09-22T18:41:58.537Z]      Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 359.999923907s. [closed=[], committed=[buffered_nanos=41642, remote_addr=fo.bar..com/xx.x.x.x.x:443]]
[2023-09-22T18:41:58.537Z]              at io.grpc.Status.asRuntimeException(Status.java:535)
[2023-09-22T18:41:58.537Z]              ... 10 more$

Which category does this issue belong to?

Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Re timeouts

Which operating system are you running Bazel on?

macOS

What is the output of bazel info release?

6.4.0rc1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response
@iancha1992
Copy link
Member

cc: @bazelbuild/triage for visibility

@jerrymarino
Copy link
Contributor Author

Please disregard this - sorry for the ping

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sgowroji sgowroji

@Pavank1992 Pavank1992

@iancha1992 iancha1992

Labels
type: bug untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jerrymarino @sgowroji @Pavank1992 @iancha1992