Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sentry: windows: io error failed to remove dir #37819

Closed
cockroach-teamcity opened this issue May 26, 2019 · 1 comment · Fixed by #41160
Closed

sentry: windows: io error failed to remove dir #37819

cockroach-teamcity opened this issue May 26, 2019 · 1 comment · Fixed by #41160
Assignees
Labels
A-kv Anything in KV that doesn't belong in a more specific category. B-os-windows Issues specific to the Windows OS. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/1043556559/?referrer=webhooks_plugin

Panic message:

replica_raft.go:606: unable to finish destroying %s while applying snapshot: %s | *storage.Replica; io error failed to remove dir no

Stacktrace (expand for inline code snippets):

if err := r.applySnapshot(ctx, inSnap, rd.Snapshot, rd.HardState, subsumedRepls); err != nil {
const expl = "while applying snapshot"
in pkg/storage.(*Replica).handleRaftReadyRaftMuLocked

cockroach/pkg/storage/store.go

Lines 3525 to 3527 in 25dd36f

if _, expl, err := r.handleRaftReadyRaftMuLocked(ctx, inSnap); err != nil {
fatalOnRaftReadyErr(ctx, expl, err)
in pkg/storage.(*Store).processRaftSnapshotRequest.func1

cockroach/pkg/storage/store.go

Lines 3294 to 3296 in 25dd36f

r.setLastReplicaDescriptors(req)
return f(ctx, r)
}
in pkg/storage.(*Store).withReplicaForRequest

cockroach/pkg/storage/store.go

Lines 3350 to 3352 in 25dd36f

) *roachpb.Error {
return s.withReplicaForRequest(ctx, &snapHeader.RaftMessageRequest, func(
ctx context.Context, r *Replica,
in pkg/storage.(*Store).processRaftSnapshotRequest
}
if err := s.processRaftSnapshotRequest(ctx, header, inSnap); err != nil {
return sendSnapshotError(stream, errors.Wrap(err.GoError(), "failed to apply snapshot"))
in pkg/storage.(*Store).receiveSnapshot

cockroach/pkg/storage/store.go

Lines 3172 to 3174 in 25dd36f

ctx := s.AnnotateCtx(stream.Context())
return s.receiveSnapshot(ctx, header, stream)
}
in pkg/storage.(*Store).HandleSnapshot
}
return handler.HandleSnapshot(req.Header, stream)
}()
in pkg/storage.(*RaftTransport).RaftSnapshot.func1.1
return handler.HandleSnapshot(req.Header, stream)
}()
}); err != nil {
in pkg/storage.(*RaftTransport).RaftSnapshot.func1
f(ctx)
}()
in pkg/util/stop.(*Stopper).RunAsyncTask.func1

pkg/storage/replica_raft.go in pkg/storage.(*Replica).handleRaftReadyRaftMuLocked at line 606
pkg/storage/store.go in pkg/storage.(*Store).processRaftSnapshotRequest.func1 at line 3526
pkg/storage/store.go in pkg/storage.(*Store).withReplicaForRequest at line 3295
pkg/storage/store.go in pkg/storage.(*Store).processRaftSnapshotRequest at line 3351
pkg/storage/store_snapshot.go in pkg/storage.(*Store).receiveSnapshot at line 666
pkg/storage/store.go in pkg/storage.(*Store).HandleSnapshot at line 3173
pkg/storage/raft_transport.go in pkg/storage.(*RaftTransport).RaftSnapshot.func1.1 at line 386
pkg/storage/raft_transport.go in pkg/storage.(*RaftTransport).RaftSnapshot.func1 at line 387
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTask.func1 at line 325
Tag Value
Cockroach Release v19.1.0
Cockroach SHA: 25dd36f
Platform windows amd64
Distribution CCL
Environment v19.1.0
Command server
Go Version go1.11.6
# of CPUs 4
# of Goroutines 190
@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels May 26, 2019
@tbg
Copy link
Member

tbg commented May 28, 2019

This is on windows, so it's probably hitting some behavior that's different to what we're used to.

@tbg tbg changed the title sentry: replica_raft.go:606: unable to finish destroying %s while applying snapshot: %s | *storage.Replica; io error failed to remove dir <redacted> <redacted> <redacted> no <redacted> <redacted> <redacted> <redacted> <redacted> sentry: windows: io error failed to remove dir May 28, 2019
@tbg tbg added the B-os-windows Issues specific to the Windows OS. label May 28, 2019
@kenliu kenliu added A-kv-gossip A-kv Anything in KV that doesn't belong in a more specific category. and removed A-kv-gossip labels Jul 10, 2019
@kenliu kenliu added this to the 19.2 milestone Jul 10, 2019
darinpp added a commit to darinpp/cockroach that referenced this issue Sep 27, 2019
RocksDB provides different delete folder methods depending on the
environment (windows, posix etc). Unfortunatelly the implementations
treat any error returned as I/O error and convert the error
codes to strings. So on CockroachDB side we can only distinguish a
serious error (disk corruption) from a harmless error
(file or folder doesn't exist) by parsing the error as a string.
Different platforms will return different strings and in case of Windows
the string wasn't in the list that we used to recognize folder not found.

Fixes cockroachdb#37819
Fixes cockroachdb#37427

Release justification: bug fix for existing functionality

Release note: None
craig bot pushed a commit that referenced this issue Sep 27, 2019
41160: rocksdb: incorrectly identifying not found folder error on windows r=darinpp a=darinpp

RocksDB provides different delete folder methods depending on the
environment (windows, posix etc). Unfortunatelly the implementations
treat any error returned as I/O error and convert the error
codes to strings. So on CockroachDB side we can only distinguish a
serious error (disk corruption) from a harmless error
(file or folder doesn't exist) by parsing the error as a string.
Different platforms will return different strings and in case of Windows
the string wasn't in the list that we used to recognize folder not found.

Fixes #37819
Fixes #37427

Release justification: bug fix for existing functionality

Release note: None

Co-authored-by: Darin <[email protected]>
@craig craig bot closed this as completed in 1005192 Sep 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-kv Anything in KV that doesn't belong in a more specific category. B-os-windows Issues specific to the Windows OS. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants