Skip to content

Commit

Permalink
Improve unwind info persisting failure handling
Browse files Browse the repository at this point in the history
In some production machines, persisting the unwind info fails.
We are currently investigating this and so far we don't know
what the culprit is. Typically after some attempts, persisting
eventually succeeds.

On those hosts we get pretty much 100% unwind errors, which
should not happen. This leads me to notice that errors persisting
the unwind info aren't handled properly.

For example, once a shard is full, the current code ignores this
wipes the in-memory shard and assigns a new BPF shard. This is not correct.

Test Plan
=========

Forced some errors in this logic and the current in-memory state
wasn't wiped. We need failure injection during testing to ensure
all these cases are covered and don't regress.
  • Loading branch information
javierhonduco committed Apr 23, 2024
1 parent a8094d1 commit fdebff4
Showing 1 changed file with 8 additions and 6 deletions.
14 changes: 8 additions & 6 deletions src/profiler.rs
Original file line number Diff line number Diff line change
Expand Up @@ -355,12 +355,14 @@ impl Profiler<'_> {
}
}

let due_to_persist =
self.native_unwind_state.last_persisted.elapsed() > Duration::from_millis(100);

if self.native_unwind_state.dirty
&& self.native_unwind_state.last_persisted.elapsed() > Duration::from_millis(100)
&& due_to_persist
&& self.persist_unwind_info(&self.native_unwind_state.live_shard)
{
if self.persist_unwind_info(&self.native_unwind_state.live_shard) {
self.native_unwind_state.dirty = false;
}
self.native_unwind_state.dirty = false;
self.native_unwind_state.last_persisted = Instant::now();
}
}
Expand Down Expand Up @@ -812,9 +814,9 @@ impl Profiler<'_> {

if self.persist_unwind_info(&self.native_unwind_state.live_shard) {
self.native_unwind_state.dirty = false;
self.native_unwind_state.live_shard.truncate(0);
self.native_unwind_state.shard_index += 1;
}
self.native_unwind_state.live_shard.truncate(0);
self.native_unwind_state.shard_index += 1;
continue;
}

Expand Down

0 comments on commit fdebff4

Please sign in to comment.