Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Thread 'tokio-runtime-worker' panicked #1566

Open
cryevecry opened this issue Jan 24, 2025 · 11 comments
Open

Thread 'tokio-runtime-worker' panicked #1566

cryevecry opened this issue Jan 24, 2025 · 11 comments
Assignees
@aaronmondal
Labels
bug Something isn't working

Comments

@cryevecry
Copy link

cryevecry commented Jan 24, 2025 •
edited
Loading

When using the cache via reclient, we encounter hanging builds.
The logs contain the following:

thread 'tokio-runtime-worker' panicked at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/bytes-1.7.1/src/bytes.rs:281:9:
range start must not be greater than end: 65536 <= 65535
thread 'tokio-runtime-worker' panicked at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/bytes-1.7.1/src/bytes.rs:281:9:
range start must not be greater than end: 65536 <= 65535
thread 'tokio-runtime-worker' panicked at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/bytes-1.7.1/src/bytes.rs:281:9:
range start must not be greater than end: 65536 <= 65535

Version v0.5.3
Nativelink started in the docker of the VM
@MarcusSorealheis
Copy link
Collaborator

@cryevecry thank you for opening the issue. Others have reported the same issue with version 0.5.3. @aaronmondal is preparing a new release now that should resolve this issue.

@cryevecry
Copy link
Author

cryevecry commented Jan 24, 2025 •
edited
Loading

@MarcusSorealheis Good news!
Can we downgrade to 0.5.1 for now so we don't have to deal with this issue?

@aaronmondal
Copy link
Member

@cryevecry Just to make sure that this is the same issue as #1565, could you check whether this image (as the cas image) fixes it?

https://github.com/TraceMachina/nativelink/pkgs/container/nativelink/339450226?tag=7zwvd8wsbsyfa4nnam8aq0953vbcgzqp

docker pull ghcr.io/tracemachina/nativelink:7zwvd8wsbsyfa4nnam8aq0953vbcgzqp

Initially this looks like a new issue to me. Could you also post the nativelink config that triggers this?

@cryevecry
Copy link
Author

@aaronmondal Okay, I'll try with this image and I'll write back to you.

config:

{
  "stores": {
    "cas_main_main": {
      "compression": {
        "compression_algorithm": {
          "lz4": {}
        },
        "backend": {
          "filesystem": {
            "content_path": "/root/.cache/nativelink/main/content-cas",
            "temp_path": "/root/.cache/nativelink/main/tmp-cas",
            "eviction_policy": {
              // 100gb.
              "max_bytes": 100000000000
            }
          }
        }
      }
    },
    "ac_main_main": {
      "filesystem": {
        "content_path": "/root/.cache/nativelink/main/content-aс",
        "temp_path": "/root/.cache/nativelink/main/tmp-aс",
        "eviction_policy": {
          // 5gb.
          "max_bytes": 5000000000
        }
      }
    },
    "cas_main2_main": {
      "compression": {
        "compression_algorithm": {
          "lz4": {}
        },
        "backend": {
          "filesystem": {
            "content_path": "/root/.cache/nativelink/main2/content-cas",
            "temp_path": "/root/.cache/nativelink/main2/tmp-cas",
            "eviction_policy": {
              // 100gb.
              "max_bytes": 100000000000
            }
          }
        }
      }
    },
    "ac_main2_main": {
      "filesystem": {
        "content_path": "/root/.cache/nativelink/main2/content-aс",
        "temp_path": "/root/.cache/nativelink/main2/tmp-aс",
        "eviction_policy": {
          // 5gb.
          "max_bytes": 5000000000
        }
      }
    }
  },
  "servers": [{
    "listener": {
      "http": {
        "socket_address": "0.0.0.0:50051"
      }
    },
    "services": {
      "cas": {
        "main": {
          "cas_store": "cas_main_main"
        },
        "main2": {
          "cas_store": "cas_main2_main"
        }
      },
      "ac": {
        "main": {
          "ac_store": "ac_main_main"
        },
        "main2": {
          "ac_store": "ac_main2_main"
        }
      },
      "capabilities": {},
      "bytestream": {
        "cas_stores": {
          "main": "cas_main_main",
          "main2": "cas_main2_main"
        }
      }
    }
  }, {
    // Only publish metrics on a private port.
    "listener": {
      "http": {
        "socket_address": "0.0.0.0:50061"
      }
    },
    "services": {
      "experimental_prometheus": {
        "path": "/metrics"
      }
    }
  }]
}

@cryevecry
Copy link
Author

@aaronmondal Alas, freezes are still present with this image. ghcr.io/tracemachina/nativelink:7zwvd8wsbsyfa4nnam8aq0953vbcgzqp

The nativelink logs only show this

thread 'tokio-runtime-worker' panicked at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/bytes-1.9.0/src/bytes.rs:390:9:
range start must not be greater than end: 65536 <= 65535
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: bytes::bytes::Bytes::slice
   3: <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
   4: <nativelink_store::compression_store::CompressionStore as nativelink_util::store_trait::StoreDriver>::get_part::{{closure}}
   5: nativelink_service::bytestream_server::ByteStreamServer::inner_read::{{closure}}::{{closure}}
   6: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
   7: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
   8: <tonic::codec::encode::EncodeBody<T,U> as http_body::Body>::poll_frame
   9: <http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_frame
  10: tokio::runtime::task::core::Core<T,S>::poll
  11: tokio::runtime::task::raw::poll
  12: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  13: tokio::runtime::scheduler::multi_thread::worker::run
  14: tokio::runtime::task::raw::poll
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

@aaronmondal
Copy link
Member

aaronmondal commented Feb 4, 2025 •
edited
Loading

Ah so it's a precision issue, likely in the compression store. Will look into creating a reproducer and fix.

@aaronmondal aaronmondal self-assigned this Feb 4, 2025
@aaronmondal aaronmondal added the bug Something isn't working label Feb 4, 2025
@MarcusSorealheis
Copy link
Collaborator

MarcusSorealheis commented Feb 4, 2025 via email

@allada
Copy link
Member

allada commented Feb 5, 2025 •
edited
Loading

Hi @cryevecry,

I'm looking into this issue and I am having a really difficult time trying to getting it to reproduce. For a while, I thought the issue was related to this line:

nativelink/nativelink-store/src/compression_store.rs

Line 523 in dcba3f5

.send(uncompressed_data.freeze().slice(start_pos..end_pos))

But I'm starting to question this now, as I'm pretty sure it is impossible for that line to ever fail assert!(start_pos <= end_pos);.

Given this, I'm starting to think maybe the issue is somewhere else, but the code was inlined so the stack trace might be missleading.

Would it be possible to run with RUST_BACKTRACE=full and share the exact image/version you are using? This might help. Failing that, would it be terribly difficult to run with a debug build?

I tested your exact config on a few bazel invocations that are very substance, but it doesn't trigger it. I'm thinking maybe reclient makes the calls in a way bazel does not (ie: using the offset and limit field more).

Thanks!

@cryevecry
Copy link
Author

@allada Hi!
RUST_BACKTRACE=full
image - ghcr.io/tracemachina/nativelink:7zwvd8wsbsyfa4nnam8aq0953vbcgzqp

thread 'tokio-runtime-worker' panicked at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/bytes-1.9.0/src/bytes.rs:390:9:
range start must not be greater than end: 65536 <= 65535
stack backtrace:
   0:     0x7f575513429c - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h1b0a18077ca6f698
   1:     0x7f575422c4b3 - core::fmt::write::hd43e6d3ae51a933a
   2:     0x7f5755102562 - std::io::Write::write_fmt::h4ee05bd9b08ac54b
   3:     0x7f575513a673 - std::sys::backtrace::BacktraceLock::print::ha3954179cd001b41
   4:     0x7f575513ab9c - std::panicking::rust_panic_with_hook::h9fe787d554af4a59
   5:     0x7f575513a748 - std::panicking::begin_panic_handler::{{closure}}::h8ebf843250dc76c5
   6:     0x7f575513a6a9 - std::sys::backtrace::__rust_end_short_backtrace::hc79e3420d69100d5
   7:     0x7f575513a69c - rust_begin_unwind
   8:     0x7f575422b20f - core::panicking::panic_fmt::h97f8f4814adeada4
   9:     0x7f5754d30bd4 - bytes::bytes::Bytes::slice::h4615a56696317bbe
  10:     0x7f5754b6e22a - <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll::h9fa4b327ec063e4b
  11:     0x7f5754cc50f5 - <nativelink_store::compression_store::CompressionStore as nativelink_util::store_trait::StoreDriver>::get_part::{{closure}}::hee560c515afa8b64
  12:     0x7f575490af54 - nativelink_service::bytestream_server::ByteStreamServer::inner_read::{{closure}}::{{closure}}::h59f21dbf8de658d1
  13:     0x7f575494a954 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::hc3d5e93b3fce9773
  14:     0x7f575494a411 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h691913861a9db7ec
  15:     0x7f575469fbb1 - <tonic::codec::encode::EncodeBody<T,U> as http_body::Body>::poll_frame::h25a79d8830937ae6
  16:     0x7f57545b24cf - <http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_frame::h53f4d8701cf9bb70
  17:     0x7f5753eb618b - tokio::runtime::task::core::Core<T,S>::poll::h561b43cd1939c2f3
  18:     0x7f5753ea3db0 - tokio::runtime::task::raw::poll::h3b31667daae22e1f
  19:     0x7f57551501eb - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h362fdf5b3e089bc0
  20:     0x7f575514e03e - tokio::runtime::scheduler::multi_thread::worker::run::h5c36a815a869170b
  21:     0x7f5755158596 - tokio::runtime::task::raw::poll::h849a925fc4beb827
  22:     0x7f5755153416 - std::sys::backtrace::__rust_begin_short_backtrace::hc6a61a1a5a40e9d1
  23:     0x7f5755152ec7 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h0d025b9aef4d580d
  24:     0x7f575513cddb - std::sys::pal::unix::thread::Thread::new::thread_start::ha3c33c56abdc36eb

If this is not enough, we can run the nativelink debugging image.

@aaronmondal
Copy link
Member

@cryevecry We haven't been able to figure this out so far. Looks like we'll need to do this with debug symbols.

We initially thought this was an issue in the compression store, but haven't been able to reproduce. We're now thinking that this might be caused by data corruption in the filesystem store which then causes unintended behavior in the bytestream server, but we're also not certain of that.

Could you try this image? That should give us a better stack trace.

ghcr.io/aaronmondal/nativelink:mf3jvdcm1n0w231xyrzpaxdr66hdvha0

(was built here: https://github.com/aaronmondal/nativelink/actions/runs/13189047660/job/36817988723)

@cryevecry
Copy link
Author

@aaronmondal

thread 'tokio-runtime-worker' panicked at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/bytes-1.9.0/src/bytes.rs:390:9:
range start must not be greater than end: 65536 <= 65535
stack backtrace:
   0:     0x7fad5d78114c - std::backtrace_rs::backtrace::libunwind::trace::hf17a76ed918bef43
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:     0x7fad5d78114c - std::backtrace_rs::backtrace::trace_unsynchronized::h4ce821b4eb9a2e70
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7fad5d78114c - std::sys::backtrace::_print_fmt::h2f254dc68e8f093d
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/sys/backtrace.rs:66:9
   3:     0x7fad5d78114c - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h1b0a18077ca6f698
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/sys/backtrace.rs:39:26
   4:     0x7fad5c822973 - core::fmt::rt::Argument::fmt::hb15418308d66d265
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/fmt/rt.rs:177:76
   5:     0x7fad5c822973 - core::fmt::write::hd43e6d3ae51a933a
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/fmt/mod.rs:1189:21
   6:     0x7fad5d74f472 - std::io::Write::write_fmt::h4ee05bd9b08ac54b
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/io/mod.rs:1884:15
   7:     0x7fad5d787523 - std::sys::backtrace::BacktraceLock::print::ha3954179cd001b41
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/sys/backtrace.rs:42:9
   8:     0x7fad5d787a4c - std::panicking::default_hook::{{closure}}::h2616e6aececc1d57
   9:     0x7fad5d787a4c - std::panicking::default_hook::h802ff2df8b0845d7
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:295:9
  10:     0x7fad5d787a4c - std::panicking::rust_panic_with_hook::h9fe787d554af4a59
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:801:13
  11:     0x7fad5d7875f8 - std::panicking::begin_panic_handler::{{closure}}::h8ebf843250dc76c5
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:674:13
  12:     0x7fad5d787559 - std::sys::backtrace::__rust_end_short_backtrace::hc79e3420d69100d5
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/sys/backtrace.rs:170:18
  13:     0x7fad5d78754c - rust_begin_unwind
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:665:5
  14:     0x7fad5c8216cf - core::panicking::panic_fmt::h97f8f4814adeada4
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/panicking.rs:76:14
  15:     0x7fad5d3153f4 - bytes::bytes::Bytes::slice::hd93d67b7157c1942
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/bytes-1.9.0/src/bytes.rs:390:9
  16:     0x7fad5d1dd125 - <nativelink_store::compression_store::CompressionStore as nativelink_util::store_trait::StoreDriver>::get_part::{{closure}}::{{closure}}::h08dc43c59615c962
                               at /build/source/nativelink-store/src/compression_store.rs:523:39
  17:     0x7fad5d1dd125 - <tokio::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll::h01a6d18e51b8376a
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/future/maybe_done.rs:65:56
  18:     0x7fad5d0ea3c2 - <nativelink_store::compression_store::CompressionStore as nativelink_util::store_trait::StoreDriver>::get_part::{{closure}}::{{closure}}::h3ac93d487ee821cc
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/macros/join.rs:138:24
  19:     0x7fad5d0ea3c2 - <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll::h68a3db2795a19234
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/future/poll_fn.rs:151:9
  20:     0x7fad5d0ea3c2 - <nativelink_store::compression_store::CompressionStore as nativelink_util::store_trait::StoreDriver>::get_part::{{closure}}::hc10fec2dd8a43e11
                               at /build/source/nativelink-store/src/compression_store.rs:600:50
  21:     0x7fad5cf3afc4 - <core::pin::Pin<P> as core::future::future::Future>::poll::h8dcdc701e29ef434
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/future/future.rs:124:9
  22:     0x7fad5cf3afc4 - nativelink_util::store_trait::StoreLike::get_part::{{closure}}::h6b449cf00f01ce40
                               at /build/source/nativelink-util/src/store_trait.rs:540:18
  23:     0x7fad5cf3afc4 - nativelink_service::bytestream_server::ByteStreamServer::inner_read::{{closure}}::{{closure}}::h757fcd511ca7b928
                               at /build/source/nativelink-service/src/bytestream_server.rs:306:22
  24:     0x7fad5cfa59a4 - <core::pin::Pin<P> as core::future::future::Future>::poll::h8dcdc701e29ef434
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/future/future.rs:124:9
  25:     0x7fad5cfa59a4 - <&mut F as core::future::future::Future>::poll::h88f77166c2c1cb8d
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/future/future.rs:112:9
  26:     0x7fad5cfa59a4 - nativelink_service::bytestream_server::ByteStreamServer::inner_read::{{closure}}::{{closure}}::{{closure}}::{{closure}}::h5a43da1d9d1ba232
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/macros/select.rs:557:49
  27:     0x7fad5cfa59a4 - <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll::haf84173d5b2bee72
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/future/poll_fn.rs:151:9
  28:     0x7fad5cfa59a4 - nativelink_service::bytestream_server::ByteStreamServer::inner_read::{{closure}}::{{closure}}::{{closure}}::h8566325e31d74ddb
                               at /build/source/nativelink-service/src/bytestream_server.rs:320:21
  29:     0x7fad5cfa59a4 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h1ead516eecfae110
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tracing-0.1.41/src/instrument.rs:321:9
  30:     0x7fad5cfa59a4 - <futures_util::stream::unfold::Unfold<T,F,Fut> as futures_core::stream::Stream>::poll_next::hde63e3ea957ba15e
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/futures-util-0.3.31/src/stream/unfold.rs:107:33
  31:     0x7fad5cfa59a4 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::hd50ef0dd81042f3f
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/futures-core-0.3.31/src/stream.rs:130:9
  32:     0x7fad5cfa5461 - <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next::h79eddb022a3246e7
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/futures-core-0.3.31/src/stream.rs:130:9
  33:     0x7fad5cc7d791 - <core::pin::Pin<P> as core::future::future::Future>::poll::h09d3015b8a662e25
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/future/future.rs:124:9
  34:     0x7fad5cc7d791 - <tokio_stream::stream_ext::fuse::Fuse<T> as futures_core::stream::Stream>::poll_next::h2b5ce54a9285c82f
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-stream-0.1.17/src/stream_ext/fuse.rs:35:36
  35:     0x7fad5cc7d791 - <tonic::codec::encode::EncodedBytes<T,U> as futures_core::stream::Stream>::poll_next::hc86362e7eb7b8aa7
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tonic-0.12.3/src/codec/encode.rs:94:19
  36:     0x7fad5cc7d791 - <tonic::codec::encode::EncodeBody<T,U> as http_body::Body>::poll_frame::h9cc00b3911e47cef
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tonic-0.12.3/src/codec/encode.rs:320:22
  37:     0x7fad5cd101bf - <http_body_util::combinators::box_body::UnsyncBoxBody<D,E> as http_body::Body>::poll_frame::h3b73072b46df2db2
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/http-body-util-0.1.2/src/combinators/box_body.rs:103:9
  38:     0x7fad5cd101bf - <http_body_util::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_frame::hf84e28e483e6ee59
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/http-body-util-0.1.2/src/combinators/map_err.rs:62:15
  39:     0x7fad5c4b1336 - <http_body_util::combinators::box_body::UnsyncBoxBody<D,E> as http_body::Body>::poll_frame::hf7757930cef98ce0
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/http-body-util-0.1.2/src/combinators/box_body.rs:103:9
  40:     0x7fad5c4b1336 - <axum_core::body::Body as http_body::Body>::poll_frame::h8b3edf430faf8016
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/axum-core-0.4.5/src/body.rs:122:31
  41:     0x7fad5c4b1336 - <hyper::proto::h2::PipeToSendStream<S> as core::future::future::Future>::poll::h7d20f6549f8afb40
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/hyper-1.5.2/src/proto/h2/mod.rs:151:26
  42:     0x7fad5c4b1336 - hyper::proto::h2::server::H2Stream<F,B>::poll2::h03e29a84c9882b66
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/hyper-1.5.2/src/proto/h2/server.rs:520:28
  43:     0x7fad5c4b1336 - <hyper::proto::h2::server::H2Stream<F,B> as core::future::future::Future>::poll::hb2a6051e49ecad28
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/hyper-1.5.2/src/proto/h2/server.rs:539:9
  44:     0x7fad5c4b1336 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::ha67cad478c0329f9
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tracing-0.1.41/src/instrument.rs:321:9
  45:     0x7fad5c4b1336 - <nativelink_util::origin_context::ContextAwareFuture<T> as core::future::future::Future>::poll::h655117664b950ee5
                               at /build/source/nativelink-util/src/origin_context.rs:364:22
  46:     0x7fad5c4b1336 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h11deca1598d32f38
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/core.rs:331:17
  47:     0x7fad5c4b1336 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h136af470cba38c9b
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/loom/std/unsafe_cell.rs:16:9
  48:     0x7fad5c4b1336 - tokio::runtime::task::core::Core<T,S>::poll::h2004873c79116a62
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/core.rs:320:30
  49:     0x7fad5cb32340 - tokio::runtime::task::harness::poll_future::{{closure}}::ha58ab7305da841af
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/harness.rs:532:19
  50:     0x7fad5cb32340 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hcb413edcaa7cb9fc
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/panic/unwind_safe.rs:272:9
  51:     0x7fad5cb32340 - std::panicking::try::do_call::h399dd9567d2a0c69
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:557:40
  52:     0x7fad5cb32340 - std::panicking::try::hd45a91217a47aea0
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:520:19
  53:     0x7fad5cb32340 - std::panic::catch_unwind::h0d1314d23b27825b
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panic.rs:358:14
  54:     0x7fad5cb32340 - tokio::runtime::task::harness::poll_future::h07ecf12638037cb6
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/harness.rs:520:18
  55:     0x7fad5cb32340 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h7261301e8bc4cccf
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/harness.rs:209:27
  56:     0x7fad5cb32340 - tokio::runtime::task::harness::Harness<T,S>::poll::h0863bab2ffbc8b5f
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/harness.rs:154:15
  57:     0x7fad5cb32340 - tokio::runtime::task::raw::poll::h59a9477c0dcec7d2
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/raw.rs:271:5
  58:     0x7fad5d79a77b - tokio::runtime::task::raw::RawTask::poll::hd6d3a5dabddbd2d3
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/raw.rs:201:18
  59:     0x7fad5d79a77b - tokio::runtime::task::LocalNotified<S>::run::hfc7850f360a0c2ea
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/mod.rs:449:9
  60:     0x7fad5d79a77b - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h3b35570a0df2105f
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/scheduler/multi_thread/worker.rs:659:22
  61:     0x7fad5d79a77b - tokio::runtime::coop::with_budget::h92ed07d3a8013a2f
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/coop.rs:107:5
  62:     0x7fad5d79a77b - tokio::runtime::coop::budget::hcc456b15d70e0cef
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/coop.rs:73:5
  63:     0x7fad5d79a77b - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::he32a5ade4010c2c2
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/scheduler/multi_thread/worker.rs:595:9
  64:     0x7fad5d79816e - tokio::runtime::scheduler::multi_thread::worker::Context::run::hb44de53becffbac0
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/scheduler/multi_thread/worker.rs:546:24
  65:     0x7fad5d79816e - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::h5dbeebd839d3395c
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/scheduler/multi_thread/worker.rs:511:21
  66:     0x7fad5d79816e - tokio::runtime::context::scoped::Scoped<T>::set::h4e2ee950cb5968c3
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/context/scoped.rs:40:9
  67:     0x7fad5d79816e - tokio::runtime::context::set_scheduler::{{closure}}::hfdab58ef85e10aae
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/context.rs:180:26
  68:     0x7fad5d79816e - std::thread::local::LocalKey<T>::try_with::ha753d9eb4ab91741
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/thread/local.rs:283:12
  69:     0x7fad5d79816e - std::thread::local::LocalKey<T>::with::ha4664025120a7457
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/thread/local.rs:260:9
  70:     0x7fad5d79816e - tokio::runtime::context::set_scheduler::hc387fa65b241a64e
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/context.rs:180:17
  71:     0x7fad5d79816e - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h9bf8d8be558cf3d1
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/scheduler/multi_thread/worker.rs:506:9
  72:     0x7fad5d79816e - tokio::runtime::context::runtime::enter_runtime::h5b4bd46d0bc36482
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/context/runtime.rs:65:16
  73:     0x7fad5d79816e - tokio::runtime::scheduler::multi_thread::worker::run::hb34f5ba0dd46cc9e
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/scheduler/multi_thread/worker.rs:498:5
  74:     0x7fad5d78f546 - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::hc6185ccb7208d936
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/scheduler/multi_thread/worker.rs:464:45
  75:     0x7fad5d78f546 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h2a6fc5cc1c3b7768
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/blocking/task.rs:42:21
  76:     0x7fad5d78f546 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hc95e8ef42e68d2b7
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/core.rs:331:17
  77:     0x7fad5d78f546 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h98b9ff109b95bc45
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/loom/std/unsafe_cell.rs:16:9
  78:     0x7fad5d78f546 - tokio::runtime::task::core::Core<T,S>::poll::h351cbbf57fe8e951
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/core.rs:320:30
  79:     0x7fad5d78f546 - tokio::runtime::task::harness::poll_future::{{closure}}::h9add26d538729ff0
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/harness.rs:532:19
  80:     0x7fad5d78f546 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h3d08896780520d7d
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/panic/unwind_safe.rs:272:9
  81:     0x7fad5d78f546 - std::panicking::try::do_call::h17136d953fee946b
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:557:40
  82:     0x7fad5d78f546 - std::panicking::try::ha62d05a31ed1df73
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:520:19
  83:     0x7fad5d78f546 - std::panic::catch_unwind::hda51cce5b8fb8ced
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panic.rs:358:14
  84:     0x7fad5d78f546 - tokio::runtime::task::harness::poll_future::h2ae68ac370bb4e25
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/harness.rs:520:18
  85:     0x7fad5d78f546 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::heea48226b3d57766
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/harness.rs:209:27
  86:     0x7fad5d78f546 - tokio::runtime::task::harness::Harness<T,S>::poll::h03598e3f65c7464e
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/harness.rs:154:15
  87:     0x7fad5d78f546 - tokio::runtime::task::raw::poll::h38e67b5f49c25e62
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/raw.rs:271:5
  88:     0x7fad5d7b627a - tokio::runtime::task::raw::RawTask::poll::hd6d3a5dabddbd2d3
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/raw.rs:201:18
  89:     0x7fad5d7b627a - tokio::runtime::task::UnownedTask<S>::run::ha5cdf51c4c7f8749
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/task/mod.rs:486:9
  90:     0x7fad5d7b627a - tokio::runtime::blocking::pool::Task::run::h31b5d3e2f26fcdde
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/blocking/pool.rs:161:9
  91:     0x7fad5d7b627a - tokio::runtime::blocking::pool::Inner::run::ha6141012d73f2645
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/blocking/pool.rs:511:17
  92:     0x7fad5d7b627a - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h1a221c6aaae9ea74
                               at /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/tokio-1.43.0/src/runtime/blocking/pool.rs:469:13
  93:     0x7fad5d7b627a - std::sys::backtrace::__rust_begin_short_backtrace::h876cb4fb84e39701
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/sys/backtrace.rs:154:18
  94:     0x7fad5d7b5d47 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hb72c118745920600
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/thread/mod.rs:561:17
  95:     0x7fad5d7b5d47 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h6f6ff8d91d50f040
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/panic/unwind_safe.rs:272:9
  96:     0x7fad5d7b5d47 - std::panicking::try::do_call::h20e1fcbb147a3eb2
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:557:40
  97:     0x7fad5d7b5d47 - std::panicking::try::h766c55b5e98edeef
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panicking.rs:520:19
  98:     0x7fad5d7b5d47 - std::panic::catch_unwind::h51766ea5ab67ca67
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/panic.rs:358:14
  99:     0x7fad5d7b5d47 - std::thread::Builder::spawn_unchecked_::{{closure}}::h4bbf47d52b4e6207
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/thread/mod.rs:559:30
 100:     0x7fad5d7b5d47 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h4e2aa9fadf28c72c
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/core/src/ops/function.rs:250:5
 101:     0x7fad5d789c8b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h876f74e54d3da903
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/boxed.rs:1972:9
 102:     0x7fad5d789c8b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h301fad8ef3d0bbb7
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/boxed.rs:1972:9
 103:     0x7fad5d789c8b - std::sys::pal::unix::thread::Thread::new::thread_start::ha3c33c56abdc36eb
                               at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/sys/pal/unix/thread.rs:105:17

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aaronmondal aaronmondal

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@allada @MarcusSorealheis @aaronmondal @cryevecry