Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors when profiling application with large debuginfo #2208

Open
Ilmarii opened this issue Dec 1, 2022 · 0 comments
Open

Errors when profiling application with large debuginfo #2208

Ilmarii opened this issue Dec 1, 2022 · 0 comments

Comments

@Ilmarii
Copy link

Ilmarii commented Dec 1, 2022

Hi,
When trying to profile our Vector deployments, I found that although the samples are displayed normally, there are no function names.
And i found this error in logs:

level=warn name=parca ts=2022-12-01T18:49:13.161878448Z caller=store.go:423 component=debuginfo buildid=971415fa19d528ee99a74555d4ce0646039eee59 msg="failed to validate debug information" err=EOF
level=warn name=parca ts=2022-12-01T18:49:13.162012268Z caller=store.go:435 component=debuginfo buildid=971415fa19d528ee99a74555d4ce0646039eee59 msg="failed to fetch debuginfod file" err="failed to fetch from debuginfod: debug info not found"
level=error name=parca ts=2022-12-01T18:49:13.162494329Z caller=store.go:232 component=debuginfo msg="failed to validate object file" buildid=971415fa19d528ee99a74555d4ce0646039eee59
level=error name=parca ts=2022-12-01T18:49:13.162527249Z caller=logger.go:38 protocol=grpc grpc.component=server grpc.service=parca.debuginfo.v1alpha1.DebugInfoService grpc.method=Upload grpc.method_type=client_stream peer.address=100.96.37.20:60480 grpc.start_time=2022-12-01T18:49:13Z grpc.request.deadline=2022-12-01T18:50:54Z grpc.code=Internal grpc.error="rpc error: code = Internal desc = EOF" grpc.time_ms=0.826 finishedcall=(MISSING)
level=warn name=parca ts=2022-12-01T18:49:13.464684326Z caller=store.go:423 component=debuginfo buildid=971415fa19d528ee99a74555d4ce0646039eee59 msg="failed to validate debug information" err=EOF
level=warn name=parca ts=2022-12-01T18:49:13.464816583Z caller=store.go:435 component=debuginfo buildid=971415fa19d528ee99a74555d4ce0646039eee59 msg="failed to fetch debuginfod file" err="failed to fetch from debuginfod: debug info not found"
level=error name=parca ts=2022-12-01T18:49:13.465469569Z caller=store.go:232 component=debuginfo msg="failed to validate object file" buildid=971415fa19d528ee99a74555d4ce0646039eee59
level=error name=parca ts=2022-12-01T18:49:13.465508493Z caller=logger.go:38 protocol=grpc grpc.component=server grpc.service=parca.debuginfo.v1alpha1.DebugInfoService grpc.method=Upload grpc.method_type=client_stream peer.address=100.96.24.75:37130 grpc.start_time=2022-12-01T18:49:13Z grpc.request.deadline=2022-12-01T18:51:04Z grpc.code=Internal grpc.error="rpc error: code = Internal desc = EOF" grpc.time_ms=0.989 finishedcall=(MISSING)
level=warn name=parca ts=2022-12-01T18:49:13.481096585Z caller=store.go:423 component=debuginfo buildid=971415fa19d528ee99a74555d4ce0646039eee59 msg="failed to validate debug information" err=EOF
level=warn name=parca ts=2022-12-01T18:49:13.481212601Z caller=store.go:435 component=debuginfo buildid=971415fa19d528ee99a74555d4ce0646039eee59 msg="failed to fetch debuginfod file" err="failed to fetch from debuginfod: debug info not found"
level=error name=parca ts=2022-12-01T18:49:13.481889726Z caller=store.go:232 component=debuginfo msg="failed to validate object file" buildid=971415fa19d528ee99a74555d4ce0646039eee59
level=error name=parca ts=2022-12-01T18:49:13.481924931Z caller=logger.go:38 protocol=grpc grpc.component=server grpc.service=parca.debuginfo.v1alpha1.DebugInfoService grpc.method=Upload grpc.method_type=client_stream peer.address=100.96.36.219:60862 grpc.start_time=2022-12-01T18:49:13Z grpc.request.deadline=2022-12-01T18:51:04Z grpc.code=Internal grpc.error="rpc error: code = Internal desc = EOF" grpc.time_ms=0.981 finishedcall=(MISSING)

When I tried to load debuginfo with cli, an error also occurred:

parca-debuginfo upload --store-address=localhost:7070 --insecure /tmp/vector-x86_64-unknown-linux-gnu/bin/vector 
level=error err="1 error occurred:\n\t* failed to upload debug information: send chunk: rpc error: code = Internal desc = EOF\n\n"

Founded workaround is to stop all parca agents, then clear the object store on the parca server in /tmp, and then load the debuginfo with parca-debuginfo.
After these steps, symbolization proceeds normally.
The size of Vector debuginfo is about 2.1 gigabytes. (Just build release with debug symbols enabled.)

Parca version: 0.14
Parca-agent version: 0.10
Parca and Vector deployed in k8s.
Vector: https://github.com/vectordotdev/vector

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant