Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vsyscall mapping with remote gperftools profile causes error #339

Closed
toddlipcon opened this issue Mar 21, 2018 · 3 comments
Closed

vsyscall mapping with remote gperftools profile causes error #339

toddlipcon opened this issue Mar 21, 2018 · 3 comments

Comments

@toddlipcon
Copy link

Please answer these questions before submitting your issue. Thanks!

What version of pprof are you using?

latest github (a74ae6f)

What operating system and processor architecture are you using?

Ubuntu 16 x64

What did you do?

Tried to fetch a profile from a gperftools-based profile handler:
pprof http://my-host/pprof/profile?seconds=10

This produced the following:
pprof: unexpected negative adjusted address, mapping &{7 18446744073699065856 18446744073699069952 0 [vsyscall] false false false false 0 0} source 18446744073699066188, offset 0

This seems to be an issue handling the following vsyscall mapping from /proc/self/maps:

ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

If possible, provide a recipe for reproducing the error.
Attaching a profile you are trying to analyze is good.

Attached the profile. It works without symbolization or if I download the binary and pass it on the command line. When I try to use the remote symbolz endpoint, though, it fails as above.
broken-profile.bin.gz

toddlipcon added a commit to toddlipcon/pprof that referenced this issue Mar 21, 2018
This fixes google#339 by skipping
attempts to symbolize anything from system mappings like [vdso],
[vsyscall], etc.
@aalexand
Copy link
Collaborator

Tried to fetch a profile from a gperftools-based profile handler.

How can I get a simple server with such handler up and running?

@toddlipcon
Copy link
Author

Unfortunately that's a bit easier said than done, since gperftools doesn't itself provide an HTTP server. I hit this while running against an Apache Kudu server (http://github.com/apache/kudu) but the build takes a while so may not be convenient for you.

In order to repro I wrote a quick little Python HTTP server that responds with the bad profile on any GET requests, and with some junk on any POST request (for symbols). Here's a tarball with a little Python HTTP server:
test-server.tar.gz

You can run it with 'python server.py 12345' and then reproduce the bug:

todd@todd-laptop:~/gopath/src/github.com/google/pprof$ ./pprof http://localhost:12345/
Fetching profile over HTTP from http://localhost:12345/
Local symbolization failed for kudu-tserver: stat /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.205/lib/kudu/sbin-release/kudu-tserver: no such file or directory
Local symbolization failed for libpthread-2.12.so: stat /lib64/libpthread-2.12.so: no such file or directory
Local symbolization failed for libc-2.12.so: stat /lib64/libc-2.12.so: no such file or directory
Local symbolization failed for libstdc++.so.6.0.13: stat /usr/lib64/libstdc++.so.6.0.13: no such file or directory
Local symbolization failed for libssl.so.1.0.1e: stat /usr/lib64/libssl.so.1.0.1e: no such file or directory
Local symbolization failed for libcrypto.so.1.0.1e: stat /usr/lib64/libcrypto.so.1.0.1e: no such file or directory
Local symbolization failed for librt-2.12.so: stat /lib64/librt-2.12.so: no such file or directory
Some binary filenames not available. Symbolization may be incomplete.
Try setting PPROF_BINARY_PATH to the search path for local binaries.
pprof: unexpected negative adjusted address, mapping &{7 18446744073699065856 18446744073699069952 0 [vsyscall]  false false false false 0 0} source 18446744073699066176, offset 0

toddlipcon added a commit to toddlipcon/pprof that referenced this issue Mar 22, 2018
This fixes google#339 by skipping
attempts to symbolize anything from system mappings like [vdso],
[vsyscall], etc.
@aalexand
Copy link
Collaborator

aalexand commented May 3, 2018

This was fixed by #368.

@aalexand aalexand closed this as completed May 3, 2018
aalexand added a commit to aalexand/pprof that referenced this issue Oct 18, 2024
We added skipping unsymbolizable mappings in symbolz long time ago in
PR google#368 to address google#339 where we saw error like "unexpected negative
adjusted address".

But that error was fixed in a more proper way in subsequent google#397 to fix
another reported issue google#280 (and internal b/32020573). So skipping
unsymbolized mappings shouldn't be needed anymore.

I tested this by verifying that the test case from google#339 still works fine
with the proposed change. And that it does fail if I roll back google#397
locally.

This change is useful as we experiment with using symbolz to symbolize
JIT locations from //anon (which is unsymbolizable per current terminology).
aalexand added a commit that referenced this issue Oct 23, 2024
We added skipping unsymbolizable mappings in symbolz long time ago in
PR #368 to address #339 where we saw error like "unexpected negative
adjusted address".

But that error was fixed in a more proper way in subsequent #397 to fix
another reported issue #280 (and internal b/32020573). So skipping
unsymbolized mappings shouldn't be needed anymore.

I tested this by verifying that the test case from #339 still works fine
with the proposed change. And that it does fail if I roll back #397
locally.

This change is useful as we experiment with using symbolz to symbolize
JIT locations from //anon (which is unsymbolizable per current terminology).

Co-authored-by: Maggie Nolan Edmonds <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants