"No probes to attach" with uretprobe in a shared library

[phmarek]

What reproduces the bug?

All these as root:

$ lsof -p 615444 | grep libssl
...
sbcl    615444 user  mem    REG               0,26          13945975 /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (path dev=0,27)

$ bpftrace -l 'uretprobe:/usr/lib/x86_64-linux-gnu/libssl.so.1.1:SSL_*' | wc -l
456
$ bpftrace -p 615444 -v -e 'uretprobe:/usr/lib/x86_64-linux-gnu/libssl.so.1.1:SSL_* { printf("%7d %-40s %12d 0x%8x\n", tid, func, retval, retval); }'
INFO: node count: 10
No probes to attach

$ bpftrace -l 'uretprobe:/usr/lib/x86_64-linux-gnu/libssl.so.1.1:SSL_write'
uretprobe:/usr/lib/x86_64-linux-gnu/libssl.so.1.1:SSL_write
$ bpftrace -p 615444 -v -e 'uretprobe:/usr/lib/x86_64-linux-gnu/libssl.so.1.1:SSL_write { printf("%7d %-40s %12d 0x%8x\n", tid, func, retval, retval); }'
INFO: node count: 10
No probes to attach

Seems that uprobes in a shared library aren't found for this process.

There are probes, though -- but only ones in the binary:

$ bpftrace -p 615444 -l | cut -f 1-2 -d: | sort | uniq -c
    582 uprobe:/proc/615444/root/.../sbcl

bpftrace --info

ERROR: bpftrace currently only supports running as the root user.

That might be another thing to fix... can't I get --info as a user??

System
  OS: Linux 5.14.0-2-amd64 #1 SMP Debian 5.14.9-2 (2021-10-03)
  Arch: x86_64

Build
  version: v0.14.0
  LLVM: 11.1.0
  ORC: v2
  foreach_sym: yes
  unsafe uprobe: no
  bfd: no
  bpf_attach_kfunc: yes
  bcc_usdt_addsem: yes
  bcc bpf_attach_uprobe refcount: yes
  bcc library path resolution: yes
  libbpf: yes
  libbpf btf dump: yes
  libbpf btf dump type decl: yes
  libdw (DWARF support): no

Kernel helpers
  probe_read: yes
  probe_read_str: yes
  probe_read_user: yes
  probe_read_user_str: yes
  probe_read_kernel: yes
  probe_read_kernel_str: yes
  get_current_cgroup_id: yes
  send_signal: yes
  override_return: no
  get_boot_ns: yes
  dpath: yes

Kernel features
  Instruction limit: 1000000
  Loop support: yes
  btf (depends on Build:libbpf): yes
  map batch (depends on Build:libbpf): yes
  uprobe refcount (depends on Build:bcc bpf_attach_uprobe refcount): yes

Map types
  hash: yes
  percpu hash: yes
  array: yes
  percpu array: yes
  stack_trace: yes
  perf_event_array: yes

Probe types
  kprobe: yes
  tracepoint: yes
  perf_event: yes
  kfunc: yes
  iter:task: yes
  iter:task_file: yes

bpftrace=0.14.0-1, Debian package

[fbs]

Was able to reproduce this, just spin up the ubuntu-21.10 vagrant box. In one terminal run nc -l 1234, in the other run curl localhost:1234 and SIGSTOP both.

We definitly have some confusing behaviour, the program part of the uprobe filter is ignored, e.g.:

$ sudo bpftrace -p $(pgrep curl) -l 'u:/usr/lib/x86_64-linux-gnu/libssl.so.1.1:*' | head -n2
uprobe:/proc/11449/root/usr/bin/curl:GetFileAndPassword
uprobe:/proc/11449/root/usr/bin/curl:GetSizeParameter

$ sudo bpftrace -p $(pgrep curl) -l 'u:abc:*tool_*'
uprobe:/proc/11449/root/usr/bin/curl:tool_create_output_file.isra.0
uprobe:/proc/11449/root/usr/bin/curl:tool_debug_cb

This behaviour makes sense as we use the namespaced version but its also annoying

We don't seem to look at shared libs at all.

[fbs]

// gcc curlie.c -l curl -o curlie
// curlie.c
#include <stdio.h>
#include <curl/curl.h>
#include <unistd.h>
#include <sys/types.h>

FILE *devnull;

int curler(void)
{
  CURL *curl;
  CURLcode res;
  curl = curl_easy_init();
  if(curl) {
    curl_easy_setopt(curl, CURLOPT_URL, "https://google.com");
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, devnull);
    res = curl_easy_perform(curl);
    curl_easy_cleanup(curl);
  }
  return 0;
}

int main() {
    devnull = fopen("/dev/null", "w+");
    printf("Pid: %d\n", getpid());
    while(1) { printf("Doing request\n"), curler(); sleep(2); }
}

Running this in a terminal somewhere.

$ ./curlie
Pid: 21296
Doing request

$ sudo bpftrace -p $(pgrep curlie) -l 'u:libssl:SSL_wr*'
uprobe:/usr/lib/x86_64-linux-gnu/libssl.so.1.1:SSL_write
uprobe:/usr/lib/x86_64-linux-gnu/libssl.so.1.1:SSL_write_early_data
uprobe:/usr/lib/x86_64-linux-gnu/libssl.so.1.1:SSL_write_ex

$ sudo bpftrace -p $(pgrep curlie) -e 'u:libssl:SSL_write { printf("hit %d\n", pid) }' -q
hit 21296
hit 21296

So the short libname seems to work as expected.

Now to use the full lib path:

$ sudo bpftrace -p $(pgrep curlie) -e 'u:libssl:SSL_write { printf("hit %s %d\n", probe, pid) }' -q
hit uprobe:/usr/lib/x86_64-linux-gnu/libssl.so.1.1:SSL_write 21296
hit uprobe:/usr/lib/x86_64-linux-gnu/libssl.so.1.1:SSL_write 21296

$ sudo bpftrace -p $(pgrep curlie) -e 'uprobe:/usr/lib/x86_64-linux-gnu/libssl.so.1.1:SSL_write { printf("hit %s %d\n", probe, pid) }'
No probes to attach

So somethings going wrong there, have to dig into the code

[fbs]

ah so we have this bit of code:

#ifdef HAVE_BCC_WHICH_SO
  if (!has_wildcard(parts_[1]) && parts_[1].find("lib") == 0)
  {
    // Automatic resolution of shared library paths.
    // If the target has form "libXXX" then we use BCC to find the correct path
    // to the given library as it may differ across systems.
    auto libname = parts_[1].substr(3);
    const char *lib_path = bcc_procutils_which_so(libname.c_str(),
                                                  bpftrace_.pid());
    if (lib_path)
      ap_->target = lib_path;
  }
#endif

  if (ap_->target.empty())
  {
    if (bpftrace_.pid() > 0)
    {
      ap_->target = get_pid_exe(bpftrace_.pid());
      ap_->target = path_for_pid_mountns(bpftrace_.pid(), ap_->target);
    }
    else
      ap_->target = parts_[1];
  }

So for u:libX:sym we set the target to the full path of libX, i.e. /usr/lib/x86_64-linux-gnu/libX.so.1. However if we don't find lib at position 0 we use the binary through /proc/PID/exe which in this case is curlie. If we don't specify the pid target is set correctly but then we end up attaching system wide, not to the process you're interested in.

So the proper way to attach now is to do -p PID -e 'u:libX:sym { prog.. }, using just the lib name. Although by the looks of it that will fail with namespaces.

I think it makes sense to transform this into:

diff --git a/src/ast/attachpoint_parser.cpp b/src/ast/attachpoint_parser.cpp
index f0b96243..348da3bf 100644
--- a/src/ast/attachpoint_parser.cpp
+++ b/src/ast/attachpoint_parser.cpp
@@ -351,7 +351,8 @@ AttachPointParser::State AttachPointParser::uprobe_parser(bool allow_offset,
   {
     // For PID, the target may be skipped
     parts_.push_back(parts_[1]);
-    parts_[1] = "";
+    auto target = get_pid_exe(bpftrace_.pid());
+    parts_[1] = path_for_pid_mountns(bpftrace_.pid(), target);
   }
   if (parts_.size() != 3)
   {
@@ -380,13 +381,7 @@ AttachPointParser::State AttachPointParser::uprobe_parser(bool allow_offset,

   if (ap_->target.empty())
   {
-    if (bpftrace_.pid() > 0)
-    {
-      ap_->target = get_pid_exe(bpftrace_.pid());
-      ap_->target = path_for_pid_mountns(bpftrace_.pid(), ap_->target);
-    }
-    else
-      ap_->target = parts_[1];
+    ap_->target = parts_[1];
   }
   // Handle uprobe:/lib/asdf:func+0x100 case

That way we use the pid exe if the probe only has 2 parts (u:SSL_write) but uses what the user specified if the probe has 3 parts, even if a pid is also specified.

The manual states

+When tracing libraries, it is sufficient to specify the library name instead of
+a full path. The path will be then automatically resolved using `/etc/ld.so.cache`:

sufficient is not required so both behaviours should work.

Although this behaviour was there before we added that feature:

  if (bpftrace_.pid() > 0)
  {
    ap_->target = get_pid_exe(bpftrace_.pid());
    ap_->target = path_for_pid_mountns(bpftrace_.pid(), ap_->target);
  }
  else
    ap_->target = parts_[1];

@viktormalik @danobi , thoughts on this?

[viktormalik]

Yes, this is definitely a problem that should be fixed.

The proposed change looks good, the only problem might be that we are breaking the existing (although incorrect) behaviour. If someone had a script that used a PID and a custom target which didn't match the pid exe, then the script will stop working now (it was overridden by the pid exe before).

We should at least properly document the PID behaviour.

[danobi]

I'd need to look at the code more, but I think the reported issue does need fixing.

If someone had a script that used a PID and a custom target which didn't match the pid exe

I'm not concerned about this case. It's kind of nonsensical anyways

[phmarek]

Well, I'd see that as a feature - better to quit on typos immediately than to sit in the terminal and not telling the user that something is wrong...