Disable a symbol name cache if ASLR is enabled and -c option is not…

… given

`resolve_usym()` caches a `bcc_symbol` object using an executable name
as a key, but on the ASLR-enabled platform, symbol addresses change with
each execution. Disable (discard) a cache, in this case, to resolve
symbol names properly.

Introduce `BPFTRACE_CACHE_USER_SYMBOLS` env variable to force caching.
Caching is fine if only trace one program execution.

Note and known issues:

- A cache is discarded whenever resolve_usym is called even if a pid is
the same as the old one. This is because pid may be reused.
- This does not check whether a binary is PIE ASLAR or not. Note that
even if a binary is not PIE ASLR, addresses of shared libraries are
randomized if ASLR is enabled.  (If a binary is not PIE ASLR and
`resolve_usym()` resolves symbol in a binary, we can utilize a cache.)
- If ASLR is disabled on the first execution but enabled on the second
execution, `resolve_usym()` for the second run will use the previous
cache.
- As discussed in bpftrace#246, symbolizing will fail after process termination
(this is a separate issue). For example:

```
% bpftrace -e 'u:/lib/x86_64-linux-gnu/libc.so.6:*nanosleep* /comm == "sleep"/ { @[ustack] = count(); }'
Attaching 7 probes...
^C

@[
    0x7ff1917cb990
]: 3
@
[
    0x7fea4211c990
]: 3
@
[
    0x7f32bc51a990
]: 3
```

-----

Closes bpftrace#1031 and solves the second part of bpftrace#75.

docs/reference_guide.md

@@ -140,12 +140,13 @@ OPTIONS:
     --version      bpftrace version
 
 ENVIRONMENT:
-    BPFTRACE_STRLEN           [default: 64] bytes on BPF stack per str()
-    BPFTRACE_NO_CPP_DEMANGLE  [default: 0] disable C++ symbol demangling
-    BPFTRACE_MAP_KEYS_MAX     [default: 4096] max keys in a map
-    BPFTRACE_MAX_PROBES       [default: 512] max number of probes bpftrace can attach to
-    BPFTRACE_VMLINUX          [default: None] vmlinux path used for kernel symbol resolution
-    BPFTRACE_BTF              [default: None] BTF file
+    BPFTRACE_STRLEN             [default: 64] bytes on BPF stack per str()
+    BPFTRACE_NO_CPP_DEMANGLE    [default: 0] disable C++ symbol demangling
+    BPFTRACE_MAP_KEYS_MAX       [default: 4096] max keys in a map
+    BPFTRACE_MAX_PROBES         [default: 512] max number of probes bpftrace can attach to
+    BPFTRACE_CACHE_USER_SYMBOLS [default: auto] enable user symbol cache
+    BPFTRACE_VMLINUX            [default: none] vmlinux path used for kernel symbol resolution
+    BPFTRACE_BTF                [default: none] BTF file
 
 EXAMPLES:
 bpftrace -l '*sleep*'
@@ -464,15 +465,24 @@ This is the maximum number of probes that bpftrace can attach to. Increasing the
 memory, increase startup times and can incur high performance overhead or even freeze or crash the
 system.
 
-### 9.5 `BPFTRACE_VMLINUX`
+### 9.5 `BPFTRACE_CACHE_USER_SYMBOLS`
+
+Default: 0 if ASLR is enabled on system and `-c` option is not given; otherwise 1
+
+By default, bpftrace caches the results of symbols resolutions only when ASLR (Address Space Layout
+Randomization) is disabled. This is because the symbol addresses change with each execution with ASLR.
+However, disabling caching may incur some performance. Set this env variable to 1 to force bpftrace to
+cache. This is fine if only trace one program execution.
+
+### 9.6 `BPFTRACE_VMLINUX`
 
 Default: None
 
 This specifies the vmlinux path used for kernel symbol resolution when attaching kprobe to offset.
 If this value is not given, bpftrace searches vmlinux from pre defined locations.
 See src/attached_probe.cpp:find_vmlinux() for details.
 
-### 9.6 `BPFTRACE_BTF`
+### 9.7 `BPFTRACE_BTF`
 
 Default: None
 

src/bpftrace.cpp

@@ -8,6 +8,7 @@
 #include <arpa/inet.h>
 
 #include <fcntl.h>
+#include <sys/personality.h>
 #include <sys/prctl.h>
 #include <sys/stat.h>
 #include <sys/types.h>
@@ -1703,6 +1704,51 @@ std::string BPFtrace::resolve_inet(int af, const uint8_t* inet) const
   return addrstr;
 }
 
+// /proc/sys/kernel/randomize_va_space >= 1 and        // system-wide
+// (/proc/<pid>/personality & ADDR_NO_RNDOMIZE) == 0   // this pid
+// if pid == -1, then only check system-wide setting
+bool BPFtrace::is_aslr_enabled(int pid)
+{
+  std::string randomize_va_space_file = "/proc/sys/kernel/randomize_va_space";
+  std::string personality_file = "/proc/" + std::to_string(pid) +
+                                 "/personality";
+
+  {
+    std::ifstream file(randomize_va_space_file);
+    if (file.fail())
+    {
+      if (bt_verbose)
+        std::cerr << strerror(errno) << ": " << randomize_va_space_file
+                  << std::endl;
+      // conservatively return true
+      return true;
+    }
+
+    std::string line;
+    if (std::getline(file, line) && std::stoi(line) < 1)
+      return false;
+  }
+
+  if (pid == -1)
+    return true;
+
+  {
+    std::ifstream file(personality_file);
+    if (file.fail())
+    {
+      if (bt_verbose)
+        std::cerr << strerror(errno) << ": " << personality_file << std::endl;
+      return true;
+    }
+    std::string line;
+    if (std::getline(file, line) &&
+        ((std::stoi(line) & ADDR_NO_RANDOMIZE) == 0))
+      return true;
+  }
+
+  return false;
+}
+
 std::string BPFtrace::resolve_usym(uintptr_t addr, int pid, bool show_offset, bool show_module)
 {
   struct bcc_symbol usym;
@@ -1717,16 +1763,23 @@ std::string BPFtrace::resolve_usym(uintptr_t addr, int pid, bool show_offset, bo
 
   if (resolve_user_symbols_)
   {
-    std::string pid_exe = get_pid_exe(pid);
-    if (exe_sym_.find(pid_exe) == exe_sym_.end())
+    if (cache_user_symbols_)
     {
-      // not cached, create new ProcSyms cache
-      psyms = bcc_symcache_new(pid, &symopts);
-      exe_sym_[pid_exe] = std::make_pair(pid, psyms);
+      std::string pid_exe = get_pid_exe(pid);
+      if (exe_sym_.find(pid_exe) == exe_sym_.end())
+      {
+        // not cached, create new ProcSyms cache
+        psyms = bcc_symcache_new(pid, &symopts);
+        exe_sym_[pid_exe] = std::make_pair(pid, psyms);
+      }
+      else
+      {
+        psyms = exe_sym_[pid_exe].second;
+      }
     }
     else
     {
-      psyms = exe_sym_[pid_exe].second;
+      psyms = bcc_symcache_new(pid, &symopts);
     }
   }
 
@@ -1748,6 +1801,9 @@ std::string BPFtrace::resolve_usym(uintptr_t addr, int pid, bool show_offset, bo
       symbol << " ([unknown])";
   }
 
+  if (psyms && !cache_user_symbols_)
+    bcc_free_symcache(psyms, pid);
+
   return symbol.str();
 }
 

src/bpftrace.h

@@ -121,6 +121,7 @@ class BPFtrace
   virtual pid_t child_pid() { return child_pid_; };
   int spawn_child();
   void kill_child();
+  bool is_aslr_enabled(int pid);
 
   std::string cmd_;
   int pid_{0};
@@ -153,6 +154,7 @@ class BPFtrace
   uint64_t log_size_ = 409600;
   bool demangle_cpp_symbols_ = true;
   bool resolve_user_symbols_ = true;
+  bool cache_user_symbols_ = true;
   bool safe_mode_ = true;
   bool force_btf_ = false;
 

src/main.cpp

@@ -58,15 +58,16 @@ void usage()
   std::cerr << "    --info         Print information about kernel BPF support" << std::endl;
   std::cerr << "    -V, --version  bpftrace version" << std::endl << std::endl;
   std::cerr << "ENVIRONMENT:" << std::endl;
-  std::cerr << "    BPFTRACE_STRLEN           [default: 64] bytes on BPF stack per str()" << std::endl;
-  std::cerr << "    BPFTRACE_NO_CPP_DEMANGLE  [default: 0] disable C++ symbol demangling" << std::endl;
-  std::cerr << "    BPFTRACE_MAP_KEYS_MAX     [default: 4096] max keys in a map" << std::endl;
-  std::cerr << "    BPFTRACE_CAT_BYTES_MAX    [default: 10k] maximum bytes read by cat builtin" << std::endl;
-  std::cerr << "    BPFTRACE_MAX_PROBES       [default: 512] max number of probes" << std::endl;
-  std::cerr << "    BPFTRACE_LOG_SIZE         [default: 409600] log size in bytes" << std::endl;
-  std::cerr << "    BPFTRACE_NO_USER_SYMBOLS  [default: 0] disable user symbol resolution" << std::endl;
-  std::cerr << "    BPFTRACE_VMLINUX          [default: None] vmlinux path used for kernel symbol resolution" << std::endl;
-  std::cerr << "    BPFTRACE_BTF              [default: None] BTF file" << std::endl;
+  std::cerr << "    BPFTRACE_STRLEN             [default: 64] bytes on BPF stack per str()" << std::endl;
+  std::cerr << "    BPFTRACE_NO_CPP_DEMANGLE    [default: 0] disable C++ symbol demangling" << std::endl;
+  std::cerr << "    BPFTRACE_MAP_KEYS_MAX       [default: 4096] max keys in a map" << std::endl;
+  std::cerr << "    BPFTRACE_CAT_BYTES_MAX      [default: 10k] maximum bytes read by cat builtin" << std::endl;
+  std::cerr << "    BPFTRACE_MAX_PROBES         [default: 512] max number of probes" << std::endl;
+  std::cerr << "    BPFTRACE_LOG_SIZE           [default: 409600] log size in bytes" << std::endl;
+  std::cerr << "    BPFTRACE_NO_USER_SYMBOLS    [default: 0] disable user symbol resolution" << std::endl;
+  std::cerr << "    BPFTRACE_CACHE_USER_SYMBOLS [default: auto] enable user symbol cache" << std::endl;
+  std::cerr << "    BPFTRACE_VMLINUX            [default: none] vmlinux path used for kernel symbol resolution" << std::endl;
+  std::cerr << "    BPFTRACE_BTF                [default: none] BTF file" << std::endl;
   std::cerr << std::endl;
   std::cerr << "EXAMPLES:" << std::endl;
   std::cerr << "bpftrace -l '*sleep*'" << std::endl;
@@ -444,6 +445,28 @@ int main(int argc, char *argv[])
     }
   }
 
+  if (const char* env_p = std::getenv("BPFTRACE_CACHE_USER_SYMBOLS"))
+  {
+    std::string s(env_p);
+    if (s == "1")
+      bpftrace.cache_user_symbols_ = true;
+    else if (s == "0")
+      bpftrace.cache_user_symbols_ = false;
+    else
+    {
+      std::cerr << "Env var 'BPFTRACE_CACHE_USER_SYMBOLS' did not contain a "
+                   "valid value (0 or 1)."
+                << std::endl;
+      return 1;
+    }
+  }
+  else
+  {
+    // enable user symbol cache if ASLR is disabled on system or `-c` option is
+    // given
+    bpftrace.cache_user_symbols_ = cmd_str || !bpftrace.is_aslr_enabled(-1);
+  }
+
   if (cmd_str)
     bpftrace.cmd_ = cmd_str;