arguments not correctly parsed on auditbeat 7.17.5

[jalogisch]

When you run Auditbeat to collect system events, data does not look right and might not be picked up completely. Not sure if this is a cosmetical issue or something serious as I did the testing with fields that might get redacted at some point.

Running against Elasticsearch 7.17.5 in the cloud and the following configuration:

cloud.id: "test:werwerwerwerwer="
cloud.auth: "elastic:werwerwerwer"

# =========================== Modules configuration ============================
auditbeat.modules:

- module: auditd
  audit_rules: |
    ## Define audit rules here.
    ## Create file watches (-w) or syscall audits (-a or -A). Uncomment these

    ## Stunnel
    -w /usr/sbin/stunnel -p x -k stunnel
    -w /usr/bin/stunnel -p x -k stunnel
    ## Reconnaissance
    -w /usr/bin/whoami -p x -k recon
    -w /usr/bin/id -p x -k recon
    -w /bin/hostname -p x -k recon
    -w /bin/uname -p x -k recon
    -w /etc/issue -p r -k recon
    -w /etc/hostname -p r -k recon
    ## Suspicious activity
    -w /usr/bin/wget -p x -k susp_activity
    -w /usr/bin/curl -p x -k susp_activity
    -w /usr/bin/base64 -p x -k susp_activity
    -w /bin/nc -p x -k susp_activity
    -w /bin/netcat -p x -k susp_activity
    -w /usr/bin/ncat -p x -k susp_activity
    -w /usr/bin/ssh -p x -k susp_activity
    -w /usr/bin/scp -p x -k susp_activity
    -w /usr/bin/sftp -p x -k susp_activity
    -w /usr/bin/ftp -p x -k susp_activity
    -w /usr/bin/socat -p x -k susp_activity
    -w /usr/bin/wireshark -p x -k susp_activity
    -w /usr/bin/tshark -p x -k susp_activity
    -w /usr/bin/rawshark -p x -k susp_activity
    -w /usr/bin/rdesktop -p x -k susp_activity
    -w /usr/bin/nmap -p x -k susp_activity



- module: file_integrity
  paths:
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc
  hash_types: [sha1,sha256,md5]

  recursive: true

- module: system
  datasets:
    - host
    - login
    - user
    - package
  period: 2m

- module: system
  datasets:
    - process
    - socket
  period: 1s
  socket.include_localhost: false

  state.period: 12h


# ================================== General ===================================

fields:
  os.type: linux
fields_under_root: true

# ================================== Outputs ===================================


# ================================= Processors =================================

processors:
  - add_host_metadata:
  - add_process_metadata:
      match_pids: [process.ppid]
      target: process.parent
  - drop_event:
      when:
        or:
          - equals:
              network.direction: unknown
          - and:
              - equals:
                  network.direction: ingress
              - or:
                  - equals:
                      destination.port: 1234
                  - equals:
                      destination.port: 1234
                  - equals:
                      destination.port: 1234
          - and:
              - equals:
                  network.direction: egress
              - equals:
                  server.port: 1234
          - and:
              - equals:
                  server.domain: "server.domain"
              - equals:
                  server.port: 1234
          - and:
              - equals:
                  server.ip: 11.11.11.11
              - equals:
                  server.port: 1234
          - and:
              - equals:
                  server.ip: 22.22.22.22
              - equals:
                  server.port: 1234

On the system and forcing a process that is recognized echo "encryptme" | openssl enc -aes-256-cbc -md sha256 -k TOP_SECRET -nosalt created the following entry in process.args

openssl, enc, -aes-256-cbc, -md, sha256, -k, (empty), (empty), (empty), (empty), (empty), (empty), (empty), (empty), (empty), (empty), (empty), -nosalt

Full message

{
  "_index": "auditbeat-7.17.5-2022.08.08-000001",
  "_type": "_doc",
  "_id": "VBEmfoIBiNC6JVjyVuFZ",
  "_version": 1,
  "_score": 1,
  "_source": {
    "@timestamp": "2022-08-08T15:50:17.923Z",
    "process": {
      "executable": "/usr/bin/openssl",
      "working_directory": "/home/jan.doberstein",
      "args": [
        "openssl",
        "enc",
        "-aes-256-cbc",
        "-md",
        "sha256",
        "-k",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "-nosalt"
      ],
      "pid": 4806,
      "parent": {
        "process": {
          "pid": 4291,
          "start_time": "2022-08-08T14:41:08.920Z",
          "name": "bash",
          "executable": "/usr/bin/bash",
          "ppid": 4290,
          "parent": {
            "pid": 4290
          },
          "title": "-bash",
          "args": [
            "-bash"
          ]
        }
      },
      "start": "2022-08-08T15:50:15.280Z",
      "entity_id": "9Qj1v0mgckLREPSS",
      "name": "openssl",
      "hash": {
        "sha1": "bc5d38d71c83920f58a152b68b04f7a2fe2dd89a"
      },
      "ppid": 4291
    },
    "host": {
      "architecture": "x86_64",
      "os": {
        "type": "linux",
        "platform": "rhel",
        "version": "7.9 (Maipo)",
        "family": "redhat",
        "name": "Red Hat Enterprise Linux Server",
        "kernel": "3.10.0-1160.71.1.el7.x86_64",
        "codename": "Maipo"
      },
      "id": "bcfaa0286a21d5af46057ce387ab204b",
      "containerized": false,
      "ip": [
        "10.200.0.5",
        "fe80::ed6c:e9c7:947b:a1b3"
      ],
      "mac": [
        "42:01:0a:c8:00:05"
      ],
      "name": "jan.doberstein.auditbeats.7175",
      "hostname": "jan.doberstein.auditbeats.7175"
    },
    "agent": {
      "name": "jan.doberstein.auditbeats.7175",
      "type": "auditbeat",
      "version": "7.17.5",
      "hostname": "jan.doberstein.auditbeats.7175",
      "ephemeral_id": "cfa3caec-bb36-4b4b-b04b-1bd203014b19",
      "id": "0d2eb7f3-ee2d-48ac-991c-22cf207d0674"
    },
    "message": "Process openssl (PID: 4806) by user jan.doberstein STOPPED",
    "event": {
      "type": [
        "end"
      ],
      "action": "process_stopped",
      "module": "system",
      "dataset": "process",
      "kind": "event",
      "category": [
        "process"
      ]
    },
    "user": {
      "name": "jan.doberstein",
      "id": "1080",
      "group": {
        "id": "1081",
        "name": "jan.doberstein"
      },
      "effective": {
        "id": "1080",
        "group": {
          "id": "1081"
        }
      },
      "saved": {
        "group": {
          "id": "1081"
        },
        "id": "1080"
      }
    },
    "service": {
      "type": "system"
    },
    "os": {
      "type": "linux"
    },
    "ecs": {
      "version": "1.12.0"
    }
  },
  "fields": {
    "process.parent.process.executable": [
      "/usr/bin/bash"
    ],
    "event.category": [
      "process"
    ],
    "process.name.text": [
      "openssl"
    ],
    "host.os.name.text": [
      "Red Hat Enterprise Linux Server"
    ],
    "host.hostname": [
      "jan.doberstein.auditbeats.7175"
    ],
    "process.pid": [
      4806
    ],
    "host.mac": [
      "42:01:0a:c8:00:05"
    ],
    "service.type": [
      "system"
    ],
    "host.os.version": [
      "7.9 (Maipo)"
    ],
    "user.group.name": [
      "jan.doberstein"
    ],
    "host.os.name": [
      "Red Hat Enterprise Linux Server"
    ],
    "agent.name": [
      "jan.doberstein.auditbeats.7175"
    ],
    "host.name": [
      "jan.doberstein.auditbeats.7175"
    ],
    "event.kind": [
      "event"
    ],
    "user.id": [
      "1080"
    ],
    "host.os.type": [
      "linux"
    ],
    "process.ppid": [
      4291
    ],
    "agent.hostname": [
      "jan.doberstein.auditbeats.7175"
    ],
    "process.parent.process.start_time": [
      "2022-08-08T14:41:08.920Z"
    ],
    "process.parent.process.title": [
      "-bash"
    ],
    "process.parent.process.pid": [
      4291
    ],
    "user.saved.group.id": [
      "1081"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "process.name": [
      "openssl"
    ],
    "agent.id": [
      "0d2eb7f3-ee2d-48ac-991c-22cf207d0674"
    ],
    "host.containerized": [
      false
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "agent.version": [
      "7.17.5"
    ],
    "user.group.id": [
      "1081"
    ],
    "host.os.family": [
      "redhat"
    ],
    "process.start": [
      "2022-08-08T15:50:15.280Z"
    ],
    "user.saved.id": [
      "1080"
    ],
    "user.effective.group.id": [
      "1081"
    ],
    "process.parent.process.name": [
      "bash"
    ],
    "user.effective.id": [
      "1080"
    ],
    "user.name": [
      "jan.doberstein"
    ],
    "process.parent.process.parent.pid": [
      4290
    ],
    "os.type": [
      "linux"
    ],
    "process.working_directory": [
      "/home/jan.doberstein"
    ],
    "process.entity_id": [
      "9Qj1v0mgckLREPSS"
    ],
    "host.ip": [
      "10.200.0.5",
      "fe80::ed6c:e9c7:947b:a1b3"
    ],
    "agent.type": [
      "auditbeat"
    ],
    "process.executable.text": [
      "/usr/bin/openssl"
    ],
    "event.module": [
      "system"
    ],
    "process.parent.process.ppid": [
      4290
    ],
    "host.os.kernel": [
      "3.10.0-1160.71.1.el7.x86_64"
    ],
    "host.id": [
      "bcfaa0286a21d5af46057ce387ab204b"
    ],
    "process.executable": [
      "/usr/bin/openssl"
    ],
    "process.parent.process.args": [
      "-bash"
    ],
    "process.working_directory.text": [
      "/home/jan.doberstein"
    ],
    "host.os.codename": [
      "Maipo"
    ],
    "process.args": [
      "openssl",
      "enc",
      "-aes-256-cbc",
      "-md",
      "sha256",
      "-k",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "",
      "-nosalt"
    ],
    "message": [
      "Process openssl (PID: 4806) by user jan.doberstein STOPPED"
    ],
    "event.action": [
      "process_stopped"
    ],
    "@timestamp": [
      "2022-08-08T15:50:17.923Z"
    ],
    "host.os.platform": [
      "rhel"
    ],
    "event.type": [
      "end"
    ],
    "agent.ephemeral_id": [
      "cfa3caec-bb36-4b4b-b04b-1bd203014b19"
    ],
    "process.hash.sha1": [
      "bc5d38d71c83920f58a152b68b04f7a2fe2dd89a"
    ],
    "event.dataset": [
      "process"
    ],
    "user.name.text": [
      "jan.doberstein"
    ]
  }
}

I think this is connected to #29410 and even if this is only cosmetical it creates the impression that important data is missing.

[efd6]

The logic that is causing this is as follows:

1. A security sensitive command is invoked with secrets on the command line.
2. The command is aware of the sensitivity and zeros out the secrets in argv resulting in an array (length = the length of the secret+1) of strings that are zero length.
3. In the case here, the number of command line arguments is greater than maxProgArgs (5) and so this code is run to get the command line arguments.

   if argc > maxProgArgs || truncatedArg {
   	// Attempt to get complete args list from /proc/<pid>/cmdline.
   	cmdline, err := os.ReadFile(fmt.Sprintf("/proc/%d/cmdline", e.Meta.PID))
   	if err == nil {
   		p.args = strings.Split(strings.TrimRight(string(cmdline), "\x00"), "\x00")
   	}
   }
   

4. This gives us one empty string for each zero byte, hence the multiple empty strings in the args array.

This is confirmed by the following experiment:

$ cat bigfile | openssl enc -aes-256-cbc -md sha256 -k password -nosalt
^Z
$ ps ax | grep openssl                           
31412 pts/16   T      0:00 openssl enc -aes-256-cbc -md sha256 -k          -nosalt
$ cat /proc/31412/cmdline | hexdump -C
00000000  6f 70 65 6e 73 73 6c 00  65 6e 63 00 2d 61 65 73  |openssl.enc.-aes|
00000010  2d 32 35 36 2d 63 62 63  00 2d 6d 64 00 73 68 61  |-256-cbc.-md.sha|
00000020  32 35 36 00 2d 6b 00 00  00 00 00 00 00 00 00 00  |256.-k..........|
00000030  2d 6e 6f 73 61 6c 74 00                           |-nosalt.|
00000038

Here you can see that the password (literally "password") is blanked out in the output from ps and the bytes that would correspond to the password in /proc/31412/cmdline are zeroed out.

Fixing the behaviour here would break other arguably reasonable argument lists since a "" cannot be distinguished from a NULL in argv. argc could potentially be used in conjunction with the length of the obtained arg list from /proc/<pid>/cmdline, though this would depend on heuristics on what constitutes a reasonable command line (i.e. adjacent empty string argument only appear after all flagged arguments) and would make the code more brittle.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

After discussion with others on the team, I'm going to close this as working as intended; the information is not available because a security-aware application has scrubbed the process's argv.

[andrewkroh]

The system.process dataset utilizes polling of /proc on Linux to collect data. I would expect other collection methods that are hooked into the kernel like monitoring execve syscalls via the auditd module in Auditbeat or utilizing Elastic Endpoint should be able to capture the data before argv is modified.