Metricbeat: test_filesystem fails on macOS

[sharbuz]

Flaky Test

• Test Name: test_filesystem (metricbeat/module/system/test_system.py)
  Link:

  beats/metricbeat/module/system/test_system.py

  Lines 261 to 288 in d80ab3b

  	@unittest.skipUnless(re.match("(?i)win|linux|darwin|freebsd|openbsd", sys.platform), "os")
  	def test_filesystem(self):
  	"""
  	Test system/filesystem output.
  	"""
  	self.render_config_template(modules=[{
  	"name": "system",
  	"metricsets": ["filesystem"],
  	"period": "5s",
  	# prevent permissions issues on systems with docker
  	"extras": {
  	"filesystem.ignore_types": ["nsfs",
  	"sysfs", "tmpfs", "bdev", "proc", "cgroup", "cgroup2", "cpuset",
  	"devtmpfs", "configfs", "debugfs", "tracefs", "securityfs", "sockfs",
  	"bpf", "pipefs", "ramfs", "hugetlbfs", "devpts", "autofs", "efivarfs",
  	"mqueue", "selinuxfs", "binder", "pstore", "fuse", "fusectl", "rpc_pipefs",
  	"overlay", "binfmt_misc", "unavailable"],
  	}
  	}])
  	self.run_beat_and_stop()
  	output = self.read_output_json()
  	self.assertGreater(len(output), 0)
  	for evt in output:
  	self.assert_fields_are_documented(evt)
  	filesystem = evt["system"]["filesystem"]
  	self.assert_fields_for_platform(SYSTEM_FILESYSTEM, filesystem)

• Branch:
  -- main: Enable tests that were muted during migration #40387
  -- 7.17: Enable tests that were muted during migration 7.17 #40398

NB! tests should be enabled back, since those were muted while migrating from Jenkins to Buildkite. Please check changed files in mentioned PRs:
-- metricbeat/module/system/test_system.py
-- metricbeat/tests/system/test_reload.py

• Artifact Link:
  -- main: beats-metricbeat_build_8052_mac-metricbeat-macos-x64-64-unit-tests.log
  -- 7.17: beats-metricbeat_build_8089_mac-metricbeat-macos-x64-64-unit-tests.log

• Notes: Additional details about the test. e.g. theory as to failure cause
  -- Discussion: https://elastic.slack.com/archives/C0522G6FBNE/p1705570512029519
  -- Temporary solution: Temporarily disable filesystem check for MacOS unit-tests #37687
  -- The similar issue: [metricbeat] MacOS FAIL: metricbeat/module/system/filesystem TestFileSystemList #29038
  -- Buildkite builds: main, 7.17

Stack Trace

=================================== FAILURES ===================================
_____________________________ Test.test_filesystem _____________________________
self = <test_system.Test testMethod=test_filesystem>
    @unittest.skipUnless(re.match("(?i)win|linux|darwin|freebsd|openbsd", sys.platform), "os")
    def test_filesystem(self):
        """
        Test system/filesystem output.
        """
        self.render_config_template(modules=[{
            "name": "system",
            "metricsets": ["filesystem"],
            "period": "5s",
            # prevent permissions issues on systems with docker
            "extras": {
                "filesystem.ignore_types": ["nsfs",
                                            "sysfs", "tmpfs", "bdev", "proc", "cgroup", "cgroup2", "cpuset",
                                            "devtmpfs", "configfs", "debugfs", "tracefs", "securityfs", "sockfs",
                                            "bpf", "pipefs", "ramfs", "hugetlbfs", "devpts", "autofs", "efivarfs",
                                            "mqueue", "selinuxfs", "binder", "pstore", "fuse", "fusectl", "rpc_pipefs",
                                            "overlay", "binfmt_misc", "unavailable"],
            }
        }])
        self.run_beat_and_stop()
        output = self.read_output_json()
        self.assertGreater(len(output), 0)
        for evt in output:
            self.assert_fields_are_documented(evt)
            filesystem = evt["system"]["filesystem"]
>           self.assert_fields_for_platform(SYSTEM_FILESYSTEM, filesystem)
module/system/test_system.py:288:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/system/metricbeat.py:74: in assert_fields_for_platform
    self.assertCountEqual(self.de_dot(
E   AssertionError: Element counts were not equal:
E   First has 1, Second has 0:  'options'

Tasks

Preview Give feedback

1. Fix MacOS filesystem check
2. Enable the filesystem check for MacOS unit-tests

[pazone]

Looks like merticbeat iterates over all mounted disks and gets their filesystem info.
The test expects the field options to be present in the metricbeat filesystem json output for each of the disks.
After BK agent migration to MacOS 13(Orka) the option field is not available on one(or more) mounted disks.
Is the options field critical? The fields.yml says it represents The options present on the filesystem mount.
The example output doesn't contain the options field. I might be wrong but that makes me think that the options field is optional.
As a quick solution, we can exclude the options from the list of mandatory fields for MacOS.
Alternatively, we can try to understand which disk doesn't contain any options and try to ignore/remove it.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[rowlandgeoff]

During the migration of beats-ci from Jenkins to Buildkite, a number of tests were failing consistently due to issues unrelated to the migration. Those tests were disabled to stabilize the CI, with the intent to revisit them post-migration. @oakrizan has reviewed them all in her draft PRs linked above in the description, and has opened tickets such as this one to highlight to the product teams the tests that are currently still disabled and could use some attention.