multiple `- module: name` configuration has different behaviour between `modules.d/module.yml` and `filebeat.yml` #29649

aspacca · 2021-12-30T12:56:23Z

Please post all questions and issues on https://discuss.elastic.co/c/beats
before opening a Github Issue. Your questions will reach a wider audience there,
and if we confirm that there is a bug, then you can open a new issue.

For security vulnerabilities please only send reports to [email protected].
See https://www.elastic.co/community/security for more information.

Please include configurations and logs if available.

For confirmed bugs, please report:

Version: v7.16.1, v7.16.2, master
Operating System: n/a
Discuss Forum URL: https://discuss.elastic.co/t/filebeat-multiple-vpc-logs-from-different-s3-buckets-not-working/292711/10
Steps to Reproduce:
Define in filebeat.yml the following:

filebeat.modules:
  - module: aws
    cloudtrail:
      enabled: false
    cloudwatch:
      enabled: false
    ec2:
      enabled: false
    s3access:
      enabled: false
    elb:
      enabled: false
    vpcflow:
      enabled: true
      var.queue_url: "https://sqs.eu-central-1.amazonaws.com/account-id/queue-1"
  - module: aws
    cloudtrail:
      enabled: false
    cloudwatch:
      enabled: false
    ec2:
      enabled: false
    s3access:
      enabled: false
    elb:
      enabled: false
    vpcflow:
      enabled: true
      var.queue_url: "https://sqs.eu-central-1.amazonaws.com/account-id/queue-2"

run filebeat and observe how only one of the two modules/queues is actually loaded

configure the same multiple modules in modules.d/aws.yml:

  - module: aws
    cloudtrail:
      enabled: false
    cloudwatch:
      enabled: false
    ec2:
      enabled: false
    s3access:
      enabled: false
    elb:
      enabled: false
    vpcflow:
      enabled: true
      var.queue_url: "https://sqs.eu-central-1.amazonaws.com/account-id/queue-1"
  - module: aws
    cloudtrail:
      enabled: false
    cloudwatch:
      enabled: false
    ec2:
      enabled: false
    s3access:
      enabled: false
    elb:
      enabled: false
    vpcflow:
      enabled: true
      var.queue_url: "https://sqs.eu-central-1.amazonaws.com/account-id/queue-2"

run filebeat and observe how both the two modules/queues are loaded

it's not relevant for the test that the queues actually exist

The text was updated successfully, but these errors were encountered:

elasticmachine · 2021-12-30T13:09:21Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

botelastic · 2021-12-30T13:09:31Z

This issue doesn't have a Team:<team> label.

legoguy1000 · 2022-01-23T06:08:43Z

It's because for modules defined in the filebeat.yml -> filebeat.modules:, They're all loaded at the same time and the map thats created can only support a single config for each module (should be the last one) https://github.com/elastic/beats/blob/master/filebeat/fileset/modules.go#L99. The modules in their own files are loaded individually via the reloader https://github.com/elastic/beats/blob/master/filebeat/beater/crawler.go#L89 and so when it gets to the map above, there is no conflict.

Stack Trace for modules loaded from filebeat.yml
debug.PrintStack() insterted at https://github.com/elastic/beats/blob/master/filebeat/fileset/modules.go#L54

runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/elastic/beats/v7/filebeat/fileset.newModuleRegistry({0xc0002ad0e0, 0x27}, {0xc00029f840, 0x3, 0x0}, 0x1, {{0x557326445555, 0x8}, {0x557326445555, 0x8}, ...})
        /home/alex/beats/filebeat/fileset/modules.go:56 +0xe5
github.com/elastic/beats/v7/filebeat/fileset.NewModuleRegistry({0xc0006350a0, 0x2, 0xc000121f40}, {{0x557326445555, 0x8}, {0x557326445555, 0x8}, {0x55732643e27a, 0x5}, 0x1, ...}, ...)
        /home/alex/beats/filebeat/fileset/modules.go:164 +0x5f3
github.com/elastic/beats/v7/filebeat/beater.newBeater(0xc000364600, 0x55732732a7a8, 0xc000a4f0e0)
        /home/alex/beats/filebeat/beater/filebeat.go:110 +0x287
github.com/elastic/beats/v7/filebeat/beater.New.func1(0xc000125ef0, 0x5573264411f0)
        /home/alex/beats/filebeat/beater/filebeat.go:89 +0x25
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater(0xc000364600, 0xc00026ffe0)
        /home/alex/beats/libbeat/cmd/instance/beat.go:393 +0x748
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch(0xc000364600, {{0x557326445555, 0x8}, {0x557326445555, 0x8}, {0x0, 0x0}, 0x1, 0x1, {{0x0, ...}, ...}, ...}, ...)
        /home/alex/beats/libbeat/cmd/instance/beat.go:456 +0x4ea
github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1(0xc000a4fd38, 0x8020102)
        /home/alex/beats/libbeat/cmd/instance/beat.go:210 +0x628
github.com/elastic/beats/v7/libbeat/cmd/instance.Run({{0x557326445555, 0x8}, {0x557326445555, 0x8}, {0x0, 0x0}, 0x1, 0x1, {{0x0, 0x0}, ...}, ...}, ...)
        /home/alex/beats/libbeat/cmd/instance/beat.go:211 +0x25
github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1(0xc000364300, {0x55732643d9e6, 0x5, 0x5})
        /home/alex/beats/libbeat/cmd/run.go:36 +0x58
github.com/spf13/cobra.(*Command).execute(0xc000364300, {0xc000112190, 0x5, 0x5})
        /home/alex/go/pkg/mod/github.com/spf13/[email protected]/command.go:846 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0xc000364300)
        /home/alex/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
        /home/alex/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
main.main()
        /home/alex/beats/x-pack/filebeat/main.go:22 +0x25

From modules in their own files (ex. aws.yml)

runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/elastic/beats/v7/filebeat/fileset.newModuleRegistry({0xc00005be90, 0x27}, {0xc00013a888, 0x1, 0x7}, 0x40, {{0x557326445555, 0x8}, {0x557326445555, 0x8}, ...})
        /home/alex/beats/filebeat/fileset/modules.go:56 +0xe5
github.com/elastic/beats/v7/filebeat/fileset.NewModuleRegistry({0xc0002e5998, 0x1, 0x0}, {{0x557326445555, 0x8}, {0x557326445555, 0x8}, {0x55732643e27a, 0x5}, 0x1, ...}, ...)
        /home/alex/beats/filebeat/fileset/modules.go:164 +0x5f3
github.com/elastic/beats/v7/filebeat/fileset.(*Factory).createRegistry(0xc0002e5ab0, 0x557324547d19)
        /home/alex/beats/filebeat/fileset/factory.go:134 +0xb6
github.com/elastic/beats/v7/filebeat/fileset.(*Factory).Create(0xc0002862d0, {0x55732739efb8, 0xc00000e270}, 0x557326dc0ce0)
        /home/alex/beats/filebeat/fileset/factory.go:81 +0x4b
github.com/elastic/beats/v7/libbeat/cfgfile.createRunner({0x55732739e180, 0xc0002862d0}, {0x55732739efb8, 0xc00000e270}, 0xc000dfa3d0)
        /home/alex/beats/libbeat/cfgfile/list.go:178 +0x119
github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Reload(0xc0002c8f00, {0xc000dfa3e0, 0x2, 0x1})
        /home/alex/beats/libbeat/cfgfile/list.go:93 +0x739
github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Run(0xc000c862a0, {0x55732739e180, 0xc0002862d0})
        /home/alex/beats/libbeat/cfgfile/reload.go:215 +0x388
github.com/elastic/beats/v7/filebeat/beater.(*crawler).Start.func2()
        /home/alex/beats/filebeat/beater/crawler.go:104 +0x29
created by github.com/elastic/beats/v7/filebeat/beater.(*crawler).Start
        /home/alex/beats/filebeat/beater/crawler.go:103 +0x38f

aspacca added the Filebeat Filebeat label Dec 30, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 30, 2021

aspacca added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Dec 30, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 30, 2021

aspacca added bug needs_team Indicates that the issue/PR needs a Team:* label labels Dec 30, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 30, 2021

legoguy1000 mentioned this issue Jan 23, 2022

[Filebeat] Fix multiple modules in filebeat.yml #29952

Merged

4 tasks

legoguy1000 added a commit to legoguy1000/beats that referenced this issue Feb 7, 2022

elastic#29649: Fix multiple modules in filebeat.yml

0e9dbd2

kvch closed this as completed in #29952 Feb 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multiple `- module: name` configuration has different behaviour between `modules.d/module.yml` and `filebeat.yml` #29649

multiple `- module: name` configuration has different behaviour between `modules.d/module.yml` and `filebeat.yml` #29649

aspacca commented Dec 30, 2021

elasticmachine commented Dec 30, 2021

botelastic bot commented Dec 30, 2021

legoguy1000 commented Jan 23, 2022 •

edited

Loading

multiple - module: name configuration has different behaviour between modules.d/module.yml and filebeat.yml #29649

multiple - module: name configuration has different behaviour between modules.d/module.yml and filebeat.yml #29649

Comments

aspacca commented Dec 30, 2021

elasticmachine commented Dec 30, 2021

botelastic bot commented Dec 30, 2021

legoguy1000 commented Jan 23, 2022 • edited Loading

multiple `- module: name` configuration has different behaviour between `modules.d/module.yml` and `filebeat.yml` #29649

multiple `- module: name` configuration has different behaviour between `modules.d/module.yml` and `filebeat.yml` #29649

legoguy1000 commented Jan 23, 2022 •

edited

Loading