take_over: true is ignored when using input configuration from external files

[rdner]

If inputs are set via the external file configuration using: https://www.elastic.co/guide/en/beats/filebeat/8.9/filebeat-configuration-reloading.html#load-input-config the take_over parameter is ignored and log inputs won't migrate to filestream causing data duplication and reset offsets for each affected file.

This happens because the take-over configuration is initialised only from the filebeat.inputs section and does not take into account external files:

beats/filebeat/input/filestream/takeover/takeover.go

Lines 144 to 171 in eca0a0b

	func findFilestreams(log *logp.Logger, cfg *cfg.Config) (matchers filestreamMatchers, err error) {
	matchers = make(filestreamMatchers)
	for _, input := range cfg.Inputs {
	inputCfg := defaultInputConfig()
	err := input.Unpack(&inputCfg)
	if err != nil {
	return nil, fmt.Errorf("failed to unpack input configuration: %w", err)
	}
	if inputCfg.Type != "filestream" || !inputCfg.TakeOver {
	continue
	}
	if _, exists := matchers[inputCfg.ID]; exists || inputCfg.ID == "" {
	return matchers, fmt.Errorf("filestream with ID `%s` in `take over` mode requires a unique ID. Add the `id:` key with a unique value.", inputCfg.ID)
	}
	matchers[inputCfg.ID], err = createMatcher(log, inputCfg)
	if err != nil {
	return nil, fmt.Errorf("failed to create filestream matcher: %w", err)
	}
	}
	if len(matchers) > 0 {
	log.Infof("found %d filestream inputs in `take over` mode", len(matchers))
	}
	return matchers, nil
	}

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[fixed77]

filebeat 8.15.0. problem returned

[nix-power]

I confirm: 8.15.0 problem exists. using filebeat in the container.

[nix-power]

@rdner - can this one be reopened please ?

[rdner]

@fixed77 @nix-power do you mean your use-case worked before 8.15.0 and with the 8.15.0 release it broke again? I don't recall any change in 8.15.0 that could affect this functionality.

Can you post an example of your configuration files that can verify the bug is back? Thanks in advance.

[nix-power]

Hi, thanks for the reply.
I did not try this feature in the previous versions, i try to migrate from 8.4.1 to 8.15.0 and eventually to use the change to migrate from the log to the filestream input. The following configuration is not working, and events are duplicated. But i don't think it is related to this specific bug

 filebeat.inputs:
      - type: filestream
        enabled: true
        id: myapp-log-filestream
        take_over: true
        fields_under_root: true
        fields:
          SUB_SYSTEM: 'myapp_log'
        paths:
          - 'path/to/app_log'
        close_inactive: 2m
        ignore_older: 5m
        prospector.scanner.symlinks: true
        parsers:
          - multiline:
              pattern: '^\[?[0-9]{1,}[[:space:]]'
              negate: true
              match: after

filebeat is running in the docker container, and the configuration file filebeat.yml mounted to /usr/share/filebeat/filebeat.yml inside the container.

[fixed77]

filebeat 8.15.0
there is a working filebit with the configuration:

filebeat.yml:

#=========================== Filebeat inputs =============================
filebeat.config:
  inputs:
    enabled: true
    path: inputs.d/*.yml
    reload.enabled: false
    reload.period: 60s
    clean_removed: false
    close_removed: false
    clean_renamed: false
    close_renamed: false
    close_timeout: 1h

inputs.d/nginx.yml

- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log

let's try switching to filestream. change inputs.d/nginx.yml to:

- type: filestream
  id: fs-nginx-accesslog
  take_over: true
  enabled: true
  paths:
    - /var/log/nginx/access.log

and restart filebeat.
data is duplicated and there is no mention of take_over in the logs.

if all input files move to the main filebeat.yml config, then everything goes well

[rdner]

@nix-power it does not seem like the issue is related to applying take_over: true from the external configuration files. Please check the debug logs to see if the "take over" migration actually happens. I assume you mount the Filebeat data folder as well where the file offsets are persisted.

@fixed77 it does match the description of the original issue, I'm re-opening this to investigate further.

[fixed77]

currently use filebeat version 8.15.0.
Can you tell me what version of filebeat can I downgrade to where the "take over" functionality works properly?
It seems like no one is working on the current task, and I need to switch to filestream. Duplication of events cannot be allowed.