[Automatic Import] Fixes the CSV header bug

[ilyannn]

Release Note

Automatic Import now produces correct pipeline to parse CSV files with special characters in column names.

Summary

Fixes #211911

Now the pipeline works as expected:

  - csv:
      tag: parse_csv
      field: message
      target_fields:
        - ai_202502211453.logs._timestamp
        - ai_202502211453.logs.message
      description: Parse CSV input
  - drop:
      ignore_failure: true
      if: >-
        ctx.ai_202502211453?.logs?._timestamp == '@timestamp' &&
        ctx.ai_202502211453?.logs?.message == 'message'
      tag: remove_csv_header
      description: Remove the CSV header line by comparing the values

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Details

The CSV processing is now a three-stage process:

1. Parse the samples with the temporary column names of the form column1.
2. Test parsing with the actual pipeline that parses into package.dataStream.columnName.
3. Convert the samples into JSON form {"columnName": "value", ...} for further processing.

Tests

There are unit tests tests for the CSV functionality that include a mock CSV processing pipeline.

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 176511c

Failed CI Steps

• Check Types

History

cc @ilyannn

[P1llus]

LGTM! Good one

[kibanamachine]

Starting backport for target branches: 8.16, 8.17, 8.18, 8.x

https://github.com/elastic/kibana/actions/runs/13590247953

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.16	Backport failed because of merge conflicts
❌	8.17	Backport failed because of merge conflicts You might need to backport the following PRs to 8.17: - [ResponseOps][Rules] Do not show connector not registered in action connectors modal (#212660) - [Search] Don't error out on missing connectors permissions (#212622)
❌	8.18	Backport failed because of merge conflicts You might need to backport the following PRs to 8.18: - [ResponseOps][Rules] Do not show connector not registered in action connectors modal (#212660) - [Obs AI Assistant] Show loader to fix the flicker in KB settings tab (#212678) - [Search] Don't error out on missing connectors permissions (#212622) - Fix panel to remove query filter for .NET OTel runtime metrics on curated dashboard (#212529) - [ Siem Migrations ] Upsell Siem Migrations Start (#212607) - [Cloud Security] Add upgrade agentless deployment background task (#207143)
❌	8.x	Backport failed because of merge conflicts You might need to backport the following PRs to 8.x: - [ResponseOps][Rules] Do not show connector not registered in action connectors modal (#212660) - [Obs AI Assistant] Update delete confirmation modal (#212695) - [Obs AI Assistant] Show loader to fix the flicker in KB settings tab (#212678) - [Obs AI Assistant] Enable syntax highlighting for ES|QL (#212669) - [Search] Don't error out on missing connectors permissions (#212622) - [Onboarding] Hide the semantic_text banner if there exists a semantic_text field (#210676) - [ Siem Migrations ] Upsell Siem Migrations Start (#212607) - FTR - optimize service initialization (#212421) - [Streams 🌊] Remove enablement check in PUT /api/streams/{id} for classic streams (#212289)

Manual backport

To create the backport manually run:

node scripts/backport --pr 212513

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 212513 locally