[Epic] Log Parsing Pipeline Issues

[raj-k-singh]

With completion of pipeline preview work and deployment of opamp managed otel collector, log parsing pipelines are now almost ready to be released to users.
This epic tracks the blocking bugs/cleanups that must be resolved before log parsing pipelines are ready for release to users.

Related Documents

• all known issues

Part of the larger issue for enabling log parsing from the UI

Must haves

• [Log Parsing Pipelines] Pipeline processors for unsaved pipelines are lost when closing expanded editor view #3792
• [Log Parsing Pipelines] Grok parser doesn't work in pipeline previews #3794
• [Log Parsing Pipelines] Testing TraceParsing preview from JSON doesn't work #3795
• [Log Parsing Pipelines] Older unresolved deployments should not be stuck "in progress" after a later deployment succeeds/fails. #3797
• [Log Parsing Pipelines] Trace parser should be valid as long as one of trace_id, span_id and trace_flags is specified. #3845
• [Log Parsing Pipelines] Collector error logs should not be spammed if pipeline processors target missing fields/data #3793
• [Log Parsing Pipelines] Filters referencing resources.* fields don't work #3884
• [Log Parsing Pipelines] Adding 2 processors with same name/id breaks the editor #3922
• [Log Parsing Pipelines] Emit telemetry events from Pipelines UI #3965
• [Log Parsing Pipelines] In context help in Pipelines UI #3967
• [Log Parsing Pipelines] qbtoexpr conversion breaks for string operators when targeting a nil attribute  #3915
• [Log Parsing Pipelines] pipeline preview generates wrong timestamps #4063
• [Log Parsing Pipelines] filters referencing names whose dots have been replaced with underscores don't work #4061
• [Log Parsing Pipelines] severity and timestamp parsing processors. #4060
• [Log Parsing Pipelines] User facing docs #3796
• [Log Parsing Pipelines] UI tests for logs pipelines journeys #4020
• [Log Parsing Pipelines] Expressions in Regex and Grok parser must have atleast one capture group #3952
• [Log Parsing Pipelines] Opamp client in collector should fallback to running initial config if opamp server connection can not be established within a timeout #3921
• [Log Parsing Pipelines] Pipelines filter case sensitve in otel collector. #3981
• [Log Parsing Pipelines] should be able to exit edit mode without making any changes. #3876
• [Log Parsing Pipelines] pipeline filter form becomes empty when opened after returning from logs explorer #3914
• [Log Parsing Pipelines] Pipeline preview should have more samples #4117
• [Log Parsing Pipelines] Pipeline Previews - each processed log should include information about its processing #4118
• [Log Parsing Pipelines] JSON export and import #4119
• https://github.com/SigNoz/engineering-pod/issues/1112

Maybe

• [Log Parsing Pipelines] Creating a new disabled pipeline should not create a new deployment #3798
• [Log Parsing Pipelines] Opamp client in collector should backoff before retrying server connection #3799
• [Log Parsing Pipelines] Better regex processor editing #3887
• [Log Parsing Pipelines] pipeline processor validation should use collector simulator #3868
• [Log Parsing Pipelines] trace parser pads trace id bytes to the right, which breaks logs to trace navigation #3859
• [Log Parsing Pipelines] Pipeline filter UI breaks for large filters like regexes. #3916

Features

• pipelines as code
• showing version diff in the product
• showing createdBy and editedBy in the product

[ankitnayan]

#3885

[ankitnayan]

#4444
#4298
#4199
#4176

[ankitnayan]

#2857

[ankitnayan]

Friction Points

1. https://signoz-team.slack.com/archives/C07446MCZ5X/p1720311710480109
2. https://signoz-team.slack.com/archives/C05AXQL92BT/p1720237658958349
3. https://signoz-team.slack.com/archives/C05AXQL92BT/p1719327238156559?thread_ts=1719325651.300039&cid=C05AXQL92BT
4. https://signoz-team.slack.com/archives/C05BGMUDS1L/p1718199767089559

[ankitnayan]

https://github.com/SigNoz/signoz/issues?q=is%3Aopen+is%3Aissue+label%3A%22log+pipelines%22

[pranay01]

@raj-k-singh can you create a new epic with remaining issues and close this epic ticket?

[raj-k-singh]

Remaining backlog being tracked using https://github.com/SigNoz/engineering-pod/issues/1933