Is it possible to rerun a pipeline on already ingested logs?

[jmatsushita]

I've tried to create a pipeline which has:

• a step to remove my custom attribute
• a step to Grok the custom attribute from logs

But I still see old version of the attribute (tagged as String) whereas my new Grok pattern parses them as Float, and the simulation correctly shows the attribute parsed as a float.

I'd like to avoid reingesting all my historical logs, so is there a way to "kick" a pipeline such that it deletes existing computed data and recreates it from the source without reingesting?

If not, what's the best approach to re-ingest with filelogreceiver? Is there a way I can first delete the data in Signoz/ClickHouse or will it automatically deduplicate existing entries?

[srikanthccv]

so is there a way to "kick" a pipeline such that it deletes existing computed data and recreates it from the source without reingesting?

No, this is not possible today.

what's the best approach to re-ingest with filelogreceiver? Is there a way I can first delete the data in Signoz/ClickHouse or will it automatically deduplicate existing entries?

It doesn't deduplicate. One option is to identify those logs, delete them and re-ingest using some script.

Please create a feature request here https://github.com/SigNoz/signoz/issues

[jmatsushita]

Done! #5971