Long term storage for Jaeger and Loki

[tumido]

Related to: operate-first/apps#232 for the Causter logging for DeCorus

Discussed influxDB IOx and ElasticSearch

cc @jpkrohling @RS1999ent @drbwa @eranra

[jpkrohling]

To give a bit more context, @RS1999ent mentioned a way to get historical data in a format that can be used at a later point in time, to which I mentioned that InfluxDB IOx will store data following the Apache Parquet format, allowing other tools to read the data. The files themselves can be stored using any block solution, like S3 for retention over longer periods of time. We also talked about Jaeger's possible plan to move to InfluxDB IOx in the future as the recommended storage mechanism, moving away from Elasticsearch. Whether that makes sense for other signals (logs, metrics), I don't know.

[drbwa]

I am aware of the trend to move to InfluxDB and maybe that is indeed the right thing to do.

From our experience, Elasticsearch allows for more convenient retrieval when it comes to log files.

What I don't know (yet) is if a) InfluxDB IOx will make it easier to slice and dice and retrieve your logs and b) how much more efficient InfluxDB is in terms of required storage space (guessing there it has an advantage there, but don't know if that's the case).