[pkg/stanza] file read status observability

[newly12]

Component(s)

pkg/stanza

Is your feature request related to a problem? Please describe.

We are adopting file log receiver largely for collecting logs and would like to see apart from checking registry file manually for the last read offset, trying to figure out the possibility to build this feature as part of stanza package or the receiver itself.

Describe the solution you'd like

If we are adding this, I think it could be metrics use case, for example, something like following with the file path as the metrics label(as long as the file is still a valid input source)

1. file size
2. read offset

After checking the code, the best place for this is the Manager struct from the fileconsumer package as it has the info of file path and latest offset. One alternative I can think of is build something around the storage extension so when any offset/metadata is persisted via it, it can decode the data to get the offset, but this also means overhead on the decoding work.

I appreciate all suggestions or comments.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

I generally support the idea. I know there is some ongoing work to migrate the collector's internal telemetry to native otel instrumentation but we should be looking to define component-specific metrics in the near future.

One thing to keep in mind when reporting metrics in terms of file paths is that in many cases file rotation will result in a time series being "reset". It's not immediately clear what the proper way to model this is, but as the consumer of a time series it would not necessarily be obvious what this means. For example: file size for "foo.log" [ 4, 13, 42, /* file is rotated */ 5, 27, ... ]. It may just be that we need to use a monotonic counter and document that resets may be due to file rotation.

[github-actions]

Pinging code owners for receiver/filelog: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/filelog: @djaglowski
• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[newly12]

related PRs

• [receiver/filelog] Add support for telemetry metrics #31618
• [pkg/stanza] Add monitoring metrics for open and harvested files in fileconsumer #31544

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/filelog: @djaglowski
• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/filelog: @djaglowski
• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/filelog: @djaglowski
• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.