Tracking epoch_stake distribution

[rdlrt]

Problem Report
Post an epoch transition, the only way for us to know the status for epoch_stake distribution is querying the logs for total epoch stake distributions, and checking status manually for number of entries populated for the epoch in database):

grep -r 'Inserted.*.EpochStake for EpochNo ' dbsync*
# dbsync-20230510073605.json:{"app":[],"at":"2023-05-11T01:34:53.50Z","data":{"kind":"LogMessage","message":"Inserted 1277714 EpochStake for EpochNo 411"},"env":"<unknown>:00000","host":"","loc":null,"msg":"","ns":["db-sync-node"],"pid":"31654","sev":"Info","thread":"706479"}
# dbsync-20230515073731.json:{"app":[],"at":"2023-05-16T01:40:06.39Z","data":{"kind":"LogMessage","message":"Inserted 1278735 EpochStake for EpochNo 412"},"env":"<unknown>:00000","host":"","loc":null,"msg":"","ns":["db-sync-node"],"pid":"31654","sev":"Info","thread":"706479"}

First of all, it would be nice to have epoch_stake available before epoch transition (as it's already available in ledger approximately 2/3rd of previous epoch), even if not - it is essential to be able to check status without having to check logs (eg: an entry in meta or epoch [or a seperate table] containing amount of stake additions similar to what we see in the logs.

Additional Context

Since JSON logging against a file is not a common option, it makes implementation much more difficult than it needs to be to track status before being able to consume information in epoch_stake table. This is especially true for docker/kubernetes implementations which often do not make use of file-based logging and makes things difficult to rely on.
As far as I know, this is the only information that's present in log and not database - would be nice to not involve touching/knowing environment logs in scripts

[xray-robot]

Related to #797

[kderme]

I think we can have it available earlier

as it's already available in ledger approximately 2/3rd of previous epoch

I think it's available even earlier, almost the beginning of the previous epoch. How did you infer the 2/3?

[rdlrt]

I think it's available even earlier, almost the beginning of the previous epoch. How did you infer the 2/3?

You're right indeed, it should be available in an epoch earlier (sorry the 2/3rd of epoch reference is incorrect, I confused with next epoch's nonce - which is finalised by 6*k/10 of epoch). So ye - would be great to not have to wait post epoch transition for active stake, and if we can get previous epoch's stake sooner too - that'd be very welcome.

(PS: For koios endpoints, we did end up artificially creating snapshot information by based on delta tx , MIR/reward/withdrawals/etc - which ofcourse runs once and takes ~10-20 min, but is one of those additional band-aids that we can get rid of, and something that will not be doable with prune options)