Read delta lake table history when initial transaction logs are deleted

[krvikash]

Description

This PR handles the case when transaction logs are deleted because the retention time of the transaction log is over.

This issue will occur when initial transaction logs get deleted because of log retention interval is exceeded while writing a new checkpoint file.

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
(X) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Deltalake
* Fix reading delta lake table history when initial transaction logs are removed. ({issue}`18845`)

[Praveen2112]

Any specific reason for reading backward, because we try to reverse them later in getPageSource

[krvikash]

Yes, the transaction log we intend to access through loadNewTailBackward will start from the final checkpoint file version and continue all the way back to version 0. This will remain true even if the log retention period is exceeded, leading to the removal of some of the initial transaction logs (0, 1, etc.). In this way, we can still retrieve the complete history of the table till the last checkpoint version.

[krvikash]

Thanks @findinpath for the review. I have addressed comments.

[alexjo2144]

Why do you need to do this from the middle backwards and then from the middle forward separately? Rather than from endVersion backwards in one pass.

It's not obvious to me this way that you're not potentially going to have duplicates.

[krvikash]

Thanks @alexjo2144 for the review. AC.

[krvikash]

(resolved conflicts)

[krvikash]

(resolved conflicts)

[krvikash]

(fixed CI failure)

[krvikash]

Thanks @alexjo2144 for the review. Addressed comments.

[krvikash]

ci / test (plugin/trino-delta-lake, cloud-tests) is failing in master. I am working on fixing it.

Failing for ADLS and GCS. S3 is passing.

[krvikash]

minio.removeObject is not removing files.