[HUDI-7974] Adding support for empty cleans

[nsivabalan]

Change Logs

• Add configuration for periodically creating empty clean commits when there are no files to clean.
• Added a new metadata entry to Clean plan keyed as "EarliestCommitToNotArchive". This will be leveraged by archival to ensure it will not proceed beyond the configured value for EarliestCommitToNotArchive. On rare scenarios, when cleaner and archival are executed separately, where cleaner is disabled and archival is executed, chances that archival could result in duplicates by archiving instants which are not yet cleaned by the cleaner. So, fixing that by adding this additional metadata.

Impact

Fixes an issue in append only tables where the archival does not kick in since the archival is now dependent on information in the cleaner metadata. w/o empty cleans, active timeline for append only tables kept on growing forever.

Pending:
Fixing TimelineArchiverV2 and adding tests.
Adding tests for TimelineArchiverV1

Risk level (write none, low medium or high below)

If medium or high, explain what verification was done to mitigate the risks.

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[hudi-bot]

CI report:

• afad291 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build