[HUDI-8970] Change scheduleandexecute in RunCompactionProcedure to co…

[voonhous]

…mpact all pending plans too

Change Logs

Change RunCompactionProcedure op = scheduleandexecute to run on all pending instants instead of only scheduled instant.

This is to prevent cases when scheduleandexecute fails while performing compaction, then rerunning the job again, causing a new compaction plan to be scheduled, and the origjnal plan to be stuck in the pending state.

If user triggers this job via an external system that schedules the job on a fixed frequency, such edge cases could happen, and wold require human intervention.

If users would like to control the number of plans being executed under scheduleandexecute they can still do it with the LIMIT argument, however, compaction execution will always prioritise older instants first, i.e. FIFO ordering.

Impact

None

Risk level (write none, low medium or high below)

None

Documentation Update

None

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
  ticket number here and follow the instruction to make
  changes to the website.

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[hudi-bot]

CI report:

• 5f4c071 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build