Replace flush parameters with an enum of flush purposes

[dnhatn]

Today a flush request supports two parameters: force and waitIfOngoing. However, a combination of force=true and waitIfOngoing=false (default value) makes nonsense as that force flush will be ignored if there's an ongoing flush.

@jpountz suggests replacing these two parameters with an enum of flush purposes:

• TRY_FLUSH equals force=false and waitIfOngoing=false
• FLUSH_IF_CHANGES equals force=false and waitIfOngoing=true
• FORCE_FLUSH equals force=true and waitIfOngoing=true

Note that this is a breaking change. Another option is always to execute a force flush regardless of the value of waitIfOngoing.

[elasticmachine]

Pinging @elastic/es-distributed

[jpountz]

Thanks for opening this issue Nhat. As an alternative in case this doesn't get consensus, we could reject the (force=true, waitIfOngoing=false) combination.

[s1monw]

Thanks for opening this issue Nhat. As an alternative in case this doesn't get consensus, we could reject the (force=true, waitIfOngoing=false) combination.

I think we should do this anyway as a first step.

I am not sure we actually need the force param at all. What is it used for really? Lets try to collect some usecases first before we change the API for no reason.

[bleskes]

I am not sure we actually need the force param at all.

+1. I can't think of a reason for a user to do something else then "please flush now and I will wait until it completes" (which maps to force=true, waitIfOngoing=true). I think we use those flags for internal purposes only (and it will probably be good to have a good look if we really need those too).

[ywelsch]

+1. The documentation for force even says:

This setting can be considered as internal

The docs are also out of sync with the code. The docs claim that the default for wait_if_ongoing is false, whereas it is true :-) Relates to #20597