Limitation on FlintSpark Optimistic Transaction

[noCharger]

Is your feature request related to a problem?

When transaction fails, the only action item taken is to rollback the metadata log entry. This could cause partial failures like #826. Furthermore, if the flint data index is accidentally deleted, the refreshIndex API will always fail, but the user is unaware of this because the index state has been rolled back.

What solution would you like?

Short term solution:

1. External scheduler should check the index state before fire the subsequent query

Long term solution:

1. Enhance rollback logic to handle partial failure. Enhance prepare() to consider other components which related to state.
2. Introduce a corrupted state and may recover the index deleted

[noCharger]

Regarding to 2. on long term solution, we got another feedback from Nitish that we should re create the index in such cases and customer should delete the dashboard if they no more want the refreshes to happen. Any thoughts on this? @anasalkouz @penghuo @dai-chen

[dblock]

[Catch All Triage - 1, 2, 3]