Consistency Problem with Async. Deletion

[windkit]

Description

Asynchronous deletion would cause consistency problem if another modification has occurred before the async. deletion is handled

Root Cause

Directory Deletion

• Object list under the directory is pull when leo_storage handles the deletion, the list of objects could have been changed by the time
• Time stamp of the deletion request is not taken into account, re-created file could be incorrectly deleted

Object Deletion

• Time stamp of the deletion request is recorded as the time leo_storage starts to handle, not the origin request time

Action to take

Clarify the consistency model, especially the mix of sync and async. operations, the time stamp record for reconciliation

Related Issue

Spark first cleanup the temporary folder and then start to write data into the folder #595

[windkit]

The problem here is between Request and Handle, there could be state changes (other operations)
Now we do not record the timestamp at request, we have NO ways to distinguish which state changes happen before / after the request.

[windkit]

Issue fixed with
Spark 2.2.0 + Hadoop 2.7.3 and Spark 1.6.1 + Hadoop 2.6.3 with
hadoop-aws-2.7.3.jar and aws-java-sdk-1.7.4.jar