Azure Blob CSI Driver Compatibility Issue with Seafowl

[mrchypark]

Issue Description

While testing Seafowl with local storage using the Azure Blob CSI driver in Kubernetes (due to the lack of native Azure Blob support in Seafowl), we encountered the following error:

Object Store error: Generic LocalFileSystem error: Unable to copy file from /data/3a6b2ef0-fcf6-4bc2-a75a-ad2714c3ae69/_delta_log/_commit_bdeba02b-7620-42d7-8ea8-34db0b7a0f2c.json.tmp to /data/3a6b2ef0-fcf6-4bc2-a75a-ad2714c3ae69/_delta_log/00000000000000000000.json: Function not implemented (os error 38)

No related logs were observed on the server side.

Root Cause

After investigation, we discovered that the Azure Blob CSI driver does not support hard links, which appears to be the cause of this issue. Hard links are used in the implementation of the object_store utilized by Delta Lake.

Proposed Solution

We suggest a two-step approach to address this issue:

1. Short-term: Implement a workaround in Seafowl to handle this error gracefully when it occurs.
2. Long-term: Collaborate with the upstream Azure Blobfuse project to implement hard link support.

Additional Information

We've created a test repository to confirm the lack of hard link support in the Azure Blob CSI driver: https://github.com/mrchypark/azblobcsi-debug/blob/main/src/main.rs

Thank you for your attention to this matter. We greatly appreciate your hard work on the Seafowl project and look forward to your thoughts on this issue.

[mrchypark]

Looking at this link
Azure/azure-storage-fuse#599,

it seems there are no plans to support hard links.

[mrchypark]

#650