Files get truncated

[macieii]

Which version of blobfuse was used?

2.0.3

Which OS distribution and version are you using?

ubuntu 22.04 x86-64

If relevant, please share your mount command.

blobfuse2 $1 --config-file=/home/blob/fuse_connection_portal.yml

What was the issue encountered?

When writing to the blob from the server via blobfuse, some files get truncated to a specific size (e.g. 1 MB). These files appear truncated in the blob immediately after the write operation is completed (checked real-time with Azure Storage Explorer), but when accessed from the OS level, they are still correct. and only after a few hours get corrupted. As if there was a cache.
The files are not large, e.g. 10 MB or 13 MB, but larger than 1 MB. All files that are affected have the same incorrect size afterwards.

The fuse_connection_portal.yml config file is:

allow-other: true

logging:
type: syslog
level: log_debug

components:

• libfuse
• stream
• attr_cache
• azstorage

libfuse:
attribute-expiration-sec: 120
entry-expiration-sec: 120
negative-entry-expiration-sec: 240

stream:
block-size-mb: 8
blocks-per-file: 3
cache-size-mb: 1024

attr_cache:
timeout-sec: 7200

azstorage:
type: block
account-name: XXXXX
account-key: XXXXXX
endpoint: https://XXXX.blob.core.windows.net/
mode: key
container: portal

Have you found a mitigation/solution?

Not yet. We're considering switching to file cache mode.

Please share logs if available.

blobfuse2.log

The 4276.1 is the name of the file that gets truncated.

May 9 05:00:44 jadwiga blobfuse2[1022522]: LOG_TRACE [block_blob.go (522)]: BlockBlob::GetAttr : name [/forum/21/187/4276.1](/forum/21/187/4276.1)
May 9 05:00:44 jadwiga blobfuse2[1022522]: LOG_TRACE [block_blob.go (431)]: BlockBlob::getAttrUsingRest : name [/forum/21/187/4276.1](/forum/21/187/4276.1)
May 9 05:00:44 jadwiga blobfuse2[1022522]: LOG_TRACE [libfuse_handler.go (760)]: Libfuse::libfuse_truncate : [/forum/21/187/4276.1](/forum/21/187/4276.1) size 0
May 9 05:00:44 jadwiga blobfuse2[1022522]: LOG_TRACE [read_write.go (154)]: Stream::TruncateFile : name=[/forum/21/187/4276.1](/forum/21/187/4276.1), size=0
May 9 05:00:44 jadwiga blobfuse2[1022522]: LOG_TRACE [attr_cache.go (386)]: AttrCache::TruncateFile : [/forum/21/187/4276.1](/forum/21/187/4276.1)
May 9 05:00:44 jadwiga blobfuse2[1022522]: LOG_TRACE [azstorage.go (467)]: AzStorage::TruncateFile : [/forum/21/187/4276.1](/forum/21/187/4276.1) to 0 bytes

The in the log contains just the subdomain+domain (xxx.yyy.pl), with no http nor https prefix.

[vibhansa-msft]

Can you share the full log from this run where files were truncated. Blobfuse does not truncate a file on its own unless we received a call from kernel for that. Having a full log will help us identify if there is a bug on our side or use-case itself is truncating it.

NOTE: You had shared your account name and key in the description. As this is a public forum, this might have leaked your storage account keys. I have removed those from description, kindly rotate them at the earliest.

[macieii]

Attached. The files affected are 4614.2, 4614.3, 4614.5. All three have “truncates” in the log. We also started to receive error messages “error failed to download requested data for…” – not sure if this might be related? Thanks [LOG.txt](https://github.com/Azure/azure-storage-fuse/files/11460251/LOG.txt)

[vibhansa-msft]

As the files are smaller in size can you switch to using file-cache instead of streaming. I will go through the logs and try to recreate this situation locally with streaming in the mean time. Just to unblock your use-case you can switch to file-cache and that shall create the file properly. Streaming is more of a solution suited when you are dealing with very large files.

[vibhansa-msft]

Can you change your config for streaming and try your test case again. I realised while looking into this, that config for streaming is not correct.

stream:
  block-size-mb: 8
  buffer-size-mb: 8
  # Based on how much memory you want to allocate to streaming, max-buffers will get multiplied with block-size-mb to determine the same.
  max-buffers: 200 

[macieii]

Hi

1. Switching to file-cache did help, we consider it a workaround for now,
2. The new settings still do not work. Test image files uploaded to blob via blobfuse and downloaded via Azure Explorer are broken. Not sure why this is happening, the file sizes match. It's as if they were changed in some way. We need to investigate that.

[vibhansa-msft]

File truncation is an issue with streaming I do agree, we are working from our end to fix this.

[jeffjjohnston]

We have also run into this bug where files can be corrupted when written in write-streaming mode. We prefer to use read/write streaming, as our use-case involves partial reads of large files (where read streaming helps tremendously), but also writes of both large and small files.

[vibhansa-msft]

@jeffjjohnston in your use case when you say writing is required, are we talking about random writing or more like appending to existing file ?

[jeffjjohnston]

Random writing would be rare - this would mostly be creating new files or appending to existing files.