Encrypted files do not get closed

[klassiker]

Description of the problem

When a file is opened inside an encrypted mount and the file descriptor is not closed by the application no data is ever written to the host. Since the encrypted file header is also missing (file size is zero), the application is broken on restart.

Steps to reproduce

Application:

#include <unistd.h>
#include <fcntl.h>

int main() {
    int fd = openat(AT_FDCWD, "/secrets/myfile", O_WRONLY|O_CREAT, 0666);
    write(fd, "test", 4);
}

Manifest:

libos.entrypoint = "/app/main"

loader.entrypoint = "file:{{ gramine.libos }}"

loader.log_level = "all"
loader.log_file = "log.txt"

loader.env.LD_LIBRARY_PATH = "/lib:{{ arch_libdir }}"

fs.mounts = [
    { path = "/lib",              uri = "file:{{ gramine.runtimedir() }}" },
    { path = "{{ arch_libdir }}", uri = "file:{{ arch_libdir }}" },
    { path = "/app",              uri = "file:/app" },
    { path = "/secrets",          uri = "file:/data", type = "encrypted" },
]
fs.insecure__keys.default = "ffeeddccbbaa99887766554433221100"

sgx.thread_num = 8
sgx.enclave_size = "512M"

sgx.allowed_files = [
    "file:/",
]

Expected results

File is closed by gramine when the process exits and can be opened again after a restart.

Actual results

Trace, first start:

[P1:T1:main] trace: ---- openat(AT_FDCWD, "/secrets/myfile", O_WRONLY|O_CREAT, 0666) = 0x3
[P1:T1:main] trace: ---- write(3, 0x313ae017, 0x4) ...
[P1:T1:main] trace: ---- return from write(...) = 0x4
[P1:T1:main] debug: ---- exit_group (returning 0)
[P1:T1:main] debug: clearing POSIX locks for pid 1
[P1:T1:main] debug: sync client shutdown: closing handles
[P1:T1:main] debug: sync client shutdown: waiting for confirmation
[P1:T1:main] debug: sync client shutdown: finished
[P1:libos] debug: IPC worker: exiting worker thread
[P1:T1:main] debug: process 1 exited with status 0

Trace, second start:

[P1:T1:main] warning: cb_read: EOF
[P1:T1:main] warning: encrypted_file_internal_open: pf_open failed: Callback failed
[P1:T1:main] trace: ---- openat(AT_FDCWD, "/secrets/myfile", O_WRONLY|O_CREAT, 0666) = -13

Gramine commit hash

562c6397, current master