Failed to create backup in Docker

[Aragur]

The Backup functionality isn't working on my instance.
Using the latest version of Pocketbase and Docker.

--debug-Log shows the following:

blob (key "pb_backup_20230808092633.zip") (code=Unknown): rename /tmp/pb_backup_20230808092633.zip.17795e46db5a7b21.tmp /pb_data/backups/pb_backup_20230808092633.zip: invalid cross-device link

[ganigeorgiev]

I think this was fixed in v0.16.3+. What version of PocketBase do you use?

[Aragur]

I'm using 0.17.2

[ganigeorgiev]

Hm, what was fixed before was the restore functionality (see #2519) but I forgot to check the backup creation.

I'll test it sometime later today and will try to submit a fix.

[ganigeorgiev]

Should be fixed in PocketBase v0.17.3.

If the above doesn't resolve your issue please provide more details about your Docker setup and I'll try to investigate it further.

[Aragur]

@ganigeorgiev unfortunately I'm experiencing the same issue after upgrading to v0.17.3.

Docker version 24.0.5, build ced0996

My Docker-Compose file

services:
  example-server:
    restart: always
    image: private-image
    command: ["/server", "serve", "--http=0.0.0.0:8090", "--debug"]
    volumes:
      - ./data:/pb_data:rw

My Dockerfile

## Build
FROM golang:1.20-buster as build

WORKDIR /app

COPY server/go.mod ./
COPY server/go.sum ./
COPY server/pkg ./pkg
RUN go mod download

COPY server/*.go ./

RUN CGO_ENABLED=1 go build -o /server

## Deploy
FROM gcr.io/distroless/base-debian10

WORKDIR /

COPY --from=build /server /server
COPY webapp/dist /webapp

EXPOSE 8090

ENTRYPOINT ["/server", "serve", "--http=0.0.0.0:8090"]

Also upload of files is not working

failed to upload all files: [file 0: blob (key "mpr20jprgr61u1w/ekhfom6do5bq2u2/5f7350802ee2f5d807b01ad554c3965da373f12c_4x_Baf6SMF4ZN.png") (code=Unknown): rename /tmp/5f7350802ee2f5d807b01ad554c3965da373f12c_4x_Baf6SMF4ZN.png.177967419656a71b.tmp /pb_data/storage/mpr20jprgr61u1w/ekhfom6do5bq2u2/5f7350802ee2f5d807b01ad554c3965da373f12c_4x_Baf6SMF4ZN.png: invalid cross-device link]

[ganigeorgiev]

This is a different error but I'm not able to reproduce it locally.

Could you provide a minimal reproducible repo?

[ganigeorgiev]

I guess the tmp directory errors during "upload" could happen if the pb_data volume is a separate disk partition, I'm not really sure.

As a workaround you can try to mount the /tmp dir as the special temp pb_data/.pb_temp_to_delete directory (or another directory from the same volume/partition), aka. something like:

- ./data/.pb_temp_to_delete:/tmp

[ganigeorgiev]

See the note in go-cloud/blob package that we use for the file uploads (including the backups):

// To avoid partial writes, fileblob writes to a temporary file and then renames
// the temporary file to the final path on Close. By default, it creates these
// temporary files in os.TempDir. If os.TempDir is on a different mount than
// your base bucket path, the os.Rename will fail with invalid cross-device link.
// To avoid this, either configure the temp dir to use by setting the environment
// variable TMPDIR, or set Options.NoTempDir to true (fileblob will create
// the temporary files next to the actual files instead of in a temporary directory).

So as a workaround you can try to set the TMPDIR env variable.

I'll try to experiment a little further and will see how feasible will be to define the NoTempDir option.

[ganigeorgiev]

On further investigation, it seems that the NoTempDir option was added just couple days ago (see google/go-cloud#3295; doesn't seem to be released/tagged yet). The related issue is also very similar to yours.

As mentioned in the previous comment, I'll consider updating the dependency (once tagged) and enabling the option (I'll have to find a way to reproduce it somehow first), but as a workaround for now try to set the TMPDIR env variable.

[Aragur]

Here is a repo to reproduce the bug: https://github.com/Aragur/pocketbase-invalid-cross-device-link.git

[Aragur]

The workaround

- ./data/.pb_temp_to_delete:/tmp

isn't working for me.

What should I set TMPDIR to?

[ganigeorgiev]

What should I set TMPDIR to?

To a directory path that is inside your mounted dir, aka. in your case I guess it could be /pb_data/.pb_temp_to_delete or some other directory that shares the same mount point.

[Aragur]

Thanks this worked for me

[Aragur]

@ganigeorgiev Just wanted to ask for an update.
Furthermore, I experienced issues, when restarting the docker container.
After it is restarted, the /pb_data/.pb_temp_to_delete-directory is deleted and has to be created manually.

[ganigeorgiev]

You can try to upgrade to PocketBase v0.17.4, where we downgraded the go-cloud/blob dependency to v0.32 until v0.34 is released with the NoTempDir option.

[ganigeorgiev]

If you are using PocketBase as Go framework, check your go.mod file and update the gocloud.dev entry to v0.32.0 or earlier (eg. with go get [email protected]).

[khrome83]

@ganigeorgiev this still appears to be some type of issues.

I get this same error message in the ui, but this outcome.

failed to create a temp dir: mkdir pb_data/.pb_temp_to_delete: operation not permitted

Pocketbase version - PocketBase v0.18.6

What advice do you have for giving Pocketbase permission to make the directory?

I thought about just adding it in the docker image, but after running that night, it would be deleted.

Suggestions?

This is in a Dockerfile, hosted on fly.io (so VM not container)

[ganigeorgiev]

@khrome83 The above "cross-device link" error shouldn't exist anymore with the latest release.

I have no idea why the above is causing a file permission error.
Make sure that your volume is mounted with the correct path.

If you are not able to resolve it, please consider filing a new Q&A discussion with more details about your setup and I'll try to investigate it.

[khrome83]

Done! See #3385