Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to write to bind mount #102

Open
spuder opened this issue Jan 9, 2022 · 28 comments
Open

Unable to write to bind mount #102

spuder opened this issue Jan 9, 2022 · 28 comments

Comments

@spuder
Copy link

spuder commented Jan 9, 2022

colima version 0.2.2
git commit: b2c7697bee2d73e995f156fe8e9870eb246c07e6

runtime: docker
client: v20.10.12
server: v20.10.7

uname -a
Darwin <REDACTED> 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64 x86_64

When bind mounting a volume on the host, the filesystem is read only despite using docker functionality to allow for writing to the host

Steps to reproduce

Note that :Z is added to the -v parameter.

docker run -v "$(PWD):/output:Z"  bash:latest touch /output/foo.txt
touch: cannot touch '/output/foo.txt': Read-only file system

I've also tried
4. docker run -v "$(PWD)/output:z" bash touch /output/foo.txt
4. docker run -v "$(PWD)/output:rw" bash touch /output/foo.txt
3. docker run --privileged -v "$(PWD):/output:Z" bash touch /output/foo.txt
4. docker run -v "$(PWD)/output:rw" --group-add=dialout bash touch /output/foo.txt
5. sudo docker run -v "$(PWD)/output:rw" --group-add=dialout bash touch /output/foo.txt

With the same error

Files appear to be owned by 502:dialout

docker run -v "$(PWD)/CAD:/output:rw" bash ls -al /output
total 260
-rw-r--r-- 1 502 dialout  50758 Jan  7 02:48 MPB2015 Feet v4.f3d
-rw-r--r-- 1 502 dialout 212684 Jan  7 02:42 MPB2015-Feet.stl

Is this a known limitation? A bug? Or a misconfiguration on my part? Any guidance appreciated.

@spuder
Copy link
Author

spuder commented Jan 10, 2022

I suspect this is a lima problem

lima-vm/lima#34 (comment)

Looking at ~/.lima/default/lima.yaml` the default settings are:

# Expose host directories to the guest, the mount point might be accessible from all UIDs in the guest
# Default: none
mounts:
  - location: "~"
    # CAUTION: `writable` SHOULD be false for the home directory.
    # Setting `writable` to true is possible, but untested and dangerous.
    writable: false
  - location: "/tmp/lima"
    writable: true

I don't know if this quite translates to the problem that I am seeing, but it does seem like a plausible correlation.

@abiosoft
Copy link
Owner

@spuder version 0.2.2 do not mount home directory as writeable by default, kindly install the latest development version with brew install --head colima.

@spuder
Copy link
Author

spuder commented Jan 10, 2022

colima stop
colima delete
brew unlink colima
brew install --head colima
colima start

While I no longer get the error Read-only file system, I'm finding that I'm unable to write to the host

docker run -v "$(PWD):/output" bash ls /output
docker run -v "$(PWD):/output" bash touch /output/foo.txt
ls | grep foo.txt 
# no results

@spuder
Copy link
Author

spuder commented Jan 10, 2022

I've uncovered some sort of race condition, because I'm now able to touch a file once, but if I delete the file and try again I always get an error

rm foo.txt
docker run -v "$(PWD):/output" bash touch "/output/foo.txt"
ls | grep foo.txt
foo.txt <------ notice that file exists 
rm foo.txt
docker run -v "$(PWD):/output" bash touch "/output/foo.txt"
touch: /output/foo.txt: No such file or directory

Does lima\colima sync the file system periodically? I find that If I put a sleep between the touch statements, eventually I get to the point where it can write to the volume consistently

If I write a file with a unique name, it always works

docker run --rm -v "$(PWD):/output" bash touch "/output/$(date).txt"
docker run --rm -v "$(PWD):/output" bash touch "/output/$(date).txt"
ls 
Mon Jan 10 16:22:33 MST 2022.txt   Mon Jan 10 16:22:40 MST 2022.txt

It seems like when you create/delete a file with lima/colima it is caching that file for a few seconds so you can't immediately write a file with the same name.

@jphenow
Copy link

jphenow commented Jan 11, 2022

This is sounding similar #65 maybe? I'm experiencing things more like #65 but the two sound very similar.

@jphenow
Copy link

jphenow commented Jan 11, 2022

I've been tinkering with stuff around this issue to see if I can get any better results but have been as yet unsuccessful

@yinshangwei
Copy link

When mounting Postgres data, they will be an error. Seems like the same issue.
chown: changing ownership of '/var/lib/postgresql/data': Permission denied

@vraravam
Copy link

Yes - I too have the same issue with the following docker-compose.yml file:

version: '3'
services:
  db:
    image: "postgres:13.4"
    container_name: api-db
    volumes:
      - ./postgres-data:/var/lib/postgresql/data:Z
    environment:
      - POSTGRES_USER=apiDbUser
      - POSTGRES_PASSWORD=apiDbPassword
      - POSTGRES_DB=apiDb
    ports:
      - "5433:5432"

I'm using the latest HEAD version of colima along with the above.

@abiosoft
Copy link
Owner

@spuder kindly update colima and lima and try again.

brew upgrade lima colima

@spuder
Copy link
Author

spuder commented Jan 13, 2022

I just upgraded colima and lima, but I"m not seeing that bind mounts don't work at all

brew upgrade lima colima
colima version 
  colima version 0.3.1
  git commit: 787ae5631ae8de072feef95a509c47fc93308b2e
  
  runtime: docker
  arch: x86_64
  client: v20.10.12
  server: v20.10.11
colima stop
colima delete
colima start
docker run -v "$(PWD):/output" bash touch "/output/foo.txt"
ls | wc -l
0 <-------------- This should say 1 because there should be 1 file named foo.txt in the directory

I've reproduced this twice now and I'm pretty sure this is a regression

@spuder spuder closed this as completed Jan 13, 2022
@spuder spuder reopened this Jan 13, 2022
@john-ot
Copy link

john-ot commented Jan 20, 2022

I'm able to reproduce the issue as well. I'm running:
colima start --cpu 2 --memory 8 --disk 40 --mount $HOME/IdeaProjects:/IdeaProjects:rw

But when I look at ~/.lima/colima/lima.yaml I see this:

mounts:
    - location: /Users/john/Library/Caches/colima
      writable: false
    - location: /Users/john/IdeaProjects/
      writable: false

I'm running colima 0.3.2

@abiosoft
Copy link
Owner

I'm able to reproduce the issue as well. I'm running: colima start --cpu 2 --memory 8 --disk 40 --mount $HOME/IdeaProjects:/IdeaProjects:rw

@john-ot You do not need to specify the mount actually, your $HOME directory is mounted by default, get rid of the mount flag and try again, you should not have any problems.
Nonetheless, your mount syntax is wrong, it should be --mount $HOME/IdeaProjects:w.

@john-ot
Copy link

john-ot commented Jan 20, 2022

Ack! I will fix the syntax.

Does the default mount give me write access though? I thought it was read only by default?

@abiosoft
Copy link
Owner

Does the default mount give me write access though? I thought it was read only by default?

Used to be read-only, but changed to writable in version 0.3.0.

@hungchai
Copy link

YES ! I have same issue too. Can anyone look into it?
OS:
image

colima status                                                          
INFO[0000] colima is running
INFO[0000] runtime: docker
INFO[0000] arch: x86_64

colima version                                                
colima version HEAD-784cec0
git commit: 784cec04dc126b02f5d2697a864282d5d2301afe

runtime: docker
arch: x86_64
client: v20.10.12
server: v20.10.11
docker-compose up redis                                             
[+] Running 2/2
 ⠿ Network foundation_default    Created                                                                                                             0.1s
 ⠿ Container foundation-redis-1  Created                                                                                                             0.1s
Attaching to foundation-redis-1
foundation-redis-1  | chown: changing ownership of '.': Permission denied
foundation-redis-1  | chown: changing ownership of './.DS_Store': Permission denied
foundation-redis-1  | chown: changing ownership of './dump.rdb': Permission denied
foundation-redis-1  | chown: changing ownership of './temp-1.rdb': Permission denied
foundation-redis-1 exited with code 1

docker-compose

  redis:
    image: redis
    ports:
      - "6379:6379"
    volumes:
      - ./redis/data:/data

lima.yaml

...
disk: 60GiB
mounts:
    - location: /Users/xxxx/Library/Caches/colima
      writable: false
    - location: /Users/xxxx/docker/
      writable: true

@DmitriiMukhin
Copy link

DmitriiMukhin commented Jan 27, 2022

Yes - I too have the same issue with the following docker-compose.yml file:

version: '3'
services:
  db:
    image: "postgres:13.4"
    container_name: api-db
    volumes:
      - ./postgres-data:/var/lib/postgresql/data:Z
    environment:
      - POSTGRES_USER=apiDbUser
      - POSTGRES_PASSWORD=apiDbPassword
      - POSTGRES_DB=apiDb
    ports:
      - "5433:5432"

I'm using the latest HEAD version of colima along with the above.

I'm facing the same issue.

@phaus
Copy link

phaus commented Jan 27, 2022

Just a quick update:

I was able to solve this for some use cases.
You need to use volumes instead of shared folders:

e.g. for the last docker-compose file use the volume postgres-data as follows:

version: '3'
services:
  db:
    image: "postgres:13.4"
    container_name: api-db
    volumes:
      - postgres-data:/var/lib/postgresql/data:Z
    environment:
      - POSTGRES_USER=apiDbUser
      - POSTGRES_PASSWORD=apiDbPassword
      - POSTGRES_DB=apiDb
    ports:
      - "5433:5432"

So use postgres-data:/var/lib/postgresql/data:Z instead of ./postgres-data:/var/lib/postgresql/data:Z

This issue is REALLY intereseting, becaus the error only happens, if a container tries to perform a chown to a folder shared via ssh-fs (what most DB Containers tend to do).
In my case, I changed to docker volumes for my DB container and was able to write to a shared folder via my dev container while building a specific docker image with the access rights of my current user:

build script (USER and UID are already set via the shell):

#!/bin/bash

VERSION=$(git describe --always --dirty --tags)
CONTAINER=$(basename $PWD)
GID=$(id -g)

echo "building for $USER ($UID:$GID)"
export UID
export GID
export USER

docker build --build-arg USER --build-arg UID --build-arg GID -t ${CONTAINER}:${VERSION} .

Dockerfile:

FROM ghcr.io/baosystems/postgis:13-3.1

ARG USER
ARG UID
ARG GID

RUN adduser -u ${UID} --gid ${GID} ${USER}

ENV TZ=Europe/Berlin
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt-get update \
    && apt-get upgrade -y \
    && apt-get install -y \
    curl \
    jq \
    && rm -rf /var/lib/apt/lists/*

@DmitriiMukhin
Copy link

Just a quick update:

I was able to solve this for some use cases. You need to use volumes instead of shared folders:

e.g. for the last docker-compose file use the volume postgres-data as follows:

version: '3'
services:
  db:
    image: "postgres:13.4"
    container_name: api-db
    volumes:
      - postgres-data:/var/lib/postgresql/data:Z
    environment:
      - POSTGRES_USER=apiDbUser
      - POSTGRES_PASSWORD=apiDbPassword
      - POSTGRES_DB=apiDb
    ports:
      - "5433:5432"

So use postgres-data:/var/lib/postgresql/data:Z instead of ./postgres-data:/var/lib/postgresql/data:Z

This issue is REALLY intereseting, becaus the error only happens, if a container tries to perform a chown to a folder shared via ssh-fs (what most DB Containers tend to do). In my case, I changed to docker volumes for my DB container and was able to write to a shared folder via my dev container while building a specific docker image with the access rights of my current user:

build script (USER and UID are already set via the shell):

#!/bin/bash

VERSION=$(git describe --always --dirty --tags)
CONTAINER=$(basename $PWD)
GID=$(id -g)

echo "building for $USER ($UID:$GID)"
export UID
export GID
export USER

docker build --build-arg USER --build-arg UID --build-arg GID -t ${CONTAINER}:${VERSION} .

Dockerfile:

FROM ghcr.io/baosystems/postgis:13-3.1

ARG USER
ARG UID
ARG GID

RUN adduser -u ${UID} --gid ${GID} ${USER}

ENV TZ=Europe/Berlin
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt-get update \
    && apt-get upgrade -y \
    && apt-get install -y \
    curl \
    jq \
    && rm -rf /var/lib/apt/lists/*

Please provide a solution - how to create volumes that will be mapped to the host directory. I mean to $HOME/some/directory/postgres-data.

@jklewa
Copy link

jklewa commented Jan 31, 2022

When mounting Postgres data, they will be an error. Seems like the same issue. chown: changing ownership of '/var/lib/postgresql/data': Permission denied

chown: changing ownership of 'foo': Permission denied is a separate issue (#83) than the read-only file system issue discussed in the beginning of this thread. Unfortunately, chown on a host-mounted directory isn't yet supported (upstream issue lima-vm/lima#231, further upstreamed by NixOS/nixpkgs#122420)

@yinshangwei
cc @vraravam @hungchai @DmitriiMukhin

@phaus suggested the workarounds 1) to use a VM-only volume instead of a host-mounted directory, or 2) don't use chown by making the user:group IDs of those paths match between the VM and host

@phaus
Copy link

phaus commented Feb 1, 2022

so from my side, this issue can be closed.

@HansG89
Copy link

HansG89 commented Feb 2, 2022

any ideas on how to fix this issue when running local k3d kubernetes in colima and providing a host path storage to a pod?

Colima is running with default settings, so home direcotry is mounted as writable. ALso told k3d to mount the home directory. I tested touching a file to a mounted volume inside a pod and it works as expected. Only chowning a folder seems to cause trouble.

So for instance the postgres example gives me:
chown: changing ownership of '/var/lib/postgresql/data/pgdata': Permission denied

config map:

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  namespace: basys
  labels:
    app: postgres
data:
  POSTGRES_DB: postgresdb
  POSTGRES_USER: admin
  POSTGRES_PASSWORD: test123
  PGDATA: /var/lib/postgresql/data/pgdata

storage:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-pv-volume
  labels:
    type: local
    app: postgres
spec:
  storageClassName: manual
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/Users/xxx/postgres/data"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgres-pv-claim
  labels:
    app: postgres
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:14.1
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432
          envFrom:
            - configMapRef:
                name: postgres-config
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgredb
              readOnly: false
      volumes:
        - name: postgredb
          persistentVolumeClaim:
            claimName: postgres-pv-claim

@phaus
Copy link

phaus commented Feb 2, 2022

@HansG89 maybe you can use a light storage provider (like https://rook.io or https://longhorn.io) to create k8s persistent volumes? Otherwise is should also be possible to use NFS or other network storage option for k8s volumes.
Using local mount paths in k8s is always a dev workaround IMHO, because in production you almost always go with persistent volumes.

@HansG89
Copy link

HansG89 commented Feb 2, 2022

@HansG89 maybe you can use a light storage provider (like https://rook.io or https://longhorn.io) to create k8s persistent volumes? Otherwise is should also be possible to use NFS or other network storage option for k8s volumes.

Using local mount paths in k8s is always a dev workaround IMHO, because in production you almost always go with persistent volumes.

Thank you for your ideas. I used k3d to keep the setup as simple as possible. I don't want to introduce another level of complexity because a developer, who should use the setup in his daily business should be able to focus on developing features and not on infrastructure. This is why the host path solution is preferred. Also it's almost the same like the good old docker-compose setup. Anyways would be great if root cause in NixOS gets fixed as soon as possible 😉

@Cerebus
Copy link

Cerebus commented Feb 7, 2022

Try mount --make-shared on the relevant VM mount point.

@iambudi
Copy link

iambudi commented Jan 29, 2023

I just upgraded colima and lima, but I"m not seeing that bind mounts don't work at all

brew upgrade lima colima
colima version 
  colima version 0.3.1
  git commit: 787ae5631ae8de072feef95a509c47fc93308b2e
  
  runtime: docker
  arch: x86_64
  client: v20.10.12
  server: v20.10.11
colima stop
colima delete
colima start
docker run -v "$(PWD):/output" bash touch "/output/foo.txt"
ls | wc -l
0 <-------------- This should say 1 because there should be 1 file named foo.txt in the directory

I've reproduced this twice now and I'm pretty sure this is a regression

I'm having this difficulty as well. colima version 0.5.2. I need to mount rw to try hot reloading.

@iambudi
Copy link

iambudi commented Jan 29, 2023

It's working now after running colima delete and colima start

@berniedurfee-renaissance

Just to clarify, this issue is still open and Colima doesn't allow for changing ownership of files in host mounted directories?

I'm seeing host mounted directories and files being owned by 506:dialout and need to change the ownership of those files and directoris.

I'm using the short form for mounting volumes:

...
services:
  my_service:
	...
	volumes:
      - my_files:/storage/my_files
...

Is this still expected behavior until this issue is resolved?

chown: changing ownership of '/storage/my_files': Permission denied

@rfay
Copy link
Contributor

rfay commented Mar 22, 2024

By default only your home directory is set up as writable. You'll need to adjust your colima.yml to make the mount that you want writable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests