Can't clone using vboxsf

[hdanniel]

I'm using a vagrant box (with VirtualBox) inside a Windows 7 host. I cannot clone to a vboxsf filesystem using git2go . This is the code i'm using:

_, err := git.Clone(RepoURL,"/vagrant/stage/myrepo" , CloneOptions)

/vagrant is the shared directory between the VM and the host . When doing a df -T I have this result:

vagrant     vboxsf   466295804 337593544 128702260  73% /vagrant

I'm getting the following error when cloning:

error: Failed to increase size of pack file '/vagrant/stage/myrepo/.git/objects/pack/pack_git2_Vz0SO2': Operation not permitted

Checking the permissions of the file is read-only , the parent directories (created by the clone) have 777 permissions:

ls -l /vagrant/stage/myrepo/.git/objects/pack/
total 0
-r-xr-xr-x 1 vagrant vagrant 0 Apr 10 14:30 pack_git2_Vz0SO2

When cloning into a different partition it works fine , for example in the / directory which is mounted as ext4:

/dev/mapper/VolGroup-lv_root
          ext4     8776988   8010148    320992  97% /

I can make a Fetch and Remote Ls with git2go. Also when using git clone I don't have the problem, only (as far as I know) when git2go. Why is the Clone function creating the files as read-only?

Thanks, I'm learning a lot with git2go

[carlosmn]

The code to clone calls down to fetch; it's quite odd that we would see different issues between them. One thing which could confuse things is the umask.

Though the file has read-only permissions, the file descriptor is opened read-write. If you look at a packfile which successfully downloads, you should see the same thing. I've never tried to run your particular configuration, so I don't know what peculiarities this vboxsf has as a file-system.

[hdanniel]

I'm sorry Carlos, the function I was using is ConnectFetch not Fetch. ConnectFetch doesn't create the pack files so the error doesn't appear. The error is triggered when git.Clone calls git_clone. I tried to trace it in libgit2 with no success. The error message appears in the append_to_pack function (indexer.c) but seems to me that the permissions are set when creating the pack file.

There is a similar error in a svn client: http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=81629 . Maybe a different path of action when using Linux or Windows provokes the problem when the working directory is in a windows partition mounted in linux.

Thanks!

[carlosmn]

The permissions on the file are different from the permissions on the file descriptor. It looks like the filesystem is acting differently from every other one we've tested this on in this regard.

The issue you linked to is about SMB and renaming, so I don't see the connection.

[jimmycuadra]

I'm running into the same problem from a similar use case (a shared VirtualBox volume). I'm running the Rust compiler via the Cargo package manager inside a Docker container, using boot2docker on OS X, meaning the offending volume is actually inside a VirtualBox VM. Cargo keeps a local index of the packages it downloads in a Git repository stored in a subdirectory of .cargo. In order to cache package resolution and download, I'm mounting the .cargo directory from a directory on my host filesystem into the boot2docker VM, and then again into the container.

Running cargo build makes Cargo attempt to download the packages my program depends on, which aborts with the following error:

root@59c9b7f2b312:/source# cargo build --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`
failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  [2] Failed to increase size of pack file '/source/.cargo/registry/index/github.aaakk.us.kg-1ecc6299db9ec823/.git/objects/pack/pack_git2_VzVv1F': Operation not permitted

Here are the permissions of the files in that pack directory:

root@59c9b7f2b312:/source/.cargo/registry/index/github.aaakk.us.kg-1ecc6299db9ec823/.git/objects/pack# ls -la
total 0
drwxr-xr-x 1 1000 staff 102 May  5 07:00 .
drwxr-xr-x 1 1000 staff 136 May  5 07:00 ..
-r--r--r-- 1 1000 staff   0 May  5 07:00 pack_git2_VzVv1F

And the output of df -T, for reference:

root@59c9b7f2b312:~# df -T
Filesystem     Type   1K-blocks      Used Available Use% Mounted on
rootfs         rootfs  19049892   1927336  16131832  11% /
none           aufs    19049892   1927336  16131832  11% /
tmpfs          tmpfs    1028252         0   1028252   0% /dev
shm            tmpfs      65536         0     65536   0% /dev/shm
none           vboxsf 243924880 236556804   7368076  97% /source
/dev/sda1      ext4    19049892   1927336  16131832  11% /etc/resolv.conf
/dev/sda1      ext4    19049892   1927336  16131832  11% /etc/hostname
/dev/sda1      ext4    19049892   1927336  16131832  11% /etc/hosts
tmpfs          tmpfs    1028252         0   1028252   0% /proc/kcore

I've tried it both with a .cargo directory that's already been run/populated on the the OS X host, and without it existing prior to it being created by cargo build inside the container. Both result in the same error.

What I'm doing might seem a bit convoluted, but I believe this is the same issue @hdanniel is experiencing.

[carlosmn]

It does look like the same issue, where the VirtualBox filesystem is not following the conventions every other filesystem uses (which may or may not be something posix mentions, depending on the reading).

I would consider this a bug in vboxfs as we have a file descriptor with write permissions. As a workaround, it would be interesting to see what happens with the following patch applied, which might let us make vboxfs that we do have permission to modify the file.

Do the indexer tests for libgit2 itself run under this filesystem?

diff --git a/src/odb_pack.c b/src/odb_pack.c
index 1757cf9..830e41c 100644
--- a/src/odb_pack.c
+++ b/src/odb_pack.c
@@ -579,7 +579,7 @@ static int pack_backend__writepack(struct git_odb_writepack **out,
        GITERR_CHECK_ALLOC(writepack);

        if (git_indexer_new(&writepack->indexer,
-               backend->pack_folder, 0, odb, progress_cb, progress_payload) < 0) {
+               backend->pack_folder, 0622, odb, progress_cb, progress_payload) < 0) {
                git__free(writepack);
                return -1;
        }

[jimmycuadra]

There were several failures in the libgit2 test suite running on the vboxsf volume inside the container for me: https://gist.github.com/jimmycuadra/db03949cee3b71876a5f I'm not sure if these are all related to the volume's filesystem or if I'm just missing some dependencies for libgit2's build environment. The container is based off the official Debian wheezy image for Docker which is a more stripped down version of Debian than you'd have on a typical VM.

[carlosmn]

It's not an issue of dependencies, and we don't need anything special from the system to run the suite (except for network, which doesn't get tested by default). It does look like it comes down to vboxsf being an odd filesystem.

It's quite strange that the pack::indexer tests run fine, but a clone or fetch has issues, as they should be behaving the same way.

[hdanniel]

When running the clone in vboxsf with the patch applied, I have the following error:

Failed to mmap. Could not write data: Invalid argument

Clone using an ext4 fileystem keeps working.

[carlosmn]

libgit2 master and v0.22.3 both now use lssek + write instead of truncate, which may be better supported by vboxsf.

[clns]

I'm also using git2go inside a VirtualBox VM with Docker on Windows 10. Remote.Fetch(() returns the error:

Failed to mmap. Could not write data: Invalid argument

I'm using the latest tip from the next branch.

[ktsakas]

I had the same problem using Vagrant:

Failed to mmap. Could not write data: Invalid argument

I fixed it by passing a directory other than /vagrant to git.Clone.
The problem appears to be with synced directories.

[Scorpil]

@ktsakas had this problem recently trying to compile servo. Looks like this happens because of this ancient issue in vboxfs. I've fixed it by switching to NFS.

[patrickdevivo]

I was encountering this error specifically in an AWS lambda function (with an EFS mount). I applied the patch mentioned by @carlosmn (#194 (comment)) right here: mergestat/libgit2@7f102fe and it worked like a charm. Solved the permission issue on the packfile I was encountering (similar to original question)