[bug] Permision denied on conan cache restore

[obajj-dlb]

Describe the bug

OS: Sonoma 14.4 (23E214) and lower versions. It is also failing on Windows.
Conan version: 2.0.17
Conan profiles: armv8 and x86_64

When restoring two packages to the conan cache we get the following error on the second restore:
ERROR: [Errno 13] Permission denied: '/Users/obajj/.conan2/p/pkg6360ad46104cc/s/.git/objects/pack/pack-78b4c8af2bb8d54a6e0b73a9f60ccda002c62d51.idx'

This doesn't happen always.

How to reproduce it

In armv8 machine:
conan cache save pkg:* --file=macos-arch_armv8-clang-cppstd_17.tgz

In x86_64 machine:
conan cache save pkg:* --file=macos-arch_x86_64-clang-cppstd_17.tgz

In (another) armv8 machine:
conan cache restore macos-arch_armv8-clang-cppstd_17.tgz
conan cache restore macos-arch_x86_64-clang-cppstd_17.tgz

output:
ERROR: [Errno 13] Permission denied: '/Users/obajj/.conan2/p/pkg6360ad46104cc/s/.git/objects/pack/pack-78b4c8af2bb8d54a6e0b73a9f60ccda002c62d51.idx'

[memsharded]

Hi @obajj-dlb

Thanks for your report.

I see that you are packaging the whole git repo (the .git folder) inside the package? This wouldn't be expected, sounds like unintended?

Git repos internal .git folder is not designed to be relocatable to other machines, up to my knowledge, this could be causing the issues.

[obajj-dlb]

Hi @memsharded

We're currently using git as version control system for that package indeed. We checked an older issue (#6605) and saw that this was similarly discussed, but the proposal was dismissed.

Do you know if there is any way to exclude the .git folder from getting packaged, for instance in the recipe configuration? That way we could create the package and we wouldn't break the repository folder.

[memsharded]

That ticket referred to the scm feature that was removed, this is the reason it was closed. But not dismissed as a use case, this is doable with Conan already.

Modern Conan has full control, not only with the exports_sources attribute, but with the export_source() method to be able to fully specify what gets copied and what not in the first exporting step. Likewise it is also possible to exclude and not package the .git folder in the package() method too. So it is totally possible to exclude the .git folder from getting packaged in the recipe configuration. If you want to share your conanfile.py that would help.

[memsharded]

Hi @obajj-dlb

Did you manage to solve the issue, by not copying/exporting the full .git folder?
Can we close this ticket? Thanks!

[obajj-dlb]

Hi @memsharded,

We haven't tested it yet, but it seems like you're right. Close it if you want and I can reopen it if needed. Is that okay?

Thank you!

[memsharded]

Sounds good, closing, you can always re-open or create a new ticket, thanks!

[Paultergeist]

Hi @memsharded,
Looks like the .git directory was not packaged by package(), but was part of source folder (judging by /s/ in the path).
I just hit the same bug today, and I have questions:

1. Why can't Conan properly restore cache if it has .git directory? How is it special and what prevents it from restoring it properly?
2. If it's really an unavoidable problem, how to avoid saving source directory by conan cache save? In Conan 1 I could do conan remove -s, but I don't see it in Conan 2. Should I remove .git directory in source() right after cloning? Btw, my source() method is

    def source(self):
        git = Git(self)
        git.fetch_commit(**self.conan_data["sources"][self.version])

is anything wrong with that? I'm using Conan 2.5.0.

EDIT: I see that I can remove source and build folders using conan cache clean. Still the question about why it's impossible to restore the cache with .git folder stands.

[memsharded]

Hi @Paultergeist

The source directory can be cleaned with conan cache clean. We might actually reconsider if this should be saved-restored or not, I can't recall now the reasons.

I think the .git thing is mostly about permissions and file modes not being equal in different platforms. I have had issues in the past while sharing .git folders between Windows and Linux for example. Note the conan cache save/restore do nothing special but targz and untargz from disk, so if it is failing I suspect it is likely to underlaying platform issues.