How to upload additional artifacts to an existing package?

[SirLemyDanger]

conan 1.0.4, linux, server: artifactory

I have a new plattform, I re-compile a conan package which is already on the server for another platfrom and now I want to upload the new artifact so I never have to compile it again.

Conan fails as it cannot reupload the conan_sources.tgz which are already on the server. I don't want to upload the sources again but the conan upload command doesn't let me upload just the artifact.

Uploading AppFramework/0.3.0@deployment/stable

Compressing recipe sources...                                         
Uploading conanmanifest.txt                                           

Uploading conanfile.py                                                

Uploading conan_sources.tgz                                           
ERROR: 
Error uploading file: conan_sources.tgz, '{
  "errors" : [ {
    "status" : 500,
    "message" : "Not enough permissions to overwrite artifact 'spc-conan:AppFramework/0.3.0/deployment/stable/export/conan_sources.tgz' (user 'deployment' needs DELETE permission)."
  } ]
}'
```

[memsharded]

Hi!

Have you tried specifying the package you want to upload? With the -p PACKAGE, --package PACKAGE argument, passing the packageID you want to upload.

Please try and tell me. Thanks!

[SirLemyDanger]

I have not tried that. My use case:

1. spawn a new docker container
2. install conan
3. clone git repo
4. conan install

I want to speed up step 4) So I would like to have a step 5)

5. upload all conan artifacts that are not on the server already

It doesn't sound practical to me to use --package PACKAGE as I don't know & don't care about the packageID. From my point of view the packageID is something conan internal with which I don't want to interfere.

[lippertto]

FWIW - I have a simpler use-case, and the -p PACKAGE is working for me. I have a Linux build that exports the recipes and built libraries to a repository, and want to add Windows binaries afterwards.

I can imagine that this is a common use-case. Maybe it would be helpful to add it to the howto section in the documentation.

[memsharded]

upload all conan artifacts that are not on the server already

This is already done. There is both protocol check and artifact de-duplication, so artifacts that are already there won't be uploaded again.

The rationale for always uploading the recipe/sources by default is because it is the way to keep consistency. Note that package binaries have a field that allows to check if they are "outdated" from the recipe. This is done storing the package recipe hash in the package binaries. So if a recipe has changes, and the package binaries are uploaded, those packages will be up-to-date to the recipe, and other package binaries that were already in the server will become outdated. It would be a bit weird to upload some binaries and the binaries you uploaded are already "outdated".

I guess the way to go would be an argument that explicitly allows to upload only package binaries, but not the recipe, lets consider it, thanks for your feedback.

[SirLemyDanger]

Hi,

may I ask how the situation with this issue is?

I still have the situation where I don't have DELETE permissions on the Artifactory but I can upload new files. With conan in its current state I can only upload 1 package. If I upload a package from another platform conan tries to upload the recipie again along with the package from the 2nd platform.

Maybe the bug after all is just that conan alters the recipie. The only thing I do is to install the recipie, build it and upload the resulting packages.

Could it be that a switch to a different conan version affects the recipie?

Cheers

[memsharded]

Hi @SirLemyDanger

Conan won't upload and won't cause this problem if the recipe is not modified. If the recipe is being modified, then yes, but that is exactly the desired behavior. I guess in your case it could be that some platform may be doing the checkout with different CRLF, like in windows. Make sure that you use the same line format (I recommended LF in all platforms) in all platforms.

There was a small issue with docker and the permissions, that made the .tgz files to change. This was fixed a couple of releases ago, so it is likely that upgrading could solve it.

Cheers!

[SirLemyDanger]

I don't think it's about line endings as I have never installed conan on any windows machine.

Let me discribe my setup regarding conan packages:

1. check out via git a project which contains a conan.py file
2. Go into its directory and call conan install . --build missing
3. Now all dependencies are fetched from the server and beeing build (as I never uploaded them from this particular platform)
4. If I try a conan upload of the just built dependencies conan tries to upload the recipie which will fail due to permissions

As you can see there is no git involved for the dependencies. Everything happens in the local conan cache.

Can you tell me what conan recognises as a recipie change?

[danimtb]

If you are trying to upload those built dependencies without specifying the remote, conan is probably trying to upload them to the remote they were installed from (only recipe was installed in this case as you are building those missing packages) and if you dont have permissions to upload to that repository you'll end up with the messaged reported above.

Make sure you upload with: conan upload pkg/version@user/channel --all -r your_remote

If you are the owner of the repository where the package were installed from, maybe you need to check the write permissions for your user.

[memsharded]

Thanks for the feedback @SirLemyDanger

Have you tried again with latest conan (1.4.4)? It seems there was a bug that was affecting docker builds, but now should be solved. Which version are you using?

[SirLemyDanger]

@danimtb Thanks, but I specified a remote

@memsharded I just retried with 1.4.4, but no change.

This is what I get on the console:

Are you sure you want to upload 'AppFramework/0.3.0@deployment/stable'? (yes/no): yes
/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
Uploading AppFramework/0.3.0@deployment/stable to remote 'scisys'
Uploading conan_sources.tgz                                           /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.r
eadthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

ERROR:
Error uploading file: conan_sources.tgz, '{
  "errors" : [ {
    "status" : 500,
    "message" : "Not enough permissions to overwrite artifact 'spc-conan:deployment/AppFramework/0.3.0/stable/export/conan_sources.tgz' (user 'deployment' needs DELETE permission)."
  } ]
}'
ERROR: Execute upload again to retry upload the failed files: conan_sources.tgz. [Remote: scisys]

So maybe I wasn't right when I said "recipe". It is rather the exported sources. But they as well should not be overwritten as there were not changed, not even re-exported again.

Cheers

[pvaddina]

I also have the same issue using 1.11.2. I wonder why this issue is being ignored. IMHO, this is quite critical in nature. Not all developers have DELETE/MODIFY permissions on the remote. Due to this bug, I am forced to request temporary write access to the remote from the admin.

[pvaddina]

I also have the same issue using 1.11.2. I wonder why this issue is being ignored. IMHO, this is quite critical in nature. Not all developers have DELETE/MODIFY permissions on the remote. Due to this bug, I am forced to request temporary write access to the remote from the admin.

Regarding the use of -p option. Using the package ID is deprecated according to the latest Conan documentation. It is suggested to use the complete package reference. I did that and it seems it does not help either.

[memsharded]

Hi @SirLemyDanger @pvaddina

I am revisiting this issue. I have tried to reproduce, and worked ok here:

• I am using latest Conan 1.20 in Windows and Artifactory 6.5.18, without DELETE permissions.
• I can create a package using exports_sources for my default settings and upload. It will upload the conan_sources.tgz
• I can create another binary package for a different configuration, and upload. As the conan_sources.tgz has NOT been modified, the upload is not actually executed, it keeps going and is able to upload the new binary configuration succesfully.
• If I do changes in the source code, create the package again, then the sources and the conan_sources.tgz is different, it tries to upload the new conan_sources.tgz and it fails.
• If I modify back the sources, then it I am able to upload again new binaries.

So it seems that what is happening in your cases is that you have a modified conan_sources.tgz. Could you please verify the checksums of both the local file in your Conan cache and in Artifactory? I would say that they are different.

If they are different, then Conan and Artifactory are working fine, and those are good permissions settings, no need to allow to overwrite. This is a protection mechanism, the sources must be exactly the same sources for all the different binary packages. If the sources are changed, Conan blocks the upload to maintain the consistency of the package. You need to make sure that the sources are not changed between different systems (a typical case is checking out code in git with different CRLF).

Please check that and report. Thanks!

[pvaddina]

I created two channels (one for Windows and the one for Linux) and notice that the manifest file indicates the same checksums for all the exported files in both the channels. But the timestamp is different, which apparently should not be considered by conan. So, I am not sure what else could be different here ?

I tried the upload with the option "--no-overwrite", to see what conan thinks about the package, and get:

ERROR: Local package is different from the remote package. Forbidden overwrite.

I manually tried to check the files under conan_sources.tgz under both the channels, and found identifical file with 'LF' line endings. So I am clueless now.

What bothers me is the lack of information on this topic ? And hoped conan gave more information on such error messages.

[memsharded]

I created two channels (one for Windows and the one for Linux) and notice that the manifest file indicates the same checksums for all the exported files in both the channels. But the timestamp is different, which apparently should not be considered by conan. So, I am not sure what else could be different here ?

Could you clarify what you mean with 2 channels, one for Windows and one for Linux? It shouldn't be necessary to use different channels, as Conan manages the binaries for different platforms within the same channel.

Yes, that timestamp is not in conan_source.tgz, so that file causing the error is only possible if the checksum of the final conan_source.tgz is different. That might happen in some scenarios, like docker using a shared drive, where weird user/permissions might cause a different final conan_source.tgz even if the zipped files and their checksums are the same. This is the reason it is important to check the checksum of the conan_source.tgz, and not only the conanmanifest.txt.

What bothers me is the lack of information on this topic ? And hoped conan gave more information on such error messages.

Sorry about that, the fact is that this has been a very unusual thing (just 2 times here, and the docker thing I commented, in the last 2 years), otherwise we (or often the community) try to help improving the error messages.

What I would suggest if you are in the CppLang Slack is to try to connect there, and try to debug this issue together via chat, might be more productive. Send us a note to [email protected] if interested.

[pvaddina]

Could you clarify what you mean with 2 channels, one for Windows and one for Linux? It shouldn't be necessary to use different channels, as Conan manages the binaries for different platforms within the same channel.

This is purely to check the manifest file that is generated for the two configurations. My problem is Conan not being able to manage the binaries of different platforms in the same channel

Yes, that timestamp is not in conan_source.tgz, so that file causing the error is only possible if the checksum of the final conan_source.tgz is different. That might happen in some scenarios, like docker using a shared drive, where weird user/permissions might cause a different final conan_source.tgz even if the zipped files and their checksums are the same. This is the reason it is important to check the checksum of the conan_source.tgz, and not only the conanmanifest.txt.

Checking the MD5 of the conan_sources.tgz for the two platforms, Indeed they are different. And I am running the recipe under Docker for Ubuntu and Yocto distributions. According to issue #3473 (the last three comments) Conan is only supposed to check the manifestfile and not the MD5 of the conan_sources.tgz anymore since version 1.8. And it seems based on the comments, it is integrated and tested. However, I have Version 1.11.2. So why this issue again ?

Sorry about that, the fact is that this has been a very unusual thing (just 2 times here, and the docker thing I commented, in the last 2 years), otherwise we (or often the community) try to help improving the error messages.

Thank you for your concern and I really appreciate the effort you and the community in general puts in making this a great product. Just want to suggest what can be improved.

All that Conan says is Upload failed due to missing permissions. However, it should have said, why it is trying to upload in the first place despite everything seems to be the same. Ex: If it said the manifest file is different, or may be the MD5 of the conan_sources.tgz is different, etc, then the user knows immediately what is happening.

[pvaddina]

For everyone who has this problem: If you are generating a multi platform package (Win, Linux), and the package generation in Linux is in a Docker container, then please note that do not use a repository which is shared between the host and the container. This seems to generate a different checksum for the conan_sources.tgz file even though they are identical (even in terms of line endings). So conan assumes this is a change in the recipe and will try to overwrite the one in artifactory. Unfortunately if you have no upload rights, you will be pretty much blocked here.
Solution is, if you are generating the package in Linux then make sure you are not sharing the repo between the Host and the container. Instead, in the container, create the repo in a non-shared folder and this will generate the same checksum for the conan_sources.tgz. And the package generation works fine.
This issue costed me nearly 2 days. Hope it helps someone out there.

[djeverett]

I am having problems without exporting any sources.
We build packages separately for many different Linux distributions, and also want to prevent overwrites in Artifactory in order to ensure that version numbers get updated with every commit. I've used:
conan upload --no-overwrite recipe --remote my-repo my-package/1.0@mainline/postcommit:de15790034a677e8e67c1038bec65ee732939cb1 --confirm
to only upload the package if the recipe is already present. This worked perfectly when building a simple test package, but when building a large package (with significant variation in build times) it failed. I presume the problem is timestamps. Any suggestions for how to prevent Conan trying to upload a recipe that is exactly the same in content to the one already in Artifactory?

[memsharded]

Conan has evolved a lot since last activity:

• Conan 2.0 tracks all upload/downloads based on revisions, not on checksums of the .tgz artifacts
• All packages are immutable (identified by revisions)

For the use case of wanting to add additional files that are not the package itself, (like logs, tests, etc), please follow up #13330

Closing this as outdated, please create new tickets for further questions.