--skip-existing flag not working on linux, version 1.21.3

[ajdes]

I am only seeing this issue on Linux (not OS X) and it just appeared when upgrading to conda build version 1.21 (1.21.3 specifically). Additionally, I am using a S3 bucket as my channel. 'conda search package_name' correctly locates the package, but if I issue a 'conda build --skip-existing path/to/recipe' it always rebuilds. The recipe version is the same as the existing version.

I will work on trying to get a more concrete example using a public package.

[msarahan]

Interesting. We test that functionality explicitly here: https://github.com/conda/conda-build/blob/1.21.x/tests/test_build_recipes.py#L276-L286

Any info you can provide on why that test isn't catching misbehavior you're seeing would be great. If possible, could you try to run that unit test from the conda-build source?

To be clear, the S3 bucket works OK on OS X, but not Linux? Or is the S3 bucket only used on one of the platforms?

[ajdes]

Ah. I did a bit more careful digging. It seems like its also not working on OS X, I just had a local copy cached when I tested that. My guess it has something to do with the S3 channel not being searched correctly when using the --skip-existing flag. As mentioned, packages are resolving correctly in the s3 channel when doing a 'conda search package_name' however.

So overall what ive seen:

1. --skip-existing works correctly when I have a local build of the recipe to resolve against
2. --skip-existing does NOT work correctly when resolving against an S3 channel (on both OS X and Linux)

In the test case you pointed it out, it appears to only test against scenario 1, so that is at least consistent.

[msarahan]

OK, need more info. The code in conda-build that does this is:

https://github.com/conda/conda-build/blob/master/conda_build/main_build.py#L246-L251
and
https://github.com/conda/conda-build/blob/master/conda_build/main_build.py#L296-L302

these build the index with conda's code here:
https://github.com/conda/conda/blob/4.1.x/conda/api.py#L10-L46

nothing looks obviously wrong to me here. I think we need to look at your S3 channel, and make sure that conda is recognizing it properly. I know you said that search works - that's a good start. What's the output of conda info? Can you open a Python interpreter and try:

import conda.api as api
index = api.get_index(use_local=True)

The keys of index are package names. Some of them may have prefixes like local::packagename. The bug here (if it isn't in your S3 channel) is probably in a prefix that conda-build does not handle correctly. Please search through these keys for your package name. Something like this might work:

for key in index.keys():
    if "yourpackagename" in key:
       print("found package: " + key)

In the meantime, I will add a test that uploads a package to an anaconda.org channel, and uses skip-existing after deleting that built package locally.

[ajdes]

Thanks for looking into this.
Here is my output from conda info on OS X:

             platform : osx-64
        conda version : 4.1.6
    conda-env version : 2.5.1
  conda-build version : 1.21.3
       python version : 2.7.12.final.0
     requests version : 2.9.1
     root environment : /Users/anthony/miniconda  (writable)
  default environment : /Users/anthony/miniconda
     envs directories : /Users/anthony/miniconda/envs
        package cache : /Users/anthony/miniconda/pkgs
         channel URLs : s3://conda-packages/osx-64/
                        s3://conda-packages/noarch/
                        https://repo.continuum.io/pkgs/free/osx-64/
                        https://repo.continuum.io/pkgs/free/noarch/
                        https://repo.continuum.io/pkgs/pro/osx-64/
                        https://repo.continuum.io/pkgs/pro/noarch/
          config file : /Users/anthony/.condarc
         offline mode : False
    is foreign system : False

Here are the keys in the index for an example package 'pika' (when I also have it built locally):

package local::pika-0.10.0-py27_0.tar.bz2
package s3://conda-packages::pika-0.10.0-py27_0.tar.bz2

[msarahan]

Thanks. Yes, it does look like the s3://... prefix is not going to work with the code as is. I need to think about how to generalize this, but will do so soon.

[msarahan]

This will be fixed in conda-build 1.21.5, to be released probably tomorrow.

[msarahan]

1.21.5 has been tagged, and should be available as a conda install soon. Please let me know if you have further trouble. Closing this issue for now.

[ajdes]

Thanks this is working now

[msarahan]

Excellent, thanks for the feedback, and thanks for raising this issue.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically locked because it has not had recent activity after being closed.

Please open a new issue if needed.

Thanks!