Enable CUDA support [READY]

[pearu]

Checklist

• Used a fork of the feedstock to propose changes
• Bumped the build number (if the version is unchanged)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

• It looks like the 'arrow-cpp' output doesn't have any tests.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

• The recipe must have some tests.
• The recipe must have a build/number section.

For recipe:

• License is not an SPDX identifier (or Other) nor an SPDX license expression.

[isuruf]

@pearu, your idea is a good one, but this is how you should implement it.
Skip linux64 and cuda_compiler_version != 9.2. (That means you only build with cuda 9.2 on linux64.) Have two packages arrow-cpp and arrow-cpp-cuda.

[pearu]

@isuruf ok, I'll fix to using 9.2. Two questions:

1. How to fix the following osx CI failure:

Error: Failed to render jinja template in /Users/runner/runners/2.165.2/work/1/s/recipe/meta.yaml:
'cuda_compiler_version' is undefined

2. When building arrow-cpp against cuda 9.2, will the conda package still be usable in environments with cuda > 9.2? Why not build against the latest cuda version?

[isuruf]

When building arrow-cpp against cuda 9.2, will the conda package still be usable in environments with cuda > 9.2?

Yes, it looks like arrow is using only the CUDA driver API. @xhochy should confirm.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

• License is not an SPDX identifier (or Other) nor an SPDX license expression.

[pearu]

@conda-forge-admin, please rerender

[jakirkham]

Other packages are using a mutex package.

[isuruf]

It seems like we can always change which way we are doing things if we want to in the future. arrow-cpp=*=*cuda seems cleaner to the user to me, but again, it doesn't really matter.

It's hard to change stuff in the future. Get it right first, otherwise people will complain.

That said, we can have both. We can have the build string in it and users who want to use the mutex package can do so.

We should also have a look at the name. arrow-cpp-build-ext, ucx-proc, etc.

[jakirkham]

No objections to keeping both.

Edit: Agree it would be nice to have the name align. Other packages currently do *-proc.

[pearu]

I have fixed merged conflicts and renamed arrow-cpp-build-ext to arrow-cpp-proc.

@isuruf @jakirkham please review.

[jakirkham]

Looks good @pearu! Thanks again for working on this 😄

Had some suggestions below that might simplify the syntax a bit and drop some things we don't need. Nothing critical though.

[pearu]

Hi all, could we have this PR merged?

[pearu]

Several projects using CUDA rely on arrow-cpp 0.16 and packaging those are currently blocked by this PR.
@xhochy could you merge this PR?

[xhochy]

@pearu Leaving this open until tomorrow to let @isuruf give any objections, otherwise I'm going to have a review by myself then (no time for that today). I will merge this to a separated 0.16 branch and will port things over to 0.17 afterwards.

[jakirkham]

LGTM. Thanks for your work on this @pearu! 😄

[alexbaden]

Hi all -- thanks for doing this work @pearu and to everyone else for reviewing. We just pushed out a new version of OmniSciDB which uses Arrow 0.16 and are interested in seeing this patch merged so we can push our python connector changes as well. Is there a rough ETA?

[jakirkham]

FWIW this issue may be relevant.

xref: https://issues.apache.org/jira/browse/ARROW-8608