Skip to content

Commit

Permalink
Fix moto env vars & pass AWS_SESSION_TOKEN to conda builds (#12902)
Browse files Browse the repository at this point in the history
This PR adds some additional AWS related environment variable mocks to the `cudf` `pytest`s.

These environment variables are documented by `moto` here: https://docs.getmoto.org/en/latest/docs/getting_started.html#recommended-usage.

This is important because of some recent CI changes that we made which now populate the `AWS_SESSION_TOKEN` environment variable, which was not mocked prior to the changes in this PR.

Additionally, this PR adds the `AWS_SESSION_TOKEN` value to the `script_env` lists for our conda recipes so that it's value is passed through to builds. I also alphabetized the `script_env` lists.

Authors:
  - AJ Schmidt (https://github.com/ajschmidt8)

Approvers:
  - GALI PREM SAGAR (https://github.com/galipremsagar)
  - Bradley Dice (https://github.com/bdice)
  - Jordan Jacobelli (https://github.com/jjacobelli)
  - Ray Douglass (https://github.com/raydouglass)

URL: #12902
  • Loading branch information
ajschmidt8 authored Mar 8, 2023
1 parent 97d8d12 commit 5b62562
Show file tree
Hide file tree
Showing 7 changed files with 60 additions and 49 deletions.
17 changes: 9 additions & 8 deletions conda/recipes/cudf/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,19 @@ build:
number: {{ GIT_DESCRIBE_NUMBER }}
string: cuda_{{ cuda_major }}_py{{ py_version }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }}
script_env:
- PARALLEL_LEVEL
- CMAKE_GENERATOR
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN
- CMAKE_C_COMPILER_LAUNCHER
- CMAKE_CXX_COMPILER_LAUNCHER
- CMAKE_CUDA_COMPILER_LAUNCHER
- SCCACHE_S3_KEY_PREFIX=cudf-aarch64 # [aarch64]
- SCCACHE_S3_KEY_PREFIX=cudf-linux64 # [linux64]
- CMAKE_CXX_COMPILER_LAUNCHER
- CMAKE_GENERATOR
- PARALLEL_LEVEL
- SCCACHE_BUCKET
- SCCACHE_REGION
- SCCACHE_IDLE_TIMEOUT
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- SCCACHE_REGION
- SCCACHE_S3_KEY_PREFIX=cudf-aarch64 # [aarch64]
- SCCACHE_S3_KEY_PREFIX=cudf-linux64 # [linux64]
ignore_run_exports:
# libcudf's run_exports pinning is looser than we would like
- libcudf
Expand Down
17 changes: 9 additions & 8 deletions conda/recipes/cudf_kafka/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,19 @@ build:
number: {{ GIT_DESCRIBE_NUMBER }}
string: py{{ py_version }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }}
script_env:
- PARALLEL_LEVEL
- CMAKE_GENERATOR
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN
- CMAKE_C_COMPILER_LAUNCHER
- CMAKE_CXX_COMPILER_LAUNCHER
- CMAKE_CUDA_COMPILER_LAUNCHER
- SCCACHE_S3_KEY_PREFIX=cudf-kafka-aarch64 # [aarch64]
- SCCACHE_S3_KEY_PREFIX=cudf-kafka-linux64 # [linux64]
- CMAKE_CXX_COMPILER_LAUNCHER
- CMAKE_GENERATOR
- PARALLEL_LEVEL
- SCCACHE_BUCKET
- SCCACHE_REGION
- SCCACHE_IDLE_TIMEOUT
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- SCCACHE_REGION
- SCCACHE_S3_KEY_PREFIX=cudf-kafka-aarch64 # [aarch64]
- SCCACHE_S3_KEY_PREFIX=cudf-kafka-linux64 # [linux64]

requirements:
build:
Expand Down
17 changes: 9 additions & 8 deletions conda/recipes/custreamz/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,19 @@ build:
number: {{ GIT_DESCRIBE_NUMBER }}
string: py{{ py_version }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }}
script_env:
- PARALLEL_LEVEL
- CMAKE_GENERATOR
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN
- CMAKE_C_COMPILER_LAUNCHER
- CMAKE_CXX_COMPILER_LAUNCHER
- CMAKE_CUDA_COMPILER_LAUNCHER
- SCCACHE_S3_KEY_PREFIX=custreamz-aarch64 # [aarch64]
- SCCACHE_S3_KEY_PREFIX=custreamz-linux64 # [linux64]
- CMAKE_CXX_COMPILER_LAUNCHER
- CMAKE_GENERATOR
- PARALLEL_LEVEL
- SCCACHE_BUCKET
- SCCACHE_REGION
- SCCACHE_IDLE_TIMEOUT
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- SCCACHE_REGION
- SCCACHE_S3_KEY_PREFIX=custreamz-aarch64 # [aarch64]
- SCCACHE_S3_KEY_PREFIX=custreamz-linux64 # [linux64]

requirements:
host:
Expand Down
17 changes: 9 additions & 8 deletions conda/recipes/dask-cudf/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,19 @@ build:
number: {{ GIT_DESCRIBE_NUMBER }}
string: cuda_{{ cuda_major }}_py{{ py_version }}_{{ date_string }}_{{ GIT_DESCRIBE_HASH }}_{{ GIT_DESCRIBE_NUMBER }}
script_env:
- PARALLEL_LEVEL
- CMAKE_GENERATOR
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN
- CMAKE_C_COMPILER_LAUNCHER
- CMAKE_CXX_COMPILER_LAUNCHER
- CMAKE_CUDA_COMPILER_LAUNCHER
- SCCACHE_S3_KEY_PREFIX=dask-cudf-aarch64 # [aarch64]
- SCCACHE_S3_KEY_PREFIX=dask-cudf-linux64 # [linux64]
- CMAKE_CXX_COMPILER_LAUNCHER
- CMAKE_GENERATOR
- PARALLEL_LEVEL
- SCCACHE_BUCKET
- SCCACHE_REGION
- SCCACHE_IDLE_TIMEOUT
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- SCCACHE_REGION
- SCCACHE_S3_KEY_PREFIX=dask-cudf-aarch64 # [aarch64]
- SCCACHE_S3_KEY_PREFIX=dask-cudf-linux64 # [linux64]

requirements:
host:
Expand Down
19 changes: 10 additions & 9 deletions conda/recipes/libcudf/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,20 @@ source:

build:
script_env:
- PARALLEL_LEVEL
- CMAKE_GENERATOR
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN
- CMAKE_C_COMPILER_LAUNCHER
- CMAKE_CXX_COMPILER_LAUNCHER
- CMAKE_CUDA_COMPILER_LAUNCHER
- SCCACHE_S3_KEY_PREFIX=libcudf-aarch64 # [aarch64]
- SCCACHE_S3_KEY_PREFIX=libcudf-linux64 # [linux64]
- CMAKE_CXX_COMPILER_LAUNCHER
- CMAKE_GENERATOR
- PARALLEL_LEVEL
- RAPIDS_ARTIFACTS_DIR
- SCCACHE_BUCKET
- SCCACHE_REGION
- SCCACHE_IDLE_TIMEOUT
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- RAPIDS_ARTIFACTS_DIR
- SCCACHE_REGION
- SCCACHE_S3_KEY_PREFIX=libcudf-aarch64 # [aarch64]
- SCCACHE_S3_KEY_PREFIX=libcudf-linux64 # [linux64]

requirements:
build:
Expand Down
11 changes: 7 additions & 4 deletions python/cudf/cudf/tests/test_s3.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2020-2022, NVIDIA CORPORATION.
# Copyright (c) 2020-2023, NVIDIA CORPORATION.

import os
import socket
Expand Down Expand Up @@ -59,9 +59,12 @@ def s3_base(endpoint_ip, endpoint_port):
with ensure_safe_environment_variables():
# Fake aws credentials exported to prevent botocore looking for
# system aws credentials, https://github.com/spulec/moto/issues/1793
os.environ.setdefault("AWS_ACCESS_KEY_ID", "foobar_key")
os.environ.setdefault("AWS_SECRET_ACCESS_KEY", "foobar_secret")
os.environ.setdefault("S3FS_LOGGING_LEVEL", "DEBUG")
os.environ["AWS_ACCESS_KEY_ID"] = "foobar_key"
os.environ["AWS_SECRET_ACCESS_KEY"] = "foobar_secret"
os.environ["S3FS_LOGGING_LEVEL"] = "DEBUG"
os.environ["AWS_SECURITY_TOKEN"] = "foobar_security_token"
os.environ["AWS_SESSION_TOKEN"] = "foobar_session_token"
os.environ["AWS_DEFAULT_REGION"] = "us-east-1"

# Launching moto in server mode, i.e., as a separate process
# with an S3 endpoint on localhost
Expand Down
11 changes: 7 additions & 4 deletions python/dask_cudf/dask_cudf/io/tests/test_s3.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2020-2022, NVIDIA CORPORATION.
# Copyright (c) 2020-2023, NVIDIA CORPORATION.

import os
import socket
Expand Down Expand Up @@ -55,9 +55,12 @@ def s3_base(endpoint_ip, endpoint_port):
with ensure_safe_environment_variables():
# Fake aws credentials exported to prevent botocore looking for
# system aws credentials, https://github.com/spulec/moto/issues/1793
os.environ.setdefault("AWS_ACCESS_KEY_ID", "foobar_key")
os.environ.setdefault("AWS_SECRET_ACCESS_KEY", "foobar_secret")
os.environ.setdefault("S3FS_LOGGING_LEVEL", "DEBUG")
os.environ["AWS_ACCESS_KEY_ID"] = "foobar_key"
os.environ["AWS_SECRET_ACCESS_KEY"] = "foobar_secret"
os.environ["S3FS_LOGGING_LEVEL"] = "DEBUG"
os.environ["AWS_SECURITY_TOKEN"] = "foobar_security_token"
os.environ["AWS_SESSION_TOKEN"] = "foobar_session_token"
os.environ["AWS_DEFAULT_REGION"] = "us-east-1"

# Launching moto in server mode, i.e., as a separate process
# with an S3 endpoint on localhost
Expand Down

0 comments on commit 5b62562

Please sign in to comment.