Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFE: support substitution in environment.environment in nested pipelines or actions #1402

Open
smoser opened this issue Jul 23, 2024 · 1 comment

Comments

@smoser
Copy link
Contributor

smoser commented Jul 23, 2024

I have a situation where I would like to put SETUPTOOLS_SCM_PRETEND_VERSION=${{package.version}} into the build environment.

I want to do:

environment:
  environment:
    SETUPTOOLS_SCM_PRETEND_VERSION: ${{package.version}}
    OTHER: "other is ${{vars.VARS}}"

The second 'OTHER' there i just put there. I think it is reasonable to support substitution within a string rather than only supporting the entire string. but I do not want to request shell-like behavior or expansion.

I also would lke to request that work within a 'range' section like seen here.

smoser added a commit to smoser/wolfi-os that referenced this issue Oct 8, 2024
A couple things here:
1. fix FTBFS state that apache-arrow was currently in.

  * need zstd-static because:

       CMake Error at /usr/lib64/cmake/zstd/zstdTargets.cmake:107 (message):
         The imported target "zstd::libzstd_static" references the file
            "/usr/lib/libzstd.a" uses=cmake/configure
         but this file does not exist.  Possible reasons include:

2. cherry-pick a patch (from 2016) to rapidjson that allows a user of it
   (apache-arrow) to build with gcc-14.

3. Add environment variables to build so the python build modules
   have access.  I had hoped to localize these or use substitution
   in the environment block, but that is not supported.

   chainguard-dev/melange#1548
   chainguard-dev/melange#1402

4. drop unnecessary runtime dependency from libarrow to libarrow.

5. remove pyarrow.

6. bump the epoch on apache-arrow to 3 to be newer than pyarrow was.
smoser added a commit to smoser/wolfi-os that referenced this issue Oct 8, 2024
A couple things here:
1. fix FTBFS state that apache-arrow was currently in.

  * need zstd-static because:

       CMake Error at /usr/lib64/cmake/zstd/zstdTargets.cmake:107 (message):
         The imported target "zstd::libzstd_static" references the file
            "/usr/lib/libzstd.a" uses=cmake/configure
         but this file does not exist.  Possible reasons include:

2. cherry-pick a patch (from 2016) to rapidjson that allows a user of it
   (apache-arrow) to build with gcc-14.

3. Add environment variables to build so the python build modules
   have access.  I had hoped to localize these or use substitution
   in the environment block, but that is not supported.

   chainguard-dev/melange#1548
   chainguard-dev/melange#1402

4. drop unnecessary runtime dependency from libarrow to libarrow.

5. remove pyarrow.

6. bump the epoch on apache-arrow to 3 to be newer than pyarrow was.
smoser added a commit to smoser/wolfi-os that referenced this issue Oct 8, 2024
A couple things here:
1. fix FTBFS state that apache-arrow was currently in.

  * need zstd-static because:

       CMake Error at /usr/lib64/cmake/zstd/zstdTargets.cmake:107 (message):
         The imported target "zstd::libzstd_static" references the file
            "/usr/lib/libzstd.a" uses=cmake/configure
         but this file does not exist.  Possible reasons include:

2. cherry-pick a patch (from 2016) to rapidjson that allows a user of it
   (apache-arrow) to build with gcc-14.

3. Add environment variables to build so the python build modules
   have access.  I had hoped to localize these or use substitution
   in the environment block, but that is not supported.

   chainguard-dev/melange#1548
   chainguard-dev/melange#1402

4. drop unnecessary runtime dependency from libarrow to libarrow.

5. remove pyarrow.

6. bump the epoch on apache-arrow to 3 to be newer than pyarrow was.
smoser added a commit to wolfi-dev/os that referenced this issue Oct 8, 2024
…30285)

A couple things here:
1. fix FTBFS state that apache-arrow was currently in.

  * need zstd-static because:

CMake Error at /usr/lib64/cmake/zstd/zstdTargets.cmake:107 (message):
         The imported target "zstd::libzstd_static" references the file
            "/usr/lib/libzstd.a" uses=cmake/configure
         but this file does not exist.  Possible reasons include:

2. cherry-pick a patch (from 2016) to rapidjson that allows a user of it
   (apache-arrow) to build with gcc-14.

3. Add environment variables to build so the python build modules
   have access.  I had hoped to localize these or use substitution
   in the environment block, but that is not supported.

   chainguard-dev/melange#1548
   chainguard-dev/melange#1402

4. drop unnecessary runtime dependency from libarrow to libarrow.

5. remove pyarrow.

6. bump the epoch on apache-arrow to 3 to be newer than pyarrow was.

7. provide multiple versions of the python modules, and make them
provide 'pyarrow' for backwards compat.
gdonval pushed a commit to gdonval/wolfi-os that referenced this issue Oct 9, 2024
…olfi-dev#30285)

A couple things here:
1. fix FTBFS state that apache-arrow was currently in.

  * need zstd-static because:

CMake Error at /usr/lib64/cmake/zstd/zstdTargets.cmake:107 (message):
         The imported target "zstd::libzstd_static" references the file
            "/usr/lib/libzstd.a" uses=cmake/configure
         but this file does not exist.  Possible reasons include:

2. cherry-pick a patch (from 2016) to rapidjson that allows a user of it
   (apache-arrow) to build with gcc-14.

3. Add environment variables to build so the python build modules
   have access.  I had hoped to localize these or use substitution
   in the environment block, but that is not supported.

   chainguard-dev/melange#1548
   chainguard-dev/melange#1402

4. drop unnecessary runtime dependency from libarrow to libarrow.

5. remove pyarrow.

6. bump the epoch on apache-arrow to 3 to be newer than pyarrow was.

7. provide multiple versions of the python modules, and make them
provide 'pyarrow' for backwards compat.
@smoser
Copy link
Contributor Author

smoser commented Nov 17, 2024

I think that generally it does work. I'm not sure what I was seeing when I opened this, but I suspect that it was related to nested pipelines.

Here is an example to play with:
test-me.yaml

package:
  name: test-me
  version: 1.0
  epoch: 0

vars:
  WARK: abc
  FOO: boo

environment:
  contents:
    packages:
      - busybox
  environment:
    BAR: "the other day I saw a BAR"
    WARK: ${{vars.WARK}}
    FOOFOO: "${{vars.FOO}} this should work too"

pipeline:
  - environment:
      XX: SuperX
    runs: |
      echo "did it work here? XX=$XX"
  - environment:
      XX: SuperX
    uses: test/dump-env
    with:
      non-empty: BAR WARK FOOFOO XX

  - environment:
      MYVAR: "WINNER WINNER"
    pipeline:
      - runs: |
          echo "==== FOO=$FOO ==="
          echo "==== FOOFOO=$FOOFOO ==="
          echo "==== BAR=$BAR ==="
          echo "==== WARK=$WARK ==="
          echo "==== MYVAR=$MYVAR ==="
      - uses: test/dump-env
        environment:
          XX: SuperX
        with:
          non-empty: BAR WARK FOOFOO XX MYVAR

pipelines/test/dump-env.yaml:

name: dump your environment
inputs:
  non-empty:
    required: true

pipeline:
  - name: "test-non-empty"
    runs: |
      _fails=0
      for n in ${{inputs.non-empty}} ; do
        eval v='$'"${n}"
        [ -n "$v" ] && echo "PASS: non-empty $n=$v" && continue
        echo "FAIL: empty $n"
        _fails=$((_fails+1))
      done
      exit $_fails
  - name: "show env"
    runs: |
      env

here is the relevant output of make debug/test-me

2024/11/16 21:20:07 INFO built image layer tarball as /tmp/apko-temp-615903904/apko-x86_64.tar.gz
2024/11/16 21:20:07 INFO using /tmp/apko-temp-615903904/apko-x86_64.tar.gz for image layer
2024/11/16 21:20:07 INFO ImgRef = /tmp/melange-guest-3447417309
2024/11/16 21:20:07 INFO did it work here? XX=SuperX
2024/11/16 21:20:07 INFO running step "test/dump-env"
2024/11/16 21:20:07 INFO running step "test-non-empty" uses=test/dump-env
2024/11/16 21:20:07 INFO PASS: non-empty BAR=the other day I saw a BAR uses=test/dump-env name=test-non-empty
2024/11/16 21:20:07 INFO PASS: non-empty WARK=abc uses=test/dump-env name=test-non-empty
2024/11/16 21:20:07 INFO PASS: non-empty FOOFOO=boo this should work too uses=test/dump-env name=test-non-empty
2024/11/16 21:20:07 INFO FAIL: empty XX uses=test/dump-env name=test-non-empty
2024/11/16 21:20:07 INFO running step "show env" uses=test/dump-env
2024/11/16 21:20:07 INFO SHLVL=1 uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO HOME=/home/build uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO OLDPWD=/home/build uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO GOTOOLCHAIN=local uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO GOFLAGS= uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO FOOFOO=boo this should work too uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO GOMODCACHE=/var/cache/melange/gomodcache uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO BAR=the other day I saw a BAR uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO SOURCE_DATE_EPOCH=0 uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO WARK=abc uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO GOPATH=/home/build/.cache/go uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO PWD=/home/build uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO ==== FOO= ===
2024/11/16 21:20:07 INFO ==== FOOFOO=boo this should work too ===
2024/11/16 21:20:07 INFO ==== BAR=the other day I saw a BAR ===
2024/11/16 21:20:07 INFO ==== WARK=abc ===
2024/11/16 21:20:07 INFO ==== MYVAR=WINNER WINNER ===
2024/11/16 21:20:07 INFO running step "test/dump-env"
2024/11/16 21:20:07 INFO running step "test-non-empty" uses=test/dump-env
2024/11/16 21:20:07 INFO PASS: non-empty BAR=the other day I saw a BAR uses=test/dump-env name=test-non-empty
2024/11/16 21:20:07 INFO PASS: non-empty WARK=abc uses=test/dump-env name=test-non-empty
2024/11/16 21:20:07 INFO PASS: non-empty FOOFOO=boo this should work too uses=test/dump-env name=test-non-empty
2024/11/16 21:20:07 INFO FAIL: empty XX uses=test/dump-env name=test-non-empty
2024/11/16 21:20:07 INFO FAIL: empty MYVAR uses=test/dump-env name=test-non-empty
2024/11/16 21:20:07 INFO running step "show env" uses=test/dump-env
2024/11/16 21:20:07 INFO SHLVL=1 uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO HOME=/home/build uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO OLDPWD=/home/build uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO GOTOOLCHAIN=local uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO GOFLAGS= uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO FOOFOO=boo this should work too uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO GOMODCACHE=/var/cache/melange/gomodcache uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO BAR=the other day I saw a BAR uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO SOURCE_DATE_EPOCH=0 uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO WARK=abc uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO GOPATH=/home/build/.cache/go uses=test/dump-env name="show env"
2024/11/16 21:20:07 INFO PWD=/home/build uses=test/dump-env name="show env"

@smoser smoser changed the title RFE: support substitution in environment.environment RFE: support substitution in environment.environment in nested pipelines or actions Nov 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant