Pants export doesn't work on all machines

[engnatha]

Describe the bug
I am able to run ./pants export :: successfully on my machine. When I ran this through our CI system, I got a permission denied error when trying to clone a VCS requirement from our requirements.txt file. I know the CI worker had access to the repos because it cloned a repo to get this far (the VCS requirement is another internal repo).

I already have the following added to pants.toml.

[subprocess-environment]
env_vars.add = ["SSH_AUTH_SOCK"]

I'm curious if it's possible this isn't getting passed through on my CI system and causing the export step to fail.

Pants version
2.11.0

OS
ubuntu 20.04

[engnatha]

When using --print-stacktrace, I get

[BOOTSTRAP] [WARNING] Received non-zero return code (1) from /buildbot/astranis-monorepo-python/build/pants --print-stacktrace export ::
[BOOTSTRAP] [WARNING] STDOUT: 
[BOOTSTRAP] [WARNING] STDERR: 23:06:08.22 [INFO] Initializing scheduler...
23:06:08.36 [INFO] Scheduler initialized.
23:06:09.43 [WARN] Unmatched globs from astranis-python/astranis/sw/ground/service/beacon:beacon's `sources` field: ["astranis-python/astranis/sw/ground/service/beacon/*.py", "astranis-python/astranis/sw/ground/service/beacon/*.pyi"], excludes: ["astranis-python/astranis/sw/ground/service/beacon/*_test.py", "astranis-python/astranis/sw/ground/service/beacon/*_test.pyi", "astranis-python/astranis/sw/ground/service/beacon/conftest.py", "astranis-python/astranis/sw/ground/service/beacon/test_*.py", "astranis-python/astranis/sw/ground/service/beacon/test_*.pyi", "astranis-python/astranis/sw/ground/service/beacon/tests.py", "astranis-python/astranis/sw/ground/service/beacon/tests.pyi"]
Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to https://www.pantsbuild.org/v2.11/docs/troubleshooting#pants-cannot-find-a-file-in-your-project.
23:06:24.41 [INFO] Starting: Building 98 requirements for requirements.pex from the python-default.lock resolve: Flask-SimpleLDAP==1.4.0, Flask-SocketIO==5.0.1, Flask==1.1.2, PyVISA-py>=0.3.1, PyVISA>=1.8, SQLAlchemy<2,>=1.4.31, ... (1881 characters truncated)
23:06:34.77 [INFO] Completed: Building 98 requirements for requirements.pex from the python-default.lock resolve: Flask-SimpleLDAP==1.4.0, Flask-SocketIO==5.0.1, Flask==1.1.2, PyVISA-py>=0.3.1, PyVISA>=1.8, SQLAlchemy<2,>=1.4.31, ... (1881 characters truncated)
23:06:34.77 [ERROR] 1 Exception encountered:
Engine traceback:
  in select
  in pants.core.goals.export.export
  in pants.backend.python.goals.export.export_virtualenvs
  in pants.backend.python.goals.export.export_virtualenv (python-default)
  in pants.backend.python.util_rules.pex.create_pex
  in pants.backend.python.util_rules.pex.build_pex (requirements.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/buildbot/.cache/pants/setup/bootstrap-Linux-x86_64/2.11.0_py38/lib/python3.8/site-packages/pants/engine/process.py", line 272, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building 98 requirements for requirements.pex from the python-default.lock resolve: Flask-SimpleLDAP==1.4.0, Flask-SocketIO==5.0.1, Flask==1.1.2, PyVISA-py>=0.3.1, PyVISA>=1.8, SQLAlchemy<2,>=1.4.31, aiotask_context, ansible-runner==2.0.3, ansible<3,>=2.10, asciitree>=0.3.3, asn1<3,>=2.5, astropy<6.0,>=5.0, asyncssh<2.9,>=2.4.2, awscli==1.19.0, bcrypt>=3.1.3, boto3-stubs<2,>=1.21.0, boto3-stubs[ecr,s3,sts], boto3==1.17.0, botocore<2,>=1.20.0, click-option-group>=0.5.2, click<8.0,>=7.1.2, cloud_sptheme==1.10.1.post20200504175005, coloredlogs>=6.0, conscript<1,>=0.1.3, cookiecutter, cryptography<37, docker==5.0.0, dohq-artifactory<1,>=0.8, elasticsearch<8,>=7, ephem>=3.7.6.0, eventlet==0.30.2, fire>=0.1.2, flake8-docstrings>=1.3.1, flake8-quotes, flake8>=3.5.0, flatdict>=4.0.1, freezegun>=1.1.0, google-api-python-client>=1.6.5, grafana-api>=1.0.3, graphviz>=0.8.4, gspread>=3.0.1, influxdb<6,>=5.2.0, ipython<8,>=7.20, jinja2<4.0,>=3.0, marshmallow==3.13.0, matplotlib>=0.0.1, mypy-protobuf==2.4, numpy<1.22.0,>=1.20.1, oauth2client>=4.1.2, openpyxl>=3.0.9, pandas<=1.3.5,>=1.0, paramiko<3.0,>=2.0, pexpect>=4.4.0, phabricator==0.8.1, pip-tools<7,>=6.5, portpicker<2,>=1.2.0, prompt-toolkit<4,>=2, protobuf==3.19.1, psutil>=5.8.0, psycopg2-binary==2.8.6, pyModbusTCP>=0.1.8, pyOpenSSL==20.0.1, pyftpdlib>=1.5.5, pyparsing<3, pyserial-asyncio>=0.4, pyserial>=3.4, pysofa2, pytest-cov!=2.12.1,<3.1,>=2.12, pytest-timeout<3,>=2.1.0, pytest-xdist<2, pytest<8,>=7, python-can>=3.1.1, python-dateutil<3,>=2.8.0, pytz>=2017.3, pyusb==1.2.1, pyyaml>=5.1, pyzmq<20,>=19.0.2, recommonmark, requests<3,>=2.27, rtl_tools@ git+ssh://[email protected]/source/rtl_tools.git@ea9d0ea6187058b50401b20f03b91365382f1552, scipy>=1.1.0, setuptools<62,>=61.2.0, sgp4<3,>=2, skyfield<2,>=1.42, slackclient>=2.7.2, sortedcontainers==2.4.0, sphinx-click<4,>=3.0.2, sphinx-rtd-theme<2,>=1.0, sphinx<5,>=4.0, tabulate>=0.8.1, testing.postgresql==1.3.0, typeguard==2.12.1, types-PyYAML==6.0.0, types-paramiko<3,>=2.8.13, typing-extensions<5,>=4.1.1, urllib3==1.26, yapf==0.20, yubihsm<3,>=2.1' failed with exit code 1.
stdout:
stderr:
  ERROR: Command errored out with exit status 128:
   command: git clone -q 'ssh://****@git.corp.astranis.com/source/rtl_tools.git' /tmp/process-executionCmXL5V/.tmp/pip-download-r2fh0e1a/rtl-tools
       cwd: None
  Complete output (5 lines):
  ****@git.corp.astranis.com: Permission denied (publickey,keyboard-interactive).
  fatal: Could not read from remote repository.
  
  Please make sure you have the correct access rights
  and the repository exists.
  ----------------------------------------
ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@git.corp.astranis.com/source/rtl_tools.git' /tmp/process-executionCmXL5V/.tmp/pip-download-r2fh0e1a/rtl-tools Check the logs for full command output.
pid 416692 -> /home/buildbot/.cache/pants/named_caches/pex_root/venvs/bd96d2df9aba88d86726507c234424e75d7b5960/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --use-deprecated legacy-resolver --isolated -q --cache-dir /home/buildbot/.cache/pants/named_caches/pex_root download --dest /tmp/process-executionCmXL5V/.tmp/tmp6mvg0ql4/usr.bin.python3.8 --no-binary :all: --no-deps rtl-tools @ git+ssh://****@git.corp.astranis.com/source/rtl_tools.git@ea9d0ea6187058b50401b20f03b91365382f1552 --index-url https://pypi.org/simple --retries 5 --timeout 15 exited with 1 and STDERR:
None
Use `--no-process-cleanup` to preserve process chroots for inspection.

[stuhood]

Sorry for the incredible delay in responding to this.

It appears that you have different authorization methods configured on the various machines: since Pants runs most processes in sandboxes rather than in the foreground, you'll need to ensure that your authorization method allows for keyboardless/non-interactive authentication.

I expect that you found a workaround or solution for this, but if not, please feel free to re-open after having tried a 2.12.x (latest stable) release.

[engnatha]

My workaround was to not upgrade pants :)

Turns out this issue is actually to be sorted out in the pex tooling. See the linked issue above. Good to keep this closed, though.