Stop using pytest to run our hub health checks

[sgibson91]

Context

We currently use pytest to run our hub health checks. Upon failure, pytest's output includes the parameters that were passed to the function. For us, this includes a JupyerHub API token which is secret. In order to prevent this from being leaked via CI logs, we currently do not print output when we run in CI environments. However, this adds extra toil to the engineering team since they would need to rerun the hub health check locally in order to see logs and hope that the problem can be reproduced using the deployer locally.

Proposal

Since we don't actually use any functionality pytest offers, we should remove the pytest dependency and call the functions as normal functions. This will allow diagnostic logs to be available in CI/CD without leaking the secret API token or requiring an engineer to do another run of the checks locally.

Updates and actions

No response

Related

• Improve information about the cause of failure during hub health check #1206

[yuvipanda]

Fully support!

[sgibson91]

Issue is specifically referencing this section:

infrastructure/deployer/deployer.py

Lines 390 to 413 in 75119bf

	# On failure, pytest prints out params to the test that failed.
	# This can contain sensitive info - so we hide stderr
	# FIXME: Don't use pytest - just call a function instead
	#
	# Show errors locally but redirect on CI
	gh_ci = os.environ.get("CI", "false")
	pytest_args = [
	"-q",
	"deployer/tests",
	f"--hub-url={hub_url}",
	f"--api-token={service_api_token}",
	f"--hub-type={hub.spec['helm_chart']}",
	]
	if (hub.spec["helm_chart"] == "daskhub") and check_dask_scaling:
	pytest_args.append("--check-dask-scaling")
	if gh_ci == "true":
	print_colour("Testing on CI, not printing output")
	with open(os.devnull, "w") as dn, redirect_stderr(dn), redirect_stdout(dn):
	exit_code = pytest.main(pytest_args)
	else:
	print_colour("Testing locally, do not redirect output")
	exit_code = pytest.main(pytest_args)

Note that we do use pytest to verify the deployer code itself, which is fine. This is only regarding running the hub health check.

[consideRatio]

I think this issue should be worked before #1206, and that #1206 is suitable work following this is done.