GitHub API rate limit

[achimnol]

After #116, I see now it performs retries while downloading PBS and other stuffs.
Unfortunately, there is another problem: we now hit the GitHub's API rate limit. 🫠

Would there be any better way to avoid it?

For instance,

• Could I set GITHUB_TOKEN to authenticate the fetch requests?
• How could I configure GitHub Actions cache to preserve the already fetched artifacts in previous workflow runs?

Example workflow run: https://github.com/lablup/backend.ai/actions/runs/12868611532

[jsirois]

For auth, you can do like so:

lift/.github/workflows/ci.yml

Line 62 in 2bd9199

SCIENCE_AUTH_API_GITHUB_COM_BEARER: ${{ secrets.GITHUB_TOKEN }}

Also see: https://github.com/pex-tool/pex/blob/027781802e73d68fb26a75f6e2b7d8b65e3b160b/.github/workflows/ci.yml#L10-L12

The whole auth scheme is defined here:

lift/science/fetcher.py

Lines 78 to 127 in 2bd9199

	def _configure_auth(url: Url) -> httpx.Auth | tuple[str, str] | None:
	if not url.info.hostname:
	return None
	normalized_hostname = url.info.hostname.upper().replace(".", "_").replace("-", "_")
	env_auth_prefix = f"SCIENCE_AUTH_{normalized_hostname}"
	env_auth = {key: value for key, value in os.environ.items() if key.startswith(env_auth_prefix)}
	def check_ambiguous_auth(auth_type: str) -> None:
	if env_auth:
	raise AmbiguousAuthError(
	f"{auth_type.capitalize()} auth was configured for {url} via env var but so was: "
	f"{", ".join(env_auth)}"
	)
	def get_username(auth_type: str) -> str | None:
	return env_auth.pop(f"{env_auth_prefix}_{auth_type.upper()}_USER", None)
	def require_password(auth_type: str) -> str:
	env_var = f"{env_auth_prefix}_{auth_type.upper()}_PASS"
	passwd = env_auth.pop(env_var, None)
	if not passwd:
	raise InvalidAuthError(
	f"{auth_type.capitalize()} auth requires a password be configured via the "
	f"{env_var} env var."
	)
	return passwd
	if bearer := env_auth.pop(f"{env_auth_prefix}_BEARER", None):
	check_ambiguous_auth("bearer")
	return "Authorization", f"Bearer {bearer}"
	if username := get_username("basic"):
	password = require_password("basic")
	check_ambiguous_auth("basic")
	return httpx.BasicAuth(username=username, password=password)
	if username := get_username("digest"):
	password = require_password("digest")
	check_ambiguous_auth("digest")
	return httpx.DigestAuth(username=username, password=password)
	try:
	return httpx.NetRCAuth(None)
	except (FileNotFoundError, IsADirectoryError):
	pass
	except NetrcParseError as e:
	logger.warning(f"Not using netrc for auth, netrc file is invalid: {e}")
	return None

So you could similarly define basic auth or use ~/.netrc.

For caching, I won't teach GitHub actions, but science allows you to control the cache with --cache-dir: https://science.scie.app/cli.html#science

So you can use SCIENCE_CACHE_DIR.

[jsirois]

@achimnol I've marked this as an answered question, but please confirm this answer works for you.

[achimnol]

Thanks for the detailed answer.
I'll check out with my team and leave the result.