Document Cloud limitations

[aaronsteers]

Task is to make sure all of these are documented.

As of now not confirmed for GA:

• Not supporting Airflow, Superset (due to state management and required ingress from public internet).
• Not supporting tap-google-analytics key_file_location (no 'secrets file' support in scope. recommendation: use other config inputs for auth).
• No "too old" Meltano versions.
• No custom Dockerfiles.

Confirmed for GA, but not in Alpha or Early Beta:

• No container_spec support.
• No private pip_url access. (Although may be possible to solve automatically by using an encrypted .env file.)

[WillDaSilva]

@aaronsteers I thought that Meltano permitted the use of env vars within pip_url in meltano.yml, but I just tried it and found that src/meltano/core/plugin/project_plugin.py::ProjectPlugin::formatted_pip_url only expands the MELTANO__PYTHON_VERSION env var.

Why don't we let any env var be expanded here? If that were possible, a user could easily add their private Python package index credentials to the encrypted .env file (or whatever approach we're using in the future for secret env vars), then make their pip_url something like this: https://${NEXUS_USERNAME}:${NEXUS_PASSWORD}@nexus.company.com/simple/package_name

Ignoring Meltano Cloud for the moment, this seems like a current issue with Meltano. To use a private Python package index currently within Meltano, doesn't one need to put their credentials in plaintext within meltano.yml? I suppose they could use the Python package keyring, which pip integrates with, or use a .netrc file (which also stores the credentials in plaintext, but at least it's out of meltano.yml and could be injected by whatever runner they're using), but those are both fairly niche & cumbersome.

[aaronsteers]

Why don't we let any env var be expanded here?

Not for any deliberate reason I'm aware of. We've been incrementally doing a bit of 'whack-a-mole' to expand where env vars are accepted. cc @cjohnhanson who has been making progress increasingly here and might be able to speak to relative level of effort to tackle pip_url values next.

Related topic:

• Document how to use alternative PyPI indexes meltano#3422
• From the GitLab original here: https://gitlab.com/meltano/meltano/-/issues/3506

One potential workaround would be to provide something like TAP_SNOWFLAKE__PIP_URL within the encrypted .env - which (I think?) might be able to override the pip url at runtime. But your proposed solution is probably better.

cc @edgarrmondragon who might know of other options or have additional context.

[edgarrmondragon]

I think the main reason we have not prioritized environment variables in the pip_url is because it's just a string of arguments for pip install, which can also be set with environment variables.

So

pip_url: '--only-binary=":all" --index-url https://pypi.org/simple my-package'

can be set with a simpler string

pip_url: 'my-package'

and env vars

PIP_ONLY_BINARY=":all:" \
PIP_INDEX_URL="https://pypi.org/simple" \
meltano install

Which is probably preferable in most situations.

[WillDaSilva]

@edgarrmondragon Using PIP_INDEX_URL is a good idea. I wonder why the comment suggesting it in the thread about env vars for pip install was heavily downvoted. I initially missed it.

By using per-plugin env configs we can provide different package indexes (with different credentials) for different plugins.

A downside to this approach is that it breaks our venv reuse strategy, which relies on the pip_url being enough to identify the contents of a virtual environment. That strategy is already flawed for a couple other reasons...

[tayloramurphy]

Archiving this repo since now the docs are in our main docs.