Dependencies of Plugins and OutputStreams

[prescod]

Proposal:

snowutil pip install psycopg2

Implementation:

1. Within snowfakery:

sys.path.append("~/.snowfakery/site-packages")

2. In snowutil:

os.environ["PIP_TARGET"]="~/.snowfakery/site-packages"
import pip
pip.main(sys.argv[1:])

[davisagli]

What happens if someone installs a package that conflicts with another plugin, or worse, with one of Snowfakery's own dependencies?

Why a separate command?

Can we do this without making it a requirement to have pip installed at runtime for anything that depends on Snowfakery?

[prescod]

What happens if someone installs a package that conflicts with another plugin, or worse, with one of Snowfakery's own dependencies?

There is no universal solution to dependency hell but I expect that the overall complexity of Snowfakery environments (i.e. the set of recipes and their dependencies needed on any one user's computer) will be substantially simpler than full-blow-general-purpose software engineering environments for the foreseeable future.

Why a separate command?

snowfakery wasn't designed to be a nested command structure and I think it will just over-complicate to try and become one. Perhaps not the right decision even if I could go back in time, but even more so given the backwards-compatibility constraints. "snowutil" can be the meta-command and Snowfakery itself can stay focused. My rough notes abovve might have implied that pip would be literally everything that snowutil can do.

Can we do this without making it a requirement to have pip installed at runtime for anything that depends on Snowfakery?

I think so. It's not necessarily the case that every context that embeds Snowfakery needs to support recipes that depend on packages. We can decide on a case by case basis. By analogy, Jython is a Python context that does not support NumPy.

"You can use Oracle in Snowfakery or you can use Snowfakery from a UI directly inside of your Salesforce org, but you can't do both at the same time."

Sort of like how some Postgres extensions don't work in Heroku. It's like the Rolling Stones say...

[davisagli]

There is no universal solution to dependency hell

Sure. My point is just that we should warn users things could break, and tell them how to recover.