Environment variables for installs/ include external dependencies

[hamiltont]

Currently we replace ~/.bash_profile, ~/.profile, and ~/.bashrc with our own version (from config/benchmark_config), so that the PATH is properly configured to include everything that has been installed into installs/.

This stinks, for a few reasons:

• You can't just call --install-software and have the tests run properly after software is installed. You have to Ctrl-C, log out, and log back in, all to trigger bash to load up that new PATH variable
• It only works for some shells. If the default shell is zsh, for example, it doesn't source any of these files on remote interactive login.
• We are completely blowing away any prior files, and these are the kind of files you don't just erase without asking the user...many people customize them quite heavily

I think there's a pretty simple fix too::

1. Update config/benchmark_config to print it's environment to stdout when finished
2. Run config/benchmark_config and capture this environment
3. Replace the running environment with the one that you got from (2)
4. Now any subprocesses execute with a copy of our environment, which is super!

Step 3 might cause issues on some non-standard setups, so we could just swap out the environment bits we care about like PATH, LD_LIBRARY_PATH, etc. These are all non-destructive in config/benchmark_config.

I'll try to patch this up soon, just wanted to get the problem and fix out there for comment

[msmith-techempower]

I agree that this method of setting the environment up in this was is... bad (for lack of a better word). It was done this way for historical purposes -- essentially, we had built the suite with the notion that it would only ever be set up on a 'clean' environment (such as a new EC2 instance, or a local VM). So, we compromised on how we handled the environment setup.

This causes a number of problems as the suite got larger, but the most annoying of which was having multiple versions of the same application in the environment and framework overwriting the previous with "their" required version (I'm looking pretty squarely at you, nginx).

For the long-term, we are going to try and make our suite more modular by design with regard to installed software. That is, if a given framework requires nginx and perl be installed, then it would specify the tarball to download (or w/e) from their setup.py before start is called. The suite would check that their required software exists in the installs directory (and downloads/extracts/configures/etc in the case where it is not), and then runs. The benefit to this is that there will be no environment setup that could overwrite a user's currently configured environment and each test will be in charge of their own installation and setup (rather than having a monolithic installer script, each test would -- when run -- bootstrap their requirements).

I do not mind you issuing a pull request to fix this environment-smashing for the short term, but I wanted you to be aware of our efforts to resolve this in the future.

[msmith-techempower]

@hamiltont So, I'm going through my emails this morning, and I read this one and say to myself "man, this guy has put in a lot of effort to point out a problem that I have known about and just have not had the time to fix; I should respond and be really clear about where we want to go in the future."

So, I do. I write a big post about how we want to modularize all the installation software and make each framework describe their own software dependencies. I am content that I made my point that it's a "some day" task on our list, but I am little saddened that I (personally) simply do not have the time to work on it.

Then, I go back to reading my emails.

https://github.com/TechEmpower/FrameworkBenchmarks/pull/900/files

o.O

Okay... so, I'm going to kiss you. This is going to require a lot of testing on our end, but you seem to have done the lion's share of the work. Thank you!

[hamiltont]

Ha ha - yay! You guys seem pretty on the ball for these type of big PRs, so I'll keep em coming :-)

[hamiltont]

Ok, my strategy to fix this shall be called get your hands off my ~/.profile ;-)

See my branch for details. It's basically what I said above - I delete the current environment (except for some key values like HOME and PATH), source our own config/benchmark_config, and then replace our running environment with this new one that I just built.

The main benefit here is that any subprocess, or subprocess of a subprocess, or bash in python, etc, will have all of the proper environment variables, including a handy FWROOT variable (I am slowly removing hard-coded references to ~/FrameworkBenchmarks).

Note i'm leaving dependencies alone - this branch is all about ensuring the environment is consistent across different hosts and we are not interfering with users' private files.

The branch isn't done yet because a lot of the setup scripts (e.g. go/setup.py) manually set their environment variables, which is an unfortunate side effect of there not being a consistent environment when they were being added. So I'll have to dig through each one, remove any PATH/ENV/etc changes, and ensure they still work ok.

[msmith-techempower]

I think that I would rather that each test set the environment variables upon which they rely manually, as go does. This keeps the user's environment clean and keeps problems (like go has) from occurring where one test of LanguageA requires a particular variable to be set, but another test on LanguageA requires it to be different.

[hamiltont]

@msmith-techempower the commit I just added is a proof-of-concept for what you're saying. It allows each framework to have it's own install directory and environment, all of which are isolated from the other frameworks. I've kept the installation directory strategy (unified or per-framework) inside of python, so if people don't feel like installing things separately for each test we can have a flag for choosing the current strategy of using one big unified directory

[hamiltont]

Closing in favor of #915