No module 'future' on emission environment.

[lucazpinheiro]

I'm part of a team responsible for install the e-mission server, so we can test it and analyze it for possible use on future projects.

Some info on the system used:
OS version
Linux Mint 19.2 (Tina) based on Ubuntu 18.04

Log conda info

    active env location : /home/lucas/anaconda3
            shell level : 1
       user config file : /home/lucas/.condarc
 populated config files : 
          conda version : 4.8.3
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages : __glibc=2.27
       base environment : /home/lucas/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/lucas/anaconda3/pkgs
                          /home/lucas/.conda/pkgs
       envs directories : /home/lucas/anaconda3/envs
                          /home/lucas/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.3 requests/2.23.0 CPython/3.7.6 Linux/4.15.0-54-generic linuxmint/19.2 glibc/2.27
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

We are trying to install the e-mission server through the manual installation following the guide/tutorial present on the project docs (https://github.com/e-mission/e-mission-docs/blob/master/docs/install/manual_install.md).
The process of installation was been carried on without problems until the last step when the server must be started with the command:
'./e-mission-py.bash emission/net/api/cfc_webapp.py'
By running this command we got the following message:

Traceback (most recent call last):
  File "emission/net/api/cfc_webapp.py", line 6, in <module>
    from future import standard_library
ImportError: No module named future

We make sure to check if we ware in the right conda environment by running the conda env list command and by having (emission) on the begging of our prompt

We tried to install the 'future' module manually, with 'conda install future' however, even after the installation was done, the 'future' module continuing missing.

The process of installation of the server was carried with both, root user and not root user, still, we got the same problem in both cases.

We have searched for similar problems and found three reported issues that looked related. (#327, #485 and #314) seem related. However, only issues #327 and #485 seems to have a close relation to our problem.

[shankari]

I'm part of a team responsible for install the e-mission server, so we can test it and analyze it for possible use on future projects.

That's great! If/when you use it, could you please add your experience with installation/customization/server resources, etc to #505?

[shankari]

Now to your actual issue. In general, every case of No module named future that I have seen has involved not being in the correct environment.

Can we try some additional troubleshooting steps?

• just so that I can double-check what you said, can you copy and paste the prompt, input and output for the following commands?

conda env list
./e-mission-py.bash emission/net/api/cfc_webapp.py

• What happens if you use the raw command instead?

PYTHONPATH=. python emission/net/api/cfc_webapp.py

• Are you sure your conda environment installed? There is a known issue with more recent versions of conda generating a PackageNotFound (Conda reporting ResolvePackageNotFound on source installation #502)

• Finally, have you tried the docker option? Because of the multiple regressions with newer conda versions (Conda reporting ResolvePackageNotFound on source installation #502, conda installation broken again with error installing cachetools-2.1.0-py_0 #511), I created a docker environment that allows you to mount a directory on your computer as a volume and run the server inside the container.
  https://github.com/e-mission/e-mission-docker/tree/master/examples/em-server#docker-composedevlivereloadyml-deploy-code-for-editing

• If the docker option works but you still want to use the manual install, make sure to use version 4.5.12 of miniconda. You can install it using the steps from the CI
  https://github.com/e-mission/e-mission-server/blob/master/.travis.yml#L7

  - wget https://repo.continuum.io/miniconda/Miniconda3-4.5.12-Linux-x86_64.sh -O miniconda.sh;
  - bash miniconda.sh -b -p $HOME/miniconda
  - source "$HOME/miniconda/etc/profile.d/conda.sh"
  - hash -r
  - conda config --set always_yes yes --set changeps1 no
  # Useful for debugging any issues with conda
  - conda info -a

[shankari]

Please update the bug with what the issue was so others can benefit from your experience!

[shankari]

Also, for the record, I plan to upgrade to the latest conda, and the latest version of multiple packages as part of the giant restructuring (#506). Unfortunately, that plan is on a small hiatus as I figure out if e-mission can help with contact tracing for COVID-19. But I'll get back to it in a little bit!

[lefterav]

I had the same problem. The environment was correctly a python 3.6 environment I solved it by enabling the emission environment and running conda install future

[shankari]

@lucazpinheiro any update on how you were able to resolve this?

[lucazpinheiro]

I had the same problem. The environment was correctly a python 3.6 environment I solved it by enabling the emission environment and running conda install future

I tried that but got the following error:

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/lucas/anaconda3/envs/emission

  added / updated specs:
    - future


The following NEW packages will be INSTALLED:

  future             pkgs/main/linux-64::future-0.18.2-py36_0


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: failed

EnvironmentNotWritableError: The current user does not have write permissions to the target environment.
  environment location: /home/lucas/anaconda3/envs/emission
  uid: 1000
  gid: 1000

[shankari]

hm, so the problem appears to be

EnvironmentNotWritableError: The current user does not have write permissions to the target environment.
  environment location: /home/lucas/anaconda3/envs/emission

Do you have write access to that location? Also, have you considered the docker approach?

There are two CI systems running for the server right now:

• .travis.yml, which uses a non-docker install
• .github/workflows/test-with-docker.yml which uses a docker install

Both of them pass (https://travis-ci.org/shankari/e-mission-server) and (https://github.com/e-mission/e-mission-server/actions).

You may want to follow those setup scripts exactly to setup your environment. Since they work in the CI environments (ubuntu:latest), they should work on your setup as well.

[lucazpinheiro]

hm, so the problem appears to be

EnvironmentNotWritableError: The current user does not have write permissions to the target environment.
  environment location: /home/lucas/anaconda3/envs/emission

Do you have write access to that location? Also, have you considered the docker approach?

There are two CI systems running for the server right now:

* `.travis.yml`, which uses a non-docker install

* `.github/workflows/test-with-docker.yml` which uses a docker install

Both of them pass (https://travis-ci.org/shankari/e-mission-server) and (https://github.com/e-mission/e-mission-server/actions).

You may want to follow those setup scripts exactly to setup your environment. Since they work in the CI environments (ubuntu:latest), they should work on your setup as well.

Sorry for taking to long so reply. I'm trying the steps you propose on the second answer.

When running PYTHONPATH=. python emission/net/api/cfc_webapp.py I got this:

Traceback (most recent call last):
  File "emission/net/api/cfc_webapp.py", line 6, in <module>
    from future import standard_library
ModuleNotFoundError: No module named 'future'

When I run conda install future as root, the module is installed, however, when I tried to start the it complains about another module:

Traceback (most recent call last):
  File "emission/net/api/cfc_webapp.py", line 23, in <module>
    import arrow
ModuleNotFoundError: No module named 'arrow'

I have tried to install the modules manually, I installed the other four modules util it ask for one that the conda repository didn't find

[shankari]

@lucazpinheiro can you also put in the prompt - e.g. something like

(emission) e-mission-server shankari$ ./e-mission-py.bash emission/net/api/cfc_webapp.py
storage not configured, falling back to sample, default configuration
Connecting to database URL localhost
analysis.debug.conf.json not configured, falling back to sample, default configuration
...

That helps me verify what is going on wrt the way in which the script is invoked.

[lucazpinheiro]

./e-mission-py.bash emission/net/api/cfc_webapp.py prompt

After install future manually as root

Traceback (most recent call last):
  File "emission/net/api/cfc_webapp.py", line 23, in <module>
    import arrow
ModuleNotFoundError: No module named 'arrow'

[shankari]

so it looks like there are multiple packages missing. I suspect this is the issue with the newer version of conda. Try docker, or install the older version of miniconda as described in the CI

[shankari]

@lucazpinheiro @lefterav I am thinking that maybe I will put the conda setup instructions from the automated build also into the setup.sh script, so it is more self-contained.

• Do you think this is a good idea?
• Should we focus on docker-based approaches instead?
• If we do decide to do this, will you be able to test it out and confirm that the instructions make sense?

[lucazpinheiro]

@shankari I installed miniconda version 4.5.12 as you suggest and it worked, thank you for the help. Should I close the issue now? Or there is some info that I could post to help people with a similar problem?

[lucazpinheiro]

@lucazpinheiro @lefterav I am thinking that maybe I will put the conda setup instructions from the automated build also into the setup.sh script, so it is more self-contained.

* Do you think this is a good idea?

* Should we focus on docker-based approaches instead?

* If we do decide to do this, will you be able to test it out and confirm that the instructions make sense?

Just to check if I understand it correctly, you are proposing to automate the anaconda setup process? If that is the case, yes, I could test, but I'm not very experienced with server mangemente, so maybe it won't be very helpful.

[shankari]

Or there is some info that I could post to help people with a similar problem?

@lucazpinheiro probably the most useful thing to do is to submit a PR to the server README with the instructions to install the correct version of miniconda.

If you could also test out the modified setup.sh script once I have it ready, that would be great.
Also, just curious, is there a reason why you did not want to try the docker build?

[lucazpinheiro]

Or there is some info that I could post to help people with a similar problem?

@lucazpinheiro probably the most useful thing to do is to submit a PR to the server README with the instructions to install the correct version of miniconda.

If you could also test out the modified setup.sh script once I have it ready, that would be great.
Also, just curious, is there a reason why you did not want to try the docker build?

I have tried the docker version, but the decision to go with the server manually came from a higher power, aka my boss. He wants to have more control over which feature of the project we would use.

About the PR... the change on the README would be made on my fork and then I would submit the PR? Also, let me know when you have made changes to the setup script and I will test.

[shankari]

the change on the README would be made on my fork and then I would submit the PR?

yup, https://en.wikipedia.org/wiki/Fork_and_pull_model

let me know when you have made changes to the setup script and I will test.

Will do.

[shankari]

@lucazpinheiro I added a script to install the correct version of conda, and I added a check to the setup scripts which asks you to run it if the version does not match.
e-mission/e-mission-server#739

Once you confirm that this makes sense, please close this issue.

[lucazpinheiro]

Tested it, but seems to still have a problem

ran the setup script for conda source setup/setup_conda.sh and got this:

user site dirs: 

CIO_TEST: <not set>
CONDA_ROOT: /home/usr/miniconda
CONDA_SHLVL: 0
DEFAULTS_PATH: /usr/share/gconf/cinnamon.default.path
INFOPATH: /home/linuxbrew/.linuxbrew/share/info:/home/linuxbrew/.linuxbrew/share/info
MANDATORY_PATH: /usr/share/gconf/cinnamon.mandatory.path
MANPATH: /home/usr/.nvm/versions/node/v10.18.1/share/man:/home/linuxbrew/.linuxbrew/share/man:/home/linuxbrew/.linuxbrew/share/man::
PATH: /home/usr/miniconda3/bin:/home/usr/.nvm/versions/node/v10.18.1/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/usr/Documents/Android/Sdk/tools:/home/usr/Documents/Android/Sdk/platform-tools
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>
XDG_SEAT_PATH: /org/freedesktop/DisplayManager/Seat0
XDG_SESSION_PATH: /org/freedesktop/DisplayManager/Session0


WARNING: could not import _license.show_info
# try:
# $ conda install -n root _license

Then ran the 'server setup' script source setup/setup.sh and got this:

Executing transaction: failed
ERROR conda.core.link:_execute(507): An error occurred while installing package 'conda-forge::cachetools-2.1.0-py_0'.
FileNotFoundError(2, "No such file or directory: '/home/usr/miniconda/envs/emission/bin/python3.6'")
Attempting to roll back.

Rolling back transaction: done

FileNotFoundError(2, "No such file or directory: '/home/usr/miniconda/envs/emission/bin/python3.6'")

Last ran the 'starter script' `./e-mission-py.bash emission/net/api/cfc_webapp.py``and got this:

Traceback (most recent call last):
  File "emission/net/api/cfc_webapp.py", line 6, in <module>
    from future import standard_library
ModuleNotFoundError: No module named 'future'

[shankari]

ah so you don't have the whole output, but the problem is that earlier in source setup/setup.sh there is an error

-bash: activate: No such file or directory

It's kind of buried because the script continues execution, but it's there.
And it's there because of line 12: source activate emission

It turns out that my own local version of anaconda was even older, and it only supported source activate, not conda activate. I had handled this properly in setup_test.sh that is used in the CI, but not in setup.sh.

I've now used the script on my own laptop (which meant that I had to handle Mac as well) and the setup works.

Couple of questions:

• is the setup script actually useful? Maybe it is easier to just tell people the version of miniconda and ask them to install manually instead?
• in order to avoid errors (like the source activate) getting lost, I can use set -e to error out early. But then if there is an error then the shell will just exit (since source runs the commands in the current shell). So you will know for sure that something is wrong. But you will need to run with bash XXX instead of source XXX to see what the error is. Will this be useful or just more confusing?

[lucazpinheiro]

* is the setup script actually useful? Maybe it is easier to just tell people the version of miniconda and ask them to install manually instead?

I think that an explanation about the right version of anaconda/miniconda would be better. Since that, the docker version already have everything automated, I don't see a problem in having to manually install anaconda/miniconda on the manually installed version of the server.

* in order to avoid errors (like the `source activate`) getting lost, I can use `set -e` to error out early. But then if there **is** an error then the shell will just exit (since `source` runs the commands in the current shell). So you will know for sure that something is wrong. But you will need to run with `bash XXX` instead of `source XXX` to see what the error is. Will this be useful or just more confusing?

And that thing about the set -e would probably make more hard to debug the problem.

[shankari]

@lucazpinheiro thanks for the feedback. Since I spent some time creating the setup_conda script, and it works now (https://github.com/e-mission/e-mission-server/actions?query=workflow%3Aosx-ubuntu-manual-install) , I will leave it in. But if it begins to be too much work keeping it maintained, then I may remove it in the future.

Also, can you pull from master and try again? As you can see from the passing CI, I believe everything should work now.

[lucazpinheiro]

There is probably a little error on the script, I ran it and got this:

Installing for platform 
Usage: setup_conda.sh <platform>
   Platform options are Linux-x86_64, MacOSX-x86_64
   For Windows, manually download and install https://repo.anaconda.com/miniconda/Miniconda3-4.5.12-Windows-x86_64.exe

I'm using linux mint 19.2 tina (based on Ubuntu 18.04 LTS) by the way.
Not sure if this was cause due to a little error the script.

[shankari]

@lucazpinheiro The script is asking you to specify the platform for which you want to install conda (linux or mac).
Since you are on linux mint, you would need to do

$ source setup_conda.sh Linux-x86_64

[lucazpinheiro]

Alright, ran the script and this is what happened:

Running source setup/setup_conda.sh Linux-x86_64
I got this at the end of the output:

WARNING: could not import _license.show_info
# try:
# $ conda install -n root _license

If I ran conda install -n root _license I get this at the end of the output:

Traceback (most recent call last):
  File "/home/lucas/miniconda/bin/conda", line 7, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'

And when I tried to start the server the no future module error pops up

Traceback (most recent call last):
  File "emission/net/api/cfc_webapp.py", line 6, in <module>
    from future import standard_library
ImportError: No module named future

If I didn't run conda install -n root _license and proceed to run setup.sh I can start the server without problems. However, if I close the terminal and come back, to try to activate emission environment again my system don't recognize de conda command or even source.

conda activate

conda: command not found

source activate

bash: activate: No such file or directory

Do you think the problem may be with some env variable on my system?

[shankari]

Couple of notes:

WARNING: could not import _license.show_info

the could not import _license.show_info is just a warning. As you can see from the CI, this shows up there too and it doesn't affect anything (https://github.com/e-mission/e-mission-server/runs/593958429?check_suite_focus=true)

Do you think the problem may be with some env variable on my system?

as you can see from the CI spec (https://github.com/e-mission/e-mission-server/blob/master/.github/workflows/osx-ubuntu-manual-install.yml) you have to run source "$HOME/miniconda/etc/profile.d/conda.sh" in every new terminal. Since conda was not installed manually, it didn't modify the bashrc with the correct path.

What I really want to do is to tell people to look at the CI and follow the steps there and look at the outputs there to see what is expected and what is not.

Any thoughts on how best to reflect that in the documentation? Or is this too confusing and I should just tell people to install the correct version manually?

[lucazpinheiro]

Now it works perfect, it would be very helpful to put the explanation about the thing of running source "$HOME/miniconda/etc/profile.d/conda.sh" in every new terminal on the docs.

I just have a question on how the script would work if there was an anaconda/miniconda version already installed?

I think the issue can be closed now, let me know if you agree or need something else to be tested. Once again, thanks for the help.

[shankari]

Now it works perfect,

@lucazpinheiro great to hear.

I just have a question on how the script would work if there was an anaconda/miniconda version already installed?

Great question! The user can choose a different install location and make sure to source the correct conda.sh in the terminal where they are running the server. That's why I don't install the version automatically, but rather, ask the user to install it.

it would be very helpful to put the explanation about the thing of running source "$HOME/miniconda/etc/profile.d/conda.sh" in every new terminal on the docs.

Do you think you could submit a PR with the instructions? Many times, I find it hard to write the documentation since the steps are obvious to me. It helps a lot to have somebody with fresh eyes write it instead.

You can edit the file directly in GitHub. Github will automatically create a fork + branch and allow you to submit the PR directly from the UI. It's just like editing a wiki!

[lucazpinheiro]

PR #520

English is not my first language, so probably there is some expressions wrong.

[shankari]

@lucazpinheiro your English is very good! Certainly vastly better than my Portuguese 😄 and even better than my Hindi 🙏 I didn't see any error with your expressions

Note also that if you want to use the e-mission phone app as well, the french team added i18n support, so it should be easy to add the Portuguese localization. If you do add this localization, please consider contributing it as well so that other groups can use it in the future.

[shankari]

Closing this issue now that the instructions are clear.