ImportError: No module named 'gi._gi' while runnint setup.py on linux mint

[jima80525]

I'm trying to walk through the tutorial. Here's the bash history:
2001 mkdir tutorial
2002 cd tutorial/
2003 python3.5 -m venv venv
2004 . venv/bin/activate
2005 pip install briefcase
2006 apt-get install python3-gi gir1.2-webkit2-3.0
2007 sudo apt-get install python3-gi gir1.2-webkit2-3.0
2008 pip install --upgrade pip
2009 cookiecutter https://github.com/pybee/briefcase-template
2010 cd helloworld/
2011 vir helloworld/app.py
2012 python setup.py linux -s

All commands were successful until the final one. setup produced a cascade of errors, the first one being:

Installation complete.
Starting Hello World
Creating symlink (/home/jima/coding/tutorial/helloworld/linux/app_packages/gi & /home/jima/coding/tutorial/helloworld/linux/app_packages/pygtkcompat) to system GTK+ libraries...
Creating symlink (/home/jima/coding/tutorial/helloworld/linux/app_packages/gi & /home/jima/coding/tutorial/helloworld/linux/app_packages/pygtkcompat) to system GTK+ libraries...
Traceback (most recent call last):
File "/home/jima/coding/tutorial/helloworld/linux/app_packages/toga_gtk/app.py", line 7, in
import gi
File "/home/jima/coding/tutorial/helloworld/linux/app_packages/gi/init.py", line 36, in
from ._gi import _gobject
ImportError: No module named 'gi._gi'

It indicates that other exceptions occurred during the handling of that one, but I suspect those are not relevant.

I initially attempted this with Py3.4 which failed. I saw in a different issue that I would likely be better off with 3.5. I built and installed 3.5 and that's what's running in the VM. I also had used mkproject instead of creating the venv manually as shown in the example. I backed that out and started a new project doing it the manual way (as shown above).

Looking at the /helloworld/linux/app_packages/gi/init.py file, there is an import there
from ._gi import _gobject
There is not a _gi directory in the gi dir. There are two files which might be related:
_gi_cairo.cpython-34m-i386-linux-gnu.so
_gi.cpython-34m-i386-linux-gnu.so

I'm suspecting that there's some flotsam left over from either the 3.4 attempt or the mkproject attempt that is causing me problems, but I'm not seeing how those would produce the linux directory (in a new, clean subdir) without the required module.

NOTE: just to be complete. I went back and redid the steps after I manually removed the ~/.cookiecutters/ briefcase-template and Python-Linux-template directories. Same result.

Thanks for any help!

[freakboy3742]

Thanks for the report!

It's likely the issue is an incompatibility with Linux Mint - that's not a distro we test extensively.

As a check - can you run the project outside of briefcase? That is, can you run python -m helloworld? If that fails, then it indicates a problem with Mint support in general; if it works, then the issue is with the briefcase template under Mint.

[jima80525]

Thank you for the quick response. Odd results of that experiment.

---

(venv) 06:54:07 jima ~/coding/tut3/helloworld $ python -m helloworld
Traceback (most recent call last):
File "/usr/local/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"main", mod_spec)
File "/usr/local/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/jima/coding/tut3/helloworld/helloworld/main.py", line 1, in
from helloworld.app import main
File "/home/jima/coding/tut3/helloworld/helloworld/app.py", line 1, in
import toga
ImportError: No module named 'toga'
----------- end clip ------------------------------
Not at all what I expected.

I gave pip install toga a shot and got different results of the experiment afterwards:
---------------- begin clip ----------------------------
Creating symlink (/home/jima/coding/tut3/venv/lib/python3.5/site-packages/gi & /home/jima/coding/tut3/venv/lib/python3.5/site-packages/pygtkcompat) to system GTK+ libraries...
Traceback (most recent call last):
File "/home/jima/coding/tut3/helloworld/helloworld/main.py", line 1, in
from helloworld.app import main
ImportError: No module named 'helloworld'
---------------- end clip -------------------------

OK. That's odd. I edited the project's main.py file changing:
from helloworld.app import main
to
from .app import main
and THEN I was at the same point as originally described.

So, looks like a Mint support problem. I'm surprised. I didn't think Mint strayed too far from Ubuntu, but I'll admit I don't follow that all too closely.

I'm happy to dig in more if that's desired.

Thanks again for the time and attention, and for helping to make this project!

[freakboy3742]

Thanks for that detail. As you've surmised, it looks like it's a Mint support problem.

If you're interested in some more detailed debugging, the problem code is here.

The issue is that GTK requires some system libraries; but you can't install the Python bindings for GTK using pip install. So, we have to play a little trick, and symbolically link the Python libraries that are part of the main system Python install into the virtual environment, so that the end user can import them.

If you're outside a virtual environment, you should be able to run import gi. However, this will fail if you run it inside the virtual environment.

What toga_gtk/app.py does is attempt the import; and if it fails, it tries to create the symbolic links (because the source libraries are in a known location on any given operating system) to the virtual environment directory (which is known because we're running in that virtual environment).

Taking a closer look at the exact error message you originally reported, it might be possible that there's a dependency missing - the import of gi itself isn't failing, it looks like a missing SO - which suggests theres a binary library missing somewhere. If you can poke around why this import gi isn't working, that would be incredibly helpful.

[jima80525]

Alright. I poking around and then, finally, taking your earlier suggestion of running "import gi", I think I have a new theory. This is mint 17.3 which ships with py2.7. I had, a while back, built and installed py3.4. I ran into a problem, I think having to do with the cookiecutter template not having py3.4 branch, and so just built 3.5.

It looks like the path work in the app.py file is pulling from /usr/lib/python3. Running /usr/bin/python3 gets me the 3.4 version. Outside of the venv, if I run py3.5, import gi, it fails (no module). If I run 3.4, the import works fine. I'm beginning to suspect that my python installs are not clean.
Any pointers as to how to make sure I have a clean py3.5 install?

------ more details below - just in case you're curious and so I don't lose the info ------
the import gi is failing not because the two symlinks aren't in place, but rather (as you probably already knew) that the gi package isn't loading properly (failing to find an internal module). This is what starts the cascade of exceptions. The second exception in the chain is that the symlinks we're trying to create are already present.

The except block here is what raises the "unable to automatically create...." exception.

[freakboy3742]

Ah - that would make sense. If the "outside venv" run of import gi is failing, then that's the underlying problem - if that doesn't work outside a venv, it won't work inside, either. Effectively, you don't just need to have a version of Python 3.5; you need to have a system-native install of the python3-gi package, compiled for Python 3.5 and installed in the Python 3.5 system packages directory.

However, the easier fix may be to resolve the initial problem: the absence of a Linux template for Python 3.4. The Linux template isn't that complex, so I've just pushed a naïve port of the 3.5 template that should allow you to deploy on a system native Python 3.4. Let me know if that works for you.

The Linux template is really simple, and needs a lot of work - in particular, I'd like to start using Flatpak so that apps are easily redistributable. If you're at all interested in helping out, that would be a huge help.

[jima80525]

Awesome!
I will test out your fix AND see if I can help with the templates and Flatpak, but won't get to either one until Tuesday. I'll get back to you when I get some feedback.
THANK YOU!

[jima80525]

@freakboy3742 : Tested and confirmed that it now works for Python3.4. Thank you!

So, I am interested in helping out. I've looked at the
https://github.com/pybee/briefcase-template
repo, but my guess is that this is not the template you were talking about. Can you give me some pointers as to where to start looking?

Thanks again!

[freakboy3742]

@jima80525 Awesome - thanks for testing that!

The repository that needs work is this one. https://github.com/pybee/Python-Linux-template

This is a utility repository that is used by Briefcase when it rolls out a Linux project. The underlying feature request for Linux support is this one: #2

While a PR against the template repository and briefcase would be amazing, you don't even need to go that far. If you're able to take a simple app (like the tutorial) and provide me step-by-step instructions for turning it into a "flatpak app" - whatever that means - I can turn that documentation into a code patch.

That is, assuming I only have the code as described in the tutorial:

• What do I need to install?

• What directory structure do I need? Assume I'll create a "linux" directory to keep the flatpak isolated from everything else (e.g, the macOS build, the Windows build, the iOS build,...) - what structure is needed beyond that?

• What configuration files do I need?

• What commands to I need to run to "build" the flatpak?

• If you give me a flatpak built this way, how do I install it?

As much as possible, follow the conventions that other platforms use - an "app" directory where the application code is contained, and an "app_packages" directory where any requirements are installed. However, beyond that... just document the process.

Does that make sense?

[jima80525]

@freakboy3742 -
Sorry for the slow response time. I'm in the middle of switching jobs AND lost my hard drive. I seem to be back on track. I've got the flatpak tutorial working and am now in process of getting that working on the briefcase tutorial. Just wanted to let you know I'm still here and working, albeit quite slowly. :)

Thanks again for the help eariler.

[freakboy3742]

@jima80525 No worries at all - Any help you can provide, on whatever timeline you can provide it, is gratefully accepted!

[jima80525]

OK. I'll preface with this with a "I'm in over my head" claim. I've just started
with briefcase AND flatpak (AND python packaging, for that matter), so if I sound clueless.....

That caveat aside, here goes. To you direct questions:

What do I need to install?

Everything. It looks like all the way down to python itself, but that might be included in the runtime. I've got an example below (shamelessly stolen from a pygame-flatpak integration) that pulls in py3.6 and builds it. Again, I'm not sure if that's needed. Even if it's not, you'll at least need to pull in all of the packages that are pip-installed into the virtualenv. That pygame-flatpak project (https://github.com/takluyver/pygame-flatpak-test) has a nice example in the python3.6 json file for using pip as part of flatpak to pull in dependencies.

What directory structure do I need? 

Assume I'll create a "linux" directory to keep the flatpak isolated from everything else (e.g, the macOS build, the Windows build, the iOS build,...) - what structure is needed beyond that?
The directory structure appears to be up to you. Each module needs to be listed in a config file (in json) with a path relative to the config file. I think your
plan should be fine.

What configuration files do I need?

As far as I can tell, you'll need a single json file to configure flatpak. There are details in the flatpak page, below, and at the pygame page.

What commands to I need to run to "build" the flatpak?

To build it use
flatpak-builder app-dir org.flatpak.Hello.json

If you give me a flatpak built this way, how do I install it?

This may get a little tricky, as the user can specify which "repo" it should be
installed in. Not sure how you handle something like that in briefcase.
THe command to install it into a local repo named 'repo' is:
flatpak-builder --repo=repo --force-clean app-dir org.flatpak.Hello.json

---

Some more general notes:

There are at least three one-time steps users will need to take to get flatpak to work on their systems:

1. Install flatpak. Instructions are here: https://flatpak.org/getting.html
   Instructions vary by distro.

NOTE: despite flatpak being pre-installed on mint18.3, I still had to manually install
flatpak-builder via apt-get

2. Set up flatpak to point to a Freedesktop1.6 runtime
   flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

3. install that runtime
   flatpak install flathub org.freedesktop.Platform//1.6 org.freedesktop.Sdk//1.6

Once users have done that, they via briefcase, will need to create a json config file.
This is going to be the tricky part.

Two issues I see:

a) The config file points back to the runtime installed in step 2. If that is different the json needs to change
b) the modules section should list all packages. This includes all the packages that briefcase adds, and all the packages the project itself needs.
I'm not at all clear as to how this would be built. NOTE: It MIGHT also need to include python itself.

Here's the simplest example from the flatpak demo:

{
    "app-id": "org.flatpak.Hello",
    "runtime": "org.freedesktop.Platform",
    "runtime-version": "1.6",
    "sdk": "org.freedesktop.Sdk",
    "command": "hello.sh",
    "modules": [
        {
            "name": "hello",
            "buildsystem": "simple",
            "build-commands": [
                "install -D hello.sh /app/bin/hello.sh"
            ],
            "sources": [
                {
                    "type": "file",
                    "path": "hello.sh"
                }
            ]
        }
    ]
}

That looks simple enough. The exception is, as listed in point b above, there will
need to be a module for each requirement for the project, all the way down.

This starts to look like this:

{
    "app-id": "org.flatpak.HelloWorld",
    "runtime": "org.freedesktop.Platform",
    "runtime-version": "1.6",
    "sdk": "org.freedesktop.Sdk",
    "command": "python setup.py linux -s",
    "modules": [
        {
            "name": "helloworld",
            "sources": [
                { "type": "file", "path": "linux/app/helloworld/app.py" },
                { "type": "file", "path": "linux/app/helloworld/__init__.py" },
                { "type": "file", "path": "linux/app/helloworld/__main__.py" }
            ]
        },
        {
            "name": "helloworld-egg",
            "sources": [
                { "type": "file", "path": "linux/app/helloworld-0.0.1-py3.5.egg-info/dependency_links.txt" },
                { "type": "file", "path": "linux/app/helloworld-0.0.1-py3.5.egg-info/installed-files.txt" },
                { "type": "file", "path": "linux/app/helloworld-0.0.1-py3.5.egg-info/PKG-INFO" },
                { "type": "file", "path": "linux/app/helloworld-0.0.1-py3.5.egg-info/SOURCES.txt" },
                { "type": "file", "path": "linux/app/helloworld-0.0.1-py3.5.egg-info/top_level.txt" }
            ]
        }
        ... similarly for colosseum and solosseum-egg
        ... Not sure how to handle these in a flatpak world:
linux/app_packages/gi -> /usr/lib/python3/dist-packages/gi
linux/app_packages/pygtkcompat -> /usr/lib/python3/dist-packages/pygtkcompat

        ... NOTE: Must include all packages in the virtualenv
        
    ]
}

Also to note: python itself might need to be packaged. There's a project built around pygame which appears to do this.
https://github.com/takluyver/pygame-flatpak-test

The python3.6 version of the json file which pulls in python looks like this:

{
    "command": "/usr/bin/bash",
    "base": "org.pygame.BaseApp",
    "runtime": "org.freedesktop.Platform",
    "sdk": "org.freedesktop.Sdk",
    "runtime-version": "1.4",
    "cleanup": [ "/cache",
                 "/include",
                 "/lib/pkgconfig",
                 "/man",
                 "/share/aclocal",
                 "/share/devhelp",
                 "/share/gir-1.0",
                 "/share/gtk-doc",
                 "/share/man",
                 "/share/pkgconfig",
                 "/share/vala",
                 "/lib/systemd",
                 "*.la", "*.a" ],
    "build-options" : {
        "cflags": "-O2 -g",
        "cxxflags": "-O2 -g"
    },
    "modules": [
       {
          "name": "cpython",
          "sources": [
             {
                "type": "archive",
                "url": "https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz",
                "sha256": "b0c5f904f685e32d9232f7bdcbece9819a892929063b6e385414ad2dd6a23622"
             }
          ]
       },
       {
          "name": "pygame",
          "sources": [
             {
                "type": "archive",
                "url": "https://pypi.python.org/packages/61/06/3c25051549c252cc6fde01c8aeae90b96831370884504fe428a623316def/pygame-1.9.3.tar.gz",
                "sha256": "751021819bdc0cbe5cbd51904abb6ff9e9aee5b0e8955af02284d0e77d6c9ec2"
             },
             {
                "type": "file",
                "path": "python_configure.py",
                "dest-filename": "configure"
             },
             {
                "type": "patch",
                "path": "pygame-add-search-dirs.patch"
             }
          ],
          "build-options": {
             "env": {
                "PORTMIDI_INC_PORTTIME": "1",
                "PIP_PATH": "/usr/bin/pip3",
                "LOCALBASE": "/app"
             }
          }
       }
    ]
}

That's what I have for now. I appologize that this is not more of a solution, but, as I said, I'm
still early on the learning curve for both of these packages.

Hope this helps!

[freakboy3742]

This appears to be an issue with the toga installer, which has been addressed in the 0.3 branch of Toga.