vmware.vmware_rest.vcenter_vm_info Fails in Python 3.10.2 #308
Comments
|
Thank you for the bug report. |
|
I just tested locally (with kubernetes.core), and ansible-collections/cloud.common#102 appears to solve this problem. |
|
I am also running into this bug. Fedora 35, ansible 2.9.27 and Python 3.10.3. |
softwarefactory-project-zuul
bot
closed this as completed
in
ansible-collections/cloud.common#102
softwarefactory-project-zuul bot
pushed a commit
to ansible-collections/cloud.common
that referenced
this issue
Apr 11, 2022
Fix import logic trying to load incorrect module
SUMMARY
There is currently a bug in the import logic that manifests when two
Ansible modules try to load a python module from somewhere in
module_utils. If the python module shares the same name as the second
(or subsequent) Ansible modules, turbo mode will attempt to load the
python module instead of the Ansible module.
This fixes the logic by removing the meta_path modification, which was
the root of the problem. Instead, it keeps the sys.path modification as
before, but attempts to reload any ansible_collection package modules.
We can't reload every module as this would overwrite any shared state in
the module cache, defeating the whole purpose of turbo mode. By
reloading only the package modules we force it to reexamine its
contents, which should now be pointing to the new zip archive since we
changed the loader for the package module.
One caveat to this is that if shared state were being stored in a
package's __init__.py, it would be written over from the package reload.
This can be worked around by conditionally defining any shared state in
__init__.py with something like:
try:
state
except NameError:
state = {}
ISSUE TYPE
Bugfix Pull Request
COMPONENT NAME
ADDITIONAL INFORMATION
The underlying problem with the current implementation is that it adds a zipimporter to the meta_path for every loaded module in the ansible_collections namespace. A zipimporter only examines that last portion of the module fullname. This can be seen in a simple example. Assuming a zip archive that looks like:
foo.zip
|-- foo
|-- __init__.py
|-- a
| |-- __init__.py
| |-- c.py
|-- b
|-- __init__.py
|-- c.py
Now, create a zipimporter for this, pointing to a subpackage and see that it loads the wrong module:
loader = zipimporter("foo.zip/foo/a")
mod = loader.load_module("foo.b.c")
assert mod.__name__ == "foo.b.c" # True
assert mod.__file__ == "foo.zip/foo/a/c.py" # True
By adding zipimporters to the meta_path, when we try to load a module that isn't yet in the module cache, we add finders that will load the wrong module. This is a problem, for example, when you have a collection that looks like:
collection/plugins/modules/a.py
collection/plugins/modules/b.py
collection/plugins/module_utils/b.py
If modules a and b both import b from module_utils, a playbook that first runs module a and then runs module b will fail with an error that it can't find main(), because in the second task it has loaded module_utils/b.py instead of modules/b.py.
It's not enough to just change the __path__ on a package module that has already been imported. This is only examined the first time a package is loaded. When the package is loaded any modules inside __path__ are added as attributes to the package module. Subsequent attempts to access a new module in that package that isn't in the module cache will fail because it is only checking package's current attributes. We have to reload the package modules to be able to load any new modules that may now be in that package namespace. As stated above, this does mean any shared state held on the package (__init__.py) will be overwritten.
I can't see a perfect solution for this, but I think this change is the best one.
Closes: ansible-collections/vmware.vmware_rest#308
Reviewed-by: Gonéri Le Bouder <[email protected]>
Reviewed-by: None <None>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
SUMMARY
When using the vmware.vmware_rest.vcenter_vm_info module, the module fails with a zipimporter error.
ISSUE TYPE
COMPONENT NAME
vmware.vmware_rest.vcenter_vm_info
ANSIBLE VERSION
COLLECTION VERSION
CONFIGURATION
OS / ENVIRONMENT
Ubuntu 20.04.4 LTS running in WSL2
Python 3.10.2 in a pyenv/virtualenv virtual environment
STEPS TO REPRODUCE
Activate your python virtual environment running Python 3.10.2
Upgrade Pip
Install Ansible
Install aiohttp
Execute your playbook
EXPECTED RESULTS
This is the output when running the playbook in Python 3.9.10:
ACTUAL RESULTS
The text was updated successfully, but these errors were encountered: