Failed to import module elasticsearch #9012

kmfreder1 · 2022-10-27T12:44:10Z

kmfreder1
Oct 27, 2022

So, I have had a strange issue following me through the last few updates on one and only one of the multiple Security Onion installations that I manage.

This error began, I believe, in 2.3.90 and has followed me all the way to 2.3.181. I have not noticed any functionality issues while using Security Onion,

Network traffic is appearing.
All of the features seem to work as expected.
This is an on-prem deployment with internet access
This was originally was a network installation.
I believe we started on 2.3.60, but I could be mistaken about that.
This is an Ubuntu 18.04 standalone installation.
The so-status script shows all services are running.
The SOC Grid page shows no failures.

Here is the error...During every high state, I get the following error repeated 4 times:

#salt-call state.highstate
[ERROR   ] Failed to import module elasticsearch, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 766, in _load_module
    mod = self.run(spec.loader.load_module)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 823, in load_module
  File "<frozen importlib._bootstrap_external>", line 682, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/salt/modules/elasticsearch.py", line 59, in <module>
    import elasticsearch
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/__init__.py", line 36, in <module>
    from .client import Elasticsearch
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/__init__.py", line 23, in <module>
    from ..transport import Transport, TransportError
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 23, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/__init__.py", line 19, in <module>
    from .http_requests import RequestsHttpConnection
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/http_requests.py", line 35, in <module>
    _REQUESTS_META_VERSION = _client_meta_version(requests.__version__)
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/utils.py", line 27, in _client_meta_version
    r"^([0-9][0-9.]*[0-9]|[0-9])(.*)$", version
  File "/usr/lib/python3.6/re.py", line 172, in match
    return _compile(pattern, flags).match(string)
TypeError: expected string or bytes-like object

I have reviewed the only other discussion topic that I found with this error: #7869, however, the answer given in that case does not address our issue. We do not have any custom configurations related to elasticsearch and I compared elasticsearch.yml with another SO install and found no differences whatsoever. I also compared the global.sls and standalone.sls files to a known working SO installation.

Does anyone have any other ideas about where to look or what could be causing this error? Is there another config file besides elasticsearch.yml I should be reviewing? I am not sure what impact this error has exactly since I haven't noticed any operational deficiencies.

Any help would be much appreciated.

Answered by m0duspwnens

Nov 1, 2022

@kmfreder1 Can you run the following from the command line on a box that is seeing this issue?

python3 -c "import elasticsearch; print(elasticsearch.__version__)"

View full answer

m0duspwnens · 2022-10-28T14:39:55Z

m0duspwnens
Oct 28, 2022
Maintainer

@kmfreder1 If you run a highstate, does the highstate complete, but generate this message? If the state runs, are you able to see which state it is generating the message for? From the error it appears that it is likely a state within the Elasticsearch state.

1 reply

kmfreder1 Oct 28, 2022
Author

Hi @m0duspwnens. Thanks for your response! The highstate completes successfully. The error message comes up almost immediately after I run salt-call state.highstate repeating twice and then after a pause, it repeats again twice before any of the normal state stuff appears, then there is a longer pause and the normal state output scrolls by with this as the final result:

Summary for local
--------------
Succeeded: 672 (changed=33)
Failed:      0
--------------
Total states run:     672
Total run time:   146.808 s

So there are no errors shown during the normal state output (in green and blue).

m0duspwnens · 2022-10-28T15:19:40Z

m0duspwnens
Oct 28, 2022
Maintainer

Can you run salt-call state.highstate -l debug, log it to a file, and post it here with any sensitive information redacted?

3 replies

kmfreder1 Oct 28, 2022
Author

Sure! Please see the attached file.
schs-debug.txt

m0duspwnens Oct 28, 2022
Maintainer

Thanks for the log. Can you please provide the output from salt --versions?

kmfreder1 Oct 28, 2022
Author

# salt --versions
Salt Version:
          Salt: 3004.2

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: 2.5.1
         gitdb: 2.0.3
     gitpython: 2.1.8
        Jinja2: 2.10
       libgit2: Not Installed
      M2Crypto: 0.31.0
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: 1.3.10
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.4.7
        pygit2: Not Installed
        Python: 3.6.9 (default, Jun 29 2022, 11:45:57)
  python-gnupg: 0.4.1
        PyYAML: 3.12
         PyZMQ: 17.1.2
         smmap: 2.0.3
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.5

System Versions:
          dist: ubuntu 18.04 Bionic Beaver
        locale: UTF-8
       machine: x86_64
       release: 4.15.0-194-generic
        system: Linux
       version: Ubuntu 18.04 Bionic Beaver

m0duspwnens · 2022-11-01T14:48:30Z

m0duspwnens
Nov 1, 2022
Maintainer

@kmfreder1 Can you run the following from the command line on a box that is seeing this issue?

python3 -c "import elasticsearch; print(elasticsearch.__version__)"

3 replies

kmfreder1 Nov 1, 2022
Author

I ran that command and followed that by a salt-call state-highstate. Unfortunately, the errors persisted repeating 4 times.

# python3 -c "import elasticsearch; print(elasticsearch.__version__)"
(7, 12, 0)

# salt-call state.highstate
[ERROR   ] Failed to import module elasticsearch, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 766, in _load_module
    mod = self.run(spec.loader.load_module)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 823, in load_module
  File "<frozen importlib._bootstrap_external>", line 682, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/salt/modules/elasticsearch.py", line 59, in <module>
    import elasticsearch
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/__init__.py", line 36, in <module>
    from .client import Elasticsearch
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/__init__.py", line 23, in <module>
    from ..transport import Transport, TransportError
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 23, in <module>
    from .connection import Urllib3HttpConnection
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/__init__.py", line 19, in <module>
    from .http_requests import RequestsHttpConnection
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/http_requests.py", line 35, in <module>
    _REQUESTS_META_VERSION = _client_meta_version(requests.__version__)
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/utils.py", line 27, in _client_meta_version
    r"^([0-9][0-9.]*[0-9]|[0-9])(.*)$", version
  File "/usr/lib/python3.6/re.py", line 172, in match
    return _compile(pattern, flags).match(string)
TypeError: expected string or bytes-like object

m0duspwnens Nov 1, 2022
Maintainer

I was able to replicate this issue by installing the python3 elasticsearch module version 7.12.0. Since this module isn't installed by default, you can either remove it (assuming you aren't using it) or install the appropriate version.

kmfreder1 Nov 1, 2022
Author

That did the trick! Thank you so much for your help.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to import module elasticsearch #9012

{{title}}

Replies: 3 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Failed to import module elasticsearch #9012

kmfreder1 Oct 27, 2022

Replies: 3 comments · 7 replies

m0duspwnens Oct 28, 2022 Maintainer

kmfreder1 Oct 28, 2022 Author

m0duspwnens Oct 28, 2022 Maintainer

kmfreder1 Oct 28, 2022 Author

m0duspwnens Oct 28, 2022 Maintainer

kmfreder1 Oct 28, 2022 Author

m0duspwnens Nov 1, 2022 Maintainer

kmfreder1 Nov 1, 2022 Author

m0duspwnens Nov 1, 2022 Maintainer

kmfreder1 Nov 1, 2022 Author

kmfreder1
Oct 27, 2022

Replies: 3 comments 7 replies

m0duspwnens
Oct 28, 2022
Maintainer

kmfreder1 Oct 28, 2022
Author

m0duspwnens
Oct 28, 2022
Maintainer

kmfreder1 Oct 28, 2022
Author

m0duspwnens Oct 28, 2022
Maintainer

kmfreder1 Oct 28, 2022
Author

m0duspwnens
Nov 1, 2022
Maintainer

kmfreder1 Nov 1, 2022
Author

m0duspwnens Nov 1, 2022
Maintainer

kmfreder1 Nov 1, 2022
Author