Issue with Shodan Analyzer

[treed593]

Request Type

Bug

Description

Describe your request as clearly as possible.

Steps to Reproduce

1)Run Cortex
2)Select Shodan_host ip analyzer
3)Provide IP
4)Run analyzer

Complementary information

{ "errorMessage": "Error: Invalid output\nTraceback (most recent call last):\n File \"./shodan_analyzer.py\", line 3, in <module>\n from cortexutils.analyzer import Analyzer\nImportError: No module named 'cortexutils'\n", "input": null, "success": false }

[saadkadhi]

@treed593 have you installed all the pre-requisites as described on https://github.com/CERT-BDF/CortexDocs/blob/master/installation/analyzers.md?

A lot of analyzers won't work since they require the cortexutils python lib.

[3c7]

Hey @treed593, looks like you don't have the analyzers properly installed. In order to install the required python modules you have to install them as root or as the user which should run them as follows:
pip install -r $(cat analyzers/*/requirements.txt | sort -u)
If you have multiple python versions installed, you may want to use
pip3 install -r $(cat analyzers/*/requirements.txt | sort -u)
too.

[treed593]

treed@ip-10-0-1-28:/opt/Cortex-Analyzers/analyzers$ sudo pip install $(sort -u */requirements.txt)
The directory '/home/treed/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/treed/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting git+https://github.com/AnyMaster/pehashng
  Cloning https://github.com/AnyMaster/pehashng to /tmp/pip-MlCjEg-build
  Requirement already satisfied (use --upgrade to upgrade): pehashng===1.0.1 from git+https://github.com/AnyMaster/pehashng in /usr/local/lib/python2.7/dist-packages
Collecting git+https://github.com/yeti-platform/pyeti
  Cloning https://github.com/yeti-platform/pyeti to /tmp/pip-_GuqZw-build
  Requirement already satisfied (use --upgrade to upgrade): pyeti===0.0.1 from git+https://github.com/yeti-platform/pyeti in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): abuse_finder in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): cortexutils in /home/treed/.local/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): datetime in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): ipaddress in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): nessrest in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): netaddr in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): olefile in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): oletools in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): passivetotal in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pefile2 in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): progressbar2 in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pydeep in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pyeupi in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pyexifinfo in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pymisp in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pypdns in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pypssl in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): python-magic in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pytz in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): requests in /home/treed/.local/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): shodan in /home/treed/.local/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): simplejson in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): yara-python in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pefile in /usr/local/lib/python2.7/dist-packages (from pehashng===1.0.1)
Requirement already satisfied (use --upgrade to upgrade): logging in /usr/local/lib/python2.7/dist-packages (from pyeti===0.0.1)
Requirement already satisfied (use --upgrade to upgrade): dnspython in /usr/local/lib/python2.7/dist-packages (from abuse_finder)
Requirement already satisfied (use --upgrade to upgrade): ipwhois in /usr/local/lib/python2.7/dist-packages (from abuse_finder)
Requirement already satisfied (use --upgrade to upgrade): pythonwhois in /usr/local/lib/python2.7/dist-packages (from abuse_finder)
Requirement already satisfied (use --upgrade to upgrade): tldextract in /usr/local/lib/python2.7/dist-packages (from abuse_finder)
Requirement already satisfied (use --upgrade to upgrade): zope.interface in /usr/local/lib/python2.7/dist-packages (from datetime)
Requirement already satisfied (use --upgrade to upgrade): ez_setup in /usr/local/lib/python2.7/dist-packages (from passivetotal)
Requirement already satisfied (use --upgrade to upgrade): future in /usr/local/lib/python2.7/dist-packages (from passivetotal)
Requirement already satisfied (use --upgrade to upgrade): python-utils>=2.1.0 in /usr/local/lib/python2.7/dist-packages (from progressbar2)
Requirement already satisfied (use --upgrade to upgrade): six in /usr/local/lib/python2.7/dist-packages (from pymisp)
Requirement already satisfied (use --upgrade to upgrade): jsonschema in /usr/local/lib/python2.7/dist-packages (from pymisp)
Requirement already satisfied (use --upgrade to upgrade): setuptools>=36.4 in /usr/local/lib/python2.7/dist-packages (from pymisp)
Requirement already satisfied (use --upgrade to upgrade): requests-cache in /usr/local/lib/python2.7/dist-packages (from pypdns)
Requirement already satisfied (use --upgrade to upgrade): urllib3<1.23,>=1.21.1 in /home/treed/.local/lib/python2.7/site-packages (from requests)
Requirement already satisfied (use --upgrade to upgrade): idna<2.7,>=2.5 in /home/treed/.local/lib/python2.7/site-packages (from requests)
Requirement already satisfied (use --upgrade to upgrade): chardet<3.1.0,>=3.0.2 in /home/treed/.local/lib/python2.7/site-packages (from requests)
Requirement already satisfied (use --upgrade to upgrade): certifi>=2017.4.17 in /home/treed/.local/lib/python2.7/site-packages (from requests)
Requirement already satisfied (use --upgrade to upgrade): XlsxWriter in /home/treed/.local/lib/python2.7/site-packages (from shodan)
Requirement already satisfied (use --upgrade to upgrade): click-plugins in /home/treed/.local/lib/python2.7/site-packages (from shodan)
Requirement already satisfied (use --upgrade to upgrade): colorama in /home/treed/.local/lib/python2.7/site-packages (from shodan)
Requirement already satisfied (use --upgrade to upgrade): click in /home/treed/.local/lib/python2.7/site-packages (from shodan)
Requirement already satisfied (use --upgrade to upgrade): ipaddr; python_version < "3.3" in /usr/local/lib/python2.7/dist-packages (from ipwhois->abuse_finder)
Requirement already satisfied (use --upgrade to upgrade): argparse in /usr/lib/python2.7 (from pythonwhois->abuse_finder)
Requirement already satisfied (use --upgrade to upgrade): requests-file>=1.4 in /usr/local/lib/python2.7/dist-packages (from tldextract->abuse_finder)
Requirement already satisfied (use --upgrade to upgrade): functools32; python_version == "2.7" in /usr/local/lib/python2.7/dist-packages (from jsonschema->pymisp)
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[treed593]

Just tried re-running that. All the other analyzers seem to be working.

[3c7]

You installed them for python2 while shodan analyzer uses python3 (-> the pip3 thing from above).

[treed593]

pip3 is not installed, that could be why. I will try that now, it should be added to https://github.com/CERT-BDF/CortexDocs/blob/master/installation/analyzers.md if it is required though

[treed593]

treed@ip-10-0-1-28:/opt/Cortex-Analyzers/analyzers$ sudo pip3 install $(sort -u */requirements.txt)
The directory '/home/treed/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/treed/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting git+https://github.com/AnyMaster/pehashng
Cloning https://github.com/AnyMaster/pehashng to /tmp/pip-xn5rtv7v-build
Collecting git+https://github.com/yeti-platform/pyeti
Cloning https://github.com/yeti-platform/pyeti to /tmp/pip-txosduj1-build
Collecting abuse_finder
Could not find a version that satisfies the requirement abuse_finder (from versions: )
No matching distribution found for abuse_finder
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Any ideas @3c7 or @saadkadhi

[nadouani]

You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Please upgrade your pip too

[treed593]

Done @nadouani, same issue
Collecting abuse_finder Could not find a version that satisfies the requirement abuse_finder (from versions: ) No matching distribution found for abuse_finder

[3c7]

Then just pip3 install shodan cortexutils.

[treed593]

That improved things, now I get
"errorMessage": "TLP is higher than allowed.",

When selecting 'Amber'

[treed593]

And
"errorMessage": "Unexpected Error: 'NoneType' object is not subscriptable",
With 'White'