Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Zvelo] Create a connector to download Zvelo intel #3121

Merged
merged 12 commits into from
Dec 18, 2024
Merged

Conversation

romain-filigran
Copy link
Member

@romain-filigran romain-filigran commented Dec 12, 2024

Proposed changes

  • First version of a connector for Zvelo Threat Intelligence Feeds

Related issues

Checklist

  • I consider the submitted work as finished
  • I tested the code for its functionality using different use cases
  • I added/update the relevant documentation (either on github or on notion)
  • Where necessary I refactored code to improve the overall quality

@romain-filigran romain-filigran added the filigran team use to identify PR from the Filigran team label Dec 12, 2024
@Powlinett Powlinett self-assigned this Dec 17, 2024
@Powlinett Powlinett linked an issue Dec 17, 2024 that may be closed by this pull request
Copy link
Member

@Powlinett Powlinett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @romain-filigran, please find my feedback after some tests:

⚠️ Two SyntaxError lead the connector to crash instantly

Once these errors are fixed:
✅ connector's workflow is clear
✅ README is crystal clear too
✅ no errors during ingestion
⁉️after some runs I receive responses with status code 400 Bad Request for 'phish' collection only (see logs below - idk if it's intended)

Note that not all my comments need to be addressed, only the ones with warnings (⚠️). The other ones are just to share knowledge and/or quick win improvements.

Last but not least, I didn't write a specific comment about docstrings but can you fill/write them all please? 😇 I think this connector could benefit from later refactos and this type of comments could really help us to navigate through the code. It would be even better if you could also add some basic typing 🤩

Many thanks!


400 Bad Request logs:

connector-zvelo-1  | {"timestamp": "2024-12-17T19:01:36.327422Z", "level": "ERROR", "name": "Zvelo", "message": "[API] Error while fetching data in collection 'phish': ", "exc_info": "Traceback (most recent call last):\n  File \"/opt/opencti-connector-zvelo/zvelo_connector/client_api.py\", line 78, in get_collections_entities\n    response.raise_for_status()\n  File \"/usr/local/lib/python3.12/site-packages/requests/models.py\", line 1024, in raise_for_status\n    raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.zvelo.io/v1/phish?page=0&created_date_start=2024-12-17T18%3A00%3A41.047713%2B00%3A00\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/opt/opencti-connector-zvelo/zvelo_connector/connector.py\", line 162, in process_message\n    stix_objects = self._collect_intelligence(from_date=last_run)\n                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/opencti-connector-zvelo/zvelo_connector/connector.py\", line 92, in _collect_intelligence\n    entities = self.client.get_collections_entities(\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/opt/opencti-connector-zvelo/zvelo_connector/client_api.py\", line 93, in get_collections_entities\n    raise Exception(error_msg)\nException: [API] Error while fetching data in collection 'phish': ", "taskName": null}

external-import/zvelo/docker-compose.yml Outdated Show resolved Hide resolved
external-import/zvelo/src/zvelo_connector/connector.py Outdated Show resolved Hide resolved
external-import/zvelo/src/zvelo_connector/connector.py Outdated Show resolved Hide resolved
external-import/zvelo/src/zvelo_connector/connector.py Outdated Show resolved Hide resolved
external-import/zvelo/src/zvelo_connector/connector.py Outdated Show resolved Hide resolved
external-import/zvelo/tests/test_zvelo_connector.py Outdated Show resolved Hide resolved
@Powlinett Powlinett added the do not merge Do not merge this PR until this tag will be removed label Dec 17, 2024
@Powlinett
Copy link
Member

400 - Bad request was due to bad date format on my local branch
✅ I found a STIX pattern ValidationError due to unescaped single quotes in IOC urls so I pushed a fix

The PR is ready for merge 🙌

@Powlinett Powlinett removed the do not merge Do not merge this PR until this tag will be removed label Dec 18, 2024
@Powlinett Powlinett merged commit 89781c1 into master Dec 18, 2024
4 checks passed
@Powlinett Powlinett deleted the issue/2591 branch December 18, 2024 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
filigran team use to identify PR from the Filigran team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Zvelo] Create a connector to download Zvelo intel
2 participants