[tenable_io] Initial Release for the Tenable.io #4816

vinit-chauhan · 2022-12-13T12:40:53Z

What does this PR do?

Generated the skeleton of the Tenable.io integration package.
Added data streams.
Added data collection logic for all the data streams.
Added the ingest pipeline for all the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added dashboards and visualizations.
Added test for pipeline for all the data streams.
Added system test cases for all the data streams.

Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency
when creating or updating a Package, Module or Dataset for an Integration.

All changes

Change follows the contributing guidelines
Supported versions of the monitoring target is documented
Supported operating systems are documented (if applicable)
Integration or System tests exist
Documentation exists
Fields follow ECS and naming conventions
At least a manual test with ES / Kibana / Agent has been performed.
Required Kibana version set to: ^8.6.0

New Package

Screenshot of the "Add Integration" page on Fleet added

Dashboards changes

Dashboards exists
Screenshots added or updated
Datastream filters added to visualizations

Log dataset changes

Pipeline tests exist (if applicable)
Generated output for at least 1 log file exists
Sample event (sample_event.json) exists

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/tenable_io directory.
Run the following command to run tests.

elastic-package test

Related issues

Closes Tenable.io #2338

Screenshots

elasticmachine · 2022-12-13T12:43:47Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2022-12-13T12:57:05Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-01-03T12:06:05.588+0000
Duration: 14 min 20 sec

Test stats 🧪

Test	Results
Failed	0
Passed	18
Skipped	0
Total	18

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-12-13T12:57:17Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`3/3`)	💚
Files	100.0% (`3/3`)	💚
Classes	100.0% (`3/3`)	💚
Methods	95.455% (`42/44`)	👎 -1.212
Lines	91.798% (`2104/2292`)	👎 -8.202
Conditionals	100.0% (`0/0`)	💚

jamiehynds · 2022-12-13T13:01:09Z

@vinit-elastic can we ensure the integration is shipping as Beta, not Tech Preview (as per the screenshots).

vinit-chauhan · 2022-12-14T08:48:20Z

Hey @jamiehynds, We have taken a screenshot of Tenable.io integration in stack version 8.6.0-SNAPSHOT, so that's why the Technical Preview tag may be there. Also, we have checked with the latest elastic stack version, 8.5.3, in which we are getting the beta tag. Here I am sharing the screenshot.

packages/tenable_io/data_stream/asset/elasticsearch/ingest_pipeline/default.yml

P1llus · 2022-12-27T13:43:36Z

packages/tenable_io/data_stream/asset/elasticsearch/ingest_pipeline/default.yml

+          return false;
+        }
+        dropEmptyFields(ctx);
+  - append:


This one is a nice usage of the new kind type 👍

P1llus · 2022-12-27T13:43:54Z

packages/tenable_io/data_stream/asset/elasticsearch/ingest_pipeline/default.yml

+  - append:
+      field: error.message
+      value: '{{{ _ingest.on_failure_message }}}'
+  - append:


Do we need it in multiple places?

We have added multiple on_failures on the potential failure points in the pipeline such as convert, date processors, etc.

@vinit-elastic what Marius meant was event.kind was already set to pipeline_error earlier in the pipeline.
Do we need to again append inside on_failure?

Yes, we have handled the scenario where the pipeline goes to fail due to an uncertain situation in processors(rename, script, etc.). So this situation will skip the first pipeline_error and the control goes to on_failure at last and the pipeline error will append as expected.

Makes sense! But how about the other way around? I mean is error.message populated even without any failures in the pipeline? If not, then we could remove the earlier processor and just let the on_failure set event.kind to pipeline_error.

The answer to your question is, no. We are not setting the error message in any other case than a pipeline error in this integration.
However, we have added on_failures in all the convert processors. And if that failure would be caught by the subsequent on_failure and not the on_failure on the root. Moreover, we do not want to add unnecessary processing by setting event.kind in all the on_failures. Hence, we have added one on the bottom to check if there are any error messages present and if so, append the event.kind with pipeline_error as the on_failure on the root won't run if the errors are caught by the on_failure associated with the processors.

Cool, makes sense! Thanks!

P1llus · 2022-12-27T13:44:32Z

packages/tenable_io/data_stream/asset/manifest.yml

+    description: Collect asset logs from Tenable.io.
+    template_path: httpjson.yml.hbs
+    vars:
+      - name: interval


This one seems to be missing proxy?

We have added a proxy at the package level. Here's the like to that line:

integrations/packages/tenable_io/manifest.yml

Line 51 in 64ca1af

- name: proxy_url

P1llus · 2022-12-27T13:48:25Z

packages/tenable_io/manifest.yml

+          - name: access_key
+            type: password
+            title: Access Key
+            description: Access key for the Tenable.io API.
+            multi: false
+            required: true
+            show_user: true
+          - name: secret_key
+            type: password
+            title: Secret Key
+            description: Secret key for the Tenable.io API.
+            multi: false
+            required: true
+            show_user: true
+          - name: proxy_url
+            type: text
+            title: Proxy URL
+            multi: false
+            required: false
+            show_user: false
+            description: URL to proxy connections in the form of http[s]://<user>:<password>@<server name/ip>:<port>. Please ensure your username and password are in URL encoded format.
+          - name: ssl
+            type: yaml
+            title: SSL Configuration
+            description: i.e. certificate_authorities, supported_protocols, verification_mode etc.
+            multi: false
+            required: false
+            show_user: false
+            default: |
+              #certificate_authorities:
+              #  - |
+              #    -----BEGIN CERTIFICATE-----
+              #    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
+              #    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
+              #    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
+              #    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
+              #    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
+              #    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
+              #    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
+              #    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
+              #    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
+              #    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
+              #    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
+              #    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
+              #    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
+              #    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
+              #    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
+              #    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
+              #    sxSmbIUfc2SGJGCJD4I=
+              #    -----END CERTIFICATE-----


Might be a slight nitpick, but there has been several occasions in which users are required to have different setups per type of datastream, so I am a bit hesitant to leave these at the top of the integration.
However we can leave them there for now, and modify it if necessary.

Hey @P1llus, The authorization parameter has been added at the package level. If we shift the SSL and proxy parameters at the data stream level, then we must shift the authorization parameter at the same level. Additionally, we are following the same pattern as all the previous connectors. However, If you think changes are required, we can change it.

kcreddy · 2023-01-02T11:22:42Z

packages/tenable_io/data_stream/asset/elasticsearch/ingest_pipeline/default.yml

+  - append:
+      field: error.message
+      value: '{{{ _ingest.on_failure_message }}}'
+  - append:


@vinit-elastic what Marius meant was event.kind was already set to pipeline_error earlier in the pipeline.
Do we need to again append inside on_failure?

packages/tenable_io/data_stream/asset/elasticsearch/ingest_pipeline/default.yml

packages/tenable_io/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml

infosecwatchman · 2023-01-26T14:53:43Z

I know this is still in beta, but is there a way to add this integration myself to my Elastic Stack, without screwing up the integration when it comes out via the normal release?

jamiehynds · 2023-01-26T14:55:46Z

@infosecwatchman glad to hear you're interested in using this integration. We're just wrapping up some final reviews, so it will be easier to wait until it's publicly available.

@vinit-elastic @kcreddy are there any loose ends on the PR review or are we ready to merge?

infosecwatchman · 2023-01-26T14:59:29Z

Great! Do I have to upgrade my stack once this is merged?

jamiehynds · 2023-01-26T15:10:58Z

Great! Do I have to upgrade my stack once this is merged?

Possibly. The integration will require v8.4 or later.

kcreddy

LGTM 👍🏼

kcreddy · 2023-01-27T05:52:43Z

@infosecwatchman glad to hear you're interested in using this integration. We're just wrapping up some final reviews, so it will be easier to wait until it's publicly available.

@vinit-elastic @kcreddy are there any loose ends on the PR review or are we ready to merge?

I think I was just waiting for @P1llus review to be resolved as well. Looks like we are good. I will merge it now.

elasticmachine · 2023-01-27T06:12:21Z

Package tenable_io - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=tenable_io

infosecwatchman · 2023-01-27T13:29:12Z

How long does it take for new packages to be loaded into the repository?

jamiehynds · 2023-01-27T13:31:18Z

Hey @infosecwatchman - the integration is now available, however it requires the latest version of the stack, 8.6.

infosecwatchman · 2023-02-09T15:22:44Z

What's a good way to go about troubleshooting an integration. For some reason, I have not been recollecting data on the regular intervals as defined. The tenable asset dataset has not been updated since it was created (ie. after the initial data capture).

dhruvil-elastic · 2023-02-13T12:06:19Z

What's a good way to go about troubleshooting an integration. For some reason, I have not been recollecting data on the regular intervals as defined. The tenable asset dataset has not been updated since it was created (ie. after the initial data capture).

Hello @infosecwatchman,

Integration is designed in a way that for the first call, all the asset events will be ingested, and then onwards only those assets events will be ingested whose 'updated_at' field is updated or new assets are added.
Debug logs will be helpful in confirming if the integration is working as intended or not.

Moreover, we are working on an enhancement where a user faced an issue with a very large export job. Can you please confirm if it is the same case as yours or not? If not would you mind raising a new issue in GitHub?

Thank you.

vinit-chauhan added 2 commits December 13, 2022 17:57

Initial Release for the Tenable.io

0e45a30

Update the changelog entry

f5a5d5f

vinit-chauhan added enhancement New feature or request Team:Security-External Integrations New Integration Issue or pull request for creating a new integration package. labels Dec 13, 2022

Set event.original field in custom mapping

64ca1af

P1llus reviewed Dec 27, 2022

View reviewed changes

kcreddy reviewed Jan 2, 2023

View reviewed changes

Update the PR as per the review comments

2202648

kcreddy approved these changes Jan 27, 2023

View reviewed changes

kcreddy merged commit 90bb74f into elastic:main Jan 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[tenable_io] Initial Release for the Tenable.io #4816

[tenable_io] Initial Release for the Tenable.io #4816

vinit-chauhan commented Dec 13, 2022

elasticmachine commented Dec 13, 2022

elasticmachine commented Dec 13, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Dec 13, 2022 •

edited

Loading

jamiehynds commented Dec 13, 2022

vinit-chauhan commented Dec 14, 2022

P1llus Dec 27, 2022

P1llus Dec 27, 2022

vinit-chauhan Dec 28, 2022

kcreddy Jan 2, 2023

vinit-chauhan Jan 3, 2023

kcreddy Jan 4, 2023

vinit-chauhan Jan 4, 2023 •

edited

Loading

kcreddy Jan 5, 2023

P1llus Dec 27, 2022

vinit-chauhan Dec 28, 2022

P1llus Dec 27, 2022

vinit-chauhan Dec 28, 2022

kcreddy Jan 2, 2023

infosecwatchman commented Jan 26, 2023

jamiehynds commented Jan 26, 2023

infosecwatchman commented Jan 26, 2023

jamiehynds commented Jan 26, 2023

kcreddy left a comment

kcreddy commented Jan 27, 2023

elasticmachine commented Jan 27, 2023

infosecwatchman commented Jan 27, 2023

jamiehynds commented Jan 27, 2023

infosecwatchman commented Feb 9, 2023

dhruvil-elastic commented Feb 13, 2023 •

edited

Loading

[tenable_io] Initial Release for the Tenable.io #4816

[tenable_io] Initial Release for the Tenable.io #4816

Conversation

vinit-chauhan commented Dec 13, 2022

What does this PR do?

Integration release checklist

All changes

New Package

Dashboards changes

Log dataset changes

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Dec 13, 2022

elasticmachine commented Dec 13, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Dec 13, 2022 • edited Loading

🌐 Coverage report

jamiehynds commented Dec 13, 2022

vinit-chauhan commented Dec 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinit-chauhan Jan 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

infosecwatchman commented Jan 26, 2023

jamiehynds commented Jan 26, 2023

infosecwatchman commented Jan 26, 2023

jamiehynds commented Jan 26, 2023

kcreddy left a comment

Choose a reason for hiding this comment

kcreddy commented Jan 27, 2023

elasticmachine commented Jan 27, 2023

infosecwatchman commented Jan 27, 2023

jamiehynds commented Jan 27, 2023

infosecwatchman commented Feb 9, 2023

dhruvil-elastic commented Feb 13, 2023 • edited Loading

elasticmachine commented Dec 13, 2022 •

edited

Loading

elasticmachine commented Dec 13, 2022 •

edited

Loading

vinit-chauhan Jan 4, 2023 •

edited

Loading

dhruvil-elastic commented Feb 13, 2023 •

edited

Loading