Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SIEM] Create template timeline #63136

Merged
merged 52 commits into from
Apr 29, 2020
Merged

Conversation

angorayc
Copy link
Contributor

@angorayc angorayc commented Apr 9, 2020

Summary

Timeline apis

  1. Create timeline api
  2. Update timeline api
  3. Create template timeline api
  4. Update template timeline api

Create timeline api

POST /api/timeline

Authorization

Type: Basic Auth
username: Your Kibana username
password: Your Kibana password

Request header
Content-Type: application/json
kbn-version: 8.0.0
Request body
{
	"timeline": {
	     "columns": [
		    {
	          "columnHeaderType": "not-filtered",
	          "id": "@timestamp"
	        },
	        {
	          "columnHeaderType": "not-filtered",
	          "id": "message"
	        },
	        {
	          "columnHeaderType": "not-filtered",
	          "id": "event.category"
	        },
	        {
	          "columnHeaderType": "not-filtered",
	          "id": "event.action"
	        },
	        {
	          "columnHeaderType": "not-filtered",
	          "id": "host.name"
	        },
	        {
	          "columnHeaderType": "not-filtered",
	          "id": "source.ip"
	        },
	        {
	          "columnHeaderType": "not-filtered",
	          "id": "destination.ip"
	        },
	        {
	          "columnHeaderType": "not-filtered",
	          "id": "user.name"
	        }
		  ],
	     "dataProviders": [],
	     "description": "",
	     "eventType": "all",
	     "filters": [],
	     "kqlMode": "filter",
	     "kqlQuery": {
	       "filterQuery": null
	     },
	     "title": "abd",
	     "dateRange": {
	       "start": 1587370079200,
	       "end": 1587456479201
	     },
	     "savedQueryId": null,
	     "sort": {
	       "columnId": "@timestamp",
	       "sortDirection": "desc"
	     }
	 },
	"timelineId":null, // Leave this as null
	"version":null // Leave this as null
}

Update timeline api

PATCH /api/timeline

Authorization

Type: Basic Auth
username: Your Kibana username
password: Your Kibana password

Request header
Content-Type: application/json
kbn-version: 8.0.0
Request body
{
	"timeline": {            
        "columns": [
            {
                "columnHeaderType": "not-filtered",
                "id": "@timestamp"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "message"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "event.category"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "event.action"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "host.name"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "source.ip"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "destination.ip"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "user.name"
            }
        ],
        "dataProviders": [],
        "description": "",
        "eventType": "all",
        "filters": [],
        "kqlMode": "filter",
        "kqlQuery": {
            "filterQuery": null
        },
        "title": "abd",
        "dateRange": {
            "start": 1587370079200,
            "end": 1587456479201
        },
        "savedQueryId": null,
        "sort": {
            "columnId": "@timestamp",
            "sortDirection": "desc"
        },
        "created": 1587468588922,
        "createdBy": "casetester",
        "updated": 1587468588922,
        "updatedBy": "casetester",
        "timelineType": "default"
    },
	"timelineId":"68ea5330-83c3-11ea-bff9-ab01dd7cb6cc", // Have to match the existing timeline savedObject id
	"version":"WzYwLDFd" // Have to match the existing timeline version
}

Create template timeline api

POST /api/timeline

Authorization

Type: Basic Auth
username: Your Kibana username
password: Your Kibana password

Request header
Content-Type: application/json
kbn-version: 8.0.0
Request body
{
	"timeline": {
      "columns": [
        {
          "columnHeaderType": "not-filtered",
          "id": "@timestamp"
        },
        {
          "columnHeaderType": "not-filtered",
          "id": "message"
        },
        {
          "columnHeaderType": "not-filtered",
          "id": "event.category"
        },
        {
          "columnHeaderType": "not-filtered",
          "id": "event.action"
        },
        {
          "columnHeaderType": "not-filtered",
          "id": "host.name"
        },
        {
          "columnHeaderType": "not-filtered",
          "id": "source.ip"
        },
        {
          "columnHeaderType": "not-filtered",
          "id": "destination.ip"
        },
        {
          "columnHeaderType": "not-filtered",
          "id": "user.name"
        }
      ],
      "dataProviders": [
        
      ],
      "description": "",
      "eventType": "all",
      "filters": [
        
      ],
      "kqlMode": "filter",
      "kqlQuery": {
        "filterQuery": null
      },
      "title": "abd",
      "dateRange": {
        "start": 1587370079200,
        "end": 1587456479201
      },
      "savedQueryId": null,
      "sort": {
        "columnId": "@timestamp",
        "sortDirection": "desc"
      },
      "timelineType": "template" // This is the difference between create timeline
    },
	"timelineId":null, // Leave this as null
	"version":null // Leave this as null
}

Update template timeline api

PATCH /api/timeline

Authorization

Type: Basic Auth
username: Your Kibana username
password: Your Kibana password

Request header
Content-Type: application/json
kbn-version: 8.0.0
Request body
{
	"timeline": {
         "columns": [
             {
                 "columnHeaderType": "not-filtered",
                 "id": "@timestamp"
             },
             {
                 "columnHeaderType": "not-filtered",
                 "id": "message"
             },
             {
                 "columnHeaderType": "not-filtered",
                 "id": "event.category"
             },
             {
                 "columnHeaderType": "not-filtered",
                 "id": "event.action"
             },
             {
                 "columnHeaderType": "not-filtered",
                 "id": "host.name"
             },
             {
                 "columnHeaderType": "not-filtered",
                 "id": "source.ip"
             },
             {
                 "columnHeaderType": "not-filtered",
                 "id": "destination.ip"
             },
             {
                 "columnHeaderType": "not-filtered",
                 "id": "user.name"
             }
         ],
         "dataProviders": [],
         "description": "",
         "eventType": "all",
         "filters": [],
         "kqlMode": "filter",
         "kqlQuery": {
             "filterQuery": null
         },
         "title": "abd",
         "dateRange": {
             "start": 1587370079200,
             "end": 1587456479201
         },
         "savedQueryId": null,
         "sort": {
             "columnId": "@timestamp",
             "sortDirection": "desc"
         },
         "timelineType": "template",
         "created": 1587473119992,
         "createdBy": "casetester",
         "updated": 1587473119992,
         "updatedBy": "casetester",
         "templateTimelineId": "745d0316-6af7-43bf-afd6-9747119754fb", // Please provide the existing template timeline id 
         "templateTimelineVersion": 2 // Please provide a template timeline version grater than existing one
     },
     "timelineId":"f5a4bd10-83cd-11ea-bf78-0547a65f1281", // This is a must as well
     "version":"Wzg2LDFd" // Please provide the existing timeline version
}

Implementation details:

    • Add a post and a patch endpoints to create/update timeline - (Refactor) Move away from graphQL to request handler.

      • case 1 Create timeline - POST:

        • if timeline-id is NOT provided (timeline-id is undefined / null) => Create a new timeline
        • if timeline-id is provided (timeline-id`) => Check if the timeline id exists or not?
          • If exists, then throw error UPDATE timeline with POST is not allowed, please use PATCH instead
          • If it doesn't exists, then Create a new template timeline
      • case 2 Update timeline - PATCH - Timeline is not allowed to be updated via import timeline ATM:

        • if timeline-id is NOT provided (timeline-id is undefined / null) =>CREATE timeline with PATCH is not allowed, please use POST instead

        • if timeline-id is provided (timeline-id`) => Check if the timeline id exists or not?

          • If exists, Update the timeline
          • If it doesn't exists, then throw an error CREATE timeline with PATCH is not allowed, please use POST instead
    • Update timeline's io-ts schema to actually reflect the api response
    • Update timeline/notes/pinnedEvents saved object library - Move away from class to functions
    • Add a timelineType field to timeline saved object mapping - can be default or template.
    • Share the endpoint create/update timeline with create template timeline by adding the timelineType and template timeline id argument.
      • Keep globalNote and favourite for template timeline

        • case 1 Create template timeline - POST:

          • if timelineType === 'template', template timeline id is NOT provided or timeline-id is NOT provided (timeline-id is undefined / null) => Create a new template timeline
          • if timelineType === 'template', timeline id is provided => Check if the timeline and template timeline id exists or not?
            • If template timeline exists , then throw error UPDATE template timeline with POST is not allowed, please use PATCH instead
            • If it doesn't exists, then Create a new template timeline
        • case 2 Update template timeline - PATCH - Template Timeline is not allowed to be updated via import timeline ATM:

          • if timeline-id is provided (timeline-id`) => Check if the template timeline id exists or not?
            • If timeline exists
              • check the template timeline id , if it doesn't exist, throw an error TimelineVersion conflict: The given version template timeline id not match with existing timeline
              • if the template timeline exists
              • check if the timeline id match with template timeline's timeline id, if it doesn't match the given timeline id, throw an error Timeline id doesn't match with existing template timeline
              • check the timeline version, if it doesn't match the given timeline version, throw an error CREATE template timeline with PATCH is not allowed, please use POST instead
              • check the template timeline version, if it doesn't grater than given template timeline version, throw an error "Template timelineVersion conflict: The given version is older then existing version"
              • timeline id matches existing timeline id, timeline version matches existing timeline version, template timeline version is greater than existing template timeline version => Update the template timeline
            • If it doesn't exists, then throw an error CREATE template timeline with PATCH is not allowed, please use POST instead
    • Update Import timeline api to take timelineType and template timeline id
      • case 1 timeline-id is provided

        • check if timeline exist or not
          • yes => check if template timeline id exist?
            • Error: timeline_id xxx is already exist
          • no => check if timelineType === template
            • yes => create template timeline with given info
            • no => create timeline with given info
      • case 2 timeline id is NOT provided

        • check if timelineType === template?
          • yes => create template timeline with given info
          • no => create timeline with given info
    • Update Export timeline api to include timelineType and template timeline id

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@elasticmachine
Copy link
Contributor

Pinging @elastic/siem (Team:SIEM)

@angorayc
Copy link
Contributor Author

It will be nice to avoid the full reload of the table when we are re-fetching data.

Apr-27-2020 15-53-56

Should be alright now
Kapture 2020-04-28 at 11 31 20

@angorayc
Copy link
Contributor Author

Also, I realized that we can not update a template through the import API. I think we should allow an override just for the template so Craigs's team can update their template timeline if needed it.

I think for simplicity, we can create another PR for that with more validation do not hesitate to transform this comment in issue.

Sure, I'll do that in another PR.
Issue: #64632

Copy link
Contributor

@XavierM XavierM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@angorayc
Copy link
Contributor Author

@elasticmachine merge upstream

@angorayc
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@angorayc angorayc merged commit a42c202 into elastic:master Apr 29, 2020
angorayc added a commit to angorayc/kibana that referenced this pull request Apr 29, 2020
* init routes for template timeline

* create template timeline

* add create/update timelines route

* update api entry point

* fix types

* add template type

* fix types

* add types and template timeline id

* fix types

* update import timeline to handle template timeline

* unit test

* sudo code

* remove class for savedobject

* add template timeline version

* clean up arguments

* fix types for framework request

* show filter in find

* fix create template timeline

* update mock data

* handle missing timeline when exporting

* update the order for timeline routes

* update schemas

* move type to common folder so we can re-use them on UI and server side

* fix types + integrate persist with epic timeline

* update all timeline when persit timeline

* add timeline api readme

* fix validation error

* fix unit test

* display error if unexpected format is given

* fix issue with reftech all timeline query

* fix flashing timeline while refetch

* fix types

* fix types

* fix dependency

* fix timeline deletion

* remove redundant dependency

* add i18n message

* fix unit test

Co-authored-by: Xavier Mouligneau <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
gmmorris added a commit to gmmorris/kibana that referenced this pull request Apr 29, 2020
* master: (60 commits)
  [SIEM] Create template timeline (elastic#63136)
  load react component lazily in so management section (elastic#64285)
  Cleanup .eslingignore and add target (elastic#64617)
  [Ingest] Support yaml variables in datasource (elastic#64459)
  typescript-ify portions of src/optimize (elastic#64688)
  [ngSanitize] add explicit dependencies to all uses of `ngSanitize` angular module (elastic#64546)
  Consolidate downloading plugin bundles to bootstrap script (elastic#64685)
  [Maps] disable edit layer button when flyout is open for add layer or map settings (elastic#64230)
  chore(NA): add async import into infra plugin to reduce apm bundle size (elastic#63292)
  [Maps] fix edit filter (elastic#64586)
  [SIEM][Detections] Adds large list support using REST endpoints
  Replace a number of any-ed styled(eui*) with accurate types (elastic#64555)
  [Endpoint] Recursive resolver children (elastic#61914)
  [ML] Fix new job wizard with multiple indices (elastic#64567)
  Use short URLs for legacy plugin deprecation warning (elastic#64540)
  [Uptime] Update uptime ml job id to limit to 64 char (elastic#64394)
  [Ingest] Fix GET /enrollment-api-keys/null error (elastic#64595)
  Consolidate cross-cutting concerns between region & coordinate maps in new maps_legacy plugin (elastic#64123)
  ES UI new platform cleanup (elastic#64332)
  [Event Log] use @timestamp field for queries (elastic#64391)
  ...
gmmorris added a commit to gmmorris/kibana that referenced this pull request Apr 29, 2020
* alerting/np-migration: (64 commits)
  [ML] Changes Machine learning overview UI text (elastic#64625)
  [Uptime] Migrate client to New Platform (elastic#55086)
  Slim vis type timeseries (elastic#64631)
  [Telemetry] Fix inconsistent search behaviour in Advanced Settings (elastic#64510)
  removed unneeded dep and file
  [SIEM] Create template timeline (elastic#63136)
  load react component lazily in so management section (elastic#64285)
  Cleanup .eslingignore and add target (elastic#64617)
  [Ingest] Support yaml variables in datasource (elastic#64459)
  typescript-ify portions of src/optimize (elastic#64688)
  [ngSanitize] add explicit dependencies to all uses of `ngSanitize` angular module (elastic#64546)
  Consolidate downloading plugin bundles to bootstrap script (elastic#64685)
  [Maps] disable edit layer button when flyout is open for add layer or map settings (elastic#64230)
  chore(NA): add async import into infra plugin to reduce apm bundle size (elastic#63292)
  [Maps] fix edit filter (elastic#64586)
  [SIEM][Detections] Adds large list support using REST endpoints
  Replace a number of any-ed styled(eui*) with accurate types (elastic#64555)
  [Endpoint] Recursive resolver children (elastic#61914)
  [ML] Fix new job wizard with multiple indices (elastic#64567)
  Use short URLs for legacy plugin deprecation warning (elastic#64540)
  ...
angorayc added a commit that referenced this pull request Apr 29, 2020
* init routes for template timeline

* create template timeline

* add create/update timelines route

* update api entry point

* fix types

* add template type

* fix types

* add types and template timeline id

* fix types

* update import timeline to handle template timeline

* unit test

* sudo code

* remove class for savedobject

* add template timeline version

* clean up arguments

* fix types for framework request

* show filter in find

* fix create template timeline

* update mock data

* handle missing timeline when exporting

* update the order for timeline routes

* update schemas

* move type to common folder so we can re-use them on UI and server side

* fix types + integrate persist with epic timeline

* update all timeline when persit timeline

* add timeline api readme

* fix validation error

* fix unit test

* display error if unexpected format is given

* fix issue with reftech all timeline query

* fix flashing timeline while refetch

* fix types

* fix types

* fix dependency

* fix timeline deletion

* remove redundant dependency

* add i18n message

* fix unit test

Co-authored-by: Xavier Mouligneau <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>

Co-authored-by: Xavier Mouligneau <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
@MindyRS MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@angorayc
Copy link
Contributor Author

angorayc commented Aug 7, 2023

Update timeline api

request body:

{
    "timeline": {
        "columns": [
            {
                "columnHeaderType": "not-filtered",
                "id": "@timestamp"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "message"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "event.category"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "event.action"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "host.name"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "source.ip"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "destination.ip"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "user.name"
            }
        ],
        "dataProviders": [],
        "description": "",
        "eventType": "all",
        "filters": [],
        "kqlMode": "filter",
        "kqlQuery": {
            "filterQuery": null
        },
        "title": "abd",
        "dateRange": {
            "start": 1587370079200,
            "end": 1587456479201
        },
        "savedQueryId": null,
        "sort": {
            "columnId": "@timestamp",
            "sortDirection": "desc"
        },
        "created": 1587468588922,
        "createdBy": "casetester",
        "updated": 1587468588922,
        "updatedBy": "casetester",
        "timelineType": "default"
    },
    "timelineId": "4bc294e0-3516-11ee-9f62-49614d8a84fd", // Has to match the existing timeline savedObject id
    "version": "WzE5MTUsMV0=" // Has to match the existing timeline version
}

Respose:

{
    "data": {
        "persistTimeline": {
            "code": 200,
            "message": "success",
            "timeline": {
                "savedObjectId": "4bc294e0-3516-11ee-9f62-49614d8a84fd",
                "version": "WzE5MTgsMV0=",
                "columns": [
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "@timestamp"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "message"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "event.category"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "event.action"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "host.name"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "source.ip"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "destination.ip"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "user.name"
                    }
                ],
                "dataProviders": [],
                "dataViewId": null,
                "description": "",
                "eventType": "all",
                "excludedRowRendererIds": [],
                "favorite": [],
                "filters": [],
                "kqlMode": "filter",
                "kqlQuery": {
                    "filterQuery": null
                },
                "title": "abd",
                "templateTimelineId": null,
                "templateTimelineVersion": null,
                "dateRange": {
                    "start": 1587370079200,
                    "end": 1587456479201
                },
                "savedQueryId": null,
                "created": 1587468588922,
                "createdBy": "casetester",
                "updated": 1691408201273,
                "updatedBy": "elastic",
                "timelineType": "default",
                "status": "active",
                "sort": [
                    {
                        "sortDirection": "desc",
                        "columnId": "@timestamp"
                    }
                ],
                "eventIdToNoteIds": [],
                "noteIds": [],
                "notes": [],
                "pinnedEventIds": [],
                "pinnedEventsSaveObject": []
            }
        }
    }
}

Update timeline template api

request body:

{
    "timeline": {
        "columns": [
            {
                "columnHeaderType": "not-filtered",
                "id": "@timestamp"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "message"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "event.category"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "event.action"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "host.name"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "source.ip"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "destination.ip"
            },
            {
                "columnHeaderType": "not-filtered",
                "id": "user.name"
            }
        ],
        "dataProviders": [],
        "description": "",
        "eventType": "all",
        "filters": [],
        "kqlMode": "filter",
        "kqlQuery": {
            "filterQuery": null
        },
        "title": "abd",
        "dateRange": {
            "start": 1587370079200,
            "end": 1587456479201
        },
        "savedQueryId": null,
        "sort": {
            "columnId": "@timestamp",
            "sortDirection": "desc"
        },
        "timelineType": "template",
        "created": 1587473119992,
        "createdBy": "casetester",
        "updated": 1587473119992,
        "updatedBy": "casetester",
        "templateTimelineId": "6f9a3480-bf4f-11ea-9fcd-ed4e5fd0dcd1", // Please provide the existing template timeline version 
        "templateTimelineVersion": 2 // Please provide a template timeline version grater than existing one
    },
    "timelineId": "7d7d4b60-3516-11ee-9f62-49614d8a84fd", // Timeline saved object id. This is a must as well
    "version": "WzE5MTcsMV0=" // Please provide the existing timeline version
}

Response:

{
    "data": {
        "persistTimeline": {
            "code": 200,
            "message": "success",
            "timeline": {
                "savedObjectId": "7d7d4b60-3516-11ee-9f62-49614d8a84fd",
                "version": "WzE5MTksMV0=",
                "columns": [
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "@timestamp"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "message"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "event.category"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "event.action"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "host.name"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "source.ip"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "destination.ip"
                    },
                    {
                        "columnHeaderType": "not-filtered",
                        "id": "user.name"
                    }
                ],
                "dataProviders": [],
                "dataViewId": null,
                "description": "",
                "eventType": "all",
                "excludedRowRendererIds": [],
                "favorite": [],
                "filters": [],
                "kqlMode": "filter",
                "kqlQuery": {
                    "filterQuery": null
                },
                "title": "abd",
                "templateTimelineId": "6f9a3480-bf4f-11ea-9fcd-ed4e5fd0dcd1",
                "templateTimelineVersion": 2,
                "dateRange": {
                    "start": 1587370079200,
                    "end": 1587456479201
                },
                "savedQueryId": null,
                "created": 1587473119992,
                "createdBy": "casetester",
                "updated": 1691408702104,
                "updatedBy": "elastic",
                "timelineType": "template",
                "status": "active",
                "sort": [
                    {
                        "sortDirection": "desc",
                        "columnId": "@timestamp"
                    }
                ],
                "eventIdToNoteIds": [],
                "noteIds": [],
                "notes": [],
                "pinnedEventIds": [],
                "pinnedEventsSaveObject": []
            }
        }
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs_docs release_note:enhancement Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:SIEM v7.8.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants