Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Workflow for automatic generation of index templates on changes #540

Open
2 tasks done
AlexRuiz7 opened this issue Nov 11, 2024 · 3 comments · May be fixed by #586
Open
2 tasks done

Workflow for automatic generation of index templates on changes #540

AlexRuiz7 opened this issue Nov 11, 2024 · 3 comments · May be fixed by #586
Assignees
@QU3B1M
Labels
level/task Task issue request/operational Operational requests type/enhancement Enhancement issue

Comments

@AlexRuiz7
Copy link
Member

AlexRuiz7 commented Nov 11, 2024 •
edited by QU3B1M
Loading

Description

We include many new indices in Wazuh 5 and as responsible for their maintenance and generation, we have generated some tools to automate the process, however, the invocation of these tools is manual and a bit tedious when we change many indices at one. Also, the tools need to be set up before using them, as described in this README.

It would be great to have the index templates re-generated automatically using a GitHub Workflow that triggers on PRs that modify the definitions files.

A possible solution would be to have a Docker image with the requirements to build the templates.

Functional requirements

  • The index templates are generated when their definition changes.

Implementation restrictions

  • Use a GitHub Workflow that triggers automatically on PRs that modify these files.
  • Consider using our Quay.io registry to store the image.

Plan

  • Generate Docker image to generate the index templates.
  • Generate a workflow that detects changes to files under ECS and generated the corresponding index template, using the Docker image created for that.
@AlexRuiz7 AlexRuiz7 added level/task Task issue request/operational Operational requests type/enhancement Enhancement issue labels Nov 11, 2024
@wazuhci wazuhci moved this to Backlog in Release 5.0.0 Nov 11, 2024
@wazuhci wazuhci moved this from Backlog to In progress in Release 5.0.0 Nov 29, 2024
@wazuhci wazuhci moved this from In progress to On hold in Release 5.0.0 Dec 5, 2024
@wazuhci wazuhci moved this from On hold to In progress in Release 5.0.0 Dec 9, 2024
@QU3B1M QU3B1M linked a pull request Dec 10, 2024 that will close this issue
Open
3 tasks
@QU3B1M
Copy link
Member

QU3B1M commented Dec 10, 2024 •
edited
Loading

Created initial version of the Docker image to generate the ECS index templates, it uses a modified version of the generate.sh tool, and currently its only saving the resulting templates inside the container itself.

Building the image

 docker build -f docker/ecs/images/Dockerfile . -t ecs-generator
[+] Building 1.5s (11/11) FINISHED                                                                                                                                                  docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                0.0s
 => => transferring dockerfile: 633B                                                                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/python:3.10                                                                                                                                      0.7s
 => [internal] load .dockerignore                                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                                     0.0s
 => [1/6] FROM docker.io/library/python:3.10@sha256:3ba2e48b887586835af6a0c35fc6fc6086fb4881e963082330ab0a35f3f42c16                                                                                0.0s
 => => resolve docker.io/library/python:3.10@sha256:3ba2e48b887586835af6a0c35fc6fc6086fb4881e963082330ab0a35f3f42c16                                                                                0.0s
 => [internal] load build context                                                                                                                                                                   0.1s
 => => transferring context: 231.38kB                                                                                                                                                               0.1s
 => CACHED [2/6] RUN apt-get update &&     apt-get upgrade -y &&     apt-get install  -y git jq &&     apt-get clean &&     rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* &&     git clone https:/  0.0s
 => CACHED [3/6] ADD docker/ecs/images/generate.sh /ecs/generate.sh                                                                                                                                 0.0s
 => [4/6] ADD ecs/ /source/ecs                                                                                                                                                                      0.1s
 => [5/6] RUN chmod +x /ecs/generate.sh                                                                                                                                                             0.1s
 => [6/6] WORKDIR /ecs                                                                                                                                                                              0.0s
 => exporting to image                                                                                                                                                                              0.5s
 => => exporting layers                                                                                                                                                                             0.5s
 => => exporting manifest sha256:b19ca762ecf63c7e57d7767e9f6cdb0d94ac8ae3e22b40a91e3d0595a1771e0b                                                                                                   0.0s
 => => exporting config sha256:b1b8201f80d570a2d6bb5ee4d7425cd5f5bdbdce995009d6bdb5f48fc6f97b8c                                                                                                     0.0s
 => => exporting attestation manifest sha256:5ccac2093aae0f9cf4ab65f00cb5616d9b587d790ac46a32d7ca4382dbd65762                                                                                       0.0s
 => => exporting manifest list sha256:2bfd5a1b52a921fb238685989a7f83a345dd266f6143bb700837943348548da1                                                                                              0.0s
 => => naming to docker.io/library/ecs-generator:latest                                                                                                                                             0.0s

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/z6646fy4lug9e87big311tenp

Using the tool

docker run --rm -e ECS_MODULE=agent ecs-generator              
Loading schemas from git ref v8.11.0
Running generator. ECS version 8.11.0
Loading user defined schemas: ['/source/ecs/agent/fields/custom/']
Replacing unsupported types in generated mappings
Deleting the "tags" field from the index template
Removing multi-fields from the index template
Mappings saved to /source/ecs/agent/mappings/v8.11.0

Next steps:

  • Use a volume as ECS definitions source
  • Implement Docker Compose and a script to handle the tool execution
  • Add a GHA Workflow to generate the index templates and push them on the wazuh-indexer-plugin repo

@QU3B1M
Copy link
Member

QU3B1M commented Dec 13, 2024

Implemented Docker Compose to define the required environment variables (ECS_MODULE) and the volume where the ECS definitions are stored

services:
  ecs-mapping-generator:
    image: wazuh-ecs-generator
    container_name: wazuh-ecs-generator
    build:
      context: ./../..
      dockerfile: ${REPO_PATH:-.}/docker/ecs/images/Dockerfile
    volumes:
      - ${REPO_PATH:-.}/ecs:/source/ecs
    environment:
      - ECS_MODULE=${ECS_MODULE:-default_module}

Also a bash util was developed to handle the interaction with this Docker Compose environment, making the process easier. It only requires one command

bash docker/ecs/mapping-generator.sh run <ECS_MODULE>

This script can be launch from any point inside of the repository's directories.

Example execution 
bash docker/ecs/mapping-generator.sh run alerts     
[+] Running 1/0
 ✔ Container wazuh-ecs-generator  Created                                                                                                                                                           0.0s 
Attaching to wazuh-ecs-generator
wazuh-ecs-generator  | Loading schemas from git ref v8.11.0
wazuh-ecs-generator  | Running generator. ECS version 8.11.0
wazuh-ecs-generator  | Loading user defined schemas: ['/source/ecs/alerts/fields/custom/']
wazuh-ecs-generator  | Replacing unsupported types in generated mappings
wazuh-ecs-generator  | Deleting the "tags" field from the index template
wazuh-ecs-generator  | Removing multi-fields from the index template
wazuh-ecs-generator  | Mappings saved to /source/ecs/alerts/mappings/v8.11.0
wazuh-ecs-generator exited with code 0
[+] Stopping 1/0
 ✔ Container wazuh-ecs-generator  Stopped

@QU3B1M
Copy link
Member

QU3B1M commented Dec 13, 2024

Generated new GitHub Actions Workflow that executes the utility and creates a PR on wazuh-indexer-plugin with modified template. Currently the GHA is not working due to permissions.

[update-ecs-template-alerts 95b56f3] Update ECS template for module alerts
 1 file changed, 1 insertion(+), 1 deletion(-)
remote: Permission to wazuh/wazuh-indexer-plugins.git denied to github-actions[bot].
fatal: unable to access 'https://github.com/wazuh/wazuh-indexer-plugins/': The requested URL returned error: 403
Error: Process completed with exit code 128.

@wazuhci wazuhci moved this from In progress to Pending review in Release 5.0.0 Dec 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@QU3B1M QU3B1M

Labels
level/task Task issue request/operational Operational requests type/enhancement Enhancement issue
Projects
Release 5.0.0
Status: Pending review
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Implement workflow for automatic ECS templates generation wazuh/wazuh-indexer
2 participants
@AlexRuiz7 @QU3B1M