DTT1 - Iteration 3 - Provision module - Module documentation

[fcaffieri]

Epic: #4852

---

Description

The objective of this issue is to generate the documentation of the Provision module.

Tasks

• Generate documentation on the use of the module for end users.
• Generate technical documentation about the module graph and flow.

[pro-akim]

Update

SAMPLE

Provision module

User documentation

The Provision module installs libraries, dependencies and applications on the allocated infrastructure (From Allocation module)

This module can be executed as follows:
A. Installing and using Workflow engine
B. Direct execution

A. Installing and using Workflow engine

The execution of the workflow is done through the installation of its library.

Initially, Python libraries must be installed. we recommended the use of virtual environments. Follow the technical documentation at https://docs.python.org/3/library/venv.html.

1. Create the python environment:

   python3 -m venv {environment_name}

2. Activate the environment:

    source {venv directory}/bin/activate

3. Clone the wazuh-qa repository:

   Navigate to the project directory and switch to the project branch:

   https://github.com/wazuh/wazuh-qa.git
   cd wazuh-qa
   git checkout {project-branch}

4. Install requirements:

   pip3 install -r deployability/deps/requirements.txt

5. Install the Workflow engine library and its launcher:

   While in wazuh-qa:

   cd modules
   pip3 uninstall -y workflow_engine && pip3 install .

Run the module by doing the following steps:

6. Test fixture to execute:

   It is required to create a fixture (YAML file) where the infrastructure, provisioning, and tests to be executed is declared.

   Note: You can find some fixture examples in 'deployability/modules/workflow_engine/examples/'

   Example:

   version: 0.1
   description: This workflow is used to provision agents hosts por DDT1 PoC
   variables:
     agents-os:
       - linux-ubuntu-22.04-amd64
     infra-provider: vagrant
     working-dir: /tmp/dtt1-poc
   
   tasks:
     - task: "allocate-{agent}"
       description: "Allocate resources for the {agent} agent."
       do:
         this: process
         with:
           path: python3
           args:
             - modules/allocation/main.py
             - action: create
             - provider: "{infra-provider}"
             - size: small
             - composite-name: "{agent}"
             - inventory-output: "{working-dir}/agent-{agent}/inventory.yaml"
             - track-output: "{working-dir}/agent-{agent}/track.yaml"
       cleanup:
         this: process
         with:
           path: python3
           args:
             - modules/allocation/main.py
             - action: delete
             - track-output: "{working-dir}/agent-{agent}/track.yaml"
       foreach:
         - variable: agents-os
           as: agent
   
     - task: "provision-install-{agent}"
       description: "Provision resources for the {agent} agent."
       do:
         this: process
         with:
           path: python3
           args:
             - modules/provision/main.py
             - inventory: "{working-dir}/agent-{agent}/inventory.yaml"
             - dependencies:
               - agent: "{working-dir}/agent-{agent}/inventory.yaml"
             - install:
               - component: curl
               - component: python
                 type: source
                 version: "{python-version}"
               - component: virtualenv
                 type: pip
               - component: deps/requirements.txt
                 type: pip
               - component: pyyaml
                 type: pip
                 version: 6.0.1
       depends-on:
         - "allocate-{agent}"
       foreach:
         - variable: agents-os
           as: agent

   Following the schema of the example:

   Configure the following parameters depending on your allocation:

   variables/agent-os
   variables/manager-os
   infra-provider
   working-dir
   tasks

   Pay attention to the tasks:

   args
   depends-on

   Note: In args, configure the launcher's path correctly (main.py files in each module), and to fill depends-on, consider the steps of your provision (allocation and provision)

7. Command execution (local):

   Execute the command by referencing the parameters required by the library (launcher).

   python3 -m workflow_engine {.yaml fixture path}

   Example

   python3 -m workflow_engine modules/workflow_engine/examples/dtt1-agents-poc.yaml

B. Direct execution

To execute the Allocation module without installing the Workflow engine, you can use the launcher ('module/provision/main.py'):

1. Execution

While in wazuh-qa/deployability

python3 module/provision/main.py --inventory-manager { inventory .yaml} --install "{'component': '{component}', 'type': '{type}', 'version': '{version}'}" 

Example:

python3 module/provision/main.py --inventory-manager /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml --install "{'component': 'pyyaml', 'type': 'pip', 'version' : '6.0.1'}" 

---

Technical documentation

The provision module allows for the installation and uninstallation of libraries, dependencies, and applications in the infrastructure.

It can receive instructions either through the command line or be mediated through the Workflow Engine.

In both cases, the following information is required:

  - task: "provision-install-{agent}"
    description: "Provision resources for the {agent} agent."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/provision/main.py
          - inventory: "{working-dir}/agent-{agent}/inventory.yaml"
          - dependencies:
            - agent: "{working-dir}/agent-{agent}/inventory.yaml"
          - install:
            - component: curl
            - component: python
              type: source
              version: "{python-version}"
            - component: virtualenv
              type: pip
            - component: deps/requirements.txt
              type: pip
            - component: pyyaml
              type: pip
              version: 6.0.1
    depends-on:
      - "allocate-{agent}"
    foreach:
      - variable: agents-os
        as: agent

The snippet obtained from a Workflow fixture highlights the parameters necessary for its operation. These include the inventory path, the dependency/library to install, and the installation type.

It requires an infrastructure on which to perform its actions.

The module is composed of:

• Launcher ('/wazuh-qa/deployability/modules/provision/main.py')
  Entry point for the workflow or the user who wishes to execute a provision.

• Parameter validator ('/wazuh-qa/deployability/modules/provision/models.py')
  Validator for parameters entered by the Workflow engine or the user.

• Module functions ('/wazuh-qa/deployability/modules/provision/provision.py')
  Module-specific functions responsible for triggering the provision.

• Component type categorizer ('/wazuh-qa/deployability/modules/provision/component_type.py')
  Allows obtaining and expanding data based on initially received parameters.

• Executor ('/wazuh-qa/deployability/modules/provision/actions.py')
  Executes the provision using Actions in the generic module.

• Utils ('/wazuh-qa/deployability/modules/provision/utils.py')
  Enables reading and changing the format of YAML files to executable formats.

Provision.drawio.zip

License

WAZUH Copyright (C) 2015 Wazuh Inc. (License GPLv2)

[damarisg]

Being general is ok.

However, I will mark some details:

1. You should add the branch that contains the development to the step and clarify that it is temporary.

2. Is it available to Linux and Windows? If it only works on Linux, you must identify the OSs.

3. the fixture examples have {agent} variable. To be clear, you must add a full example because the agent could be the OS, the agent hostname, the agent ID, etc.

[pro-akim]

Update

Considering the Review Notes:

1. Note adding the temporary dev branch added
2. and 3. Fixture now has different Os agents.

PR added.

[fcaffieri]

We need to adapt the documentation with this recommendations:

• We could have a short introduction about allocation in the User documentation section. In the tech documentation section under it, we can see a short intro about the allocation module but nothing similar for the user doc above.
• Styles using passive voice are vague. We could change to active voice instead. For example, "You must install Python libraries. We recommend using virtual environments" rather than "... must be installed..." and "... is recommended to use...".
• Maybe creating some subsections with subtitles to separate topics like one for setting up the repository and environment, another one for using the workflow library, another for manually running it, and so on.
• Paths could use links to easily check files content in Github. For example: Module functions /wazuh-qa/deployability/modules/allocation/allocation.py: Module-specific functions responsible for triggering the allocation.
• If required, we could review phrasing as well as expanding some topics.

[pro-akim]

Update

Sample of #4851 (comment) was changed following #4851 (comment) criteria

[rauldpm]

LGTM