Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DTT1 - Iteration 3 - Provision module - Install required dependencies on each supported OS #4859

Closed
2 of 3 tasks
Tracked by #4852
fcaffieri opened this issue Jan 16, 2024 · 16 comments · Fixed by #5015
Closed
2 of 3 tasks
Tracked by #4852
Assignees

Comments

@fcaffieri
Copy link
Member

fcaffieri commented Jan 16, 2024

Epic: #4852


Description

The objective of this issue is to ensure the installation of dependencies on each system is correct and the version is the same across all the supported OS to execute the tests.

A new approach of the dependencies installation will be required so the deps version can be controlled by the end user

OS to validate:

Operating System Version Component Architectures
RedHat 7 agents, central components x86_64, aarch64
RedHat 8 agents, central components x86_64, aarch64
RedHat 9 agents, central components x86_64, aarch64
CentOS 7 agents, central components x86_64, aarch64
CentOS 8 agents, central components x86_64, aarch64
Debian 10 agents, central components x86_64, aarch64
Debian 11 agents, central components x86_64, aarch64
Debian 12 agents, central components x86_64, aarch64
Ubuntu 18 agents x86_64, aarch64
Ubuntu 20 agents, central components x86_64, aarch64
Ubuntu 22 agents, central components x86_64, aarch64
Oracle Linux 9 agents, central components x86_64, aarch64
Amazon Linux 2 agents, central components x86_64, aarch64
Amazon Linux 2023 agents, central components x86_64, aarch64
openSUSE 15 agents, central components x86_64, aarch64
SUSE 15 agents, central components x86_64, aarch64
Fedora 38 agents x86_64, aarch64
Windows 10 agents x86_64, aarch64
Windows 11 agents x86_64, aarch64
Windows Server 2012 agents x86_64, aarch64
Windows Server 2012 R2 agents x86_64, aarch64
Windows Server 2016 agents x86_64, aarch64
Windows Server 2019 agents x86_64, aarch64
Windows Server 2022 agents x86_64, aarch64

Tasks

  • Validate that the dependencies for the OS in Vagrant work
  • Validate that the dependencies for the OS in AWS work
  • Adapt the dependencies required by the tests for each OS
@wazuhci wazuhci moved this to Triage in Release 4.9.0 Jan 30, 2024
@wazuhci wazuhci moved this from Triage to Backlog in Release 4.9.0 Feb 16, 2024
@QU3B1M QU3B1M self-assigned this Feb 20, 2024
@wazuhci wazuhci moved this from Backlog to In progress in Release 4.9.0 Feb 20, 2024
@QU3B1M
Copy link
Member

QU3B1M commented Feb 20, 2024

Update report

To correctly configure "the same" environment on each system it was decided to install python from source code, and then the python based dependencies (pip, venv, etc..) on top of that.

To do so, some other steps are required on the dependencies playbook:

  1. Install the required tools to compile python source code.
    • Redhat based:
      yum groupinstall "Development Tools"
    • Debian based:
      apt-get install build-essential
    • OpenSUSE based:
      zypper install --type pattern devel_basis
    • Alpine based:
      apk add build-base
  2. Compile python source code and install it.
  3. Configure the installed python interpreter as default.
  4. Install & upgrade pip alongisde the other dependencies.

@QU3B1M QU3B1M changed the title DTT1 - Iteration 3 - Provision module - Validate dependencies according to OS DTT1 - Iteration 3 - Provision module - Install required dependencies on each supported OS Feb 21, 2024
@QU3B1M
Copy link
Member

QU3B1M commented Feb 22, 2024

Update report

Restructured the playbooks renaming generic to package (as that's the provisioning method), this way we have a more accurate definition of the provisioning methods. The wazuh related playbooks will be kept under the wazuh/ subdirectory.

├── playbooks
│   ├── deps/
│   ├── package/
│   │   ├── install/
│   │   └── uninstall/
│   ├── sources/
│   │   └── install/
│   └── wazuh/
│       ├── aio/
│       └── package/

Replaced the ComponentType based classes with a single ProvisionHandler class that is responsible for the component privisioning method definition and validations.

class ProvisionHandler:
    _base_templates_path = Path(__file__).parent / 'playbooks'
    _actions = ['install', 'uninstall']
    _methods = ['package', 'aio', 'dependencies', 'sources']

    def __init__(self, component_info: ComponentInfo, action: str, method: str) -> None:
        if not action in self._actions:
            raise ValueError(f"Unsupported action: {action}")
        if not method in self._methods:
            raise ValueError(f"Unsupported method: {method}")
        if not "wazuh" in component_info.component and method.lower() == 'aio':
            raise ValueError(f"AIO actions is only supported for Wazuh components.")

        # We cant uninstall from sources.
        if action == "uninstall" and method.lower() == "sources":
            logger.debug(f"Uninstall from sources not supported. Using package.")
            method = "package"

        self.action = action.lower()
        self.method = method.lower()
        self.component_info = component_info
        self.templates_path = self._get_templates_path()
        self.templates_order = self._get_templates_order()
        self.variables_dict = self._generate_dict()

    def _get_templates_path(self) -> str:
        # If the component is wazuh, we need to change the templates path.
        if "wazuh" in self.component_info.component:
            self._base_templates_path = f'{self._base_templates_path}/wazuh'

        return f"{self._base_templates_path}/{self.method}/{self.action}"

    def _get_templates_order(self) -> list[str]:
        """
        Get the order of the templates to be executed.

        Returns:
            list[str]: List of templates to be executed.
        """
        if self.method == 'package' and self.action == "install":
            return ["set_repo.j2", "install.j2", "register.j2", "service.j2"]
        elif self.method == 'aio':
            return ["download.j2", f"{self.action}.j2"]

        return []

    def _generate_dict(self) -> dict:
        variables = {
            'component': self.component_info.component,
            'version': self.component_info.version,
            'type': self.component_info.type,
            'dependencies': self.component_info.dependencies or None,
            'templates_path': self.templates_path,
            'templates_order': self.templates_order or None
        }

        return variables

@QU3B1M
Copy link
Member

QU3B1M commented Feb 27, 2024

On hold

This issue will remain on hold until the release testing is finished

@wazuhci wazuhci moved this from In progress to On hold in Release 4.9.0 Feb 27, 2024
@wazuhci wazuhci moved this from On hold to In progress in Release 4.9.0 Feb 29, 2024
@QU3B1M
Copy link
Member

QU3B1M commented Mar 1, 2024

Update report

The new structure seems to work properly, now I´m facing an error at the python installation from sources, punctually at the "set default python version" step

TASK [Set default python to 3.10.0] ********************************************
fatal: [192.168.57.2]: FAILED! => changed=true 
  cmd: |-
    update-alternatives --install /usr/local/bin/python3 python3 /usr/bin/python3.10.0 1
    update-alternatives --install /usr/local/bin/pip3 pip3 /usr/bin/pip3.10.0 1
  delta: '0:00:00.014980'
  end: '2024-03-01 17:15:51.401133'
  msg: non-zero return code
  rc: 2
  start: '2024-03-01 17:15:51.386153'
  stderr: |-
    update-alternatives: error: alternative path /usr/bin/python3.10.0 doesn't exist
    update-alternatives: error: alternative path /usr/bin/pip3.10.0 doesn't exist
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

@QU3B1M
Copy link
Member

QU3B1M commented Mar 4, 2024

Update report

  • Dynamic python version installation from sources is successful.
    TASK [Set default python to 3.10.11] *******************************************
    changed: [192.168.57.2] => changed=true 
      cmd: |-
        sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.10 1
      delta: '0:00:00.011521'
      end: '2024-03-04 12:54:24.605422'
      msg: ''
      rc: 0
      start: '2024-03-04 12:54:24.593901'
      stderr: ''
      stderr_lines: <omitted>
      stdout: 'update-alternatives: using /usr/bin/python3.10 to provide /usr/bin/python3 (python3) in auto mode'
      stdout_lines: <omitted>
    
    PLAY RECAP *********************************************************************
    192.168.57.2               : ok=9    changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
    
    [2024-03-04 09:54:25] [INFO] [43479] [ThreadPoolExecutor-0_0] [workflow_engine]: [provision-install-linux-ubuntu-22.04-amd64] Finished task in 348.77 seconds.
  • Test dependencies installation in the rest of the systems (WIP)

@QU3B1M
Copy link
Member

QU3B1M commented Mar 6, 2024

Update progess

  • Fix python installation from sources, tested on Debian, CentOS and Oracle
  • Add python packages provision (install and uninstall) using pip

    The installation of the packages could be directly using the package name or with the requirements.txt
    A task that using the pip provision would look something like this

    - task: "provision-install-{agent}"
      description: "Provision resources for the {agent} agent."
      do:
        this: process
        with:
          path: python3
          args:
            - modules/provision/main.py
            - inventory: "{working-dir}/agent-{agent}/inventory.yaml"
            - dependencies:
              - agent: "{working-dir}/agent-{agent}/inventory.yaml"
            - install:
              - component: curl
              - component: python
                type: source
                version: "{python-version}"
              - component: virtualenv  # Installation of one package.
                type: pip
              - component: deps/requirements.txt  # Installation from requirements.
                type: pip
              - component: pyyaml  # Installation of one package with version.
                type: pip
                version: 6.0.1
      depends-on:
        - "allocate-{agent}"
      foreach:
        - variable: agents-os
          as: agent

@QU3B1M
Copy link
Member

QU3B1M commented Mar 7, 2024

The implementation of the sources installation method and the personalized python version brought some problems with several linux distros that uses an specific python version by default. It was discussed with @fcaffieri and @pro-akim who faced a similar issue, we decided to give up that installation method and only support (at least for this iteration) the installation by package manager for the non-wazuh packages.

Packages installation on supported OS status

OS Result Full log
Debian 10 🟢 output.log
Debian 11 🟢 output.log
Debian 12 🟢 output.log
Ubuntu 18.04 🟢 output.log
Ubuntu 20.04 🟢 output.log
Ubuntu 22.04 🟢 output.log
Centos 7 🔴 output.log

@QU3B1M
Copy link
Member

QU3B1M commented Mar 8, 2024

Update report

Update the provision module to support all the required systems.
Results of wazuh-manager and wazuh-agent installation on the different OS.

Amazon Linux was repeated on the list,.

OS Result Full log
AmazonLinux 2 🟢 output.log
Centos 7 🟢 output.log
Centos 8 🟢 output.log
Debian 10 🟢 output.log
Debian 11 🟢 output.log
Debian 12 🟢 output.log
OpenSuse 15 🟢 output.log
OpenSuse Tumbleweed 🟢 output.log
Oracle 9 🟢 output.log
Ubuntu 18.04 🟢 output.log
Ubuntu 20.04 🟢 output.log
Ubuntu 22.04 🟢 output.log
RedHat 7 🔴 output.log
RedHat 8 🟢 output.log
RedHat 9 🟢 output.log

@QU3B1M
Copy link
Member

QU3B1M commented Mar 9, 2024

Completed the executions of all the required OS, RedHat 7 raised several errors, some were fixed, but the execution is still with failures. It was discussed with @fcaffieri and we decided to generate an AMI of a fixed RedHat 7 in a future iteration, but for now, this OS will remain unsuported by the provision module.

The errors on RedHat 7 are related to the packages repository of the OS.

@wazuhci wazuhci moved this from In progress to Pending review in Release 4.9.0 Mar 9, 2024
@wazuhci wazuhci moved this from Pending review to In review in Release 4.9.0 Mar 11, 2024
@wazuhci wazuhci moved this from In review to Pending final review in Release 4.9.0 Mar 11, 2024
@wazuhci wazuhci moved this from Pending final review to In progress in Release 4.9.0 Mar 11, 2024
@QU3B1M
Copy link
Member

QU3B1M commented Mar 11, 2024

Rename AIO installation to Assistant for clarity.

@wazuhci wazuhci moved this from In progress to Pending final review in Release 4.9.0 Mar 11, 2024
@fcaffieri fcaffieri removed the level/subtask Subtask issue label Mar 18, 2024
@fcaffieri fcaffieri added the level/task Task issue label Mar 18, 2024
@fcaffieri fcaffieri changed the title DTT1 - Iteration 3 - Provision module - Install required dependencies on each supported OS DTT2 - Iteration 1 - Provision module - Install required dependencies on each supported OS Mar 18, 2024
@fcaffieri fcaffieri changed the title DTT2 - Iteration 1 - Provision module - Install required dependencies on each supported OS DTT1 - Iteration 3 - Provision module - Install required dependencies on each supported OS Mar 18, 2024
@fcaffieri fcaffieri changed the title DTT1 - Iteration 3 - Provision module - Install required dependencies on each supported OS DTT2 - Iteration 1 - Provision module - Install required dependencies on each supported OS Mar 18, 2024
@fcaffieri fcaffieri changed the title DTT2 - Iteration 1 - Provision module - Install required dependencies on each supported OS DTT1 - Iteration 3 - Provision module - Install required dependencies on each supported OS Mar 18, 2024
@wazuhci wazuhci moved this from Pending final review to In progress in Release 4.9.0 Mar 19, 2024
@wazuhci wazuhci moved this from In progress to On hold in Release 4.9.0 Mar 19, 2024
@QU3B1M
Copy link
Member

QU3B1M commented Mar 20, 2024

On hold

This issue will remain in On Hold status until we discuss about changes on the allocator module that are now allowing us to initialize AWS machines without some labels.

@rauldpm
Copy link
Member

rauldpm commented Mar 20, 2024

Meet defined: Sync allocation module - Wednesday, March 20⋅8:00 – 8:30pm ESP
Changed status to Blocked for third-party

@wazuhci wazuhci moved this from On hold to Blocked in Release 4.9.0 Mar 20, 2024
@rauldpm
Copy link
Member

rauldpm commented Mar 21, 2024

Meet resume

  • Error handling: use raise instead of sys.exit
  • Termination_date tag/parameter: the value of this parameter will be the number of expiration days instead of a fixed date.
  • Tag/parameter issue:
    • Remove mandatory tag in AWS (mainly in wazuh-qa)
    • The parameter will not be mandatory in the module.
    • SSH key:
      • In case the issue parameter does not come in the call to the module, the instance_name or some other parameter will be used to create the name of the key.
      • A key will be created per instance that is launched for the QA workflow executions.

@wazuhci wazuhci moved this from Blocked to On hold in Release 4.9.0 Mar 21, 2024
@wazuhci wazuhci moved this from On hold to In progress in Release 4.9.0 Mar 21, 2024
@QU3B1M
Copy link
Member

QU3B1M commented Mar 21, 2024

Issue resumed

Until the DevOps teams develops the final solution mentioned in the previous comment, we will work on this issue by using a workarround that let us use the allocation module with some minor changes.

@QU3B1M
Copy link
Member

QU3B1M commented Mar 23, 2024

Update report

AWS Execution

Some fixes were applied in order to ensure the correct functionality in AWS provided VMs

OS Result
AmazonLinux 2 🟢
Centos 7 🟢
Centos 8 🟢
Debian 10 🟢
Debian 11 🟢
Debian 12 🟢
Suse 15 🟢
Oracle 9 🟢
Ubuntu 18.04 🟢
Ubuntu 20.04 🟢
Ubuntu 22.04 🟢
RedHat 7 🟢
RedHat 8 🟢
RedHat 9 🟢

@wazuhci wazuhci moved this from In progress to Pending final review in Release 4.9.0 Mar 25, 2024
@fcaffieri
Copy link
Member Author

LGTM

@wazuhci wazuhci moved this from Pending final review to Done in Release 4.9.0 Mar 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants