Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DTT1 - Identify Allocator module resources uniquely #5311

Closed
3 of 4 tasks
rauldpm opened this issue Apr 30, 2024 · 6 comments · Fixed by #5342
Closed
3 of 4 tasks

DTT1 - Identify Allocator module resources uniquely #5311

rauldpm opened this issue Apr 30, 2024 · 6 comments · Fixed by #5342
Assignees
@c-bordon
Labels
level/task Task issue type/enhancement

Comments

@rauldpm
Copy link
Member

rauldpm commented Apr 30, 2024 •
edited by c-bordon
Loading

Description

We need to improve the Allocator module to allow identify each instance uniquely, as currently, each instance is being named with generic data

The instance should be identified in a way to be able to answer the following questions

  • Who created the resource
  • For what task is the resource created

We need to find a way to identify those resources for local and cloud (Jenkins/GitHub) deployments

Required changes

  • The label-issue parameter must be used to set the instance name for all the resources (as in AWS).
  • If the label-issue is not specified, the Allocator module must validate if the instance-name parameter is specified.
  • If none of those parameters were specified, the Allocator module must ask for one of those parameters and exit.
  • The directory name (for the artifacts) must maintain the current UUID.

Related
@wazuhci wazuhci moved this to Triage in Release 4.9.0 May 2, 2024
@wazuhci wazuhci moved this from Triage to Backlog in Release 4.9.0 May 2, 2024
@rauldpm rauldpm mentioned this issue May 2, 2024
Closed
11 tasks
@wazuhci wazuhci moved this from Backlog to In progress in Release 4.9.0 May 3, 2024
@c-bordon
Copy link
Member

c-bordon commented May 6, 2024

On hold by https://github.com/wazuh/internal-devel-requests/issues/1103

@wazuhci wazuhci moved this from In progress to On hold in Release 4.9.0 May 6, 2024
@wazuhci wazuhci moved this from On hold to In progress in Release 4.9.0 May 6, 2024
@c-bordon
Copy link
Member

c-bordon commented May 6, 2024 •
edited
Loading

Testing Vagrant local:

Testing without --instance-name or --label-issue

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 
[2024-05-06 14:09:27] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 70, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 60, in _create_instance
    raise ValueError("Instance name or issue label is required.")
ValueError: Instance name or issue label is required.

Testing with --label-issue

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 --label-issue https://github.com/wazuh/wazuh-qa/issues/5311
[2024-05-06 14:11:35] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-05-06 14:11:35] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-05-06 14:11:35] [DEBUG] ALLOCATOR: Generating new key pair
[2024-05-06 14:11:39] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance.
[2024-05-06 14:11:39] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-9438 created.
[2024-05-06 14:12:23] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-9438 started.
[2024-05-06 14:12:26] [INFO] ALLOCATOR: Inventory file generated at /tmp/wazuh-qa/qa-5311-ubuntu-22.04-9438/inventory.yml
[2024-05-06 14:12:26] [INFO] ALLOCATOR: SSH connection successful.
[2024-05-06 14:12:28] [INFO] ALLOCATOR: Track file generated at /tmp/wazuh-qa/qa-5311-ubuntu-22.04-9438/track.yml
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --action delete --track-output /tmp/wazuh-qa/qa-5311-ubuntu-22.04-9438/track.yml
[2024-05-06 14:13:12] [INFO] ALLOCATOR: Deleting instance from trackfile /tmp/wazuh-qa/qa-5311-ubuntu-22.04-9438/track.yml
[2024-05-06 14:13:12] [DEBUG] ALLOCATOR: Destroying instance qa-5311-ubuntu-22.04-9438
[2024-05-06 14:13:17] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-9438 deleted.

Screenshot_20240506_141251

Testing with --instance-name

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 --instance-name cbordon-testing-ubuntu
[2024-05-06 14:13:52] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-05-06 14:13:52] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-05-06 14:13:52] [DEBUG] ALLOCATOR: Generating new key pair
[2024-05-06 14:13:55] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance.
[2024-05-06 14:13:55] [INFO] ALLOCATOR: Instance cbordon-testing-ubuntu-9104 created.
[2024-05-06 14:14:36] [INFO] ALLOCATOR: Instance cbordon-testing-ubuntu-9104 started.
[2024-05-06 14:14:38] [INFO] ALLOCATOR: Inventory file generated at /tmp/wazuh-qa/cbordon-testing-ubuntu-9104/inventory.yml
[2024-05-06 14:14:38] [INFO] ALLOCATOR: SSH connection successful.
[2024-05-06 14:14:41] [INFO] ALLOCATOR: Track file generated at /tmp/wazuh-qa/cbordon-testing-ubuntu-9104/track.yml
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --action delete --track-output /tmp/wazuh-qa/cbordon-testing-ubuntu-9104/track.yml 
[2024-05-06 14:17:27] [INFO] ALLOCATOR: Deleting instance from trackfile /tmp/wazuh-qa/cbordon-testing-ubuntu-9104/track.yml
[2024-05-06 14:17:27] [DEBUG] ALLOCATOR: Destroying instance cbordon-testing-ubuntu-9104
[2024-05-06 14:17:32] [INFO] ALLOCATOR: Instance cbordon-testing-ubuntu-9104 deleted.

Screenshot_20240506_141701

@c-bordon
Copy link
Member

c-bordon commented May 6, 2024

Testing validation for github Wazuh repository

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 --label-issue https://github.com/cbordon/wazuh-qa/issues/5311
[2024-05-06 14:18:44] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 71, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 55, in _create_instance
    raise ValueError(f"The issue label was not provided or is of incorrect format, example: https://github.com/wazuh/<repository>/issues/<issue-number>")
ValueError: The issue label was not provided or is of incorrect format, example: https://github.com/wazuh/<repository>/issues/<issue-number>
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 --label-issue https://github.com/wazuh/wazuh-qa/pull/5311
[2024-05-06 14:18:57] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 71, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 55, in _create_instance
    raise ValueError(f"The issue label was not provided or is of incorrect format, example: https://github.com/wazuh/<repository>/issues/<issue-number>")
ValueError: The issue label was not provided or is of incorrect format, example: https://github.com/wazuh/<repository>/issues/<issue-number>

@c-bordon
Copy link
Member

c-bordon commented May 6, 2024

Test in AWS

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider aws --size micro --composite-name linux-ubuntu-22.04-amd64 --label-issue https://github.com/wazuh/wazuh-qa/issues/5311 --label-team devops --label-termination-date 1d
[2024-05-06 14:21:16] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-05-06 14:21:16] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-05-06 14:21:16] [DEBUG] ALLOCATOR: Generating new key pair
[2024-05-06 14:21:16] [DEBUG] ALLOCATOR: Creating base directory: /tmp/wazuh-qa/AWS-EEB0F126-993C-4769-8082-68753C90CAA0
[2024-05-06 14:21:51] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-EEB0F126-993C-4769-8082-68753C90CAA0 directory to /tmp/wazuh-qa/i-0234e7c80116e3c61
[2024-05-06 14:21:51] [INFO] ALLOCATOR: Instance i-0234e7c80116e3c61 created.
[2024-05-06 14:21:53] [INFO] ALLOCATOR: Instance i-0234e7c80116e3c61 started.
[2024-05-06 14:21:53] [INFO] ALLOCATOR: Inventory file generated at /tmp/wazuh-qa/i-0234e7c80116e3c61/inventory.yml
[2024-05-06 14:21:58] [WARNING] ALLOCATOR: Error on attempt 1 of 30: [Errno None] Unable to connect to port 2200 on 3.88.26.8
[2024-05-06 14:22:29] [INFO] ALLOCATOR: SSH connection successful.
[2024-05-06 14:22:29] [INFO] ALLOCATOR: Track file generated at /tmp/wazuh-qa/i-0234e7c80116e3c61/track.yml
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --action delete --track-output /tmp/wazuh-qa/i-0234e7c80116e3c61/track.yml 
[2024-05-06 14:24:34] [INFO] ALLOCATOR: Deleting instance from trackfile /tmp/wazuh-qa/i-0234e7c80116e3c61/track.yml
[2024-05-06 14:24:35] [DEBUG] ALLOCATOR: Deleting credentials: qa-5311-ubuntu-22.04-key-5195
[2024-05-06 14:25:22] [INFO] ALLOCATOR: Instance i-0234e7c80116e3c61 deleted.

Screenshot_20240506_142419

@c-bordon
Copy link
Member

c-bordon commented May 6, 2024

Testing macOS ARM

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name macos-ventura-13.4.1-arm64 --instance-name cbordon-testing-macos
[2024-05-06 14:37:35] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-05-06 14:37:40] [INFO] ALLOCATOR: macStadium server has less than 2 VMs running, deploying in this host.
[2024-05-06 14:37:40] [DEBUG] ALLOCATOR: Checking if instance directory exists on remote host
[2024-05-06 14:37:44] [DEBUG] ALLOCATOR: Creating instance directory on remote host
[2024-05-06 14:37:47] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-05-06 14:37:47] [DEBUG] ALLOCATOR: Generating new key pair
[2024-05-06 14:37:51] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance.
[2024-05-06 14:37:56] [INFO] ALLOCATOR: Instance cbordon-testing-macos-8137 created.
[2024-05-06 14:39:20] [INFO] ALLOCATOR: Instance cbordon-testing-macos-8137 started.
[2024-05-06 14:39:38] [INFO] ALLOCATOR: Inventory file generated at /tmp/wazuh-qa/cbordon-testing-macos-8137/inventory.yml
[2024-05-06 14:39:40] [INFO] ALLOCATOR: SSH connection successful.
[2024-05-06 14:39:51] [INFO] ALLOCATOR: Track file generated at /tmp/wazuh-qa/cbordon-testing-macos-8137/track.yml
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --action delete --track-output /tmp/wazuh-qa/cbordon-testing-macos-8137/track.yml 
[2024-05-06 14:41:13] [INFO] ALLOCATOR: Deleting instance from trackfile /tmp/wazuh-qa/cbordon-testing-macos-8137/track.yml
[2024-05-06 14:41:14] [DEBUG] ALLOCATOR: Destroying instance cbordon-testing-macos-8137
[2024-05-06 14:41:28] [DEBUG] ALLOCATOR: Deleting remote directory /Users/jenkins/testing/cbordon-testing-macos-8137
[2024-05-06 14:41:31] [DEBUG] ALLOCATOR: Killing remote process on port 43220
[2024-05-06 14:41:38] [INFO] ALLOCATOR: Instance cbordon-testing-macos-8137 deleted.

Remote host

sh-3.2# ls -la
total 0
drwxr-xr-x   8 jenkins  staff   256 May  6 10:37 .
drwxr-x---+ 34 jenkins  staff  1088 May  6 07:10 ..
drwxr-xr-x   4 root     staff   128 Jan 15 04:56 689
drwxr-xr-x   4 jenkins  staff   128 May  6 10:37 cbordon-testing-macos-8137
drwxr-xr-x   4 root     staff   128 Apr 19 07:07 macos-testing
drwxr-xr-x   4 jenkins  staff   128 Apr 26 10:57 sign-test
drwxr-xr-x   2 root     staff    64 Feb 21 05:17 testing_dir
drwxr-xr-x   4 root     staff   128 Aug 22  2023 wpk-testing
sh-3.2# prlctl list --all
UUID                                    STATUS       IP_ADDR         NAME
{e2b36366-bc32-4eba-8d56-213d0b1db078}  running      -               Test_e2e_system_283_20240506170351_macos_1400_7_0
{7c5230be-ef0d-42f6-a37e-5c5989c1af13}  running      -               cbordon-testing-macos-8137

@c-bordon
Copy link
Member

c-bordon commented May 6, 2024

Forcing instance dir concurrency

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 --label-issue https://github.com/wazuh/wazuh-qa/issues/5311
[2024-05-06 14:45:03] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-05-06 14:45:03] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-05-06 14:45:03] [DEBUG] ALLOCATOR: Generating new key pair
[2024-05-06 14:45:07] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance.
[2024-05-06 14:45:07] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-3347 created.
[2024-05-06 14:45:46] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-3347 started.
[2024-05-06 14:45:49] [INFO] ALLOCATOR: Inventory file generated at /tmp/wazuh-qa/qa-5311-ubuntu-22.04-3347/inventory.yml
[2024-05-06 14:45:49] [INFO] ALLOCATOR: SSH connection successful.
[2024-05-06 14:45:51] [INFO] ALLOCATOR: Track file generated at /tmp/wazuh-qa/qa-5311-ubuntu-22.04-3347/track.yml
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 --label-issue https://github.com/wazuh/wazuh-qa/issues/5311
[2024-05-06 14:48:14] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-05-06 14:48:14] [DEBUG] ALLOCATOR: Instance directory /tmp/wazuh-qa/qa-5311-ubuntu-22.04-3347 already exists. Assigning new suffix.
[2024-05-06 14:48:14] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-05-06 14:48:14] [DEBUG] ALLOCATOR: Generating new key pair
[2024-05-06 14:48:18] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance.
[2024-05-06 14:48:18] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-3126 created.
[2024-05-06 14:48:59] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-3126 started.
[2024-05-06 14:49:02] [INFO] ALLOCATOR: Inventory file generated at /tmp/wazuh-qa/qa-5311-ubuntu-22.04-3126/inventory.yml
[2024-05-06 14:49:02] [INFO] ALLOCATOR: SSH connection successful.
[2024-05-06 14:49:05] [INFO] ALLOCATOR: Track file generated at /tmp/wazuh-qa/qa-5311-ubuntu-22.04-3126/track.yml

In the case where an attempt was made to create 2 or more instances with the same characteristics, that is, the same issue and the same system and version, or the same --instance-name, the module will throw the following debug message and will retry creating the directory by appealing to that In the second attempt there was no concurrency, taking into account that the suffix we added is a random number between 0000 and 9999, so the probability of the same number being repeated is low.

[2024-05-06 14:48:14] [DEBUG] ALLOCATOR: Instance directory /tmp/wazuh-qa/qa-5311-ubuntu-22.04-3347 already exists. Assigning new suffix.

@c-bordon c-bordon mentioned this issue May 6, 2024
Merged
@c-bordon c-bordon linked a pull request May 6, 2024 that will close this issue
Force instance name to be defined in vagrant deploy #5342
Merged
@wazuhci wazuhci moved this from In progress to Pending review in Release 4.9.0 May 6, 2024
@wazuhci wazuhci moved this from Pending review to Done in Release 4.9.0 May 7, 2024
@davidcr01 davidcr01 mentioned this issue May 9, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@c-bordon c-bordon

Labels
level/task Task issue type/enhancement
Projects
No open projects
Release 4.9.0
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Force instance name to be defined in vagrant deploy wazuh/wazuh-qa
3 participants
@rauldpm @teddytpc1 @c-bordon