Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The allocator module fails to allocate instances in AWS due to volume size #5617

Closed
rauldpm opened this issue Jul 29, 2024 · 5 comments · Fixed by #5625
Closed

The allocator module fails to allocate instances in AWS due to volume size #5617

rauldpm opened this issue Jul 29, 2024 · 5 comments · Fixed by #5625
Assignees
@c-bordon @Enaraque
Labels
level/task Task issue type/bug

Comments

@rauldpm
Copy link
Member

rauldpm commented Jul 29, 2024 •
edited
Loading

Description

We found the following error when creating instances for Linux endpoints using the Allocator module and AWS resources

botocore.exceptions.ClientError: An error occurred (InvalidBlockDeviceMapping) when calling the RunInstances operation: Volume of size 20GB is smaller than snapshot 'snap-0e449fc6854107ee9', expect size>=30GB

This is happening for

  • linux-debian-12-amd64
  • linux-debian-12-arm64
  • linux-oracle-9-amd64

It is necessary to investigate the error and fix the volume size so those systems can be used in AWS

Related
@rauldpm rauldpm mentioned this issue Jul 29, 2024
Closed
1 task
@wazuhci wazuhci moved this to Backlog in Release 4.9.0 Jul 29, 2024
@juliamagan juliamagan mentioned this issue Jul 29, 2024
Closed
@Enaraque Enaraque self-assigned this Jul 30, 2024
@c-bordon c-bordon self-assigned this Jul 30, 2024
@c-bordon
Copy link
Member

Update report

A change is made to obtain the default storage requested by the AMI and compare it with the storage defined by us. If the storage requested by the AMI is greater than the storage we defined, we use the storage requested by the AMI, in this way, we ensure that the instance can be created correctly.

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider aws --size medium --label-team devops --label-termination-date 1d --composite-name linux-debian-12-amd64
[2024-07-30 14:45:05] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-07-30 14:45:06] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-07-30 14:45:06] [DEBUG] ALLOCATOR: Generating new key pair
[2024-07-30 14:45:06] [DEBUG] ALLOCATOR: Creating base directory: /tmp/wazuh-qa/AWS-76AC6808-D874-4198-8C4C-3914236DE1B7
Config storage: 20 is less than the AMI storage: 30.
[2024-07-30 14:45:26] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-76AC6808-D874-4198-8C4C-3914236DE1B7 directory to /tmp/wazuh-qa/i-055b513de26cf97b0
[2024-07-30 14:45:26] [INFO] ALLOCATOR: Instance i-055b513de26cf97b0 created.
[2024-07-30 14:45:28] [INFO] ALLOCATOR: Instance i-055b513de26cf97b0 started.
[2024-07-30 14:45:28] [INFO] ALLOCATOR: The inventory file generated at /tmp/wazuh-qa/i-055b513de26cf97b0/inventory.yaml
[2024-07-30 14:45:28] [INFO] ALLOCATOR: The track file generated at /tmp/wazuh-qa/i-055b513de26cf97b0/track.yaml
[2024-07-30 14:45:28] [WARNING] ALLOCATOR: Error on attempt 1 of 30: [Errno None] Unable to connect to port 2200 on 54.162.193.132
[2024-07-30 14:46:00] [INFO] ALLOCATOR: SSH connection successful.
[2024-07-30 14:46:00] [INFO] ALLOCATOR: Instance i-055b513de26cf97b0 created successfully.
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ cat /tmp/wazuh-qa/i-055b513de26cf97b0/inventory.yaml
ansible_connection: ssh
ansible_host: ec2-54-162-193-132.compute-1.amazonaws.com
ansible_port: 2200
ansible_ssh_common_args: -o StrictHostKeyChecking=no
ansible_ssh_private_key_file: /tmp/wazuh-qa/i-055b513de26cf97b0/debian-12-amd64-key-4411
ansible_user: admin
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ ssh -i /tmp/wazuh-qa/i-055b513de26cf97b0/debian-12-amd64-key-4411 -o StrictHostKeyChecking=no -p 2200 [email protected]
Warning: Permanently added '[ec2-54-162-193-132.compute-1.amazonaws.com]:2200' (ED25519) to the list of known hosts.
Linux ip-172-31-32-230 6.1.0-13-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Jul  1 14:33:46 2024 from 79.117.226.210
admin@ip-172-31-32-230:~$ sudo su -
root@ip-172-31-32-230:~# df -h
Filesystem       Size  Used Avail Use% Mounted on
udev             1.9G     0  1.9G   0% /dev
tmpfs            389M  456K  388M   1% /run
/dev/nvme0n1p1    30G  1.1G   27G   4% /
tmpfs            1.9G     0  1.9G   0% /dev/shm
tmpfs            5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p15  124M   12M  113M  10% /boot/efi
tmpfs            389M     0  389M   0% /run/user/1000

@wazuhci wazuhci moved this from Backlog to In progress in Release 4.9.0 Jul 30, 2024
@c-bordon
Copy link
Member

Windows tests

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider aws --size medium --label-team devops --label-termination-date 1d --composite-name windows-desktop-10-amd64
[2024-07-30 14:53:27] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-07-30 14:53:28] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-07-30 14:53:28] [DEBUG] ALLOCATOR: Creating temp directory: /tmp/wazuh-qa/AWS-F0BC3431-9158-4FC4-AE6A-6EC4272895D3
Config storage: 20 is less than the AMI storage: 40.
[2024-07-30 14:53:46] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-F0BC3431-9158-4FC4-AE6A-6EC4272895D3 directory to /tmp/wazuh-qa/i-07078c248024bc99e
[2024-07-30 14:53:46] [INFO] ALLOCATOR: Instance i-07078c248024bc99e created.
[2024-07-30 14:53:48] [INFO] ALLOCATOR: Instance i-07078c248024bc99e started.
[2024-07-30 14:53:48] [INFO] ALLOCATOR: The inventory file generated at /tmp/wazuh-qa/i-07078c248024bc99e/inventory.yaml
[2024-07-30 14:53:48] [INFO] ALLOCATOR: The track file generated at /tmp/wazuh-qa/i-07078c248024bc99e/track.yaml
[2024-07-30 14:54:18] [WARNING] ALLOCATOR: Error on attempt 1 of 30: HTTPSConnectionPool(host='ec2-44-200-82-183.compute-1.amazonaws.com', port=5986): Max retries exceeded with url: /wsman (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7045fd714d30>, 'Connection to ec2-44-200-82-183.compute-1.amazonaws.com timed out. (connect timeout=30)'))
[2024-07-30 14:55:18] [WARNING] ALLOCATOR: Error on attempt 2 of 30: HTTPSConnectionPool(host='ec2-44-200-82-183.compute-1.amazonaws.com', port=5986): Max retries exceeded with url: /wsman (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7045fd7172b0>, 'Connection to ec2-44-200-82-183.compute-1.amazonaws.com timed out. (connect timeout=30)'))
[2024-07-30 14:56:18] [WARNING] ALLOCATOR: Error on attempt 3 of 30: HTTPSConnectionPool(host='ec2-44-200-82-183.compute-1.amazonaws.com', port=5986): Max retries exceeded with url: /wsman (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7045fd717ca0>, 'Connection to ec2-44-200-82-183.compute-1.amazonaws.com timed out. (connect timeout=30)'))
[2024-07-30 14:56:49] [WARNING] ALLOCATOR: Error on attempt 4 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 14:57:19] [WARNING] ALLOCATOR: Error on attempt 5 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 14:57:50] [WARNING] ALLOCATOR: Error on attempt 6 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 14:58:20] [WARNING] ALLOCATOR: Error on attempt 7 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 14:58:51] [WARNING] ALLOCATOR: Error on attempt 8 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 14:59:21] [WARNING] ALLOCATOR: Error on attempt 9 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 14:59:51] [WARNING] ALLOCATOR: Error on attempt 10 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:00:22] [WARNING] ALLOCATOR: Error on attempt 11 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:00:52] [WARNING] ALLOCATOR: Error on attempt 12 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:01:22] [WARNING] ALLOCATOR: Error on attempt 13 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:01:53] [WARNING] ALLOCATOR: Error on attempt 14 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:02:23] [WARNING] ALLOCATOR: Error on attempt 15 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:02:53] [WARNING] ALLOCATOR: Error on attempt 16 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:03:24] [WARNING] ALLOCATOR: Error on attempt 17 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:03:54] [WARNING] ALLOCATOR: Error on attempt 18 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:04:24] [WARNING] ALLOCATOR: Error on attempt 19 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:04:55] [WARNING] ALLOCATOR: Error on attempt 20 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:05:25] [WARNING] ALLOCATOR: Error on attempt 21 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:05:56] [WARNING] ALLOCATOR: Error on attempt 22 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:06:56] [WARNING] ALLOCATOR: Error on attempt 23 of 30: HTTPSConnectionPool(host='ec2-44-200-82-183.compute-1.amazonaws.com', port=5986): Max retries exceeded with url: /wsman (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7045fd717790>, 'Connection to ec2-44-200-82-183.compute-1.amazonaws.com timed out. (connect timeout=30)'))
[2024-07-30 15:07:26] [WARNING] ALLOCATOR: Error on attempt 24 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:07:56] [WARNING] ALLOCATOR: Error on attempt 25 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:08:27] [WARNING] ALLOCATOR: Error on attempt 26 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:08:57] [WARNING] ALLOCATOR: Error on attempt 27 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:09:49] [WARNING] ALLOCATOR: Error on attempt 28 of 30: 
[2024-07-30 15:10:41] [WARNING] ALLOCATOR: Error on attempt 29 of 30: 
[2024-07-30 15:11:18] [INFO] ALLOCATOR: WinRM connection successful.
[2024-07-30 15:11:18] [INFO] ALLOCATOR: Instance i-07078c248024bc99e created successfully

Screenshot_20240730_151304

@c-bordon
Copy link
Member

macOS test

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider aws --size medium --label-team devops --label-termination-date 1d --composite-name macos-sonoma-14-amd64
[2024-07-30 15:02:02] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-07-30 15:02:03] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-07-30 15:02:03] [DEBUG] ALLOCATOR: Generating new key pair
[2024-07-30 15:02:03] [DEBUG] ALLOCATOR: Creating base directory: /tmp/wazuh-qa/AWS-BB2689E8-8E98-4818-A1E7-357CAD0D3FC9
[2024-07-30 15:02:05] [INFO] ALLOCATOR: Creating dedicated host: sonoma-14-amd64-Host-amd64
[2024-07-30 15:02:06] [INFO] ALLOCATOR: Dedicated host created: h-08062808a74443fb6
[2024-07-30 15:02:24] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-BB2689E8-8E98-4818-A1E7-357CAD0D3FC9 directory to /tmp/wazuh-qa/i-0e97802433b2d874d
[2024-07-30 15:02:24] [INFO] ALLOCATOR: Instance i-0e97802433b2d874d created.
[2024-07-30 15:02:26] [INFO] ALLOCATOR: Instance i-0e97802433b2d874d started.
[2024-07-30 15:02:26] [INFO] ALLOCATOR: The inventory file generated at /tmp/wazuh-qa/i-0e97802433b2d874d/inventory.yaml
[2024-07-30 15:02:26] [INFO] ALLOCATOR: The track file generated at /tmp/wazuh-qa/i-0e97802433b2d874d/track.yaml
[2024-07-30 15:04:40] [WARNING] ALLOCATOR: Error on attempt 1 of 30: [Errno 110] Connection timed out
[2024-07-30 15:07:24] [WARNING] ALLOCATOR: Error on attempt 2 of 30: [Errno 110] Connection timed out
[2024-07-30 15:07:54] [WARNING] ALLOCATOR: Error on attempt 3 of 30: [Errno None] Unable to connect to port 2200 on 52.207.191.134
[2024-07-30 15:08:27] [INFO] ALLOCATOR: SSH connection successful.
[2024-07-30 15:08:27] [INFO] ALLOCATOR: Instance i-0e97802433b2d874d created successfully.

Screenshot_20240730_151247

@c-bordon
Copy link
Member

Debian 12 arm

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider aws --size medium --label-team devops --label-termination-date 1d --composite-name linux-debian-12-arm64
[2024-07-30 15:19:13] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-07-30 15:19:13] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-07-30 15:19:13] [DEBUG] ALLOCATOR: Generating new key pair
[2024-07-30 15:19:13] [DEBUG] ALLOCATOR: Creating base directory: /tmp/wazuh-qa/AWS-595EE7D1-2A01-456A-873B-EDF3D2924797
[2024-07-30 15:19:34] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-595EE7D1-2A01-456A-873B-EDF3D2924797 directory to /tmp/wazuh-qa/i-0ceaf401e8b3e6659
[2024-07-30 15:19:34] [INFO] ALLOCATOR: Instance i-0ceaf401e8b3e6659 created.
[2024-07-30 15:19:35] [INFO] ALLOCATOR: Instance i-0ceaf401e8b3e6659 started.
[2024-07-30 15:19:35] [INFO] ALLOCATOR: The inventory file generated at /tmp/wazuh-qa/i-0ceaf401e8b3e6659/inventory.yaml
[2024-07-30 15:19:35] [INFO] ALLOCATOR: The track file generated at /tmp/wazuh-qa/i-0ceaf401e8b3e6659/track.yaml
[2024-07-30 15:19:43] [WARNING] ALLOCATOR: Error on attempt 1 of 30: [Errno None] Unable to connect to port 2200 on 3.216.126.171
[2024-07-30 15:20:14] [INFO] ALLOCATOR: SSH connection successful.
[2024-07-30 15:20:14] [INFO] ALLOCATOR: Instance i-0ceaf401e8b3e6659 created successfully.

Screenshot_20240730_152038

@c-bordon
Copy link
Member

Oracle Linux 9

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider aws --size medium --label-team devops --label-termination-date 1d --composite-name linux-oracle-9-amd64
[2024-07-30 15:21:31] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-07-30 15:21:32] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-07-30 15:21:32] [DEBUG] ALLOCATOR: Generating new key pair
[2024-07-30 15:21:32] [DEBUG] ALLOCATOR: Creating base directory: /tmp/wazuh-qa/AWS-B8071AC0-CEB2-4E13-86EE-6887D4249970
[2024-07-30 15:21:52] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-B8071AC0-CEB2-4E13-86EE-6887D4249970 directory to /tmp/wazuh-qa/i-0e343abfea62eaff0
[2024-07-30 15:21:52] [INFO] ALLOCATOR: Instance i-0e343abfea62eaff0 created.
[2024-07-30 15:21:53] [INFO] ALLOCATOR: Instance i-0e343abfea62eaff0 started.
[2024-07-30 15:21:54] [INFO] ALLOCATOR: The inventory file generated at /tmp/wazuh-qa/i-0e343abfea62eaff0/inventory.yaml
[2024-07-30 15:21:54] [INFO] ALLOCATOR: The track file generated at /tmp/wazuh-qa/i-0e343abfea62eaff0/track.yaml
[2024-07-30 15:23:03] [INFO] ALLOCATOR: SSH connection successful.
[2024-07-30 15:23:03] [INFO] ALLOCATOR: Instance i-0e343abfea62eaff0 created successfully.

Screenshot_20240730_152324

@c-bordon c-bordon mentioned this issue Jul 30, 2024
Merged
@c-bordon c-bordon linked a pull request Jul 30, 2024 that will close this issue
Fixed custom storage for AMIs #5625
Merged
@wazuhci wazuhci moved this from In progress to Pending final review in Release 4.9.0 Jul 31, 2024
@wazuhci wazuhci moved this from Pending final review to In final review in Release 4.9.0 Jul 31, 2024
@wazuhci wazuhci moved this from In final review to Done in Release 4.9.0 Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@c-bordon c-bordon

@Enaraque Enaraque

Labels
level/task Task issue type/bug
Projects
No open projects
Release 4.9.0
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixed custom storage for AMIs wazuh/wazuh-qa
4 participants
@rauldpm @c-bordon @Enaraque @juliamagan