Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Research current pipelines documentation and identify the ones to migrate #5599

Closed
3 tasks done
QU3B1M opened this issue Jul 22, 2024 · 5 comments
Closed
3 tasks done

Research current pipelines documentation and identify the ones to migrate #5599

QU3B1M opened this issue Jul 22, 2024 · 5 comments
Assignees
@pcapellan @QU3B1M
Labels
level/task Task issue type/research

Comments

@QU3B1M
Copy link
Member

QU3B1M commented Jul 22, 2024 •
edited
Loading

Description

This issue aims to read and analyze the current Jenkins pipelines documentation, identifying which ones must be migrated for v4.10.0 and listing their code location and dependencies.

We won't migrate any non-maintained pipelines, only the essential pipelines, and justify. Also, we have to validate to not lose coverage.

Tasks

  • Analyze the Jenkins pipelines documentation
  • Identify pipelines required for migration
  • Document the selected pipelines with all the related code and dependencies

Conclusion

All the pipelines were analyzed, and considering the migrations to GitHub Actions (GHA) that are planned or in progress by the development teams, and the DTT development by the QA team, this is the list of pipelines that are candidates to be removed or migrated to the new platforms.

Note

The column 'Maintain v4.X' is meant to identify the pipelines that must be maintained in the current Jenkins environment for the v4.X.X versions.

Pipeline name In use Responsible team Move to Maintain v4.X Last code update Last run Details
Build_ALAS Yes CppServer GHA No 15/05/2023 Present Package building, move to actions on correspondent repository.
Build_framework_python No PyServer GHA No 18/09/2023 26/01/2024 Package building, move to actions on correspondent repository.
Build_last_tag No --- --- No 16/01/2020 07/01/2020 Legacy pipeline. Must be removed.
Build_MSU Yes CppServer GHA No 30/10/2023 Present Package building, move to actions on correspondent repository.
Build_NVD Yes CppServer GHA No 18/07/2023 Present Package building, move to actions on correspondent repository.
Build_packages_from_branch No --- --- No 27/02/2020 08/08/2019 Legacy pipeline. Must be removed.
Packages_builder Yes CppServer - Agent - Indexer - Dashboard GHA No 20/02/2024 Present Package building, move to actions on correspondent repository.
Packages_builder_macos Yes Agent GHA No 30/10/2023 Present Package building, move to actions on correspondent repository.
Packages_builder_nightly No --- --- No --- --- Legacy pipeline. Must be removed.
Packages_builder_solaris Yes Agent GHA No 12/05/2022 19/07/2024 Package building, move to actions on correspondent repository.
Packages_builder_special Yes Agent GHA No 25/01/2024 22/07/2024 Package building, move to actions on correspondent repository.
Packages_builder_tier Yes CppServer - Agent - Indexer - Dashboard --- No 06/06/2023 Present Triggers multiple packages build, probably not required on GHA.
Packages_builder_win Yes Agent GHA No 08/06/2023 Present Package building, move to actions on correspondent repository.
Packages_uploader_repository Yes QA - Release New Jenkins Yes 09/07/2024 17/07/2024 Refactor on Packages Redesign.
PR_FIM No CppServer --- No 16/12/2020 --- Legacy pipeline. Must be removed.
PR_http_request Yes CppServer GHA No 27/09/2022 19/04/2024 Move to actions on correspondent repository.
PR_intelligence_platform No Content - CppServer GHA No 15/02/2023 12/09/2023 Move to actions on correspondent repository.
PR_jenkins (disabled) No --- --- No 08/09/2021 24/06/2020 Legacy pipeline. Must be removed.
PR_packages Yes QA - CppServer - Agent - Indexer - Dashboard --- No 24/03/2023 Present Wont be required in the new architecture.
PR_ruleset (disabled) No --- --- No 18/06/2021 09/12/2020 Legacy pipeline. Must be removed.
PR_wazuh Yes CppServer - Agent GHA No 09/02/2024 Present Move to actions on correspondent repository.
PR_wazuh_feed (disabled) No --- --- No 25/11/2021 11/02/2022 Legacy pipeline. Must be removed.
Procedure_ecs_task Yes QA New Jenkins Yes 24/04/2024 22/07/2024 Move to new jenkins, maybe replaced by Allocator.
Procedure_ecs_task_tier Yes QA New Jenkins Yes 15/09/2023 Present Move to new jenkins, could be merged with previous.
Test_code_vulnerabilities No PyServer GHA No 18/05/2023 N/A Move to actions on correspondent repository.
Test_compilation No QA --- No 30/05/2023 31/12/2023 Legacy pipeline. Covers compilation tests for main components, could be refactored.
Test_dependencies_vulnerabilities Yes QA --- No 18/05/2023 29/05/2023 Legacy pipeline. GitHub already provides a solution for this.
Test_e2e_system Yes QA New Jenkins Yes 18/06/2024 Present Move to new jenkins.
Test_feed(removed) No --- --- No 11/02/2022 25/11/2021 Legacy pipeline. Must be removed.
Test_FIM No QA --- No 29/07/2021 N/A Legacy pipeline. Covers FIM performance tests.
Test_install Yes QA DTT No 01/12/2023 Present Replaced by new DTT suite. Covers: version, process, log errors, conf, connection.
Test_install_special No QA DTT No 15/09/2022 02/05/2022 Replaced by new DTT suite. Covers: version, process, log errors, conf, connection.
Test_install_tier Yes QA DTT No 01/12/2023 Present Executes a set of tests install, replaced by DTT
Test_integration Yes CppServer - Agent - PyServer GHA No 04/06/2024 Present Move to actions on correspondent repository.
Test_integration_endpoints Yes PyServer GHA No 25/06/2024 Present Move to actions on correspondent repository.
Test_integration_launcher Yes CppServer - Agent - PyServer GHA No 11/06/2024 Present Move to actions on correspondent repository.
test_nightly (disabled) No QA New Jenkins No 23/10/2023 04/01/2024 Legacy pipeline. A new nightly will be required for the new structure.
Test_registration Yes QA DTT No 21/03/2023 Present Must be included in new DTT suite. Covers: register simple, password, manager verification (with & without), agent verification (with & without), verifications with passwords.
Test_registration_tier Yes QA DTT No 18/06/2024 Present Executes a set of test registration, replaced by DTT
Test_reliability No PyServer GHA No 21/10/2022 N/A Move to actions on correspondent repository.
Test_RTR Yes CppServer - Agent GHA No 29/09/2023 Present Move to actions on correspondent repository.
Test_RTR_external No CppServer - Agent GHA No 27/09/2022 19/04/2024 Move to actions on correspondent repository.
Test_RTR_scanbuild Yes CppServer GHA No 04/11/2022 Present Move to actions on correspondent repository.
Test_ruleset Yes CppServer GHA No 15/09/2022 28/05/2024 Move to actions on correspondent repository.
Test_selinux Yes QA DTT No 30/12/2020 Present Good to have, but the actual implementation is not great. Covers: check log 'cannot connect to socket' is not raised
Test_service Yes QA DTT No 05/01/2024 Present Must be included in new DTT suite. Covers: start, stop, restart (systemctl & sysv).
Test_stack Yes CppServer - Indexer - Dashboard GHA Yes 11/08/2023 Present Move to actions on correspondent repository. Covers: Checkfiles.
Test_stack_tier Yes CppServer - Indexer - Dashboard GHA Yes 11/08/2023 Present Executes a set of Test_stack
Test_stress (footprint) Yes QA New Stress suite Yes 01/07/2024 Present Must be refactored. Covers: syscheck, logcollector, sca, AR, rootcheck, syscollector, Docker, Cis-Cat, Osquery, Azure, VD, VD-Register.
Test_stress_AIX No QA --- No --- 02/06/2023 Legacy pipeline. Must be removed.
Test_stress_gh Yes QA New Jenkins Yes 11/01/2024 Present Must be refactored. It creates the footprint issues.
Test_stress_tier (footprint) Yes QA New Stress suite Yes 01/07/2024 Present Executes a set of Test_stress.
Test_stress_Vagrant Yes QA New Stress suite Yes 11/01/2024 Present Must be refactored, it could use JobFlow and Allocator.
Test_system No QA New System suite No 05/05/2022 05/12/2023 Must be refactored.
Test_unattended Yes QA DTT No 26/06/2024 Present Replaced by DTT suite. Covers install with unattended tools.
Test_unattended_distributed Yes QA DTT No 26/06/2024 Present Replaced by DTT suite. Covers install distributed environment with unattended tools.
Test_unattended_distributed_cases Yes QA DTT No 13/07/2022 Present Replaced by DTT suite. Covers install distributed environment with unattended tools.
Test_unattended_tier Yes QA DTT No 28/11/2022 Present Replaced by DTT suite. Executes a set of Test_unattended
Test_unit Yes CppServer - Agent - PyServer GHA No 10/06/2024 Present Move to actions on correspondent repository.
Test_unit_jenkins (Removed) No --- --- No --- 04/10/2021 Legacy pipeline. Must be removed.
Test_upgrade Yes QA DTT No 18/06/2024 Present Replaced by DTT.
Test_upgrade_nightly (Removed) No --- --- No --- 03/04/2020 Legacy pipeline. Must be removed.
Test_upgrade_tier Yes QA DTT No 18/06/2024 Present Replaced by DTT.
Test_upgrade_wpk (disabled) No QA --- No 27/06/2023 --- Legacy pipeline. Must be removed.
Tests_launcher (disabled) No --- --- No --- 29/05/2020 Legacy pipeline. Must be removed.
@QU3B1M QU3B1M self-assigned this Jul 22, 2024
@wazuhci wazuhci moved this to In progress in Release 4.10.0 Jul 22, 2024
@QU3B1M QU3B1M mentioned this issue Jul 11, 2024
Closed
11 tasks
@damarisg damarisg mentioned this issue Jul 22, 2024
Open
12 tasks
@QU3B1M
Copy link
Member Author

QU3B1M commented Jul 22, 2024 •
edited
Loading

Update progress

  • Read the pipelines definition
  • Analyze the different pipelines and categorize them
  • Create a table with the analysis made.
Initial version of table

This table is uncomplete and outdated

Pipeline Last code update Last run Team v4.X Details
Build_ALAS 15/05/2023 Present CppServer No Must be moved to GitHub Actions
Build_framework_python 18/09/2023 26/01/2024 PyServer No Must be moved to GitHub Actions
Build_last_tag 16/01/2020 07/01/2020 --- No Legacy pipeline. Must be removed
Build_MSU 30/10/2023 Present CppServer No Must be moved to GitHub Actions
Build_NVD 18/07/2023(TaskDefPushed) Present CppServer No Must be moved to GitHub Actions
Build_packages_from_branch 27/02/2020 08/08/2019 --- No Legacy pipeline. Must be removed
Packages_builder 20/02/2024 Present CppServer - Agent - Indexer - Dashboard No Must be moved to a GitHub Action for each component in correspondent repository
Packages_builder_macos 30/10/2023 Present Agent No Must be moved to GitHub Actions
Packages_builder_nightly --- --- --- No Legacy pipeline. Must be removed
Packages_builder_solaris 12/05/2022 19/07/2024 Agent No Must be moved to GitHub Actions
Packages_builder_special 25/01/2024 22/07/2024 No Agent. Must be moved to GitHub Actions
Packages_builder_tier 06/06/2023 Present No CppServer. Wont be required in the new architecture
Packages_builder_win 08/06/2023 Present No Agent. Must be moved to GitHub Actions
Packages_uploader_repository 09/07/2024 17/07/2024 Yes QA - Release. Refactor on Packages Redesign.
PR_FIM 16/12/2020 --- No CppServer. Legacy pipeline. Must be removed
PR_http_request 27/09/2022 19/04/2024 No CppServer. Must be moved to GitHub Actions
PR_intelligence_platform 15/02/2023 12/09/2023 No Content - CppServer. Must be moved to GitHub Actions
PR_jenkins (disabled) 08/09/2021 24/06/2020 No Legacy pipeline. Must be removed
PR_packages 24/03/2023 Present No Wont be required in the new architecture
PR_ruleset (disabled) 18/06/2021 09/12/2020 No Legacy pipeline. Must be removed
PR_wazuh 09/02/2024 Present No CppServer - Agent. Must be moved to GitHub Actions
PR_wazuh_feed (disabled) 25/11/2021 11/02/2022 No Legacy pipeline. Must be removed
Procedure_ecs_task 24/04/2024 22/07/2024 Yes QA. Must be moved to new jenkins, maybe replaced by Allocator
Procedure_ecs_task_tier 15/09/2023 Present Yes QA. Must be moved to new jenkins, could be merged with previous
Test_code_vulnerabilities 18/05/2023 N\A No PyServer. Must be moved to GitHub Actions
Test_compilation 30/05/2023 31/12/2023 No QA. Covers compilation tests for main components, must be refactored.
Test_dependencies_vulnerabilities 18/05/2023 29/05/2023 No QA. Legacy pipeline. GitHub already provides a solution for this.
Test_e2e_system 18/06/2024 Present Yes QA. Must be moved to new jenkins
Test_feed(removed) 11/02/2022 25/11/2021 No Legacy pipeline. Must be removed
Test_FIM 29/07/2021 N\A No QA. Legacy pipeline. Covers FIM performance tests
Test_install 01/12/2023 Present No QA. Replaced by new DTT suite. Covers: version, process, log errors, conf, connection
Test_install_special 15/09/2022 02/05/2022 No QA. Replaced by new DTT suite. Covers: version, process, log errors, conf, connection
Test_install_tier 01/12/2023 Present No QA. Replaced by new DTT suite.
Test_integration 04/06/2024 Present No CppServer - Agent - PyServer. Moved to GitHub Actions
Test_integration_endpoints 25/06/2024 Present No PyServer. Must be moved to Github Actions
Test_integration_launcher 11/06/2024 Present No CppServer - Agent - PyServer. Must be moved to GitHub Actions
test_nightly (disabled) 23/10/2023 04/01/2024 Yes QA. Legacy pipeline. A new nightly will be required for the new structure
Test_registration 21/03/2023 Present Yes QA. Must be included in new DTT suite. Covers: register simple, password, manager verification (with & without), agent verification (with & without), verifications with passwords
Test_registration_tier 18/06/2024 Present Yes QA. Must be included in new DTT suite.
Test_reliability 21/10/2022 N\A No PyServer. Must be moved to GutHub Actions
Test_RTR 29/09/2023 Present No CppServer - Agent. Must be moved to GitHub Actions
Test_RTR_external 27/09/2022 19/04/2024 No CppServer - Agent. Must be moved to GitHub Actions
Test_RTR_scanbuild 04/11/2022 Present No CppServer. Must be moved to GitHub Actions
Test_ruleset 15/09/2022 28/05/2024 No CppServer. Must be moved to GitHub Actions
Test_selinux 30/12/2020 Present No QA. Good to have, but the actual implementation is not great. Covers: check log 'cannot connect to socket' is not raised
Test_service 05/01/2024 Present No QA. Must be included in new DTT suite. Covers: start, stop, restart (systemctl & sysv)
Test_stack 11/08/2023 Present No CppServer - Indexer - Dashboard. Must be moved to GitHub Actions. Covers: Checkfiles
Test_stack_tier 11/08/2023 Present No Must be moved/replaced to actions
Test_stress (footprint) 01/07/2024 Present Yes QA. Must be refactored. Covers: syscheck, logcollector, sca, AR, rootcheck, syscollector, Docker, Cis-Cat, Osquery, Azure, VD, VD-Register
Test_stress_AIX --- 02/06/2023 No QA. Legacy pipeline. Must be removed
Test_stress_gh 11/01/2024 Present No QA. Must be refactored. It creates the footprint issues
Test_stress_tier (footprint) 01/07/2024 Present Yes QA. Could be replaced with a JobFlow implementation
Test_stress_Vagrant 11/01/2024 Present ye QA. Must be refactored, it could use JobFlow
Test_system 05/05/2022 05/12/2023 Neh QA. Must be refactored
Test_unattended 26/06/2024 Present No QA. Replaced by DTT suite. Covers install with unattended tools
Test_unattended_distributed 26/06/2024 Present No QA. Replaced by DTT suite. Covers install distributed environment with unattended tools
Test_unattended_distributed_cases 13/07/2022 Present No QA. Replaced by DTT suite. Covers install distributed environment with unattended tools
Test_unattended_tier 28/11/2022 Present No QA. Replaced by DTT suite. Covers install with unattended tools
Test_unit 10/06/2024 Present No CppServer - Agent - PyServer. Must be moved to GitHub Actions
Test_unit_jenkins (Removed) --- 04/10/2021 No Legacy pipeline. Must be removed
Test_upgrade 18/06/2024 Present No QA. Replaced by DTT
Test_upgrade_nightly (Removed) --- 03/04/2020 No Legacy pipeline. Must be removed
Test_upgrade_tier 18/06/2024 Present No QA. Replaced by DTT
Test_upgrade_wpk (disabled) 27/06/2023 --- No QA. Legacy pipeline. Must be removed
Tests_launcher (disabled) --- 29/05/2020 No Legacy pipeline. Must be removed

@damarisg damarisg changed the title Research current pipelines documentation and identify the ones to migrate MVP - Research current pipelines documentation and identify the ones to migrate Jul 23, 2024
@wazuhci wazuhci moved this from In progress to Pending review in Release 4.10.0 Jul 25, 2024
@wazuhci wazuhci moved this from Pending review to In review in Release 4.10.0 Jul 25, 2024
@damarisg
Copy link
Member

Request changes

I will request changes.

  • First change: Create a table by team. If the pipeline does not have a responsible team, that table will be called "Deprecated".

  • Second change: Add a column to each table with a brief description and a list of the use cases covered by that pipeline. It will help to identify what should be maintained or not.

Regards.

@damarisg damarisg moved this from In review to On hold in Release 4.10.0 Jul 25, 2024
@pcapellan pcapellan self-assigned this Jul 25, 2024
@wazuhci wazuhci moved this from On hold to In progress in Release 4.10.0 Jul 25, 2024
@pcapellan
Copy link

pcapellan commented Jul 26, 2024 •
edited by damarisg
Loading

31/7/2024

Pipelines analyzed:

  • PR_http_request
  • PR_package
  • PR_intelligence_platform
  • PR_wazuh
  • Test_RTR
  • Test_RTR_scanbuild
  • Build_framework_python
  • Test_unit
  • Test_reliability
  • Test_code_vulnerabilities
  • Test_unattended
  • Test_unattended_tier
  • Test_unattended_distributed
  • Test_stress
  • Test_stress_AIX
  • Test_stress_gh

File with pipelines analysis:

wazuh-pipelines.xlsx

30/7/2024

Pipelines analyzed:

  • Test_Ruleset
  • Package_builder
  • Package_builder_tier
  • Package_builder_macos
  • Package_builder_win
  • Package_builder_solaris
  • Package_builder_special
  • Test_integration
  • Test_integration_launcher
  • Test_integration_endpoints
  • Build_NVD
  • Test_stack
  • Test_stack_tier
29/7/2024

Pipelines analyzed:

  • Test_service
  • Test_system
  • Selinux
  • Test_compilation
  • package_uploader_repository
  • Test_dependencies_vulnerabilities
  • Build_ALAS
  • Build_MSU
  • PR_FIM
  • Test_FIM
26/7/2024

Pipelines analyzed:

  • Procedure_ecs_task
  • Procedure_ecs_task_tier
  • Test_e2e_system
  • Test_registration
  • Test_registration_tier
  • Test_install
  • Test_install_special
  • Test_install_tier
  • Test_upgrade
  • Test_upgrade_tier

Pipelines by team

Total of pipelines: 65

  • Deprecated pipelines: 12
  • QA pipelines 26
  • CORE pipelines: 27

pipelines_by_team

@wazuhci wazuhci moved this from In progress to Pending review in Release 4.10.0 Jul 31, 2024
@wazuhci wazuhci moved this from Pending review to Pending final review in Release 4.10.0 Aug 2, 2024
@wazuh wazuh deleted a comment from pcapellan Aug 2, 2024
@wazuh wazuh deleted a comment from pcapellan Aug 2, 2024
@wazuh wazuh deleted a comment from pcapellan Aug 2, 2024
@wazuh wazuh deleted a comment from pcapellan Aug 2, 2024
@wazuhci wazuhci moved this from Pending final review to In final review in Release 4.10.0 Aug 2, 2024
@damarisg
Copy link
Member

damarisg commented Aug 2, 2024

Review Number 2

I modified the Pipelines details.xlsx file to improve the information. I am waiting to respond to some questions and then move the status question.

@damarisg
Copy link
Member

damarisg commented Aug 6, 2024

Review 3 - Conclusion

In this first approach, we detect the pipelines to migrate details.
The director will review this to define our current approach.

@damarisg damarisg closed this as completed Aug 6, 2024
@damarisg damarisg moved this from In final review to Done in Release 4.10.0 Aug 6, 2024
@damarisg damarisg changed the title MVP - Research current pipelines documentation and identify the ones to migrate Research current pipelines documentation and identify the ones to migrate Aug 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pcapellan pcapellan

@QU3B1M QU3B1M

Labels
level/task Task issue type/research
Projects
Release 4.10.0
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@damarisg @pcapellan @QU3B1M