Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Custom hypervisor minimal changes #7692

Merged
merged 8 commits into from
Aug 16, 2023

Conversation

nvazquez
Copy link
Contributor

@nvazquez nvazquez commented Jun 26, 2023

Description

Design document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/%5BDRAFT%5D+Minimal+changes+to+allow+new+dynamic+hypervisor+type%3A+Custom+Hypervisor

This PR introduces the minimal changes to add a new hypervisor type (internally named Custom in the codebase, and configurable display name), allowing to write an external hypervisor plugin as a Custom Hypervisor to CloudStack

The custom hypervisor name is set by the setting: 'hypervisor.custom.display.name'. The new hypervisor type does not affect the behaviour of any CloudStack operation, it simply introduces a new hypervisor type into the system.

CloudStack does not have any means to dynamically add new hypervisor types. The hypervisor types are internally preset by an enum defined within the CloudStack codebase and unless a new version supports a new hypervisor it is not possible to add a host of a hypervisor that is not in part of the enum. It is possible to implement minimal changes in CloudStack to support a new hypervisor plugin that may be developed privately

This PR is an initial work on allowing new dynamic hypervisor types (adds a new element to the HypervisorType enum, but allows variable display name for the hypervisor)

Proposed Future work:

Replace the HypervisorType from a fixed enum to an extensible registry mechanism, registered from the hypervisor plugin

Feature Specifications

  • The new hypervisor type is internally named 'Custom' to the CloudStack services (management server and agent services, database records).
  • A new global setting ‘hypervisor.custom.display.name’ allows administrators to set the display name of the hypervisor type. The display name will be shown in the CloudStack UI and API.
    • In case the ‘hypervisor.list’ setting contains the display name of the new hypervisor type, the setting value is automatically updated after the ‘hypervisor.custom.display.name’ setting is updated.
  • The new Custom hypervisor type supports:
    • Direct downloads (the ability to download templates into primary storage from the hypervisor hosts without using secondary storage)
    • Local storage (use hypervisor hosts local storage as primary storage)
    • Template format: RAW format (the templates to be registered on the new hypervisor type must be in RAW format)
  • The UI is also extended to display the new hypervisor type and the supported features listed above.
  • The above are the minimal changes for CloudStack to support the new hypervisor type, which can be tested by integrating the plugin codebase with this feature.

Use cases

This PR allows the cloud administrators to test custom hypervisor plugins implementations in CloudStack and easily integrate it into CloudStack as a new hypervisor type ("Custom"), reducing the implementation to only the hypervisor supported specific storage/networking and the hypervisor resource to communicate with the management server.

  • CloudStack admin should be able to create a zone for the new custom hypervisor and add clusters, hosts into the zone with normal operations
  • CloudStack users should be able to execute normal VMs/volumes/network/storage operations on VMs/volumes running on the custom hypervisor hosts

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

Having set 'hypervisor.custom.display.name' = 'MyCustomHW', added an extra zone with the hypervisor type 'MyCustomHW' and added one host:

  • Listing hosts, one per zone. The hypervisor type is set as the display name of the custom hypervisor
Screenshot 2023-08-06 at 10 38 56
  • Listing hosts in database. The hypervisor type is set as Custom, despite the display name which is variable
mysql> select name, hypervisor_type from host where status = 'Up' and type = 'Routing' and removed is null;
+------------------------------------------+-----------------+
| name                                     | hypervisor_type |
+------------------------------------------+-----------------+
| ref-trl-4901-k-Mr8-nicolas-vazquez-kvm2  | KVM             |
| ref-trl-4901-k-Mr8-nicolas-vazquez-test1 | Custom          |
+------------------------------------------+-----------------+
2 rows in set (0.00 sec)

How Has This Been Tested?

@nvazquez
Copy link
Contributor Author

@blueorangutan package

@nvazquez
Copy link
Contributor Author

@blueorangutan package

@blueorangutan
Copy link

@nvazquez a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6342

Copy link
Member

@rohityadavcloud rohityadavcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, didn't test it but this would allow anybody to write an external hypervisor plugin

@github-actions
Copy link

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@nvazquez
Copy link
Contributor Author

@blueorangutan package

@blueorangutan
Copy link

@nvazquez a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6351

@nvazquez
Copy link
Contributor Author

@blueorangutan test

@blueorangutan
Copy link

@nvazquez a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Copy link
Contributor

@shwstppr shwstppr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code change looks okay but I fear we have some checks in code based on hypervisor type such vm migration with storage, find hosts for migration, etc. Custom hypervisor plugin may struggle there or may not give the desired results.

@codecov
Copy link

codecov bot commented Jun 27, 2023

Codecov Report

Merging #7692 (a5568b8) into main (eb31e3d) will increase coverage by 0.81%.
Report is 6 commits behind head on main.
The diff coverage is 7.35%.

@@             Coverage Diff              @@
##               main    #7692      +/-   ##
============================================
+ Coverage     13.47%   14.28%   +0.81%     
- Complexity     9423     9993     +570     
============================================
  Files          2747     2748       +1     
  Lines        258853   258944      +91     
  Branches      40311    40325      +14     
============================================
+ Hits          34873    36994    +2121     
+ Misses       219579   217152    -2427     
- Partials       4401     4798     +397     
Files Changed Coverage Δ
...in/java/com/cloud/storage/GuestOSHypervisorVO.java 42.30% <0.00%> (ø)
server/src/main/java/com/cloud/api/ApiDBUtils.java 0.16% <0.00%> (-0.01%) ⬇️
...src/main/java/com/cloud/api/ApiResponseHelper.java 3.78% <0.00%> (ø)
...m/cloud/api/query/dao/DomainRouterJoinDaoImpl.java 0.47% <0.00%> (ø)
.../java/com/cloud/api/query/dao/HostJoinDaoImpl.java 0.35% <0.00%> (-0.01%) ⬇️
...om/cloud/api/query/dao/StoragePoolJoinDaoImpl.java 0.52% <0.00%> (ø)
...ava/com/cloud/api/query/dao/UserVmJoinDaoImpl.java 3.18% <0.00%> (ø)
...ava/com/cloud/api/query/dao/VolumeJoinDaoImpl.java 5.71% <0.00%> (-0.07%) ⬇️
.../cloud/configuration/ConfigurationManagerImpl.java 16.04% <0.00%> (-0.04%) ⬇️
.../java/com/cloud/hypervisor/HypervisorGuruBase.java 9.92% <0.00%> (ø)
... and 7 more

... and 46 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@blueorangutan
Copy link

[SF] Trillian test result (tid-6883)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 36152 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7692-t6883-kvm-centos7.zip
Smoke tests completed. 93 look OK, 20 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestRouterDHCPHosts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestProjectSuspendActivate>:setup Error 8.36 test_projects.py
test_01_add_delete_kubernetes_supported_version Error 0.05 test_kubernetes_supported_versions.py
ContextSuite context=TestMetrics>:setup Error 0.00 test_metrics_api.py
ContextSuite context=TestListIdsParams>:setup Error 0.00 test_list_ids_parameter.py
ContextSuite context=TestRouterDns>:setup Error 0.00 test_router_dns.py
ContextSuite context=TestLoadBalance>:setup Error 0.00 test_loadbalance.py
ContextSuite context=TestRouterDnsService>:setup Error 0.00 test_router_dnsservice.py
ContextSuite context=TestRouterServices>:setup Error 0.00 test_routers.py
ContextSuite context=TestIsolatedNetworks>:setup Error 0.00 test_routers_network_ops.py
ContextSuite context=TestRedundantIsolateNetworks>:setup Error 0.00 test_routers_network_ops.py
ContextSuite context=TestRouterIpTablesPolicies>:setup Error 0.00 test_routers_iptables_default_policy.py
ContextSuite context=TestVPCIpTablesPolicies>:setup Error 0.00 test_routers_iptables_default_policy.py
ContextSuite context=TestCpuCapServiceOfferings>:setup Error 0.00 test_service_offerings.py
ContextSuite context=TestServiceOfferings>:setup Error 0.24 test_service_offerings.py
ContextSuite context=TestVMWareStoragePolicies>:setup Error 0.00 test_storage_policy.py
test_01_create_template Error 1.11 test_templates.py
test_CreateTemplateWithDuplicateName Error 1.13 test_templates.py
test_02_1_create_template_with_checksum_sha1_negative Error 0.12 test_templates.py
test_02_create_template_with_checksum_sha1 Error 0.10 test_templates.py
test_03_1_create_template_with_checksum_sha256_negative Error 0.09 test_templates.py
test_03_create_template_with_checksum_sha256 Error 0.09 test_templates.py
test_04_1_create_template_with_checksum_md5_negative Error 0.09 test_templates.py
test_04_create_template_with_checksum_md5 Error 0.12 test_templates.py
test_05_create_template_with_no_checksum Error 0.08 test_templates.py
ContextSuite context=TestTemplates>:setup Error 259.76 test_templates.py
test_01_migrate_VM_and_root_volume Error 80.80 test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks Error 52.38 test_vm_life_cycle.py
test_10_attachAndDetach_iso Error 0.07 test_vm_life_cycle.py
test_06_download_detached_volume Error 301.78 test_volumes.py
test_13_migrate_volume_and_change_offering Error 127.86 test_volumes.py
ContextSuite context=TestIpv6Vpc>:setup Error 0.00 test_vpc_ipv6.py
ContextSuite context=TestVPCRedundancy>:setup Error 0.00 test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup Error 0.00 test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup Error 0.00 test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup Error 0.00 test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup Error 0.00 test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup Error 0.00 test_vpc_vpn.py

@borisstoyanov
Copy link
Contributor

@blueorangutan test

@blueorangutan
Copy link

@borisstoyanov a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6659

@nvazquez
Copy link
Contributor Author

nvazquez commented Aug 5, 2023

@blueorangutan package

@blueorangutan
Copy link

@nvazquez a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6673

@nvazquez
Copy link
Contributor Author

nvazquez commented Aug 5, 2023

@blueorangutan test matrix

@blueorangutan
Copy link

@nvazquez a [SF] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian Build Failed (tid-7292)

@blueorangutan
Copy link

[SF] Trillian test result (tid-7290)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40101 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7692-t7290-xenserver-71.zip
Smoke tests completed. 113 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

@blueorangutan
Copy link

[SF] Trillian test result (tid-7291)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 74108 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7692-t7291-vmware-67u3.zip
Smoke tests completed. 109 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_02_upgrade_kubernetes_cluster Failure 591.23 test_kubernetes_clusters.py
test_list_system_vms_metrics_history Failure 0.23 test_metrics_api.py
test_list_vms_metrics_admin Error 3607.24 test_metrics_api.py
test_list_vms_metrics_history Error 19.01 test_metrics_api.py
test_list_volumes_metrics_history Error 3607.11 test_metrics_api.py
test_01_deploy_vm_on_specific_host Error 11.52 test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster Error 3602.61 test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod Error 4.40 test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster Error 2.36 test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod Error 2.31 test_vm_deployment_planner.py
test_09_expunge_vm Failure 424.66 test_vm_life_cycle.py

@nvazquez
Copy link
Contributor Author

nvazquez commented Aug 6, 2023

@blueorangutan package

@blueorangutan
Copy link

@nvazquez a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✖️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6680

@nvazquez
Copy link
Contributor Author

nvazquez commented Aug 7, 2023

@blueorangutan package

@blueorangutan
Copy link

@nvazquez a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6691

@nvazquez
Copy link
Contributor Author

nvazquez commented Aug 7, 2023

@blueorangutan package

@blueorangutan
Copy link

@nvazquez a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6693

Copy link
Contributor

@DaanHoogland DaanHoogland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the code looks good to me. I really don't like the design of it though. This hack of adding a custom enum type for HypervisorType is ugle. I would really hope this is the start of a longer term project to make hypervisors truly plugable.

Copy link
Contributor

@borisstoyanov borisstoyanov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rohityadavcloud rohityadavcloud marked this pull request as ready for review August 16, 2023 15:21
Copy link
Member

@rohityadavcloud rohityadavcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, relying on @borisstoyanov 's testing - this would enable hypervisor plugin developers to be able to build/ship custom plugins outside of CloudStack release cycle, much like it's possible with storage plugins. While the implementation can be improved to allow a more pluggable model, this is good enough for the first iteration.

@rohityadavcloud rohityadavcloud merged commit 8b5ba13 into apache:main Aug 16, 2023
@rohityadavcloud rohityadavcloud deleted the upstream-custom-hw-pr branch August 16, 2023 15:26
shwstppr pushed a commit to shapeblue/cloudstack that referenced this pull request Oct 12, 2023
Design document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/%5BDRAFT%5D+Minimal+changes+to+allow+new+dynamic+hypervisor+type%3A+Custom+Hypervisor

This PR introduces the minimal changes to add a new hypervisor type (internally named Custom in the codebase, and configurable display name), allowing to write an external hypervisor plugin as a Custom Hypervisor to CloudStack

The custom hypervisor name is set by the setting: 'hypervisor.custom.display.name'. The new hypervisor type does not affect the behaviour of any CloudStack operation, it simply introduces a new hypervisor type into the system.

CloudStack does not have any means to dynamically add new hypervisor types. The hypervisor types are internally preset by an enum defined within the CloudStack codebase and unless a new version supports a new hypervisor it is not possible to add a host of a hypervisor that is not in part of the enum. It is possible to implement minimal changes in CloudStack to support a new hypervisor plugin that may be developed privately

This PR is an initial work on allowing new dynamic hypervisor types (adds a new element to the HypervisorType enum, but allows variable display name for the hypervisor)

Replace the HypervisorType from a fixed enum to an extensible registry mechanism, registered from the hypervisor plugin

- The new hypervisor type is internally named 'Custom' to the CloudStack services (management server and agent services, database records).
- A new global setting ‘hypervisor.custom.display.name’ allows administrators to set the display name of the hypervisor type. The display name will be shown in the CloudStack UI and API.
   - In case the ‘hypervisor.list’ setting contains the display name of the new hypervisor type, the setting value is automatically updated after the ‘hypervisor.custom.display.name’ setting is updated.
- The new Custom hypervisor type supports:
   - Direct downloads (the ability to download templates into primary storage from the hypervisor hosts without using secondary storage)
   - Local storage (use hypervisor hosts local storage as primary storage)
   - Template format: RAW format (the templates to be registered on the new hypervisor type must be in RAW format)
- The UI is also extended to display the new hypervisor type and the supported features listed above.
- The above are the minimal changes for CloudStack to support the new hypervisor type, which can be tested by integrating the plugin codebase with this feature.

This PR allows the cloud administrators to test custom hypervisor plugins implementations in CloudStack and easily integrate it into CloudStack as a new hypervisor type ("Custom"), reducing the implementation to only the hypervisor supported specific storage/networking and the hypervisor resource to communicate with the management server.

- CloudStack admin should be able to create a zone for the new custom hypervisor and add clusters, hosts into the zone with normal operations
- CloudStack users should be able to execute normal VMs/volumes/network/storage operations on VMs/volumes running on the custom hypervisor hosts

(cherry picked from commit 8b5ba13)
Signed-off-by: Rohit Yadav <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants