Enhance CI automation #106
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Allows ServiceTelemetry overrides to be expressed via Ansible extra-vars. Adds the four main overrides you would expect in a ServiceTelemetry object, with appropriate defaults set. Will also allow passing in the service_telemetry_manifest as a whole object like what we do with the Service Telemetry Operator.
Allow for per-repo branch overrides for the Smart Gateway Operator and Smart Gateway repositories via sgo_branch and sg_branch (respectively).
Add some functionality that was replaced when I dropped the quickstart.sh. Adds some of this functionality back in and also adds some new stuff.
Adds back a quickstart.sh that simulates the same result as the old quickstart.sh
Also adds some tags to make skipping over builds for testing much easier.
|
Don't merge this yet. I'm going to make a few more enhancements today to prep for handling CI in the SG3 branches. |
csibbitt
reviewed
Jul 9, 2020
|
Re-running locally to debug issues with smoke test. |
|
My smoketests run fine locally, I don't understand why they are failing in CI... |
|
I tested this in quicklab with |
csibbitt
approved these changes
Jul 9, 2020
On subsequent runs the in-place modification of the CSV can cause issues either in the development environment, or re-runs of the CI system. Copying the CSV out of the in-place repo into a working location, and then modifying in-place results in a cleaner setup. By doing the copy of the CSV files, we can drop the need to force clone the supporting repositories. Also cleans up some shell commands that were commented out now that they are being dealt with via the replace module. Removes the extra commands added to ci.yml.
leifmadsen
added a commit
that referenced
this pull request
Jul 13, 2020
* Enhance CI automation (#106) * Add ServiceTelemetry overrides Allows ServiceTelemetry overrides to be expressed via Ansible extra-vars. Adds the four main overrides you would expect in a ServiceTelemetry object, with appropriate defaults set. Will also allow passing in the service_telemetry_manifest as a whole object like what we do with the Service Telemetry Operator. * Allow for per-repo branch overrides Allow for per-repo branch overrides for the Smart Gateway Operator and Smart Gateway repositories via sgo_branch and sg_branch (respectively). * Add functionality around quickstart.sh Add some functionality that was replaced when I dropped the quickstart.sh. Adds some of this functionality back in and also adds some new stuff. * Fix syntax error * Add back quickstart.sh Adds back a quickstart.sh that simulates the same result as the old quickstart.sh * Better CSV modification support Also adds some tags to make skipping over builds for testing much easier. * Debugging ci.yml firing * Make sure namespace is set before using it * Fix syntax and documentation * Test locally first kids * Drop CI debug lines * Clean up working repo clones * Copy CSV into working directory On subsequent runs the in-place modification of the CSV can cause issues either in the development environment, or re-runs of the CI system. Copying the CSV out of the in-place repo into a working location, and then modifying in-place results in a cleaner setup. By doing the copy of the CSV files, we can drop the need to force clone the supporting repositories. Also cleans up some shell commands that were commented out now that they are being dealt with via the replace module. Removes the extra commands added to ci.yml. * Changes to infrared-openstack.sh for OSP13 (#102) * Migrate OSP16 script to OSP13 directory Uses a multi-cloud stf-connectors.yaml style configuration which directly loads the resource lists rather than a list of environment files. Uses the same script as used in OSP16 but subs out the network configuration for a vlan type setup and the latest paths for async puddle. * Working deployment of OSP13 * Migrate changes to align to existing docs Update PR to align to existing documentation and testing the group has been working on. Adjust the stf-connectors.yaml.template to better reflect what we've been testing. Deployment by default will result in presettle: true which is bad for reliability of message delivery. * Get closer alignment to OSP16 setup * Enable deployment of metric SG for Ceilometer data (#93) * Enable deployment of metric SG for Ceilometer data Depends-On: infrawatch/smart-gateway#83 Depends-On: infrawatch/smart-gateway-operator#48 * Add smoketest for Ceilometer data * Listen on correct channel * Ceilometer smoketest tuning Makes smoketest_ceilometer_entrypoint.sh being executed during smoketest job. * Use data source setting for metrics too # Please enter the commit message for your changes. Lines starting # with '#' will be ignored, and an empty message aborts the commit. # # On branch mmagr-amqp10connections # Changes to be committed: # modified: roles/servicetelemetry/templates/manifest_smartgateway_metrics.j2 # * Finish ceilometer events smoke test * Do not use hardcoded timestamps * Finish ceilometer metrics smoketests * Increase timeout * Add container names * Validate also Ceilometer metrics SG * Update tests/smoketest/smoketest_ceilometer_entrypoint.sh * Update tests/smoketest/smoketest_collectd_entrypoint.sh Co-authored-by: Martin Magr <[email protected]> Co-authored-by: Leif Madsen <[email protected]> * Implement CI updates for SG3 * Correct value for SG bridge image path * Lock operator-courier to 2.1.7 Lock operator-courier to 2.1.7 until we can figure out what is wrong with our CSV/CRD setup or until the operator-courier issue noted in the related issue is resolved. Related: #108 * Update build/stf-run-ci/tasks/main.yml Co-authored-by: Chris Sibbitt <[email protected]> * Adjust README to match run-ci.yaml methods Co-authored-by: Martin Mágr <[email protected]> Co-authored-by: Martin Magr <[email protected]> Co-authored-by: Chris Sibbitt <[email protected]>
leifmadsen
added a commit
that referenced
this pull request
Jul 13, 2020
* Changes to match new internal SG metrics names * Changing exported_instance to host * plugin_instance relabelling no longer being done in sg2 * type becomes type_instance * Applied relevant changes to the new alerts * Fixing plugin vs type instance on memory alarm * Adjusted order of labels in smoketest * Need longer interval for rates with default 10s scrape interval * OCP metrics label change from pod_name to pod * Raise the linkCapacity of QDR->bridge * Adds a new edgeListener on 5673 with linkCapacity 25000 * Adjusts metrics SG bridge to connect to new listener * Minimizes presettled metrics loss during throughput testing * Should also provide performance improvements in unsettled mode bursts * Other SG modes can keep using 5672 until we converge the SG code * New CRD and CSV for v1.0.3 * Updated scorecard paths * Added amqpDataSource to metrics SG template * This is so I can test changes here: infrawatch/smart-gateway-operator#54 * In preparation to merge with: https://github.com/infrawatch/service-telemetry-operator/pull/93/files#diff-33675527951f20f9727fb4be5c84a746R8 * Putting quickstart.sh back until build/run-ci.yaml is a true replacement * No ability to deploy published artifacts without building * No support for ephemeral storage * Adding back quickstart configs * Setup SG3 CI system (#107) * Enhance CI automation (#106) * Add ServiceTelemetry overrides Allows ServiceTelemetry overrides to be expressed via Ansible extra-vars. Adds the four main overrides you would expect in a ServiceTelemetry object, with appropriate defaults set. Will also allow passing in the service_telemetry_manifest as a whole object like what we do with the Service Telemetry Operator. * Allow for per-repo branch overrides Allow for per-repo branch overrides for the Smart Gateway Operator and Smart Gateway repositories via sgo_branch and sg_branch (respectively). * Add functionality around quickstart.sh Add some functionality that was replaced when I dropped the quickstart.sh. Adds some of this functionality back in and also adds some new stuff. * Fix syntax error * Add back quickstart.sh Adds back a quickstart.sh that simulates the same result as the old quickstart.sh * Better CSV modification support Also adds some tags to make skipping over builds for testing much easier. * Debugging ci.yml firing * Make sure namespace is set before using it * Fix syntax and documentation * Test locally first kids * Drop CI debug lines * Clean up working repo clones * Copy CSV into working directory On subsequent runs the in-place modification of the CSV can cause issues either in the development environment, or re-runs of the CI system. Copying the CSV out of the in-place repo into a working location, and then modifying in-place results in a cleaner setup. By doing the copy of the CSV files, we can drop the need to force clone the supporting repositories. Also cleans up some shell commands that were commented out now that they are being dealt with via the replace module. Removes the extra commands added to ci.yml. * Changes to infrared-openstack.sh for OSP13 (#102) * Migrate OSP16 script to OSP13 directory Uses a multi-cloud stf-connectors.yaml style configuration which directly loads the resource lists rather than a list of environment files. Uses the same script as used in OSP16 but subs out the network configuration for a vlan type setup and the latest paths for async puddle. * Working deployment of OSP13 * Migrate changes to align to existing docs Update PR to align to existing documentation and testing the group has been working on. Adjust the stf-connectors.yaml.template to better reflect what we've been testing. Deployment by default will result in presettle: true which is bad for reliability of message delivery. * Get closer alignment to OSP16 setup * Enable deployment of metric SG for Ceilometer data (#93) * Enable deployment of metric SG for Ceilometer data Depends-On: infrawatch/smart-gateway#83 Depends-On: infrawatch/smart-gateway-operator#48 * Add smoketest for Ceilometer data * Listen on correct channel * Ceilometer smoketest tuning Makes smoketest_ceilometer_entrypoint.sh being executed during smoketest job. * Use data source setting for metrics too # Please enter the commit message for your changes. Lines starting # with '#' will be ignored, and an empty message aborts the commit. # # On branch mmagr-amqp10connections # Changes to be committed: # modified: roles/servicetelemetry/templates/manifest_smartgateway_metrics.j2 # * Finish ceilometer events smoke test * Do not use hardcoded timestamps * Finish ceilometer metrics smoketests * Increase timeout * Add container names * Validate also Ceilometer metrics SG * Update tests/smoketest/smoketest_ceilometer_entrypoint.sh * Update tests/smoketest/smoketest_collectd_entrypoint.sh Co-authored-by: Martin Magr <[email protected]> Co-authored-by: Leif Madsen <[email protected]> * Implement CI updates for SG3 * Correct value for SG bridge image path * Lock operator-courier to 2.1.7 Lock operator-courier to 2.1.7 until we can figure out what is wrong with our CSV/CRD setup or until the operator-courier issue noted in the related issue is resolved. Related: #108 * Update build/stf-run-ci/tasks/main.yml Co-authored-by: Chris Sibbitt <[email protected]> * Adjust README to match run-ci.yaml methods Co-authored-by: Martin Mágr <[email protected]> Co-authored-by: Martin Magr <[email protected]> Co-authored-by: Chris Sibbitt <[email protected]> Co-authored-by: Chris Sibbitt <[email protected]> Co-authored-by: Martin Mágr <[email protected]> Co-authored-by: Martin Magr <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements a few features that I stripped when migrating to an Ansible based deployment method for CI setup, which was also being used for local testing.
quickstart.shscript which wraps a call toansible-playbookwith appropriate overridesServiceTelemetrymanifest overrides (metrics, events, high availability, ephemeral storage)ServiceTelemetrymanifest overrideAlso adds some new features that were requested:
-e __local_builds_enabled=false)-e sg_branch=<branch_name>,-e sgo_branch=<branch_name>)Should beget the need for #105 (hopefully)