This plugin allows the user to create Single and Pipeline Jobs to build and execute VectorCAST Projects. Coverage is displayed using the VectorCAST Coverage Plugin.
This plugin adds a new top-level menu item to the Jenkins sidebar. Select the VectorCAST menu item to create a new VectorCAST Job.
There are two build/execute job types and one diagnostic job type.
- Single Job
- Pipeline Job
- Diagnostic Job
Single Job creates a Single Jenkins Job to build/execute and (optionally) report on all environments in a VectorCAST Project.
Pipeline Job creates a Pipeline Jenkins Job to build/execute and (optionally) report on all environments in a VectorCAST Project in parallel. A Pipeline script (Jenkinsfile) is created to run a build/execute job for each environment and an overall job to combine the results.
There are two options for running tests:
- Using an SCM system (any that is supported by Jenkins).
- Jenkins will check out the source code and tests into the workspace for each Jenkins Job from the user's repository.
- The Pipeline Job will then combine the coverage and test results from all of the individual machines/nodes.
- The VectorCAST Project should be specified as relative to the root of the checkout.
- There is now an option to use the main Pipeline Job's Workspace as a dedicated single checkout directory. This checkout directory must be available to all executors across all nodes either by having all executors running on the same computer or having the main Pipeline Job's Workspace on a shared network drive.
- Using an existing drive/directory for the VectorCAST Project.
- The VectorCAST Project should be specified as an absolute path that is available on all machines/nodes.
- Each job can optionally clean up the working directory which will have no effect on the VectorCAST Project since it is located elsewhere.
- The reports are generated into the workspace and archived as part of the Jenkins Job.
The user will be able to disable the use of Change Based Testing to perform a complete run of their VectorCAST Project. By default, Change Based Testing is enabled but this option can be disabled by unchecking the Use Change Based Testing box.
Users with Continuous Integration Licenses can access those licenses by checking the Use Continuous Integration License checkbox. If you do not have Continuous Integration Licenses, do not check this box as you will encounter licensing errors.
If the user wishes to call the Jenkins Job from another Pipeline Job, check the Use Parameterize the Jenkinsfile box. This will add parameters to the Pipeline Job that will be used by the VectorCAST Pipeline Job to locate an external VectorCAST Project (VCAST_PROJECT_DIR) and force the VectorCAST Jobs to be executed on a specific node (VCAST_FORCE_NODE_EXEC_NAME) instead of using the compiler as a node label.
If addition processing is required to a repository after checkout, the user can add Post SCM Checkout Commands. These commands would be native to a shell or batch file depending on which platform the jobs is running on.
Calling the build command will return Failed, Unstable, or Success corresponding to the results of the VectorCAST Pipeline Job. To allow the main Pipeline Job to continue on error, the user can surround the build command without a catchError block as demonstrated below.
catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE', catchInterruptions : false ) {
build job: 'UnitTestingProject_vcast_pipeline', parameters: [string(name: 'VCAST_PROJECT_DIR', value: 'C:\\UnitTesting\\Project'), string(name: 'VCAST_FORCE_NODE_EXEC_NAME', value: 'MyTestNode')]
}
For Pipeline Jobs, the plugin processes the build log to determine which tests have been skipped by VectorCAST's Change Based Testing feature. Any test cases that previously passed and were skipped on the current execution will be marked as skipped for JUnit and displayed in yellow on the test result trend chart and denoted as Skipped in the test results list.
The user can customize additional parameters for job creation by selecting the Job Creation Options:
Use the Job Name setting for changing the created pipeline job name. Default is VectorCASTProjectName_vcast_pipeline
The Shared Artifact Directory option allows VectorCAST Project's build artifacts to be stored in a different location from the default VectorCASTProjectName/build. If stored in a location that can be maintained between builds, VectorCAST's Change Based Testing can accelerate testing
Maximum Parallel Queued Jobs allows the job to specify the maximum number of unit test jobs to queue up at any one time for parallel execution. To queue all jobs, leave blank or set to zero (0). The use case for this option would be if you don't want to queue the Jenkins server with all VectorCAST environment build/execute jobs; thus, allowing other jobs to queue up as well.
Use Coverage History marks build as failed if statement or branch coverage decrease by comparing the previous non-failing build's statement and branch coverage to the current build's statement and branch coverage. If either of the coverages have decreased the job will be marked as failed
Use Strict Test Case Importing allows the job to mark individual test cases as failures if errors encountered during test script import. This option enables strict testcase importing for the VectorCAST Project. This option give a more reliable metrics for pass/fail test cases as without strict test case import, bad test cases are just ignored. The default is enabled.
Use Imported Results allows jobs to previous test results as input for the current job execution phase. This option allows VectorCAST Change Based Testing to have a known result to work from. This option works in conjunction with Use Change Based Testing. The user can selected between internal imported results or external result files
When using imported results and the Use Local Imported Results option, the job will export results from the current build and save that result file as an job artifact. The next build will pull the result archive from the last SUCCESS or UNSTABLE build and use change based testing against that result baseline. If existing build artifacts exist, the combined results will be used for change based testing.
When using imported results and the Use External Result File option, the job will use an external results from the a specified location to use as a baseline for the the current job. If existing build artifacts exist, the combined results will be used for change based testing. The user will need to specify an external result filename (.vcr) in the External Result Filename option. This file would be typically stored in the repository. This external result file could be updated by a main branch or periodic build to establish a current baseline.
Other Vector tool integrations are supported by this plugin.
- PC-lint Plus
- Squore
- TESTinsights
- For PC-lint Plus, the user must provide the command or script that executes PC-lint Plus analysis. It is important that the following format properties are included to ensure the results file is readable by the Warnings-NG plugin:
-v // turn off verbosity
-width(0) // don't insert line breaks (unlimited output width)
-"format=%f(%l): %t %n: %m"
-hs1 // The height of a message should be 1
- For Squore analysis, the user must provide the Squore command found on the last page of the Squore project's configuration/build page.
- For TESTinsights, if the user is using Git or Subversion for SCM, the plugin will attempt to have the SCM version linked to the TESTinsights project for team access and distributed change based testing. The user must provide the following:
- TESTinsights URL - The URL to TESTinsights server and project (Use Copy Team Area Url).
- TESTinsights Project - The Project Name in TESTinsights to push the results (leave blank to use the Jenkins Job Name).
- TESTinsights Credential ID - The Credential ID from Jenkins for TESTinsights.
- The proxy to push to TESTinsights server in the format proxy.address:port (optional)
When using a Pipeline Job, the sub jobs are created to run on specific node related to the compiler chosen for the environment. For example:
Make sure to set the labels on the relevant Jenkins nodes. Manage Jenkins -> Manage Nodes -> configure appropriate node and set the ‘Labels’ field. In this example the labels have been set to GNU_Native_5.1_C and GNU_Native_4.8_C
- VectorCAST must be installed and setup on each node.
- The environment variables VECTORCAST_DIR and VECTOR_LICENSE must be set.
- Jenkins must be installed and setup.
- BUILD_URL must be defined (in Jenkins->Manage Jenkins->Configure System and define 'Jenkins URL' and save the settings).
VectorCAST HTML reports for metrics were updated to use cascading style sheets (CSS) in the 2019 release and 2020 for top level project metrics. This was done to give users more flexibility in what and how metrics are displayed. To maintain single file HTML format, VectorCAST Reports used inline CSS. Inline CSS was disallowed under Jenkins more restrictive CSP.
"Jenkins 1.641 / Jenkins 1.625.3 introduce the Content-Security-Policy header to static files served by Jenkins (specifically, DirectoryBrowserSupport). This header is set to a very restrictive default set of permissions to protect Jenkins users from malicious HTML/JS files in workspaces, /userContent, or archived
artifacts."
The result of this combination incorrectly formatted the VectorCAST reports.
Numerous options are available to correct this:
- Use the Jenkins Resource Root URL (Manage Jenkins > Configure System)
- Reconfigure VectorCAST Jobs to use external CSS (VCAST_RPTS_SELF_CONTAINED=FALSE) combined with enabling anonymous reads from the Manage Jenkins > Configure Global Security > Authorization
- Reconfigure the Jenkins Content Security Policy
- Download the archives and view reports locally
For more information on the Jenkins CSP, please see Configuring Content Security Policy
For more information on VectorCAST Reports and Jenkins Content Security Policy, please see the article VectorCAST Reports and Jenkins Content Security Policy
For non-pipeline jobs, JUnit publisher will fail any environments published without test results. If you have an environment with no test results, you will need to manually check the box "Do not fail the build on empty test results" in the Publish JUnit test result report configuration.
For customers using VectorCAST's requirements gateway, there's a potential for loss of requirements data when running test environments in parallel while using a shared requriments database.
For environments that use imported results with versions of VectorCAST before 2020, reporting of test results and code coverage will not properly generate because of the lack of required build information.
VectorCAST/QA projects cannot use imported results for change based testing
In rare cases, VectorCAST projects will have disabled environment with results stored before they were disabled. In cases where the disabled environments share source file with enabled environments, this may lead addition coverage metrics. It is recommended to clean the environment before disabling. This takes into account enviornments that are directly disabled or disabled at the Compiler or TestSuite Nodes.
- Updated for findbugs to spotbugs
- Fixed changed based testing analysis for compound tests being shown as Skipped
- Fixed getjobs.py script to account for disabled compiler/testsuite nodes
- Added option for adding strict test importing
- Added option for using imported results for change based Testing
- Local results will be saved as a Jenkins Build artifact
- External results will be used from the repository
- Added support for DataAPI VCProjectApi for reporting imported results
- Bump json from 20220320 to 20230227
- Added maximum number of jobs to be queued so that server won't get flood with jobs and stall other Jenkins jobs
- Removed Squore script as customer should use script from their Squore installation.
- Fixed copy_build_dir.py ability to handle relative paths in the database.
- Removed duplicate code/variables in NewPipeline.java.
- Switched from CustomReport.report_from_api to api.report to solve corner case
- Added support in VectorCAST version 2023 Projects for environments that are not in a group
- Decoupled the two parameters used for parameterized pipeline
- Fixed skipped test analysis issue for overloaded C++ methods
- Fixed error in pipeline syntax "continue" changed to "return" for each loop
- Fixed single job error when not using CBT for missing incremental build rebuild report
- Added option to have pipeline job be marked as failed if statement or branch coverage decreases
- Added post-SCM step to run commands after SCM checkout before the project build/execute
- Fixed error identifying unbuilt migrated system test projects
- findbugs-related warnings in the output from maven are fixed in the VectorCAST/Execution plugin v0.74
- The Jenkinsfile base has been restructured and commented for pipeline jobs
- Added catchError to additional tools stage
- Additional Java tests incorporated into the plugin build
- The VectorCAST icon has been updated to match the Jenkins design aesthetic
- Previously, errors indicated that calls cannot run in the Jenkins sandbox. VectorCAST/Execution v0.74 fixed errors caused by the updated version of Java required for the Jenkins plugin.
- Fixed support for entering a Windows path for PC Lint Plus or Squore commands.
- Fixed support for Continuous Integration Licenses.
- Fixed support for System Test Environments with Squore.
- Fixed issue when TESTinsights project name is not given, it is named after the job.
- Support was removed for the deprecated VectorCAST Multi-Jobs. Users should create VectorCAST Pipeline Jobs in place of VectorCAST Multi-Jobs.
- Previously, VectorCAST Jobs were executed on the built-in node by default. Now, when creating new VectorCAST Job, the user must provide the name of the node that the job should be executed on. Jenkins advises against running builds on the built-in node. See Controller Isolation for more information.
- A new stage was added to VectorCAST Pipeline jobs that runs the system tests in serial after running all unit tests. This prevents multiple instances of VectorCAST from modifying the same instrumented source files at the same time.
- Support for additional external tools was added for VectorCAST jobs. See the Additional Tools options on the job creation page.
- PC-lint Plus
- Squore
- TESTinsights
- Options were reorganized on the VectorCAST Pipeline Job creation page.
- The VectorCAST Execution Plugin version is now displayed on the VectorCAST Jobs page.
- Previously, custom reports that used the VCAST_MANAGE_PROJECT_DIRECTORY environment variable would fail to generate. Now, the environment variable is set when the reports are generated.
- Updated baseline pipeline script to reflect changes in the code coverage plugin
- Updated to skip CBT analysis on migrated system test projects
- Updated script to use import to detect DataAPI.
- Updated scripts to use io.open for all opens
- Updated parse_traceback to display all tracebacks
- Updated pipeline job to handle continuous integration licenses
- New HTML reports match current Jenkins CSP
- Encoding read for GBK characters
- Bug fixes for compatibility issues with VectorCAST/2021
- Bug fix for counting run system test cases as skipped
- Handle uppercase VectorCAST Project .vcm extension
- Add support for Python 3.9
- Fixed bug caught by updated to Python 3
- Fixed command capture for processing later
- Changed side panel display of VectorCAST to not be shown when anonymous user is using Jenkins
- Single jobs not displaying full report and rebuild report correctly in summary (VC2020)
- Capturing stdout for skipped analysis causing console output to be delayed
- Pipeline job fails unless user specific .vcm for VectorCAST Project
- Problem with CBT analysis when not using single checkout directory
- Using lowercase %workspace% causes single checkout directory error
- Updated managewait.py to overcome race condition causing script to hang on readline()
- Added skipped test analysis to Single Jobs
- Added detection of disabled environment to DataAPI generated results
- Added addition error detection
- Skipped tests not detected for cover environments
- Check for illegal characters in pipeline job names
- Add Option to not use CBT in Jenkins pipeline plugin
- VC2020 Manage reports have no formatting in Jenkins job summary
- Remove xUnit reporting
- Remove truncated output during the build
- "Could not acquire a read lock on the project's vcm file" should raise an error, but it does not
- Add option to active CI licenses
- Regression Scripts with identical ending directories show testcase as skipped
- Add option to make job parameterized for VectorCAST Project location and Force Node Execution rather than compiler
- Update error thrown when user creates absolute path with SCM snippet
- Need to add more error detection
- Expand JUnit test case name display to include file.subprogram
- Change reports to not use VCAST_RPTS_SELF_CONTAINED=FALSE
- Support Multi-Branch Pipelines
- Include plugin version in Pipeline Job Jenkins Script and console log
- Check for illegal characters in Pipeline Job names
- Update Single and Multi-Jobs to use same reporting as Pipeline Jobs
- Remove CombinedReport.html from previous Pipeline job build at beginning of new build
- Make aggregate coverage results consistent with VectorCAST Project aggregate coverage results
- Always use new VectorCAST report API for VectorCAST/2019 and later
- Fix potential file access race condition when generating xml
- Fix exception when building VectorCAST/Unit regression script environments
- Support for VectorCAST/vpython 3
- Introduce pipeline job
- Updated support for Jenkins Pipeline
- Updated python scripts to work with VectorCAST 2019 SP3
- Added fix for Ada environments not building properly
- Update support for Manage API and VCAST_RPTS_SELF_CONTAINED
- Update for VECTORCAST_DIR rollback
- Update for different drive than workspace
- Updates for duplicate results in multijob
- Revert changes from 0.43 and 0.44 to remove need for VECTORCAST_DIR. VECTORCAST_DIR will need to be defined when running Jenkins for now.
- Add support for new VCAST_RPTS_SELF_CONTAINED option added in VC19 SP2
- Add support for using Manage API to generate XML reports if available
- Correct variable expression on Linux platforms
- Updated scripts to handle compound only in compound case
- Updates for F+FC coverage and updating database pathing
- Updates for changed to aggregate coverage report metrics heading change
- Additional debug logging
- Updates for scripts not handling Ada operator "+"
- Update exception handling when using DataAPI
- Updated to support VectorCAST 2019 SP1
- Update to get complete display name for C++ functions
- Update to fix characters that need escaping in XML attributes
- Fix for disabled environments
- Fix for printing non UTF-8 compliant failure code
- Updates for using external CSS/images
- Removed xUnit dependency and converted to JUnit.
- Added better legacy support for VectorCAST installations that do not have their executables on the system PATH
- Removed all uses of the environment variable VECTORCAST_DIR. From now on it is assumed that VectorCAST executables are on the system PATH. Legacy support is still maintained for older versions of VectorCAST.
- Additional cleaning up of old files
- Updates for corner cases, verbose out issue, and cleaning up previous build's files
- Problem when function coverage enabled, but not function call
- Function coverage format incorrect in XML causing plugin to throw an error
- Added catch for additional licensing errors
- Added catch for all (E) Line: errors in the console log
- Fix for function and basis path coverage when using VectorCAST 2019
- Update to fix auto job updates (where path to VectorCAST Project was being removed)
- Update to make the management report generate for a cover project
- Fix for spurious newline characters in report title in XML for Jenkins with VectorCAST 2019
- Corrected missing " that may affect running multi-job on Linux
- Added support for generating reports using VectorCAST 2019
- Support overlapping version 17 VectorCAST Projects
- Updates to support long directory paths in VectorCAST Project reporting
- Support newer versions of xUnit plugin
- Support MultiJob plugin up to 0.29 and later, 0.30 onwards
-
Don't create intermediate CSV file for bad test case management report
-
Raise post-groovy alert for bad test case management report
- Improve support for unit without coverage, avoiding corrupt xml files
- Removed spurious " in Linux single job
- Corrected link from xUnit graph to report
- Correct regression with report naming for archived artifacts with shorter names
- Added environment variable (VCAST_VC_SCRIPTS) to provide optional source of vc_scripts
- Improve support for long VectorCAST Project names, environment names and compiler names
- Correct regression with windows variable names being used in Unix script
- Correct regression with missing space in commands for single job
- Option to set the name of the single job or multi-job (name is pre-pended to sub-job in the case of multi-jobs)
- Option to configure (at creation/update time) the node to run the single job or top-level multi-job on
- Allow license retries for single jobs
- Update summary/detailed text written by the groovy scripts
- Update to retry functionality to support jobs created with earlier plugin versions
- Update to store and use job details when auto-updating
- Added (optional) ability to retry a command if it fails due to unavailable license
- Added a job that can be used to update an existing multi-job
- Added support for new version of VectorCAST Manage that uses 2 levels instead of 4
- Improved groovy script to mark failing builds as failed rather than unstable
- Allow conversion script to accept report that has a missing or incomplete Function Coverage column
- Added --force option to use of --release-locks
- Added option to use either HTML or TEXT format for the build description
- Add execution report link to all test cases
- Added update to pulling in both the full report and incremental build report into the job build description
- Added update to pulling in both the full report and incremental build report into the job build description
- Always display the VectorCAST menu and leave permission checking/reporting to Jenkins
- Corrected processing checking if BUILD_URL has been set
- Corrected processing to support function and function call coverage
- Corrected typos in Diagnostics job and pattern for files to copy
- Add support for spaces in paths
- Add support to keep or clean the working directory
- Support added for multi-job with SCM and for calculating correctly aggregated coverage for the top-level display in the VectorCAST coverage plugin
- Initial release (no support for using SCM with multi-job)