Skip to content

Commit

Permalink
Update quality declaration and coverage (#674)
Browse files Browse the repository at this point in the history
* Update quality level and coverage

Signed-off-by: ahcorde <[email protected]>

* Added feedback

Signed-off-by: ahcorde <[email protected]>

* Fixed wording and links

Signed-off-by: ahcorde <[email protected]>

* Updated QD

Signed-off-by: ahcorde <[email protected]>
  • Loading branch information
ahcorde committed Oct 27, 2020
1 parent fa4e9c6 commit e5fbbb4
Show file tree
Hide file tree
Showing 8 changed files with 76 additions and 27 deletions.
10 changes: 6 additions & 4 deletions rcl/QUALITY_DECLARATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ This document is a declaration of software quality for the `rcl` package, based

# `rcl` Quality Declaration

The package `rcl` claims to be in the **Quality Level 4** category.
The package `rcl` claims to be in the **Quality Level 3** category.

Below are the rationales, notes, and caveats for this claim, organized by each requirement listed in the [Package Quality Categories in REP-2004](https://index.ros.org/doc/ros2/Contributing/Developer-Guide/#package-quality-categories).

Expand Down Expand Up @@ -101,7 +101,9 @@ Currently nightly test results can be seen here:

### Public API Testing [4.ii]

Most of the functionality of the declared API in this package is covered in its unit tests. Currently it has a line coverage of [77%](https://ci.ros2.org/job/ci_linux_coverage/85/cobertura/src_ros2_rcl_rcl_src_rcl/).
Each part of the public API has tests, and new additions or changes to the public API require tests before being added. The tests aim to cover both typical usage and corner cases, but are quantified by contributing to code coverage.

The following functions are partially supported: `rcl_take_loaned_message`, `rcl_return_loaned_message_from_subscription`, `rcl_borrow_loaned_message`, `rcl_return_loaned_message_from_publisher` and `rcl_publish_loaned_message` because they are not currently supported on Tier 1 RMW providers.

### Coverage [4.iii]

Expand All @@ -114,7 +116,7 @@ This includes:

Changes are required to make a best effort to keep or increase coverage before being accepted, but decreases are allowed if properly justified and accepted by reviewers.

Current coverage statistics can be viewed (77%) [here](https://ci.ros2.org/job/ci_linux_coverage/85/cobertura/src_ros2_rcl_rcl_src_rcl/). Current coverage statistics need to be improved to reach a higher quality level.
Current coverage statistics can be viewed [here](https://ci.ros2.org/job/nightly_linux_coverage/lastSuccessfulBuild/cobertura/src_ros2_rcl_rcl_src_rcl/). A description of how coverage statistics are calculated is summarized in this page ["ROS 2 Onboarding Guide"](https://index.ros.org/doc/ros2/Contributing/ROS-2-On-boarding-Guide/#note-on-coverage-runs).

### Performance [4.iv]

Expand Down Expand Up @@ -167,7 +169,7 @@ It is **Quality Level 4**, see its [Quality Declaration document](https://github

The `rcutils` package provides an API which contains common utilities and data structures needed when programming in C.

It is **Quality Level 4**, see its [Quality Declaration document](https://github.com/ros2/rcutils/blob/master/QUALITY_DECLARATION.md).
It is **Quality Level 2**, see its [Quality Declaration document](https://github.com/ros2/rcutils/blob/master/QUALITY_DECLARATION.md).

#### `rmw_implementation`

Expand Down
2 changes: 1 addition & 1 deletion rcl/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ Library to support implementation of language specific ROS Client Libraries.

Features are described in detail at [http://docs.ros2.org](http://docs.ros2.org/latest/api/rcl/index.html)

This package claims to be in the **Quality Level 4** category, see the [Quality Declaration](./QUALITY_DECLARATION.md) for more details.
This package claims to be in the **Quality Level 3** category, see the [Quality Declaration](./QUALITY_DECLARATION.md) for more details.
28 changes: 21 additions & 7 deletions rcl_action/QUALITY_DECLARATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ This document is a declaration of software quality for the `rcl_action` package,

# `rcl_action` Quality Declaration

The package `rcl_action` claims to be in the **Quality Level 4** category.
The package `rcl_action` claims to be in the **Quality Level 3** category.

Below are the rationales, notes, and caveats for this claim, organized by each requirement listed in the [Package Requirements for Quality Level 4 in REP-2004](https://www.ros.org/reps/rep-2004.html).
Below are the rationales, notes, and caveats for this claim, organized by each requirement listed in the [Package Quality Categories in REP-2004](https://index.ros.org/doc/ros2/Contributing/Developer-Guide/#package-quality-categories).

## Version Policy [1]

Expand Down Expand Up @@ -88,15 +88,29 @@ The results of the test can be found [here](https://ci.ros2.org/view/nightly/job

`rcl_action` has feature tests, which test for proper node state transitions.
The tests are located in the [test](test) subdirectory.
New features are required to have tests before being added.
Currently nightly test results can be seen here:
* [linux-aarch64_release](https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_release/lastBuild/testReport/rcl_action/)
* [linux_release](https://ci.ros2.org/view/nightly/job/nightly_linux_release/lastBuild/testReport/rcl_action/)
* [mac_osx_release](https://ci.ros2.org/view/nightly/job/nightly_osx_release/lastBuild/testReport/rcl_action/)
* [windows_release](https://ci.ros2.org/view/nightly/job/nightly_win_rel/lastBuild/testReport/rcl_action/)

### Public API Testing [4.ii]

Much of the API in `rcl_action` is tested.
The tests are located in the [test](test) subdirectory.
Each part of the public API has tests, and new additions or changes to the public API require tests before being added. The tests aim to cover both typical usage and corner cases, but are quantified by contributing to code coverage.

### Coverage [4.iii]

`rcl_action` does not currently track test coverage.
`rcl_action` follows the recommendations for ROS Core packages in the [ROS 2 Developer Guide](https://index.ros.org/doc/ros2/Contributing/Developer-Guide/#code-coverage), and opts to use line coverage instead of branch coverage.

This includes:

- tracking and reporting line coverage statistics
- no lines are manually skipped in coverage calculations

Changes are required to make a best effort to keep or increase coverage before being accepted, but decreases are allowed if properly justified and accepted by reviewers.

Current coverage statistics can be viewed [here](https://ci.ros2.org/job/nightly_linux_coverage/lastSuccessfulBuild/cobertura/src_ros2_rcl_rcl_action_src_rcl_action/). A description of how coverage statistics are calculated is summarized in this page ["ROS 2 Onboarding Guide"](https://index.ros.org/doc/ros2/Contributing/ROS-2-On-boarding-Guide/#note-on-coverage-runs).

### Performance [4.iv]

Expand Down Expand Up @@ -129,13 +143,13 @@ It is **Quality Level 4**, see its [Quality Declaration document](https://github

`rcl` is the ROS 2 client library in C.

It is **Quality Level 4**, see its [Quality Declaration document](../rcl/QUALITY_DECLARATION).
It is **Quality Level 3**, see its [Quality Declaration document](../rcl/QUALITY_DECLARATION).

#### `rcutils`

`rcutils` provides commonly used functionality in C.

It is **Quality Level 4**, see its [Quality Declaration document](https://github.com/ros2/rcutils/blob/master/QUALITY_DECLARATION.md).
It is **Quality Level 2**, see its [Quality Declaration document](https://github.com/ros2/rcutils/blob/master/QUALITY_DECLARATION.md).

#### `rmw`

Expand Down
2 changes: 1 addition & 1 deletion rcl_action/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ http://design.ros2.org/articles/actions.html

## Quality Declaration

This package claims to be in the **Quality Level 4** category, see the [Quality Declaration](./QUALITY_DECLARATION.md) for more details.
This package claims to be in the **Quality Level 3** category, see the [Quality Declaration](./QUALITY_DECLARATION.md) for more details.
26 changes: 21 additions & 5 deletions rcl_lifecycle/QUALITY_DECLARATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ This document is a declaration of software quality for the `rcl_lifecycle` packa

# `rcl_lifecycle` Quality Declaration

The package `rcl_lifecycle` claims to be in the **Quality Level 4** category.
The package `rcl_lifecycle` claims to be in the **Quality Level 3** category.

Below are the rationales, notes, and caveats for this claim, organized by each requirement listed in the [Package Requirements for Quality Level 4 in REP-2004](https://www.ros.org/reps/rep-2004.html).
Below are the rationales, notes, and caveats for this claim, organized by each requirement listed in the [Package Quality Categories in REP-2004](https://index.ros.org/doc/ros2/Contributing/Developer-Guide/#package-quality-categories).

## Version Policy [1]

Expand Down Expand Up @@ -88,14 +88,30 @@ The results of the test can be found [here](https://ci.ros2.org/view/nightly/job

`rcl_lifecycle` has feature tests, which test for proper node state transitions.
The tests are located in the [test](test) subdirectory.
New features are required to have tests before being added.
Currently nightly test results can be seen here:
* [linux-aarch64_release](https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_release/lastBuild/testReport/rcl_lifecycle/)
* [linux_release](https://ci.ros2.org/view/nightly/job/nightly_linux_release/lastBuild/testReport/rcl_lifecycle/)
* [mac_osx_release](https://ci.ros2.org/view/nightly/job/nightly_osx_release/lastBuild/testReport/rcl_lifecycle/)
* [windows_release](https://ci.ros2.org/view/nightly/job/nightly_win_rel/lastBuild/testReport/rcl_lifecycle/)


### Public API Testing [4.ii]

Much of the API in `rcl_lifecycle` is tested in the aforementioned feature tests, but it is not tested explicitly.
Each part of the public API has tests, and new additions or changes to the public API require tests before being added. The tests aim to cover both typical usage and corner cases, but are quantified by contributing to code coverage.

### Coverage [4.iii]

`rcl_lifecycle` does not currently track test coverage.
`rcl_lifecycle` follows the recommendations for ROS Core packages in the [ROS 2 Developer Guide](https://index.ros.org/doc/ros2/Contributing/Developer-Guide/#code-coverage), and opts to use line coverage instead of branch coverage.

This includes:

- tracking and reporting line coverage statistics
- no lines are manually skipped in coverage calculations

Changes are required to make a best effort to keep or increase coverage before being accepted, but decreases are allowed if properly justified and accepted by reviewers.

Current coverage statistics can be viewed [here](https://ci.ros2.org/job/nightly_linux_coverage/lastSuccessfulBuild/cobertura/src_ros2_rcl_rcl_lifecycle_src/). A description of how coverage statistics are calculated is summarized in this page ["ROS 2 Onboarding Guide"](https://index.ros.org/doc/ros2/Contributing/ROS-2-On-boarding-Guide/#note-on-coverage-runs).

### Performance [4.iv]

Expand Down Expand Up @@ -128,7 +144,7 @@ It is **Quality Level 4**, see its [Quality Declaration document](https://github

`rcl` is the ROS 2 client library in C.

It is **Quality Level 4**, see its [Quality Declaration document](../rcl/QUALITY_DECLARATION).
It is **Quality Level 3**, see its [Quality Declaration document](../rcl/QUALITY_DECLARATION).

#### `rcutils`

Expand Down
2 changes: 1 addition & 1 deletion rcl_lifecycle/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ https://design.ros2.org/articles/node_lifecycle.html

## Quality Declaration

This package claims to be in the **Quality Level 4** category, see the [Quality Declaration](./QUALITY_DECLARATION.md) for more details.
This package claims to be in the **Quality Level 3** category, see the [Quality Declaration](./QUALITY_DECLARATION.md) for more details.
31 changes: 24 additions & 7 deletions rcl_yaml_param_parser/QUALITY_DECLARATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ This document is a declaration of software quality for the `rcl_yaml_param_parse

# `rcl_yaml_param_parser` Quality Declaration

The package `rcl_yaml_param_parser` claims to be in the **Quality Level 4** category.
The package `rcl_yaml_param_parser` claims to be in the **Quality Level 3** category.

Below are the rationales, notes, and caveats for this claim, organized by each requirement listed in the [Package Requirements for Quality Level 4 in REP-2004](https://www.ros.org/reps/rep-2004.html).
Below are the rationales, notes, and caveats for this claim, organized by each requirement listed in the [Package Quality Categories in REP-2004](https://index.ros.org/doc/ros2/Contributing/Developer-Guide/#package-quality-categories).

## Version Policy [1]

Expand Down Expand Up @@ -59,10 +59,12 @@ All pull requests must resolve related documentation changes before merging.

### Feature Documentation [3.i]

`rcl_yaml_param_parser` does not have feature documentation.
`rcl_yaml_param_parser` provides the main elements of its API listed using doxygen and is hosted [here](http://docs.ros2.org/latest/api/rcl_yaml_param_parser/index.html).

### Public API Documentation [3.ii]

`rcl_yaml_param_parser` has embedded API documentation and it is generated using doxygen. Currently, its latest version is hosted [here](http://docs.ros2.org/latest/api/rcl_yaml_param_parser/index.html). Latest version has to be generated before considering this item fully resolved.

All of `rcl_yaml_param_parser` has embedded API documentation. It is not yet hosted publicly.

### License [3.iii]
Expand All @@ -85,15 +87,30 @@ The results of the test can be found [here](https://ci.ros2.org/view/nightly/job

### Feature Testing [4.i]

`rcl_yaml_param_parser` has feature tests, which ensure that a wide variety of yaml files are checked for parsing.
Most features in `rcl_yaml_param_parser` have corresponding tests which simulate typical usage, and they are located in the [`test`](./test) directory.
New features are required to have tests before being added.
Currently nightly test results can be seen here:
* [linux-aarch64_release](https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_release/lastBuild/testReport/rcl_yaml_param_parser/)
* [linux_release](https://ci.ros2.org/view/nightly/job/nightly_linux_release/lastBuild/testReport/rcl_yaml_param_parser/)
* [mac_osx_release](https://ci.ros2.org/view/nightly/job/nightly_osx_release/lastBuild/testReport/rcl_yaml_param_parser/)
* [windows_release](https://ci.ros2.org/view/nightly/job/nightly_win_rel/lastBuild/testReport/rcl_yaml_param_parser/)

### Public API Testing [4.ii]

Much of the API in `rcl_yaml_param_parser` is tested in the aforementioned feature tests, but it is not tested explicitly.
Each part of the public API has tests, and new additions or changes to the public API require tests before being added. The tests aim to cover both typical usage and corner cases, but are quantified by contributing to code coverage.

### Coverage [4.iii]

`rcl_yaml_param_parser` does not currently track test coverage.
`rcl_yaml_param_parser` follows the recommendations for ROS Core packages in the [ROS 2 Developer Guide](https://index.ros.org/doc/ros2/Contributing/Developer-Guide/#code-coverage), and opts to use line coverage instead of branch coverage.

This includes:

- tracking and reporting line coverage statistics
- no lines are manually skipped in coverage calculations

Changes are required to make a best effort to keep or increase coverage before being accepted, but decreases are allowed if properly justified and accepted by reviewers.

Current coverage statistics can be viewed [here](https://ci.ros2.org/job/nightly_linux_coverage/lastSuccessfulBuild/cobertura/src_ros2_rcl_rcl_yaml_param_parser_src/). A description of how coverage statistics are calculated is summarized in this page ["ROS 2 Onboarding Guide"](https://index.ros.org/doc/ros2/Contributing/ROS-2-On-boarding-Guide/#note-on-coverage-runs).

### Performance [4.iv]

Expand All @@ -118,7 +135,7 @@ It also has several test dependencies, which do not affect the resulting quality

`rcutils` provides commonly used functionality in C.

It is **Quality Level 4**, see its [Quality Declaration document](https://github.com/ros2/rcutils/blob/master/QUALITY_DECLARATION.md).
It is **Quality Level 3**, see its [Quality Declaration document](https://github.com/ros2/rcutils/blob/master/QUALITY_DECLARATION.md).

#### `libyaml_vendor`

Expand Down
2 changes: 1 addition & 1 deletion rcl_yaml_param_parser/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,4 @@ This package depends on C libyaml.

## Quality Declaration

This package claims to be in the **Quality Level 4** category, see the [Quality Declaration](./QUALITY_DECLARATION.md) for more details.
This package claims to be in the **Quality Level 3** category, see the [Quality Declaration](./QUALITY_DECLARATION.md) for more details.

0 comments on commit e5fbbb4

Please sign in to comment.