Skip to content

Home

Jump to bottom
Till Maas edited this page Mar 18, 2020 · 6 revisions

Improve Network Linux System Role 2020 GSoC project

  • Difficulty: Intermediate
  • Technology: Python, Ansible
  • Mentors: Till Maas (@tyll) for Python/Ansible, Rich Meggison (richm) for Python/Ansible/CI, Pavel Cahyna (cahyna) for Python/Ansible/CI, Thomas Haller (thom311) for NetworkManager/initscripts/Linux networking
  • Project URL: https://github.com/linux-system-roles/network/

Description

The Network Linux System Role provides a uniform configuration interface for network-scripts and NetworkManager. In this project, the testing framework for the network role should be improved.

Improve the testing framework

Currently, the role is using unit tests for a subset of the code and playbooks as integration tests. They are run by a custom CI tool and are inspired by the Standard Test Interface. Additionally some static code analyzers are used. This project is focused on improving the integration tests.

There are several problems with the integration tests:

  1. Originally, the standard test interface is meant as a way to define how to call a test suite and not to be the test suite itself. Also, since Ansible is a automation framework and not a real programming language/testing framework, writing tests in pure Ansible is a complicated task.
  2. The log output from the integration tests is very verbose and there is no clear error reporting
  3. When testing several variations of a network configuration, the test runner runs all steps for the role for each variation. This makes testing slow.
  4. It does not provide an easy way to ignore failures

Pytest is a tool for testing python code with good reporting capabilities. To improve the integration tests, integration tests using pytest for the network role should be implemented. They should still be runnable by the current test runner. A possible implementation could be:

  1. Write an Ansible playbook that
    1. copies the test with all required files to the test system, possibly using the standard test roles
    2. install all dependencies for the test such as pytest
    3. Runs integration tests with pytest
  2. Create at least one example integration test that
    1. Includes a pytest fixture to setup and teardown veth interfaces and network profiles to be used for testing
    2. Calls code from the Ansible module for the network role and asserts that the profile exists on the system

Since the whole Ansible stack should be tested, too, there needs to be a better way to run tests with Ansible. A solution to simplify tests is to create custom Ansible modules that combine or simplify tasks that are needed for testing:

  1. Creating and removing test interfaces
  2. Asserting certain conditions such as
    1. that an interface is present or absent
    2. that a network profile is present or absent
    3. certain network settings are set, such as ethtool options
  3. Add support for the assertions to provide a good error message (ideally at the end of the playbook run) that includes the test name if the assertion failed
  4. Add support to ignore errors for assertions under certain conditions or to actually expect a failure (see xfail in pytest). This should allow to provide new tests to show a bug or a feature request to support test driven development.

A rough outline for the project is:

  1. Write an integration test to become familiar with the role, for example for https://github.com/linux-system-roles/network/issues/124
  2. Make the tests more uniform (they developed over the time. Adjust old tests to use the conventions of newer tests).
  3. Add support to test the ansible module directly via pytest instead of only via ansible-playbook, this should work for all supported platforms
  4. Add an integration tests for a profile type and for several state transitions such as removing an existing profile, taking an existing profile down, updating an existing profile
  5. Simplify test playbooks by writing custom Ansible modules that simplify test setup/preparation and assertions

The remaining time should be used to increase the coverage of the tests by adding tests that ideally test all relevant snippets from the README, the example playbooks or any untested feature.

What are we looking for

  • Interest in writing high-quality code in Python for Ansible
  • Personal accountability with regular, clear and open communication
  • Ability to independently transfer feedback into code
  • Support for more features or a better test framework
  • Effective collaboration via GitHub, IRC and video conferences

First Steps

  • Subscribe to the system roles mailing list and send an introduction message. Tell us which area interests you and why you would like to participate
  • If you are using IRC, join the #systemroles IRC channel on freenode and say Hi (it is rather new, therefore there is not much going on, yet)
  • Take a look at the integration tests and try to get them running.
  • Get the unit and linting tests running using tox
  • Optional: Get the molecule tests running by running tox -e molecule (this requires being able to run docker as the current user)
  • Study the recently added integration tests: tests_ethtool_features_nm.yml and tests_ethtool_features_initscripts.yml
  • Possible initial tasks:
    • Port one of the old tests to use the same format as the new tests:
      • Pick one of tests_bridge.yml, tests_default.yml, tests_ethernet.yml and create a new issue in the issue tracker to indicate that you plan to work on it and submit a PR with the new test that references the issue
      • Once you did it, create a new PR that references your issue and request a review from @tyll
    • Create a new test using the example files from https://github.com/linux-system-roles/network/issues/124#issuecomment-572250996
    • Look through the README for examples and write a simple test for them if there is none already
  • Watch this wiki page for updates
Clone this wiki locally