Skip to content

Latest commit

 

History

History
268 lines (208 loc) · 13.9 KB

README.md

File metadata and controls

268 lines (208 loc) · 13.9 KB

Selenium tests using Avocado

These use seleniumlib.py wrappers around selenium

Libraries are accessible in testlib_avocado module.

How to run tests

The default way to run selenium tests is to use the testing images that CI uses and test cockpit installation inside a test VM.

Currently, these tests run on Fedora 31. Other images don't have selenium and avocado installed.

$ test/image-prepare fedora-31 # Install code to test

Run the run-tests script with appropriate parameters.

  • bots/image-download services # Download a VM image with pre-installed selenium
  • TEST_OS=fedora-31 test/selenium/run-tests --browser firefox -v

Although this is the default way to run selenium tests the run-tests script is configurable and can be changed to run tests against different machines. This can be useful for developing or debugging tests. Check bellow the HACKING section for more details.

Debugging tests:

When running selenium tests with run-tests you can debug them in the following ways.

  • Run just selected (one or more) tests via listing them on the command line, tests are relative to test/selenium/ directory (for example: TEST_OS=fedora-31 test/selenium/run-tests --browser firefox -v selenium-base.py) test filename is relative to test/selenium/ directory
  • Pass --sit parameter to run-tests which will leave all test machines running after the tests finish.
  • Use own selenium grid via option --hub

Use selenium grid in container

Start own selenium grid in container (Original description).

  • Create selenium grid with chrome browser with VNC enabled (you can replace it by firefox and change commands according to that): 
    podman run --rm -p 4444:4444 -p 5555:5900 -v /dev/shm:/dev/shm selenium/standalone-chrome-debug
  • Start VNC session to that machine (password is secret): 
    vncviewer 127.0.0.1:5555
  • Enable port forwarding of port 127.0.0.2:9091 to publicly accessible address, because selenium hub in podman container will have trouble to contact host port because it is forwarded right to 127.0.0.2. YOUR_IP is the hostname or IP address of your machine, that the selenium HUB will be able to access. 
    ssh -L $YOUR_IP:9991:127.0.0.2:9091 $(whoami)@localhost
  • Run the test. HUB_IP is IP address of your selenium grid, has to be able to see cockpit instance via port 9991 on GUEST_IP 
    test/selenium/run-tests --hub $HUB_IP:$GUEST_IP:9991 -v -b chrome selenium-base.py --sit

Selenium grid on your machine with your local browsers

Steps are described in selenium page

  • Download selenium standalone: 
    curl -f -L https://selenium-release.storage.googleapis.com/3.141/selenium-server-standalone-3.141.59.jar > selenium.jar
  • Run hub and attach the browsers here: 
    java -jar selenium.jar -role hub
    • Then attach your local browsers there. You have browsers, for example, firefox or chrome installed, and also ensure you have proper selenium drivers for the selected the browser 
      java -jar selenium.jar -role node -hub https://localhost:4444/grid/register
  • Run the test. HUB_IP is the IP address of your selenium grid 
    test/selenium/run-tests --hub $HUB_IP -v -b chrome selenium-base.py --sit

Reschedule the the test/s again

It is possible to reschedule tests if you used --sit option for run-tests, in case of failure, machines won't be destroyed. Command-line options depend on which selenium grid you used. It is described on the output of the run-tests script. For example:

$test/selenium/run-tests --hub $HUB_IP -v -b chrome selenium-navigate.py --sit

...
To rerun tests connect to VM with avocado installed via
    ssh -p 2202 [email protected]

and run the command
    PYTHONPATH=/tmp/avocado_library HUB=$HUB_IP_ADDR GUEST=127.0.0.2
PORT=9091 SSH_PORT=22 SSH_GUEST=10.111.113.1 BROWSER=chrome timeout 297
python3 -m avocado run --show-job-log /tmp/avocado_tests/selenium-navigate.py:NavigateTestSuite.testNavigateNoReload 2>&1

It is possible also to run the tests on your machine, without test VM (bots/images) that contains the installed avocado testing framework.

  • You have to install the avocado framework for python3 (for example via pip: pip3 install avocado-framework)
  • Then a little bit modify command what you see on the output of the run-tests command and it depends how you run the selenium grid. 
    HUB=$HUB_IP GUEST=$GUEST_IP PORT=9991 SSH_PORT=2021 SSH_GUEST=127.0.0.2 BROWSER=chrome python3 -m avocado --show=test run  test/selenium/selenium-navigate.py
  • Remember
    • An avocado runner is:
      • for run-tests script it is dedicated VM (different from GUEST and is not configurable from outside this script)
      • but could be your machine, or anywhere, in case it can fulfill the next conditions
    • GUEST has to be accessible via port PORT from the HUB
    • HUB has to be accessible via port 4444 from the avocado runner
    • SSH_GUEST has to be accessible via port SSH_PORT from avocado runner (GUEST and SSH_GUEST is the same machine, but may have another address, because of port forwarding and HUB location)
Description of example output
  • PYTHONPATH=/tmp/avocado_library - path to libraries
  • HUB=$HUB_IP_ADDR - IP address of your selenium grid
  • GUEST=127.0.0.2 - IP address of cockpit machine (HUB has to see GUEST cockpit port)
  • PORT=9091 - port of cockpit machine on GUEST IP address
  • SSH_PORT=22 - ssh port number on SSH_GUEST
  • SSH_GUEST=10.111.113.1 - guest ip how to connect to ssh (GUEST has to be able to connect SSH_GUEST:SSH_PORT)
  • BROWSER=chrome - name of browser to use (possible options: chrome, firefox, edge)
  • timeout 297 - timeout to kill the avocado process (could be removed in case of manual rescheduling)
  • python3 -m avocado run --show-job-log - avocado test scheduler (in case you have newer avocado (>70.0) there is little bit different sytanx python3 -m avocado --show=test run )
  • /tmp/avocado_tests/selenium-navigate.py:NavigateTestSuite.testNavigateNoReload name of test, it consists of filename_path:Class_name.tests_method_name
  • 2>&1 - could be removed in case of manual rescheduling

Hacking

Selenium tests

run-tests script can change its behavior by the environment variables specified bellow.

Where cockpit is running

Defines where is the cockpit instance that you want to test. This cannot be used together with the TEST_OS variable.

  • GUEST (default: localhost) - defines hostname or IP of the machine where cockpit-ws is running
    • This machine has to have enabled ssh for execution remote commands for purpose of self.machine.execute
  • PORT (default: 9090) - defines the port where cockpit-ws component accepts connections in GUEST machine
  • URL_BASE (default: http) - defines what protocol to use, http or https.
  • SSH_GUEST (default: same as GUEST) - define name or ip of cockpit machine, it is important for the run-tests scheduler for debugging, where is used port forwarding
  • SSH_PORT (default: 22) - use another port to connect to cockpit machine

Leads to address URL_BASE//GUEST:PORT

Where is selenium running

There are two ways to use selenium:

The following environment variables can be used to configure selenium options inrun-tests script:

  • HUB (default: localhost) - location of selenium grid (on port 4444)
  • LOCAL (default: no) - if you set to yes, it will use installed browsers directly
    • WARN - if LOCAL option set to yes HUB option is ignored

Which browser to use

It is possible to test 3 browsers - Firefox, Google Chrome or Microsoft Edge. It is highly connected with the previous section, depends what your selenium grid or your local machine supports and has installed or registered. Browser selection can be done by the BROWSER environment variable (default is firefox).

BROWSER variable can take one of the following values:

  • firefox - it will use Firefox as a browser
    • WARN please ensure that you have installed gecko driver in your PATH. New Firefox browsers are not working without this driver
  • chrome - Will use Google Chrome browser
  • edge - Will use Microsoft Edge browser

There are several possibilities how to do it:

  • Directly - Use local browser when used LOCAL=yes you have to have your browser browser installed
  • Via selenium grid directly (will register you local browser to hub on port 4444
$ java -jar selenium-server-standalone-2.44.0.jar -role hub
$ java -jar selenium-server-standalone-2.44.0.jar -role node  -hub
  • via docker selenium grid - it will redirect HUB port 4444 to your machine
$ docker run -d -p 4444:4444 --name selenium-hub selenium/hub
$ docker run -d --link selenium-hub:hub selenium/node-chrome
$ docker run -d --link selenium-hub:hub selenium/node-firefox

Other options

  • IDENTITY (default: testlib_avocado/identity symlink) - private key to run commands on GUEST machine to execute commands there via self.machine.execute. Please do not use subprocess modules or similar way, because then commands will every time run locally not on defined machine.

How to run tests against local cockpit installation with local browser

Generic dependencies

When not using the selenium image the following dependencies need to be installed on the system.

sudo pip3 install selenium
sudo pip3 install avocado-framework

Firefox with gecko driver

sudo dnf install firefox
curl -L -f https://github.com/mozilla/geckodriver/releases/download/v0.23.0/geckodriver-v0.23.0-linux64.tar.gz > geckodriver.tar.gz
tar xzvf geckodriver.tar.gz
cp geckodriver /usr/local/bin

Other requirements

When not using selenium image you need to have the created an test user with credentials defined in seleniumlib

And finally run selected test(s) LOCAL=yes BROWSER=firefox avocado-3 run test/selenium/selenium-base.py

How to write tests

The rules for tests are a bit different:

  • One or more machines are dedicated to running all our tests: The machines are initialized specifically for this purpose, and will not be reused for anything else after all tests have been run.

  • A test runs as root directly in one of the machines dedicated to the test run.

  • A test is allowed to drastically modify the machine, as long as that doesn't break other tests.

  • The tests should aim to be independent from each other: Although a the test run is deterministic and the tests are always run in the same order, implicit or explicit dependencies between tests should be avoided. If they can't easily be avoided, they need to be documented, of course.

  • This means that tests should clean up after themselves, and restore the machine into the same configuration as it was before the test.

    Of course, a test should not aim for rootkit-level stealthiness: It can and should leave entries in the journal behind, etc.

More concretely:

  • When a test starts, the cockpit socket is already active, but the cockpit itself is not running. After a test, the cockpit will be stopped.

  • A test is a Python file in this directory modeled after example-check-foo.py.

  • Previously, we have combined multiple tests into one file, like check-storage, which has 12 tests in it. Now we need to split them out into 12 individual Python files, maybe with a common support module.

  • Each test is a class derived from cockpit.Test, which in turn is derived from avocado.test.Test.

  • The test class normally shouldn't define the usual 'setup', 'action', and 'cleanup' methods.

    Instead, it should define a 'test' method, and use cockpit.Test.atcleanup to register any cleanup actions it wants. The cockpit.Test class has some convenience functions such as 'replace_file' that should make this easier.

  • If you do want to define 'setup' or 'cleanup', make sure to call cockpit.Test.setup and cockpit.Test.cleanup as appropriate as part of it.

  • Use 'self.browser' to get an instance of the old Browser class. It works exactly as before.

  • The old 'self.allow_journal_messages' and 'self.allowed_restart_messages' are also still there.

  • The old 'self.machine' is gone. Use 'self.run_shell_command' or the more general 'avocado.utils.process'. (But make sure to undo the actions.)

  • To change files, use 'self.replace_file'. It will put the old content back during cleanup.

  • You can't use 'print' statements in the 'test' method. Avocado swallows the output. Use 'self.log.debug' or similar.

  • Use lib/var.env to pass information from setup.sh to the tests. It contains, for example, the IP of the DNS server to use when discovering the default domain.

    In a test, use 'self.environment' to access it.