-
Notifications
You must be signed in to change notification settings - Fork 17
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into data-grid-update-cell
- Loading branch information
Showing
17 changed files
with
174 additions
and
87 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,137 +1,196 @@ | ||
--- | ||
title: End-to-end testing | ||
title: End-to-End Testing Your App | ||
editor: | ||
markdown: | ||
wrap: sentence | ||
--- | ||
|
||
End-to-end testing is like checking your app from start to finish, just as a user would. | ||
### What is End-to-End Testing (and Why Should You Care)? | ||
|
||
Imagine you're using your Shiny app. You click buttons, enter data, and see results on a graph or a dashboard. End-to-end tests mimic these actions. | ||
Instead of manually clicking around, we write code to do this for us. | ||
The code interacts with your app like a user, checking if everything works as expected. | ||
Imagine you've built a beautiful, interactive Shiny app. You want to make sure everything works exactly as expected, every time, for every user. That's where end-to-end testing comes in. | ||
|
||
#### Benefits | ||
- End-to-end tests find issues early, like broken links or unexpected behavior. | ||
- As your app grows, it becomes harder to keep track of all parts. Tests help ensure nothing breaks. | ||
**What it is:** | ||
|
||
* End-to-end testing checks your *entire* Shiny app, from start to finish, as if a real person were using it. | ||
* It simulates user actions like clicking buttons, filling in forms, and navigating between different parts of your app. | ||
* It verifies that the app's outputs (like graphs, tables, and text) are correct based on those actions. | ||
|
||
### Playwright | ||
**Why it's awesome:** | ||
|
||
***Playwright*** is an open-source library developed by Microsoft. It enables developers to automate browser interactions and perform end-to-end testing of web applications. | ||
* **Early bug detection:** Find problems *before* your users do! No more embarrassing surprises. | ||
* **Confidence in changes:** When you update your app, tests make sure you haven't accidentally broken anything. | ||
* **Time saver:** Instead of manually clicking through your app every time you make a change, tests automate the process. | ||
* **Peace of mind:** Know that your app is working reliably, so you can focus on building new features. | ||
|
||
Benefits of using Playwright for Shiny App testing | ||
### Introducing Playwright: A Comprehensive Automated Testing Solution for Web Applications | ||
|
||
- **End-to-End Testing**: Playwright allows you to simulate real user interactions with your Shiny app, ensuring that the reactive components and user flows work as expected. | ||
- **Cross-Browser Testing**: Playwright supports multiple browsers like Chromium, Firefox, and Safari(Webkit), enabling you to test your Shiny app's compatibility across different browser environments. | ||
- **Dynamic wait times** Playwright provides dynamic wait times, automatically waiting for elements to be ready before interacting with them, which eliminates the need for explicit waits and reduces flakiness caused by timing issues. | ||
***Playwright*** is a robust, open-source automation framework developed by Microsoft that enables programmatic control of web browsers. This tool provides developers with the capability to automate interactions with web applications across Chrome, Firefox, and Safari, simulating user behavior in a controlled, reproducible environment. | ||
|
||
For detailed information and guidance, check out the [official Playwright documentation](https://playwright.dev/python/). | ||
**Why Playwright is perfect for Shiny:** | ||
|
||
### How it works: a basic example | ||
* **Handles interactivity:** It can interact with all those cool Shiny widgets like sliders, dropdowns, and buttons. | ||
* **Cross-browser testing:** Make sure your app works flawlessly on different browsers. | ||
* **Smart waiting:** Playwright automatically waits for your app to load and for elements to be ready, so your tests are reliable. | ||
* **Easy to learn:** The code is relatively straightforward, and we'll walk you through it. | ||
|
||
Consider the following app that simply displays a message with double the slider value: | ||
Learn more at the [official Playwright documentation](https://playwright.dev/python/). | ||
|
||
```{.python filename="app.py"} | ||
### Let's Build and Test a Simple Shiny App! | ||
|
||
We'll start with a super simple example to show you the basics. Follow along, and you'll be writing your own tests in no time! | ||
|
||
#### Step 1: Create Your First Shiny App | ||
|
||
First, let's create a tiny Shiny app with just a slider and some text. | ||
|
||
1. **Create a new file:** Create a file named `app.py`. | ||
2. **Copy and paste this code:** | ||
|
||
```python | ||
from shiny import render, ui | ||
from shiny.express import input | ||
|
||
ui.panel_title("Hello Shiny!") | ||
ui.input_slider("n", "N", 0, 100, 20) | ||
|
||
|
||
@render.text | ||
def txt(): | ||
return f"n*2 is {input.n() * 2}" | ||
|
||
``` | ||
|
||
If we want to test that the shiny app works for the following scenario: | ||
3. **What this app does:** This app displays a slider (labeled "N") that goes from 0 to 100. Below the slider, it shows the text "n*2 is \[value]", where \[value] is twice the current slider value. | ||
|
||
#### Step 2: What Are We Testing? | ||
|
||
Our goal is to write a test that does the following: | ||
|
||
1. Wait for the Shiny app to finish loading | ||
1. Drag the slider to value as `55` | ||
1. Verify the output text changes to reflect the value of `n*2 is 110` | ||
1. **Opens the app:** Starts the Shiny app in a browser. | ||
2. **Moves the slider:** Sets the slider to a specific value (*55* in this case). | ||
3. **Checks the output:** Verifies that the text below the slider displays the correct result ("n*2 is 110"). | ||
|
||
The test code to test the shiny app to emulate the above scenario would be as following: | ||
#### Step 3: Write Your First Test! | ||
|
||
```{.python filename="test_basic_app.py"} | ||
Now for the exciting part – writing the test code! | ||
|
||
1. **Create a new file:** Create a new file named `test_basic_app.py` in the same directory as your `app.py` file. Remember, test file names must start with `test_`. | ||
2. **Copy and paste this code:** | ||
|
||
```python | ||
from shiny.playwright import controller | ||
from shiny.run import ShinyAppProc | ||
from playwright.sync_api import Page | ||
from shiny.pytest import create_app_fixture | ||
|
||
app = create_app_fixture("../app.py") | ||
def test_basic_app(page: Page, local_app: ShinyAppProc) -> None: | ||
# Navigate to the app URL when it's ready | ||
page.goto(local_app.url) | ||
|
||
def test_basic_app(page: Page, app: ShinyAppProc): | ||
page.goto(app.url) | ||
# Controller objects for interacting with specific Shiny components | ||
txt = controller.OutputText(page, "txt") | ||
slider = controller.InputSlider(page, "n") | ||
|
||
# Move the slider to position 55 | ||
slider.set("55") | ||
|
||
# Verify that the output text shows "n*2 is 110" | ||
# (since 55 * 2 = 110) | ||
txt.expect_value("n*2 is 110") | ||
``` | ||
|
||
Here's a breakdown of what's happening in the test code: | ||
|
||
1. The code begins by importing the `controller` module. This module provides classes and methods for controlling Shiny [components](https://shiny.posit.co/py/components/). With these classes, you can mimic user interactions (e.g., `.set()`) and verify state (e.g., `.expect_value()`). [See here](https://shiny.posit.co/py/api/testing/) for more information on the available classes and methods. | ||
- **Understand role of Fixtures** | ||
- **ShinyAppProc**: Manages a Shiny application subprocess, handling lifecycle (startup, shutdown) and providing access to output streams. | ||
- **page**: Playwright object representing the browser tab. | ||
- **local_app**: Running instance of the Shiny application. | ||
|
||
2. Defines ***test_basic_app*** function with ***page*** and ***app*** parameters. *page* is an instance of the Page class from the Playwright library, which represents a single tab in a browser, and *app* is an instance of the `ShinyAppProc` class, which represents the Shiny app being tested. | ||
- **Understand role of Controllers** | ||
|
||
3. Navigates to the app's URL. | ||
Controllers such as `OutputText` and `InputSlider` provide abstraction over Playwright's low-level interactions by: | ||
|
||
4. Creates instances of `OutputText` and `InputSlider` for UI elements. | ||
|
||
5. Sets the slider value to `55`. | ||
|
||
6. Checks if the output text displays `n*2 is 110` as expected. | ||
- Automatically handling element waiting and state changes | ||
- Offering specialized interfaces for specific Shiny component types | ||
- Managing Shiny-specific behaviors without additional code | ||
- Providing consistent patterns for testing similar components | ||
|
||
And visually, this is what happens when the test runs: | ||
|
||
|
||
 | ||
|
||
#### Step 4: Run Your Test! | ||
|
||
### Run the test | ||
Before you run the test, you need to install a couple of things: | ||
|
||
To run end-to-end tests, you'll first want to make sure `pytest` and the `pytest-playwright` plugin are installed: | ||
1. **Install pytest and pytest-playwright**: Open your terminal (or command prompt) and type: | ||
|
||
```bash | ||
pip install pytest pytest-playwright | ||
``` | ||
|
||
Now, if you have the `app.py` and `test_basic_app.py` files in the same directory, from that same directory, run the `pytest` command: | ||
2. **Navigate to your app's directory**: In the terminal, use the `cd` command to go to the folder where you saved `app.py` and `test_basic_app.py`. | ||
|
||
3. **Run the test**: Type the following command and press Enter: | ||
|
||
```bash | ||
pytest | ||
``` | ||
|
||
You should see output similar to this: | ||
|
||
```text | ||
======== test session starts ======== | ||
platform darwin -- Python 3.10.12, pytest-7.4.4, pluggy-1.4.0 | ||
configfile: pytest.ini | ||
plugins: asyncio-0.21.0, timeout-2.1.0, Faker-20.1.0, cov-4.1.0, playwright-0.4.4, rerunfailures-11.1.2, xdist-3.3.1, base-url-2.1.0, hydra-core-1.3.2, anyio-3.7.0, syrupy-4.0.5, shiny-1.0.0 | ||
asyncio: mode=strict | ||
12 workers [1 item] | ||
... (some details about your setup) | ||
. | ||
======== 1 passed in 3.05s ======== | ||
``` | ||
|
||
What does this mean? | ||
|
||
======== 1 passed in 3.05s ======== | ||
- The `.` (dot) means your test passed! | ||
- If you see an `F`, it means the test failed. Double-check your code and make sure you followed all the steps. | ||
|
||
#### Visualize Your Test (Optional) | ||
|
||
If you want to see what Playwright is doing, you can run the test in "headed" mode. This will open a browser window and show you the interactions. | ||
|
||
```bash | ||
pytest --headed | ||
``` | ||
|
||
Each test inside the file is shown by a single character in the output: | ||
You can also specify a particular browser: | ||
|
||
- `.` for passing | ||
- `F` for failure. | ||
```bash | ||
pytest --browser firefox | ||
``` | ||
|
||
For more information on different options for running tests (like running tests in headful mode or in different browsers), check out the [Playwright documentation](https://playwright.dev/python/docs/test-runners). | ||
### Adding Tests to an Existing Shiny App | ||
|
||
### Add tests an existing app | ||
If you already have a Shiny app, you can easily add tests: | ||
|
||
If you already have a shiny app and want to add a test file to it, enter the following command in your terminal/console: | ||
1. Open your terminal: Navigate to your app's directory. | ||
1. Run the shiny add test command: | ||
|
||
```bash | ||
shiny add test | ||
``` | ||
|
||
This command will ask you for a path to the app file as well as a name for the test file. | ||
Just make sure to follow `pytest` conventions for naming the test file (should start with `test_`). | ||
Note that the test file this command provides will need to be updated with the test logic you want to add. | ||
1. Answer the prompts: It will ask for the path to your app file (e.g., `app.py`) and a name for your test file (e.g., `test_myapp.py`). Remember, the test file name must start with `test_`. | ||
|
||
1. Edit the generated test file: This command creates a basic test file. You'll need to modify it to add your specific test scenarios (like the slider example above). | ||
|
||
### Troubleshooting Common Issues | ||
|
||
- Test fails with an error about finding an element: Make sure the IDs you're using in your test code (e.g., "txt", "n") match the IDs in your Shiny app code. Inspect your app's HTML in the browser's developer tools if you're unsure. | ||
|
||
- Test is flaky (sometimes passes, sometimes fails): This can happen if your app takes a while to load or if there are timing issues. Playwright has built-in waiting mechanisms, but you might need to add explicit waits in some cases. See the [Playwright documentation](https://playwright.dev/python/docs/events#waiting-for-event) on waiting. | ||
|
||
### Keep Exploring! | ||
|
||
You've taken your first steps into the world of Shiny app testing! Here are some resources to help you learn more: | ||
|
||
- [Shiny testing API documentation](https://shiny.posit.co/py/api/testing/) - This is your go-to guide for all the available testing methods in Shiny. | ||
- [Playwright documentation](https://playwright.dev/python/) - Learn more about Playwright's powerful features. | ||
- [pytest documentation](https://docs.pytest.org/en/stable/) | ||
|
||
### Learn more | ||
|
||
For more information about the testing methods available to the user, read the reference documentation about shiny testing API [here](https://shiny.posit.co/py/api/testing/). | ||
Happy testing! You're now well-equipped to build more robust and reliable Shiny apps. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.