[FEATURE] Expose Playwright async/await flavor #74

maratori · 2021-09-08T21:31:39Z

I need to run aiohttp server and use playwright in tests at the same time.
Am I right that pytest-playwright right now can't help me with that?

It provides fixtures only for sync API. But I need async API.
So I have to do something like that.

from aiohttp import web
from playwright.async_api import async_playwright

async def index_html(request):
    return web.Response(body="<html><body>Hello world</body></html>", content_type="text/html")

async def test_without_page_fixture_passes(aiohttp_server):
    app = web.Application()
    app.router.add_get("/", index_html)
    server = await aiohttp_server(app)
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        context = await browser.new_context()
        page = await context.new_page()
        await page.goto(str(server.make_url("/")))
        assert await page.text_content("body") == "Hello world"

The text was updated successfully, but these errors were encountered:

mxschmitt · 2021-09-09T11:28:08Z

Exactly, currently the pytest plugin does only provide the sync APIs. So you need to launch browser etc. yourself, something similar to https://github.com/microsoft/playwright-python/blob/master/tests/async/conftest.py

Shall I rename this issue title into a feature request for providing also the async flavor?

maratori · 2021-09-09T11:36:07Z

Thanks.

Shall I rename this issue title into a feature request for providing also the async flavor?

Sure, go ahead.

dwiyatci · 2021-12-12T09:23:51Z

Exactly, currently the pytest plugin does only provide the sync APIs. So you need to launch browser etc. yourself, something similar to https://github.com/microsoft/playwright-python/blob/master/tests/async/conftest.py

Shall I rename this issue title into a feature request for providing also the async flavor?

@mxschmitt I'm on Playwright v1.17 and I did try to copy-paste all the fixtures in the file you pointed here (except the utils), but then I got:

E       fixture 'launch_arguments' not found
>       available fixtures: _verify_url, base_url, browser, browser_channel, browser_context_args, browser_factory, browser_name, browser_type, browser_type_launch_args, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, context, context_factory, delete_output_dir, device, doctest_namespace, event_loop, is_chromium, is_firefox, is_webkit, launch_browser, monkeypatch, page, password, playwright, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, report_name, selectors, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, username
>       use 'pytest --fixtures [testpath]' for help on them.

I quickly fixed it by changing launch_arguments to browser_type_launch_args, but then again, I got:

playwright = <async_generator object playwright at 0x111459d40>, browser_name = 'chromium'

    @pytest.fixture(scope="session")
    def browser_type(playwright, browser_name: str):
        if browser_name == "chromium":
>           return playwright.chromium
E           AttributeError: 'async_generator' object has no attribute 'chromium'

conftest.py:20: AttributeError

I'm pretty new to Python and pytest but I guess async_generator of playwright_object instead of playwright_object itself being "injected" to browser_type – any pointer on how I should remedy this? And is this async gonna be planned to be supported in the end? 😅 Thank you.

mxschmitt · 2021-12-12T15:35:37Z

@dwiyatci why do you prefer async over sync? Understanding this use-case for the testing scenario would us help to prioritize it.

dwiyatci · 2021-12-12T22:22:35Z

@mxschmitt I would like to implement a custom wait_for_network_settled function like this (microsoft/playwright#7856 (comment)), and I believe it'd be much easier if I could have asyncio in place.

fscherf · 2021-12-13T13:30:56Z

@mxschmitt: I want to use this plugin together with aiohttp-pytest, because my project is based on aiohttp. Thats the reason why @maratori and i prefer (in this case) async over sync.

baiyyee · 2022-09-17T03:10:16Z

any updates for this feature?

nathanielobrown · 2023-06-19T11:01:25Z

It is very odd that there are async/sync versions of Playwright Python but the integration with pytest only supports sync. If the primary purpose of Playwright is to write tests, but the async version cannot be used with the dominant test framework, then what is the async version's purpose? If the async version is intended for applications other than testing, then this is still odd as we'd likely want to use pytest to test that code.

On a more pragmatic level, as more and more applications are written using async libraries, more and more tests will need to be async. Also, having async tests allows us to use some cleaner semantics for things like processes that happen in parallel.

There's also the issue (microsoft/playwright-python#1339 (comment)) I ran into that we can't even mix async/sync tests in the same test suite and use playwright-pytest.

@mxschmitt, do you have pointers on how hard this would be/what would need to be done?

mxschmitt · 2023-06-19T11:13:10Z

@nathanielobrown I totally agree that providing an async layer would be cool! The reason why we didn't do it yet was that its more idiomatic to write tests in sync mode (debugging, you don't need to install other pytest plugins, REPL just works etc).

Regarding how hard it is to implement, I think it hardly depends on the question how much code we want to duplicate. In general the pytest-playwright plugin is not a lot of code, to provide an async version, we could in theory just copy it and use the async version of Playwright and it should just work™️ . Something where I'm not sure about is if we need a dedicated plugin (PIP package) for the async version or if we can internally detect somehow if its async or sync and then do conditional logic. This requires some investigation.

(happy to accept patches for it, just not something we prioritise, since not a lot of users have requested it so far.)

nathanielobrown · 2023-06-19T11:43:49Z

@mxschmitt, that's great to hear there is no technical blocker. My initial thought would be to add new fixtures to the same package with _async suffixes, like page_async. Then users can mix/match/migrate to their hearts content. Auto detection would be cool, but even if there's a way to do that I feel like that would be more likely to cause issues and it's not to much load to just use different fixture names.

Also this fix (microsoft/playwright-python#1339 (comment)) for the multiple event loops issue just worked from me and is topical.

I'll probably work something up locally and if it works well can create a PR.

m9810223 · 2023-06-25T16:18:23Z

Hey guys!
I made a plugin https://github.com/m9810223/playwright-async-pytest that support async playwright for my projects.
It refers to original plugin, welcome to use it if you need!

balazs-ita-epam · 2023-07-24T09:17:56Z

Waiting for the async feature in pytest!

blooser · 2024-06-10T13:01:41Z

Hey!

What is the status of the feature?

louisabraham · 2024-08-12T12:43:04Z

Hi! Any update? We need this as well :)

Lendemor · 2024-08-14T21:39:51Z

We also need this feature to fully adopt playwright in our integration tests.

balazs-ita-epam · 2024-09-26T08:30:30Z

Please prioritize higher to implement the python - Playwright with async.

mxschmitt · 2024-11-14T12:15:57Z

While we prepare things for releasing it as a separate package, here are instructions to install it from source for now, curious to hear your thoughts!

virtualenv env && source env/bin/activate
pip install -e "git+https://github.com/microsoft/playwright-pytest#egg=pytest-playwright-asyncio&subdirectory=pytest-playwright-asyncio"
playwright install
Create test_foo.py

import pytest

@pytest.mark.asyncio(loop_scope="session")
async def test_foo(page):
    await page.goto("https://github.com")

I saw that https://github.com/m9810223/playwright-async-pytest is based on anyio, not sure whats the standard nowadays.

m9810223 · 2024-11-14T14:08:52Z

That's great news! 🎉

@mxschmitt

I later modified https://github.com/m9810223/playwright-async-pytest to use anyio for the following reasons:

I prefer not to add @pytest.mark.asyncio to every async test, although I'm not debating which approach is better.
Using anyio also allows compatibility with Python's built-in asyncio.

ref: m9810223/playwright-async-pytest#8

mxschmitt · 2024-11-14T15:46:20Z

I'm curious to what the community currently is mostly using when testing with Pytest. I tried to find anyio in this survey but didn't see any mentions. I understand the benefit of not having to wrap them into custom fixtures / mark them.

I think the mark issue is solved by this - thats what projects are usually using nowadays afaik.

Using anyio also allows compatibility with Python's built-in asyncio.

Do you mind elaborating on that?

bartfeenstra · 2024-11-18T22:54:16Z

I tried the instructions from #74 (comment) just now, and as soon as a test requests the page fixture, pytest reports the following error:

____________________________ ERROR collecting betty/tests/project/extension/cotton_candy/test_search_ui.py _____________________________
venv13/lib/python3.13/site-packages/pluggy/_hooks.py:513: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
venv13/lib/python3.13/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
venv13/lib/python3.13/site-packages/pytest_asyncio/plugin.py:560: in pytest_pycollect_makeitem_convert_async_functions_to_subclass
    ] = hook_result.get_result()
venv13/lib/python3.13/site-packages/_pytest/python.py:245: in pytest_pycollect_makeitem
    return list(collector._genfunctions(name, obj))
venv13/lib/python3.13/site-packages/_pytest/python.py:462: in _genfunctions
    self.ihook.pytest_generate_tests.call_extra(methods, dict(metafunc=metafunc))
venv13/lib/python3.13/site-packages/pluggy/_hooks.py:574: in call_extra
    return self._hookexec(self.name, hookimpls, kwargs, firstresult)
venv13/lib/python3.13/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
venv13/lib/python3.13/site-packages/pytest_asyncio/plugin.py:742: in pytest_generate_tests
    raise MultipleEventLoopsRequestedError(
E   pytest_asyncio.plugin.MultipleEventLoopsRequestedError: Multiple asyncio event loops with different scopes have been requested
E   by betty/tests/project/extension/cotton_candy/test_search_ui.py::TestSearch::test. The test explicitly requests the event_loop fixture, while
E   another event loop with session scope is provided by .
E   Remove "event_loop" from the requested fixture in your test to run the test
E   in a session-scoped event loop or remove the scope argument from the "asyncio"
E   mark to run the test in a function-scoped event loop.

Without the marker (I use asyncio_mode=auto in my pytest configuration, and would like not to need a marker) pytest reports another error:

ScopeMismatch: You tried to access the function scoped fixture event_loop with a session scoped request object. Requesting fixture stack:
venv13/src/pytest-playwright-asyncio/pytest-playwright-asyncio/pytest_playwright_asyncio/pytest_playwright.py:263:  def browser(launch_browser: Callable[[], Awaitable[playwright.async_api._generated.Browser]]) -> AsyncGenerator[playwright.async_api._generated.Browser, NoneType]
venv13/src/pytest-playwright-asyncio/pytest-playwright-asyncio/pytest_playwright_asyncio/pytest_playwright.py:250:  def launch_browser(browser_type_launch_args: Dict, browser_type: playwright.async_api._generated.BrowserType) -> Callable[..., Awaitable[playwright.async_api._generated.Browser]]
venv13/src/pytest-playwright-asyncio/pytest-playwright-asyncio/pytest_playwright_asyncio/pytest_playwright.py:245:  def browser_type(playwright: playwright.async_api._generated.Playwright, browser_name: str) -> playwright.async_api._generated.BrowserType
venv13/src/pytest-playwright-asyncio/pytest-playwright-asyncio/pytest_playwright_asyncio/pytest_playwright.py:238:  def playwright() -> AsyncGenerator[playwright.async_api._generated.Playwright, NoneType]
Requested fixture:
venv13/lib/python3.13/site-packages/pytest_asyncio/plugin.py:1024:  def event_loop(request: _pytest.fixtures.FixtureRequest) -> Iterator[asyncio.events.AbstractEventLoop]

Now, my configuration also includes asyncio_default_fixture_loop_scope=function (which will be the pytest default in upcoming versions), and without that, this test case passes.

mxschmitt · 2024-11-19T09:03:16Z

@bartfeenstra thanks for giving it a try - we are still figuring out the bits - so didn't release it as a version on Pypi yet.

I think in your case you'd have to either put asyncio_default_fixture_loop_scope in the config or mark your tests (depending if you use pytestmark or on module level as async with event_loop=session setting so Playwright can stay alive for the whole test session.

Without the marker (I use asyncio_mode=auto in my pytest configuration, and would like not to need a marker) pytest.

Would a file-wide marker or doing it in conftest be fine for you or do you prefer asyncio_mode=auto?

Due to the recent changes in the pytest-asyncio plugin I think we we have no other option as setting loop_scope=session in the Fixtures and the tests. If we rely on asyncio_default_fixture_loop_scope or set it explictly in the plugin is up to us I think. In my opinion its slightly better to set it in the plugin directly with the pytest_asyncio.fixture(loop_scope="session") while the first would allow us to not depend on pytest-asyncio but requires the user to use asyncio_mode=auto what I'm hesitate about. Maybe someone else has thoughts about that. This would allow @m9810223 to use e.g. anyio tho.

I'm curious why it works for you with asyncio_default_fixture_loop_scope=function I was under impression that we need "session".

Thank you all.

bartfeenstra · 2024-11-19T10:11:25Z

Would a file-wide marker or doing it in conftest be fine for you or do you prefer asyncio_mode=auto?

What I don't want is to have to decorate every other asynchronous test method in my codebase. That's what pytest provides these configuration settings for. What I would not mind doing, is adding extra decorators to those few Playwright tests (they are a minority in my code base, so adding extras is manageable). Also, if playwright-pytest(-asyncio) does not play well with pytest's built-in configuration, then that's a pretty damning first impression if enabling playwright-pytest-asyncio causes every other non-Playwright test to fail. Something that is ideally avoided. E.g. it would look better if the Playwright tests fail (because that's literally what a developer would be in the process of setting up) than to make it look like the setup broke existing code (which would be much more confusing, and potentially harder to work around).

I'm curious why it works for you with asyncio_default_fixture_loop_scope=function I was under impression that we need "session".

I'm not sure what you mean by "it" in this question. Having a function-scoped event loop is the default in pytest-asyncio as it provides the most isolation between tests, so out of all the options it's the one most likely to work. However, it does not work with pytest-playwright-asyncio.

I was unable to find where Playwright's Pytest fixtures are defined. Could you point me in the right direction?

mxschmitt · 2024-11-20T12:53:47Z

Also, if playwright-pytest(-asyncio) does not play well with pytest's built-in configuration, then that's a pretty damning first impression if enabling playwright-pytest-asyncio causes every other non-Playwright test to fail. Something that is ideally avoided. E.g. it would look better if the Playwright tests fail (because that's literally what a developer would be in the process of setting up) than to make it look like the setup broke existing code (which would be much more confusing, and potentially harder to work around).

Definitely! Thats why we didn't release it yet. One thing to note is that the pytest-asyncio default mode is strict - so I think we should aim to support both, auto and strict.

I'm not sure what you mean by "it" in this question. Having a function-scoped event loop is the default in pytest-asyncio as it provides the most isolation between tests, so out of all the options it's the one most likely to work. However, it does not work with pytest-playwright-asyncio.

Playwright would like to re-use the Playwright instance, so we need a loop which is session wide - I think there is no way around that. Either via setting it globally via the asyncio_default_fixture_loop_scope setting or the other variants (pytest_collection_modifyitems or pytestmark).

I was unable to find where Playwright's Pytest fixtures are defined. Could you point me in the right direction?

pytest-playwright-asyncio fixtures are defined here: https://github.com/microsoft/playwright-pytest/blob/main/pytest-playwright-asyncio/pytest_playwright_asyncio/pytest_playwright.py

storenth · 2024-11-25T13:07:06Z

While we prepare things for releasing it as a separate package, here are instructions to install it from source for now, curious to hear your thoughts!

virtualenv env && source env/bin/activate

pip install -e "git+https://github.com/microsoft/playwright-pytest#egg=pytest-playwright-asyncio&subdirectory=pytest-playwright-asyncio"

playwright install

Create test_foo.py

import pytest

@pytest.mark.asyncio(loop_scope="session")
async def test_foo(page):
await page.goto("https://github.com")
I saw that https://github.com/m9810223/playwright-async-pytest is based on anyio, not sure whats the standard nowadays.

I use poetry and start own async pytest-playwright, so based on your suggestions I provide steps I test the approach:

remove old sync-based plugin pytest-playwright
install async plugin poetry add "git+https://github.com/microsoft/playwright-pytest#subdirectory=pytest-playwright-asyncio"
run a tests (specify a dir to avoid interference with ./src) poetry run pytest ./test
That works! Thx. But how I can handle semver in the future? I found https://pypi.org/project/pytest-playwright-asyncio/ will you push updates to?

mxschmitt · 2024-11-25T17:52:20Z

We published pytest-playwright-asyncio as v0.6.1:

PyPi: https://pypi.org/project/pytest-playwright-asyncio/
Conda: https://anaconda.org/Microsoft/pytest-playwright-asyncio

Docs will follow.

bartfeenstra · 2024-11-27T10:41:45Z

Thank you all for making this work! I am about to merge the PR that implements all this in my own project. Being able to use Python instead of JS/TS with Playwright's own test runner saves a lot of code duplication and makes writing tests a lot easier.

I did run into the problem that if run as part of the main pytest test suite of ~4500 tests, these Playwright tests would hang indefinitely. I did not manage to trace the source of the problem (it could be an event loop issue). However, I worked around it by running the Playwright tests separately from the main suite. There are many ways to do this, but I opted to organize my Playwright tests in a separate root test directory, yet inherit from the main test suite's pytest configuration (conftest). See https://github.com/bartfeenstra/betty/pull/2185/files. I hope that this comment may save people with similar issues some time in the future.

martinkirch · 2024-12-27T21:39:58Z

I also migrated to playwright-pytest-asyncio successfully, thanks for the release !

For anyone also running into tests hanging indefinitely: apparently we really have to put all tests in the same loop.

As far as I understand, this is at least because Playwright's Browser is a session-wide fixture that tries to access the event loop... but setting asyncio_default_fixture_loop_scope = "session" was not enough.

maratori mentioned this issue Sep 8, 2021

Lona has no browser tests lona-web-org/lona#74

Closed

mxschmitt added the triaging label Sep 9, 2021

mxschmitt changed the title ~~pytest-playwright is not compatible with aiohttp~~ Expose Playwright async/await flavor Sep 9, 2021

mxschmitt added p3-collecting-feedback and removed triaging labels Sep 9, 2021

roniemartinez mentioned this issue Apr 25, 2022

feat(codegen): pytest support in codegen microsoft/playwright#13746

Merged

seifertm mentioned this issue Jun 19, 2022

RuntimeError: Cannot run the event loop while another loop is running pytest-dev/pytest-asyncio#359

Closed

mxschmitt changed the title ~~Expose Playwright async/await flavor~~ [Feature] Expose Playwright async/await flavor Jul 6, 2022

mxschmitt changed the title ~~[Feature] Expose Playwright async/await flavor~~ [FEATURE] Expose Playwright async/await flavor Jul 6, 2022

mxschmitt mentioned this issue Jul 12, 2022

[Question]: RuntimeError: Cannot run the event loop while another loop is running microsoft/playwright-python#1339

Closed

rwoll mentioned this issue Aug 15, 2022

Verify files status in parallel while using bulk upload microsoft/playwright-python#1480

Closed

This was referenced Feb 6, 2023

[Question] microsoft/playwright#20683

Closed

how to playwright async working pytest microsoft/playwright#20682

Closed

mxschmitt mentioned this issue Jul 21, 2023

[Question]: Best practices for using page-objects as pytest fixtures and for implementing async methods microsoft/playwright-python#2008

Closed

mxschmitt mentioned this issue Sep 1, 2023

chore: introduce asyncio mode #184

Closed

jamesbraza mentioned this issue Jan 25, 2024

Request: async support #201

Closed

mxschmitt added the help wanted Extra attention is needed label Jun 14, 2024

m9810223 mentioned this issue Sep 26, 2024

Test suite freezes during teardown m9810223/playwright-async-pytest#8

Open

mxschmitt mentioned this issue Nov 14, 2024

feat: pytest-playwright-asyncio as separate plugin #257

Merged

mxschmitt self-assigned this Nov 14, 2024

mxschmitt mentioned this issue Nov 14, 2024

Playwright does not play right with asyncio #240

Closed

mxschmitt mentioned this issue Nov 25, 2024

docs(python): add note about async fixtures microsoft/playwright#33760

Merged

mxschmitt closed this as completed in microsoft/playwright#33760 Nov 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Expose Playwright async/await flavor #74

[FEATURE] Expose Playwright async/await flavor #74

maratori commented Sep 8, 2021

mxschmitt commented Sep 9, 2021 •

edited

Loading

maratori commented Sep 9, 2021

dwiyatci commented Dec 12, 2021

mxschmitt commented Dec 12, 2021

dwiyatci commented Dec 12, 2021

fscherf commented Dec 13, 2021

baiyyee commented Sep 17, 2022

nathanielobrown commented Jun 19, 2023

mxschmitt commented Jun 19, 2023

nathanielobrown commented Jun 19, 2023

m9810223 commented Jun 25, 2023

balazs-ita-epam commented Jul 24, 2023

blooser commented Jun 10, 2024

louisabraham commented Aug 12, 2024

Lendemor commented Aug 14, 2024

balazs-ita-epam commented Sep 26, 2024

mxschmitt commented Nov 14, 2024 •

edited

Loading

m9810223 commented Nov 14, 2024

mxschmitt commented Nov 14, 2024

bartfeenstra commented Nov 18, 2024 •

edited

Loading

mxschmitt commented Nov 19, 2024 •

edited

Loading

bartfeenstra commented Nov 19, 2024

mxschmitt commented Nov 20, 2024

storenth commented Nov 25, 2024 •

edited

Loading

mxschmitt commented Nov 25, 2024 •

edited

Loading

bartfeenstra commented Nov 27, 2024 •

edited

Loading

martinkirch commented Dec 27, 2024

[FEATURE] Expose Playwright async/await flavor #74

[FEATURE] Expose Playwright async/await flavor #74

Comments

maratori commented Sep 8, 2021

mxschmitt commented Sep 9, 2021 • edited Loading

maratori commented Sep 9, 2021

dwiyatci commented Dec 12, 2021

mxschmitt commented Dec 12, 2021

dwiyatci commented Dec 12, 2021

fscherf commented Dec 13, 2021

baiyyee commented Sep 17, 2022

nathanielobrown commented Jun 19, 2023

mxschmitt commented Jun 19, 2023

nathanielobrown commented Jun 19, 2023

m9810223 commented Jun 25, 2023

balazs-ita-epam commented Jul 24, 2023

blooser commented Jun 10, 2024

louisabraham commented Aug 12, 2024

Lendemor commented Aug 14, 2024

balazs-ita-epam commented Sep 26, 2024

mxschmitt commented Nov 14, 2024 • edited Loading

m9810223 commented Nov 14, 2024

mxschmitt commented Nov 14, 2024

bartfeenstra commented Nov 18, 2024 • edited Loading

mxschmitt commented Nov 19, 2024 • edited Loading

bartfeenstra commented Nov 19, 2024

mxschmitt commented Nov 20, 2024

storenth commented Nov 25, 2024 • edited Loading

mxschmitt commented Nov 25, 2024 • edited Loading

bartfeenstra commented Nov 27, 2024 • edited Loading

martinkirch commented Dec 27, 2024

mxschmitt commented Sep 9, 2021 •

edited

Loading

mxschmitt commented Nov 14, 2024 •

edited

Loading

bartfeenstra commented Nov 18, 2024 •

edited

Loading

mxschmitt commented Nov 19, 2024 •

edited

Loading

storenth commented Nov 25, 2024 •

edited

Loading

mxschmitt commented Nov 25, 2024 •

edited

Loading

bartfeenstra commented Nov 27, 2024 •

edited

Loading