Option --output not working with a manually launched context

[rafutek]

I am trying to output test results in another directory with the --output option. But it seems to not be working : the default directory (test-results) contains test results at each run, even if I delete the folder.

Using pytest-playwright 0.3.0 .

[mxschmitt]

I was trying it out locally and it works for me. How do you specify the command to pytest? Do you have a small repro repository which you can share?

[rafutek]

For example : pytest --output test_dir tests/admin/test_group_management.py
And sorry, I don't have a repo to share..
Pytest is up to date also (7.1.2)

[mnovait]

Same here with something like:

$ pytest path/to/some/test_playwright.py --tracing=on --output pytest-tracing

even if adding or changing the tracing with screenshots or video or completely omitting the --output param the results is the same: no folder and no output

[mxschmitt]

how does your test_playwright.py look like?

[mnovait]

I created a couple of fixtures like that:

@pytest.fixture(scope="session")
def mypage(browser, session_cookies):
    context = browser.new_context(storage_state=session_cookies)
    return context.new_page()

@pytest.fixture(scope="session")
def myfeature(mypage, backend):
    myfeature = MyFeaturePage(
        page= mypage,
    )
    yield myfeature
    myfeature.close()

where MyFeaturePage is only an helper that wraps a page and make easier tests development

the test.py is irrelevant, it just take myfeature and does some action on myfeature.page.something()

looking at the code here I'd say that the params related to the artifacts (video, screenshot, tracing) are set only if the context feature is called, which is not in my case.

does it make sense to you?

[mxschmitt]

It's intended that the --output only works with the given context and page fixture. If you manually launch a browser or create a browser context, --output will have no effect.

[rafutek]

I also modified browser context with

@pytest.fixture(scope="session")
def browser_context_args(browser_context_args):
    return {
        **browser_context_args,
        "ignore_https_errors": True
    }

Removing this fixtures enables --output option, indeed. Thank you !

[mnovait]

It's intended that the --output only works with the given context and page fixture. If you manually launch a browser or create a browser context, --output will have no effect.

ok. we could add few documentation lines about it.

in theory it would be possibile to get artifacts even if the called fixture is browser, but it could be tricky because in a browser you can initialize more than one context.
@mxschmitt do you see this as a possibile improvement of the plugin?

[mxschmitt]

Yeah, I think upstream we also support it.

In what artifacts are you exactly interested in? (video, screenshot, trace)?

[mnovait]

I'd say everything is feasible 😄
An idea could be iterate over all the contexts defined in browser and print everything.
I could work a bit on this, if it can help

[rafutek]

I do not understand why --output option is not compatible with a modified browser_context_args fixture. Is it possible to make them compatible ?

[rafutek]

Finally I just used pytest --junit-xml instead of --output, it works like a charm

[dmeecs]

Yeh this is quite annoying tbh. It would be nice if it was fixed.

[dmeecs]

Suggested this fix for it #118

[mnovait]

Finally I just used pytest --junit-xml instead of --output, it works like a charm

I'm not sure about what you mean but --output provides a folder where the artifacts should be stored, --junit-xml is for the xunit xml file path.

[rafutek]

Yes, sorry, --output and --junit-xml are completely different things.

Is there any update on this issue ? Modifying context and outputing artifacts should be possible no ?

[soprano8086]

Our use case:

I need to define context. context = browser.new_context(ignore_https_errors=True). I have it in conftest.

We are trying to move from Selenium to playwright. Not being able to take screenshot when tests fail is the only step that blocking us from moving(#139). In our Selenium framework, the screenshot is taken when test fails. The screenshot then being added to Allure report. It is very convenient to look at the screenshot on the Allure report page especially for team members who are not that familiar with certain tests. I am currently using pytest-bdd and Allure report. Everything works but this.

[soprano8086]

Hi,
Is there update or plan or work around for this issue? I have paused the Playwright project for a couple of months because screenshot does not work with custom context.

[TonyKorolev]

Hello! Is there any progress on this issue? When is it planned to release adding screenshots with new_context?

[strfx]

I stumbled upon this issue but was able to solve it for me by modifying the browser_context_args: you can configure the creation of context by modifying the browser_context_args fixture, either per test (as shown in the docs) or by masking the fixture like here #118 (comment).

screenshots (--screenshot) and videos (--video) are only recorded when using the provided context fixture, see https://github.com/microsoft/playwright-pytest/blob/main/pytest_playwright/pytest_playwright.py#L227.

Documenting this here as there seem to be some open questions about it.

[APCBoston]

It is worth noting that nothing in the documentation at https://playwright.dev/python/docs/test-runners#cli-arguments even hints that any of the CLI flags are dependent on using the default context fixture, or that the suggestion on the same page for modifying context to session-scope would be incompatible with one or more of those flags.

(ETA: at some point last week I came across the browser context args documentation at https://playwright.dev/python/docs/api/class-browser#browser-new-context, but without any mention or cross-reference of the pytest CLI flags, I was left scratching my head as to how these args would interact with CLI flags, and why I would need to set record-video-dir if I had already set --output at the CLI...)

Worse, the --output flag DELETES the provided directory when (because of custom context) no artifacts are stored there. That behavior is also undocumented.

[mxschmitt]

This should be fixed in v0.5.0: https://pypi.org/project/pytest-playwright/

[shisrafilov]

The issue is still there

[Ragav86]

test_results folder is not getting created with context. This issue still exists

[mxschmitt]

Please re-file with more details, thank you. Make sure to use the new_context fixture in order to benefit from the artifact collection.

from playwright.sync_api import BrowserContext
from pytest_playwright.pytest_playwright import CreateContextCallback

def test_foo(new_context: CreateContextCallback) -> None:
    context = new_context()

We should probably improve the docs around it, I think the misconception is that you are using browser.new_context (assumption).