Add support for third-party GUI framework plugins

[rmartin16]

Changes

• Allow new projects to be created using arbitrary third-party GUI frameworks using plugins
  • Any package can register a class as a GUI bootstrap using the briefcase.bootstraps entry point
  • To work with beeware/briefcase-template, the bootstrap must implement BaseGuiBootstrap interface
  • Alternatively, a registered bootstrap can implement an arbitrary interface to work in tandem with a different template specified via --template
• Drop support for PySide2 as its compatibility with the modern Python ecosystem has diminished

Dependencies

• Generalize template for GUI framework plugins briefcase-template#82
• Add reusable actions to create and/or verify a new project  .github#68
• https://github.com/beeware/.github/pull/1542
• Remove testing apps since CI will generate them now briefcase-android-gradle-template#73
• Remove testing apps since CI will generate them now briefcase-iOS-Xcode-template#29
• Remove testing apps since CI will generate them now; disable PySide2 testing briefcase-linux-appimage-template#45
• Remove testing apps since CI will generate them now briefcase-linux-flatpak-template#30
• Remove testing apps since CI will generate them now briefcase-linux-system-template#17
• Remove testing apps since CI will generate them now briefcase-macOS-app-template#29
• Remove testing apps since CI will generate them now briefcase-macOS-Xcode-template#34
• Remove testing apps since CI will generate them now briefcase-web-static-template#13
• Remove testing apps since CI will generate them now briefcase-windows-VisualStudio-template#26
• Remove testing apps since CI will generate them now briefcase-windows-app-template#26

BRIEFCASE_REPO: https://github.com/rmartin16/briefcase
BRIEFCASE_REF: run-app-in-ci-testing
BRIEFCASE_TEMPLATE_REPO: https://github.com/rmartin16/briefcase-template
BRIEFCASE_TEMPLATE_REF: gui-plugin-support

PR Checklist:

• All new features have been tested
• All new features have been documented
• I have read the CONTRIBUTING.md file
• I will abide by the code of conduct

[rmartin16]

POC Design

• This all fundamentally assumes that cookiecutter will be responsible for creating the app project
• A GUI framework plugin is declared via the briefcase.wizard.frameworks entry point
  • Such a plugin currently defines:
    • GUI_TOOLKIT_NAME: a string defining the GUI toolkit such as Toga
      • This value is passed in the cookiecutter context dict as gui_framework
    • app_source: a string defining the content of app.py for the app project
  • This is a very preliminary interface for such a plugin
    • I am potentially imagining the ability for a plugin to manage the entire control flow for the briefcase new command and would be responsible for providing a cookiecutter template path and context dict
    • Of course, much of this could be deferred to Briefcase's current functionality by default

Notes

• Insofar as actually running the app in CI, I'm not sure if manifesting an entire display is necessary....
  • Or if just running the app without drawing it is sufficient...
  • It looks like Toga goes through quite the effort to draw the testbed in a display...
• Adding support for GUI framework plugins could remove the need for beeware/briefcase-template altogether
  • This would mean that any Framework plugin must wholly define the cookiecutter template for the app
  • Although, this would mean we basically create several "briefcase-template" repos....just each one would be dedicated to a specific GUI framework....but that's exactly what I'm suggesting we undo for the output format repos...sooo...
• Related pieces:
  • POC plugin that would ultimately become the package that's installed in CI to get the needed app code
    • https://github.com/rmartin16/briefcase-plugin-test
  • Briefcase template support
    • beeware/briefcase-template@main...rmartin16:briefcase-template:gui-plugin-support
  • CI changes
    • beeware/.github@main...rmartin16:.github-beeware:run-app-during-verify

[rmartin16]

PoC success!!

[verify-toga] Starting app...
===========================================================================
Install path: /home/runner/work/briefcase/briefcase/tests/apps/verify-toga/build/verify-toga/ubuntu/jammy/verify-toga-0.0.1/usr
Pre-initializing Python runtime...
PYTHONPATH:
- /usr/lib/python3.10
- /usr/lib/python3.10/lib-dynload
- /home/runner/work/briefcase/briefcase/tests/apps/verify-toga/build/verify-toga/ubuntu/jammy/verify-toga-0.0.1/usr/lib/verify-toga/app
- /home/runner/work/briefcase/briefcase/tests/apps/verify-toga/build/verify-toga/ubuntu/jammy/verify-toga-0.0.1/usr/lib/verify-toga/app_packages
Configure argc/argv...
Initializing Python runtime...
Running app module: verify_toga
---------------------------------------------------------------------------
(verify-toga:3530): dbind-WARNING **: 22:12:10.048: AT-SPI: Error retrieving accessibility bus address: org.freedesktop.DBus.Error.ServiceUnknown: The name org.a11y.Bus was not provided by any .service files
WARNING: Can't find icon resources/verify_toga; falling back to default icon
>>> successfully started...exiting <<<

Notes

• A display was necessary to do the test (as well as installing gtk first)
  • The requirements here aren't as stringent as Toga so xvfb was enough
    • xvfb-run briefcase run linux system
• As I noticed with Toga testbed CI, you can't let actions/setup-python run or the app won't work (for Linux)
  • This may end up requiring some kind of refactoring in app-build-verify.yml....

Next Steps

• Formalizing the plugin's interface
  • As I said, I'm imagining the plugin could ultimately manage everything about briefcase new to produce a template path and context dict.
  • However, I thinking of limiting the scope for this first go to:
    • Plugin registers a module via an entry point
    • Interface specifies a module variable as the name of the GUI toolkit for the app
    • Interface specifies a function that returns a dictionary to be merged with the existing context dict for cookiecutter
      • Such an approach would include a dict entry for the source code to put in to the app; however, I don't think that content could contain jinja template syntax to be replaced....although, it might be reasonable to say a full replacement cookiecutter template should be used in that case...
• Establishing a plan for the "built-in" GUI frameworks
  • We could go ahead and break these off in to their separate packages and everything that goes with that
  • Or we could leave them mostly as-is and effectively become built-in plugins

Thoughts welcome on this approach....especially on this high-level design and scope.

[freakboy3742]

POC Design

• This all fundamentally assumes that cookiecutter will be responsible for creating the app project

That seems like a reasonable assumption, unless we fully divest all responsibility for template app generation to the backend.

• A GUI framework plugin is declared via the briefcase.wizard.frameworks entry point

  • Such a plugin currently defines:

    • GUI_TOOLKIT_NAME: a string defining the GUI toolkit such as Toga

      • This value is passed in the cookiecutter context dict as gui_framework

    • app_source: a string defining the content of app.py for the app project

  • This is a very preliminary interface for such a plugin

The GUI framework is probably needed for identification purposes, but I'm not 100% convinced it's needed in the template context. If we have it in the template context, then we're still going to have "if toga..." logic in the app template, which won't actually make our lives that much easier. If the framework plugin exposes all the places that are currently {% if gui_framework == %} template blocks, that will make for a much cleaner template that is more closely aligned with what the template actually needs to describe (the basic shape of a Briefcase project) rather than needing to encode framework/platform specifics.

* I am potentially imagining the ability for a plugin to manage the entire control flow for the `briefcase new` command and would be responsible for providing a cookiecutter template path and context dict
* Of course, much of this could be deferred to Briefcase's current functionality by default

I can see how this could work, but it would essentially require picking the framework first, so the right framework backend would run the app generation logic.

I'm also a little wary of YAGNI here. Yes, we could allow any backend to completely customise the app generation workflow... but is that actually required? Asking additional questions, maybe - but changing the logic of how an app_name is selected? I'm having difficulty imagining why this would be required.

Notes

• Insofar as actually running the app in CI, I'm not sure if manifesting an entire display is necessary....

  • Or if just running the app without drawing it is sufficient...
  • It looks like Toga goes through quite the effort to draw the testbed in a display...

I'm not sure "without drawing" is actually a test. We need to validate that a GUI window can be displayed by the running app.

• Adding support for GUI framework plugins could remove the need for beeware/briefcase-template altogether

  • This would mean that any Framework plugin must wholly define the cookiecutter template for the app
  • Although, this would mean we basically create several "briefcase-template" repos....just each one would be dedicated to a specific GUI framework....but that's exactly what I'm suggesting we undo for the output format repos...sooo...

An interesting idea, but I wonder if that just bifurcates our maintenance overhead. For example - let's say we add a new top-level app property. Right now, we add that to the template once. If we give each framework update its own base template, we'd need to update that property on every base template.

I think the better approach is to provide the mechanism to cross-mix the two. Provide a base app template that covers the common structure, but also provides the extension points so that frameworks can inject platform-specific details as required.

Next Steps

• Formalizing the plugin's interface

  • As I said, I'm imagining the plugin could ultimately manage everything about briefcase new to produce a template path and context dict.

  • However, I thinking of limiting the scope for this first go to:

    • Plugin registers a module via an entry point

    • Interface specifies a module variable as the name of the GUI toolkit for the app

    • Interface specifies a function that returns a dictionary to be merged with the existing context dict for cookiecutter

      • Such an approach would include a dict entry for the source code to put in to the app; however, I don't think that content could contain jinja template syntax to be replaced....although, it might be reasonable to say a full replacement cookiecutter template should be used in that case...

I agree that it would be good to avoid meta-templating... but I'm not sure how we can avoid at least some meta templating, because we need to include the app name in the generate app content.

• Establishing a plan for the "built-in" GUI frameworks

  • We could go ahead and break these off in to their separate packages and everything that goes with that
  • Or we could leave them mostly as-is and effectively become built-in plugins

Thoughts welcome on this approach....especially on this high-level design and scope.

My initial inclination is to provide them in the Briefcase repo, and register them as part of installing Briefcase. As you've noted in #1523, the one thing we don't need is more repositories :-)

[rmartin16]

If the framework plugin exposes all the places that are currently {% if gui_framework == %} template blocks, that will make for a much cleaner template that is more closely aligned with what the template actually needs to describe (the basic shape of a Briefcase project) rather than needing to encode framework/platform specifics.

Thanks for this bit; it's definitely helped evolve my thinking on this.

To that end, a lot of the plugin needs to provide content for pyproject.toml. Since the TOML datatypes have more or less a one-to-one correspondence with Python datatypes, we could just expect the plugin to return them.

For instance, requires is just a list, so the plugin could just return ["PySide6-Essentials~=6.5", "PySide6-Addons~=6.5"] as a Python list and Briefcase could format that in to the TOML.

Alternatively, the plugin could return a string of the same information and Briefcase could insert it verbatim in to the TOML. The advantage of this approach is metadata that will be difficult to capture in a Python datatype.

For instance, consider system_requires under the linux.system.debian section:

system_requires = [
    # Needed to compile pycairo wheel
    "libcairo2-dev",
    # Needed to compile PyGObject wheel
    "libgirepository1.0-dev",
]

These comments are valuable to end-users to understand why it's included in their rolled out project.

Therefore, my inclination is to allow plugins to do either and return the format that makes sense for them.

@freakboy3742, do you have any opinions or additional insight for this before I move forward? Thanks.

[freakboy3742]

Therefore, my inclination is to allow plugins to do either and return the format that makes sense for them.

@freakboy3742, do you have any opinions or additional insight for this before I move forward? Thanks.

str | list (or str | dict[str, str], or whatever is appropriate) seems entirely workable, but I wonder if it's worth the effort to support both. The use case for returning str is "this definition is complex and needs explanatory notes"; that implies the use case for list is "this definition is simple and self evident" - which also means it's the case that will benefit least from the "strong typing" benefit of returning a list.

The only other thought I've had is that we could switch to tomlkit, which is a style-preserving TOML library. It allows you to annotate comments onto nodes in the TOML document... but it (a) only works on TOML, and (b) would require all plugins to also use tomlkit. However, this gets us to the point of supporting str, native types and tomlkit "DOM" nodes as inputs... which is even more complexity for even less gain.

Honestly - returning a simple str seems like the best, and most flexible option.

[rmartin16]

I pivoted this PR to just focus on adding support for third-party GUI framework plugins. A subsequent PR will build on this to support running apps from app-build-verify in CI.

[rmartin16]

A few initial discussion points:

• Entry point name: briefcase.wizard.frameworks
  • Need to confirm this name. It originated in Create new project with Positron template (Django or Static) #1288.
  • I'm not sure I'm a fan of "wizard" per se....
  • Perhaps along the lines of briefcase.platforms....this could be briefcase.gui_frameworks
• New directory hierarchy
  • I created a top-level directory named plugins....but it does seem a bit weird given this is the first-party repo
  • I'm considering just getting rid of it and moving the frameworks directory up under src/briefcase
• GUI framework plugin implementation
  • I wanted a way to define a prototype for the plugin...hence the BaseGuiPlugin class
  • I didn't add abstractmethod to every method because I didn't want to force every subclass to implement every single method
• Defaulting for briefcase-template context fields
  • I intentionally left meaningful defaults out of briefcase-template....because otherwise any changes there would be picked up by third-party plugins....and that doesn't seem appropriate...they should fully define their requirements

[freakboy3742]

A few initial discussion points:

• Entry point name: briefcase.wizard.frameworks

  • Need to confirm this name. It originated in Create new project with Positron template (Django or Static) #1288.
  • I'm not sure I'm a fan of "wizard" per se....
  • Perhaps along the lines of briefcase.platforms....this could be briefcase.gui_frameworks

FWIW, I don't hate wizard as a name - my biggest objection is that it's not clear that it's the "new project" wizard (not that we have other wizards, but in theory we could).

• New directory hierarchy

  • I created a top-level directory named plugins....but it does seem a bit weird given this is the first-party repo
  • I'm considering just getting rid of it and moving the frameworks directory up under src/briefcase

Agreed that it would make more sense in the src folder, rather than something standalone.

Whatever name we choose, I think it should align with the name we use for the entry point.

I'm a little hesitant on "frameworks", because (a) it's arguable whether a GUI library is a "framework", and (b) what we're doing here is a little more that just the framework. The "Positron" use case is "Toga, but with a very specific project layout and bootstrap". "Toga" is a GUI framework; "Positron" is then a layer on top of Toga. But we need to be able to distribute "Positron" independent of "Toga", and the two need to be able to co-exist. For that matter, Toga itself might end up offering multiple plugins - one for a "simple" app, one for a Document app, one for a system tray app, and so on.

I'm wondering if "bootstrap" (briefcase.bootstraps / src/briefcase/bootstraps or src/bootstraps) might be a good name - talking about a "Toga bootstrap" or a "Positron bootstrap" seems a reasonable description of what its doing in the context of a Briefcase project lifecycle, without being tied to the implementation (wizard) or being tied to a the name of a project/framework. Having multiple "Toga bootstraps" with different options wouldn't be out of place either.

• GUI framework plugin implementation

  • I wanted a way to define a prototype for the plugin...hence the BaseGuiPlugin class
  • I didn't add abstractmethod to every method because I didn't want to force every subclass to implement every single method

Honestly, I'd be surprised if there wasn't a base class :-) Using the "default empty implementation" approach seems a reasonable approach.

[rmartin16]

I'm wondering if "bootstrap" (briefcase.bootstraps / src/briefcase/bootstraps or src/bootstraps) might be a good name - talking about a "Toga bootstrap" or a "Positron bootstrap" seems a reasonable description of what its doing in the context of a Briefcase project lifecycle, without being tied to the implementation (wizard) or being tied to a the name of a project/framework. Having multiple "Toga bootstraps" with different options wouldn't be out of place either.

This is a good choice to me; getting away from both "wizard" and "framework" sheds a lot of out-of-place connotations.

This PR is stable at this point. I may add some more tests or tweak things with more testing. I now need to get briefcase-template tests working again.....I don't think I even knew it had tests...

I've been using this example plugin for my testing....which, IMO, also nicely demonstrates how easy it'll be to extend the default Toga project.

I'm not super happy with the "bootstrap prompt" code....it isn't too terrible but it's certainly a little awkward.

[freakboy3742]

The broad strokes of this look really promising.

One idea/suggestion: What happens when a bootstrap needs an option that isn't on this list, or doesn't want to include a setting that is? Say they need to inject a Dockerfile fragment on Linux, or omit the web_style_framework setting completely - this setup won't let the bootstrap do that. Unless our API blesses a specific setting, they can't include it; and if we bless it, it must exist.

I'm wondering if it might be better to make the backends provide the entire configuration block for a platform, rather than explicitly enumerating each setting we think is important. And then, if the bootstrap doesn't define a configuration endpoint (e.g., PySide on iOS/Android/Web), the template outputs "not supported". That way we've only got 1 API point per deployment target (6... or 12 if we explicitly include the various Linux options), rather than one for each possible setting; and the expansion path for the future will be "one API endpoint for each new platform we support", rather than "one per configuration item that we think is important".

It might also be worth passing in the Briefcase version somewhere, so that third-party plugins can adapt to settings that need to exist (or not exist) on different Briefcase versions. I guess this could be extracted from the context that is passed to constructor, but for that particular detail, it might be better to be explicit.

[freakboy3742]

This one should hopefully disappear - #1166 includes code that makes the style framework used by web deployments a feature of the GUI toolkit.

However that raises another question - details in the overall comment.

[rmartin16]

Revamped bootstrap interface based on feedback.

[rmartin16]

Those are good points.

I was also considering if there should be a plugin method for asking additional questions....although, technically, a plugin could drop in more user prompts anywhere.

And was thinking about a plugin adding entirely new sections to the TOML....probably most useful if a plugin is also being used for a custom platform/format as well.

Mostly note to myself....but I'll want to confirm how Briefcase responds to commands for a section if a plugin leaves that section blank....probably want to avoid Briefcase bombing out too badly.

[rmartin16]

Status of the dependency among PRs

All of the PRs are dependent on each other and likely will fail if they are not all on the same branch. The CI workflows in beeware/.github#68 are currently hardcoded to always use the relevant pieces of all this development; this will obviously all need to be replaced prior to merging and I've tried to clearly mark them all in the PRs. But it should mean that the CI running for these PRs is accurate insofar as meaning everything's compatible.

As for merging in to main, these three PRs all need to go first and at once:

• Generalize template for GUI framework plugins briefcase-template#82
• Add support for third-party GUI framework plugins #1524
• Add reusable actions to create and/or verify a new project  .github#68

Once those repos look good, the template PRs can go.

[freakboy3742]

Except for the beeware/.github update, this is all looking good. Marking as approved, conditional on the changes that are needed as part of the landing strategy.

In terms of the landing strategy - this is what I understand the landing strategy will be:

1. Land Add the ability to override app configurations at the command line #1542, to expose the --config option.
2. Address the feedback on Add reusable actions to create and/or verify a new project  .github#68. That should (I think) clear up the issues currently causing test failures on briefcase and briefcase-template; but there will still be test failures on the .github PR because the briefcase install tests won't work in their current state.
3. Rerun CI on briefcase and briefcase-template. With the fixes from (2), they should pass clean.
4. Push the updates to briefcase, briefcase-template and .github to reset the references to in-progress beeware/.github repos. We know this will break CI (because the repos are co-dependent), so we don't wait for a green board.
5. Merge briefcase, briefcase-template and .github in rapid succession. There's a minor risk that .github might fail because we couldn't see a green board prior to merging Briefcase, but we can hot fix that as needed.
6. Update the template repos to reset the beeware/.github references, wait for green CI, then merge.
7. Open beverage of choice :-)

Have I missed or misunderstood anything in this?

[rmartin16]

Have I missed or misunderstood anything in this?

Notwithstanding the inevitable issues from merging CI changes (especially this extensive), yeah, this is the plan. Definitely feel free to ping me to be around during the merges.

[rmartin16]

note to self: add changenote for dropping PySide2 support.

[freakboy3742]

Have I missed or misunderstood anything in this?

Notwithstanding the inevitable issues from merging CI changes (especially this extensive), yeah, this is the plan. Definitely feel free to ping me to be around during the merges.

Ack. I'm guessing #1542 won't be merged until early next week (Monday my time, Sunday for you); but once that's there, I'll ping you before I start merging anything. If we're lucky, that will be Monday morning after the dependabot updates; but failing that, I'll put it as my "first thing in the morning" set of merges on a day that works for you.

[freakboy3742]

Approved for final ship.