[WEP] Service tooling proposal

[leaanthony]

Description

This proposal outlines new CLI functionality for the new "Services" concept in v3, which can be user to create new services in a standard way. It offers a foundation for future service management, such as installing 3rd party services.

I'm willing to implement this myself.

[cloudflare-workers-and-pages]

Deploying wails with    Cloudflare Pages

Latest commit:	2866852
Status:	✅  Deploy successful!
Preview URL:	https://b110c04f.wails.pages.dev
Branch Preview URL:	https://v3-alpha-proposals-service-t.wails.pages.dev

View logs

[fbbdev]

First of all thank you for your work on this and sorry for the very late feedback.

It is true, as you wrote in the proposal, that people can manage services by hand without too much effort.

Nonetheless, I think this contribution fits very well wails' mentality of providing ergonomic, moderately opinionated development tools on top of the go distribution, and it is definitely going to play a part, however small, in making the developer experience smoother and simpler.

I would like to object to the currently proposed code layout, and suggest an alternative.

The main problem IMO is that it does not fit very well the layout of generated JS code, and it is going to introduce friction for the developers on the frontend side.

A secondary problem is that I don't think having a per-service go.mod file makes sense for application-level services, which are most probably going to import internal subpackages of the application module, and are probably never going to be reused.

Regarding the main problem, the binding generator currently outputs:

• per-service files <servicename>.js/ts;
• per-package index files index.js;

Service files can be imported with the usual syntax

import * as ServiceName from "<package>/<servicename>.js"

whereas index files are designed to facilitate imports like this

import {ServiceName1, ServiceName2} from "<package>";

Having one folder per service is gonna make indexes redundant and practically useless, as well as complicate JS imports, which need to be typed by hand. The former pattern is going to become:

import * as ServiceName from "<package>/<servicename>/<servicename>.js"

The latter will look like this:

import {ServiceName1} "<package>/<servicename1>";
import {ServiceName2} "<package>/<servicename2>";

Not nice...

I understand that having per-service folders was meant to facilitate method handling, and I am gonna propose a couple alternative solutions below.

Regarding the secondary problem, I explained my reasons above. I'd just like to add this: I remember reading around the internet that having nested go.mod files should always be avoided, except for versioning purposes. I always try to abide by this rule, but maybe it was just someone's personal opinion? Aren't there any drawbacks to this?

On to the alternatives.

I'd like to propose the following layout for applications:

<application root>
  +- services
     +- <servicename1>.go
     +- <servicename2>.go
     +- ...
     +- index.go

For reusable, non application-specific services, I propose adding a new plugin template to the init command that would result in the following layout:

<plugin root>
  +- <servicename1>.go
  +- <servicename2>.go
  +- ...
  +- index.go
  +- go.mod
  +- go.sum
  +- plugin.yml
  +- Taskfile.yml

The service tool would use the former layout if the current package is main, the latter if it is non-main. This logic has the added benefit of making the service tool work seamlessly in the services folder, as well as in any other subpackage that might want to export its own services.

The plugin.yml file replaces the proposed service.yml file, keeping the same content. I propose delegating this entirely to the template system, the service tool in my view should not be concerned with this detail.

A service file <servicename>.go would contain by default the type and lifecycle methods (names are hypothetical)

type ServiceName struct { /* ... */ }

func (s *ServiceName) InitService() { /* ... */ }
func (s *ServiceName) ShutdownService() { /* ... */ }

as well as any user-defined methods.

The index.go file would contain the following function:

func Services() application.Service {
    return application.CombineServices(
        application.NewService(&ServiceName1{}),
        application.NewService(&ServiceName2{}),
    )
}

The name Services is tentative, alternative names could be Instances, ServiceInstances, GetServices, GetInstances, GetServiceInstances...

Then application templates would be amended to include the following default configuration:

import "<module path>/services"

// ...

func main() {
    // ...
    application.New(application.Options{
        // ...
        Bind: []application.Service{
            services.Services(),
        },
        // ...
    })
    // ...
}

application.CombineServices would be a new function that combines multiple service instances into one application.Service value. Thanks to the struct being opaque, this can be done without any disruption to current code.

Let me explain the rationale behind this approach:

• it would be nice for the tool to automatically keep bound services in sync with the services folder;
• however, editing the main.go file directly would be cumbersome, more error-prone and less user-friendly then editing the body of a simple secondary function;
• the reason why the Services thingy has to be a function is that users might want to add some parameters there for service configuration;
• as an added benefit, plugins (i.e. reusable service packages) gain better encapsulation and present a uniform interface to consumers.

Two problems still have to be handled:

1. how to edit the Services function, but still allow users to customise it;
2. how to manage service methods.

Problem 1 can be easily solved by parsing the package, then editing the Go AST and printing it back instead of manipulating source code directly. The only constraint being that the Services function must have exactly one return statement, whose expression is a call to application.CombineServices, and if not we throw an error. Moreover, one could consider using go/types to retrieve the list of services defined by any package and its methods. This would be very easy.

Problem 2 can be solved in the same way, but I was actually thinking that maybe method handling could be omitted entirely. Service methods are not the same as routes/actions in a web framework: they are just plain simple Go methods, with zero boilerplate. I don't see this feature being used much, and I feel like it would be a bit overkill.

The only case where we'd still need to edit methods would be service renaming and removal, and it can be easily done with the type-checker + AST editing.

I am of course available for help or discussion in implementing AST/type manipulation. In this regard, I was having a look at things to understand how hard it would be and I found this interesting lib: https://pkg.go.dev/github.com/dave/dst

[leaanthony]

Thanks @fbbdev for the insightful feedback 🙏
I guess the reason I'm keen for one service per package is that it makes it exactly the same as Go modules, which everyone is familiar with. However there are 2 use cases we need to consider:

I'm making services for myself

This is almost certainly the default use case and the one we should consider the most. In this scenario, there is no need for go.mod as local imports are sufficient via the base module path.

I'm making a sharable service

This is where someone has written a service and would like to share it. Sharing should absolutely be done via Go modules. So then the question is, how do we get from a local service to a shareable one? Perhaps it's just adding a go.mod and hosting it.

The only issue with composing multiple services into the same folder is that each 3rd party library you pull in may have different package names (they might not all be package service).

We should also be clear on terms 😅 For me a service is discrete & shareable code that can be given to the Services (was Bind) list and be callable from the front end. Each service could, in theory, have different namespaces or contain multiple pieces of differing functionality and using the "Combine" idea you presented would be fine for that.

I would be very wary of manipulating source code or copying code from 3rd party packages to local directories as, honestly, I think it's going to be very hard with lots of opportunities to go wrong. The ideal scenario is where someone uses import to pull in a 3rd party service and calls whatever method to create a new instance of it for the Services field. I think, so long as we don't attempt to combine 3rd party and local services at the source code level, then we should be pretty good. The parser would generate the code for the 3rd party lib so I think it's ok that the structure of that is whatever is best for front end. It's ok if the 2 don't match, we should just do whatever feels the most natural from a dev perspective.

Again, thanks for taking the time to provide feedback. What a great community we have! 🚀

[atterpac]

Having one folder per service is gonna make indexes redundant and practically useless, as well as complicate JS imports, which need to be typed by hand. The former pattern is going to become:

I'm not sure if I understand the concern on this, it would essentially just be acting as if created a package for something and generate as such.

I was under the impression this move was mostly going to be handled with the golangs build system for instance I can
go get github.com/wails/sqliteplugin which exposes a Sqlite struct that implements a service to be bound and goes through the parser as already defined. I think this would solve the concerns of nested go mods because it would just be version controlled through the projects go.mod. It wouldn't allow for adjustments inside of a plugin but ideally the plugin developer would build in a way that the constructor exposes enough configuration options that editing source isn't a concern. Beyond that it would be forking the git and using your own personal repo.

There would be a concern of imports here being fairly lengthy on the frontend but could that be solved with the plugin.yml? maybe parser checks if there's a "package" field that standard should be similar to neovim package managers for example user/repo so imports are only services/user/repo and any other sub packages it might expose

[atterpac]

I've been working an an implementation of this idea with the thoughts listed above, I've reached a semi-functioning point and wanted to garner some feedback before fully committing.

The PR below implements plugins as a service without much effort by the end user and no pain to anyone to conform to the plugin interface for every bound service.

There is a Plugin Interface that has the following requirements to satisfy

type Plugin interface {
     Shutdown() error
     Init() error
     Name() error
}

Any service that is provided to the application.NewService function that implements this interface will be treated as a plugin and passed through the lifecycle methods required. This means that

1. Local plugins that a developer makes for themselves and would like to move between projects is just a go module that exposes a service the implements the interface
2. Remote plugins can be downloaded and version controlled via go.mod and go get for example the sqlite plugin below you simply go get github.com/atterpac/wails-plugin-sqlite add the neccessary struct to your Services option and it will work as is
3. Existing or local services that you'd like bound are not required to conform to the interface they are generated, and handled with no altercation to the existing code meaning developers dont have to worry about every single service implementing the interface as discussed above.

I believe this targets most of the concerns in this conversation mainly

• Nested go mods: Remote modules will be handled by go's build system, local plugins can use the projects go mod so no nested mods necessary
• Requiring all services to conform: Detecting if a service implements means Services can be a mix of plugins and services
  NOTE: Plugin template suggest and provides a struct.NewPlugin() that returns the instance, this is ideal for developers to recognize what is a plugin versus what isnt
• Binding Conflicts: Given bindings will have no change there should be no more concerns about conflicts then already exist which I am pretty sure we arent too concerned about and have the needed warning logs to cover that

Below ill link the PR to my changes, this is a draft PR as there is still some things to sort out if this is the desired route, ie; CLI tooling mostly, would love feedback on what kind of tooling we would want? if any? There is some desire to have CLI commands to read the plugin.yml for debugging/support reasons on the maintainers end but maybe listing plugins that were registered on the last run? pulling some version information out of the plugin.yml to better help plugin developers?
Who knows.

Notes on the PR:
wails3 plugin init -n myplugin will init a template plugin this will provide all things necessary to implement a plugin

Notes on the SqlitePlugin

• Not fully functioning but enough to get the point across not for use in production but for use in POC
• Steps to test

go get github.com/atterpac/wails-plugin-sqlite

// main.go
	sqliteConf := sqlite.Sqlite{
		DbName: "wailsPlugin",
	}
	app := application.New(application.Options{
		Name:        "v3-tutorial",
		Description: "A demo of using raw HTML & CSS",
		Services: []application.Service{
			application.NewService(&sqliteConf),
		},
		Assets: application.AssetOptions{
			Handler: application.AssetFileServerFS(assets),
		},
	})

See PR: #3570
Example Sqlite Plugin: https://github.com/atterpac/wails-plugin-sqlite

Thank you all for your input, looking forward to your thoughts/concerns

[atterpac]

Update: Pushed new commit to #3570, this supports detecting if a service implements an http.Handler

Previous discussion in the discord threads had pointed towards using FQN for the path for handlers. For example if I created an sqlite plugin at github.com/atterpac/sqlite handlers would be registered to services/github.com/atterpac/sqlite/<path> this ran into some complications during implementation due to the varying number of parameters that path could consume. wails.io/plugins/fileserver for example would be very different to match inside a map and would require looping through each section of the path until we hopefully found a matched segment

In discussion with other maintainers we opted for another solution, an optional parameter inside NewBindings, this will allow for users to define their own prefix only if they want to use that service handler and they choose the prefix. This would look like.

// if sqlite implements http.Handler it will be registered under services/sqlite
application.NewService(&sqlite{}, "sqlite")
// sqlite can still implement the handler but will not be registered if the prefix isn't provided
application.NewService(&sqlite{})
// another service handler for fileserver /services/files/<filepath> to request a file
application.NewService(&fileserver{}, "files")

Necessary adjustments have been made to the bindings generator to accommodate this change. I intend to recreate the "dynamic assets" functionality of v2 with a plugin as POC in the coming days
Loading the go mod can be done with the following setup, use test cases haven't been tested yet

type FileServer struct {}

func (p *FileServer) ServeHTTP(res http.ResponseWriter, req *http.Request) {
	app := application.Get()
	var err error
	app.Logger.Warn("Attempting Serving file", "file", req.URL.Path)
	fileData, err := os.ReadFile(req.URL.Path)
	if err != nil {
		app.Logger.Warn("Could not load file", "file", req.URL.Path)
		res.WriteHeader(http.StatusBadRequest)
		res.Write([]byte(fmt.Sprintf("Could not load file %s", req.URL.Path)))
		return
	}
	app.Logger.Info("Serving dynamic asset", "file", req.URL.Path)
	res.Write(fileData)
}

application.NewService(&FileServer{}, "files")

Inside the JavaScript console

let mod = await fetch("/services/files/go.mod")

[leaanthony]

I'm going to close this as it was an old proposal and we have since implemented some of it.