v5 discussion

[System-Glitch]

I'm starting to think a bit more about v5 and would like to discuss my ideas with the community. This post is mostly a place where I brainstorm and you are invited to join if you want to, your feedback is always welcome!

• Move from a global-based design to a struct-based design (would allow to isolate DB instances, config, and more. This would also remove some concurrency locks that are currently necessary) Instances #166
• Add / modify how the validation system handles errors (especially for validation rules that use the database)
• Re-think validation to make it less error-prone, with compile-time checks. That is using functions instead of strings. This would also improve rule parameters types (instead of all strings).
• Change the conventions for controllers so they are structs instead of standalone functions. We could use the power of interfaces for all sorts of useful features. Like this, we could also decide if we want to make the route definitions for a controller inside the controller package rather than putting everything in the route package (better feature segregation).
• The parse request middleware is convenient most of the time, but prevents some use-cases such as streaming or custom body reading. Allowing to disable it (or make it opt-in) would be a good change for more flexibility.
• Re-think how routes are made. I would like routes to expose more information by centering their design around metadata. This could let us make an even better OpenAPI generator for example. Middleware would use this metadata to function, just like the validation and CORS middleware currently work. There is no reason custom ones cannot use this kind of system. Middleware could also inject meta into a route (to tell that a route is authenticated for example, and with the method and everything)
• I would like to try and take advantage of the generics without falling into the over-engineering part and without losing the simplicity of the framework.
  • Add a generic parameter to controllers so the request can automatically be transformed into a struct (DTO style) if it passes validation. I don't want to use dependency injection.
  • Validation and config could take advantage of generics but I'm not entirely sure how for now
• See how we could use embedded resources for language files, config and such. Having to include those files into the production package is not always convenient. (Embedded static resources #133)
• See how we could take advantage of io.FS for all file-related features.
• In validation, allow validating any type (not just objects) at root-level. This will also allow JSON request bodies that are not objects. This would be done by allowing "CurrentElement" at root level in validation ruleset (among other things)
• Remove the sliceutil package and replace it with samber/lo.
• Lock config in read-only mode when the server runs so there is less slowdowns caused by mutexes. I've never see any use-case where config really needs to be modified at runtime.
• Remove TLS support as it adds unnecessary complexity for a very rare use-case. APIs are very often deployed behind a proxy that handles that for them already.
• Remove maintenance mode for the same reason.

The changes would be so big that a big part of the code-base would have to be rewritten and projects using v4 would probably not be able to upgrade. In short, there would be big design shifts. Therefore whenever v5 releases, I would like to keep maintaining a v4 branch.

[Wulfheart]

Btw, any v5 release ETA?

[System-Glitch]

No ETA yet. Progress is a little bit slow because of busy times at the company I'm working for. So I work on Goyave only when I have some time. I keep everyone updated regularly on the Goyave discord channel.

[System-Glitch]

Log of the progress notices on Goyave's Discord

29/09/2022

As you may know, v5 is a rewrite of a big portion of the framework. I am working on each core package and try to improve them.

• Design and basic implementation of the new validation package is progressing well. I reached a satisfying state and I think I will keep most of the redesigned aspects so far.
• The language package has been improved and the changes are not very large. Implementation is complete and I am currently working on proper testing and documentation of the package. I am still looking into using embed resources for language files though.
• Overall we are now in a early state of v5. The framework's core works and is usable. A lot of testing, documentation and polishing has to be done, but there's no point in writing that yet since nothing is finalised and design may still change.

25/10/2022

Yesterday and today I managed to add support for map validation. That means that you will be able to validate an object and its properties without knowing the properties name (like a map[string]any)

In the following example: all properties of the object "object" in the body must be integers (and there must be at least one property in the object)

func WildcardValidation(r *goyave.RequestV5) v.RuleSetV5 {
    return v.RuleSetV5{
        "object.*": v.ListV5{v.Required(), v.Integer()},
    }
}

26/10/2022

When using numeric validation rules (integer or float validation), you will be able to specify the target bit size. For example, you may want an int that fits into an uint8 and use it as uint8 into your controller. In v5 you will use the Uint8() validator instead of having to implement a custom rule.
It is still recommended to use the min and max validators alongside these new ones.

I had the idea to do this when I encountered an error when inserting a DB record with a smallint type, but the input was larger than what smallint can store, leading to a DB error. It will be easier to make sure this doesn't happen with these new validators.

12/11/2022

Today I implemented the auth package for v5. It is still untested though. Although the development of v5 is very slow, we are one step closer to a preview release.

Once the existing packages are upgraded, I would like to start creating preview releases to let everyone try it out and hopefully get some feedback. This process would be very flexible because I would still be able to change things according to feedback before deciding to freeze one design aspect.

Preview releases won't be documented as well as the current v4 release, and won't be stable. Tests will be written with time and documentation when I am certain the design won't change anymore. Breaking changes between two preview releases are to be expected.

13/11/2022

Today I settled for a new concept: Components. A Component is simply a structure that has access to all server resources (config, loggers, db, lang). Here is a list of examples of structures that you already are familiar with that are now components:

• controllers
• middleware
• chained writers
• authenticators
• etc...

This new concept was necessary since we moved on from an all-global system. As all components of a Goyave app may need access to those resources, I decided to add some convenience to avoid having to pass config and DB instances around all the time.

I also implemented the log package for v5. This package contains the middleware for common and combined logs, as well as the interfaces necessary for custom loggers.

23/11/2022

I had some time this week to work on Goyave. I decided to rewrite the core part of the validation system again to solve some issues with it. Namely the validation order and comparison. Rule sets are now a slice instead of a map, which lets developers define the exact order they want fields to be validated. This reduces the validation package complexity and reduces the non-explicit behaviors.

The syntax is a little bit heavier than before (due to the necessity to add keyed fields to struct literals), but I think the change is worth it.

Example:

func BodyValidation(r *goyave.Request) v.RuleSet {
	return v.RuleSet{
		{Path: "field", Rules: v.Lis{
			v.Required(),
			v.Int(),
		}},
		{Path: "array", Rules: v.List{
			v.Required(),
			v.Array(),
			v.Between(3, 50),
		}},
		{Path: "array[]", Rules: v.List{
			v.Int(),
			v.Between(4, 5),
		}},
		{Path: "foo", Rules: v.List{
			v.Required(),
			v.GreaterThan("array[]"),
		}},
		{Path: "composed", Rules: OtherBodyValidation(r)},
	}
}

I successfully implemented RequiredIf. I also added some useful features to the walk package to help retrieve data easily and safely for this kind of use-case or for comparison rules. Here is an example usage:

func BodyValidation(r *goyave.Request) v.RuleSet {
	return v.RuleSet{
		{Path: "a", Rules: v.List{
			v.Required(),
			v.Bool(),
		}},
		{Path: "b", Rules: v.List{
			v.RequiredIf(func(ctx *v.Context) bool {
				// Using a path here is not necessary since it's a very simple case
				// But in a more complex scenario, using a path like this makes retrieving data
				// much easier and safer.
				c := walk.MustParse("a").First(ctx.Data)
				if c.Found != walk.Found {
					return false
				}
				return c.Value.(bool) // Type-assert is safe since we know "a" has been validated already
			}),
			v.Float64(),
		}},
	}
}

I am happy with the new implementation of the validator. It feels more robust and more explicit while removing a lot of code complexity. I still need to write a lot of tests and implement all the already existing validation rules.

I also wanted to talk about a new schedule at my company, which will help me get more time to develop the framework. Every 4 weeks I will have more time to work on the framework from now on. Outside of these weeks, don't expect to see any progress made for v5.

[System-Glitch]

06/01/2023

I haven't been able to work a lot on Goyave this week because of vacations. But I still made some progress.
Today I finished implementing (excluding tests) all the validation rules and I updated the default language accordingly.

Next I would like to work on improving the testing tools.

18/01/2023

The ratelimiter package is now DEPRECATED: This implementation couldn't be used in a realistic production environment and didn't cover critical aspects of a real application.

• Slow because of too many mutex locks
• Memory hungry
• Completely resets if the application restarts (which is very common when deployed on a cloud service)
• Not distributed (meaning it is impossible to use when working with multiple instances and load balancing)
• Weak policy (simple time window) whereas more efficient policies such as the leaky bucket could be used
• Outdated because based on an older IETF draft

For these reasons, this package will be removed in v5 to avoid developers relying on it. It is recommended to use better alternatives such as redis_rate.

[System-Glitch]

10/02/2023

This week was allocated to work on the framework. I made very good progress as well as some polishing. Here is a list of what's been achieved this week:

• Compress middleware: a middleware replacing the gzip middleware, and supporting multiple encodings. Uses the encoding according to request's preference reading the Accept header. It currently only supports gzip out of the box but it's very easy to add any type of encoder yourself.
• Some improvements on usability (import paths, function call conciseness, exported some accessors etc)
• Removed auto migrations. They can be handy for prototyping but have too many caveat and are not practical in real projects. Removing them leaves developers with the choice of the tool they want to use for migrations. It will still be very easy to automigrate like before if needed, but going forward I would recommend writing SQL migrations instead. Also allows to remove Registered models since they were only used for that purpose.
• Improved the fsutil package to return errors instead of panicking
• Improved config so it properly supports slices of int, float64, bools and strings.
• Added typeutil.Undefined, a type helping to differentiate between a missing field and a null value in a request when converting the request.Data to a struct using the typeutil.Convert function. This also simplifies updates using GORM by a lot since you can know just pass it your request structure easily.
• More small changes

The highlight of this week though is the revamp of testing. TestSuite has been removed and replaced by the new testutil package. Here is a list of what it brings:

• Remove a lot of friction when writing tests because you're not forced to use a TestSuite anymore for everything that needs config, lang, etc.
• Allows parallelisation of tests, making them much faster
• You are not forced to use testify anymore, you can use the library of your choice (or no library)
• Doesn't require an HTTP server to be running and listening to a port, even for HTTP tests.
• Facilitates the load and modification of everything (config, lang, DB, etc)
• Closer to standard Go tests with less framework-specific functions without compromising on ease of writing
• Allows all SQL queries (including those run by your tested code) to be run inside a transaction that will be rolled back at the end of the test without having to mock anything. This makes tests involving DB much easier and concurrently safe.

[System-Glitch]

16/03/2023

• Added a few utils such as MustConvert (same as Convert but doesn't return an error), route.BuildProxyURL() for URL generation behind a proxy, server.Host() and more.
• Added a Valid() function on validation context so some rules can be skipped if prior rules didn't pass
• Added Trim validation rule (which trims strings)
• Started writing tests (for validators mostly for now)
• Added a default (configurable) timeout for all SQL queries executed.

[System-Glitch]

15/04/2023

This week a ton of tests have been written. There are still plenty more to write but I made good progress.

[System-Glitch]

27/05/2023

A lot has happened this week! First and foremost, I am almost done with the tests. This means I will be able to go forward with confidence on the rest of what has still to be done. I also wrote a bit of documentation.

General

• Added a few more testing utilities and improved some existing ones.
• The time the request has been received is now stored in the request and is accessible.
• log.Context doesn't have access to the response anymore, only the response status code.
• Removed the route registrer function from the server's Start() method parameters. Routes are now registered through server.RegisterRoutes() beforehand.
• All startup hooks are now executed in the same goroutine in order of registration.

Services

I introduced a new concept to the framework: services. Those represent a new layer that would be a bridge between the HTTP/REST layer, the domain layer and the database layer. This is done in an effort to encourage more separation of concerns for cleaner code in larger applications. Services are linked to a server instance and registered on startup.

The general idea is to structure a project more like so:

HTTP/REST <-> Services <-> Repository <-> Database

Auth package

The auth package has gone through some refactoring and minor improvements.

• A new JWTService has been implementing for managing signing keys and token generation.
• Dropped support for RSA keys with passwords. The std lib only supports an older format of keys. It is still possible to do so by generating your tokens without using the built-in JWTService.
• The JWTController and auth middleware now take a generic parameter defining the user model.

Validation

• Some validation error message improvements. One of them is the renaming of ".array" validation message to ".element" to make it less confusing that they are messages that happen when the field under validation is an array element.
• Added ExistArray and UniqueArray to efficiently validate an entire array against a database.
• Added a way for a Validator validating an array to add errors on its elements instead of on itself.
• Improved number validators to make them safer. They make sure the source data can fit in the desired numeric type without overflow or loss of precision.

Database

• Removed DB initializers. They are not useful anymore as the DB customization can be done before starting the server now.
• Some refactoring.
• Improved Paginator . It now takes a generic parameter for the model. UpdatePageInfo() now returns the executed statement.

[System-Glitch]

I'm proud to present the first preview version! I'm looking forward to hearing your feedback!