Breaking Changes moving towards 3.0

[faustbrian]

This issue will be updated as more breaking changes reveal on the way to 3.0.

---

Core 2.0 has been a rewrite of ark-node which meant we had to support a lot of legacy functionality which will be gradually removed.

As we move towards 3.0 more breaking changes will occur as we remove those legacy parts and 2.4, 2.5 and 2.6 will contain the majority of those breaking changes with 3.0 introducing breaking changes to the container and plugin system.

2.3

• Removed the wallets table from the database as core only relies on in-memory data
• Replaced sqlite3 with lowdb as storage in core-webhooks
• Replaced core-logger-winston with core-logger-pino due to performance
• Merged core-debugger-cli into core-tester-cli and deprecated it
• eventemitter3 has been replaced with the native node.js event emitter

2.4

• Restructuring of the @arkecosystem/crypto package in preparation of AIP11/18/29. This means certain imports change and possibly method signatures.
• The old P2P API has been replaced with WebSockets.

2.5

• Removal of the legacy API from ark-node that is available as v1 in core-api. This includes the removal of certain methods from repositories in core-database and core-database-postgres.

2.6

• Final restructuring of the @arkecosystem/crypto package when the implementation of AIP11/18/29 is finalised.

3.0

• Rework of core-container and the plugin system as those have quite a few flaws because they weren't part of core from the start which resulted in a wonky implementation for certain things.
• Events will be reworked from scratch and start to follow a fixed naming scheme as they also weren't part of core from the start and suffer from the same symptoms as the container.

[faustbrian]

Kernel

core-kernel will be the central entry point for every plugin and the main application. It will be a combination of core-container, core-interfaces, core-event-emitter, core-logger and core-utils.

All of those packages at the moment are split up even though they are required to be always available and used by pretty much every plugin. Combining all of those will remove the need to maintain different packages, reduce complexity and architectural issues that currently exist during the application start up.

This also means that interfaces are always available to every package without having to require an additional package which is pretty much always required together with core-container and thus reduces the setup process for new plugins to get started.

An initial PoC can be seen at https://github.com/ArkEcosystem/core/tree/3.0/packages/core-kernel/src.

[faustbrian]

Container

When Core 2.0 was initially developed there was no container because it was started as a simple rewrite without any plans on expanding or altering how the system itself worked and interfaced with developers.

A few months into development I decided that Core was becoming too big and had to be split up into smaller parts that can be easier maintained, tested and replaced. Without much of a surprise this resulted in some structural and implementation issues when a larger application receives such a big changed over half-way into development as it has to be done fast.

The result of the pace at which it had to be completed is an implementation that has few services that have a lot of responsibilities they shouldn't have and knowledge about things outside of their responsibility they also shouldn't know about.

For example the container knows how to load configuration, setup the environment and load plugins while a container should only be concerned with setting and getting values based on keys and values it is provided. In 3.0 there will be 2 different classes that take on those tasks.

Application

The application class will be what all plugins interact with and extends the Container class to serve as a proxy to the container.

It's purpose is to provide easy access to all relevant application information like configuration, paths, environment and state without leaking any of the underlying functions like the current "container" class does.

An initial PoC can be seen at https://github.com/ArkEcosystem/core/blob/3.0/packages/core-kernel/src/application.ts.

Container

The new container is only responsible for setting and getting based on keys and values. Those bindings can be either singletons which means they only get created once and we receive the same instance or value every time we request it by its key or just a simple binding of which we get a fresh instance every time.

The container no longer has any special treatment for plugins, instead there now are Service Providers which requires plugin developers to register all of their services with the container.

An initial PoC can be seen at https://github.com/ArkEcosystem/core/blob/3.0/packages/core-kernel/src/container.ts.

[faustbrian]

Contracts

Contracts will be what core-interfaces is at the moment. It will be part of core-kernel as a contracts sub-directory and expose all interfaces of official packages that serve as implementation contracts.

An initial PoC can be seen at https://github.com/ArkEcosystem/core/tree/3.0/packages/core-kernel/src/contracts.

[faustbrian]

Services

As previously mentioned the container and with it the plugin system were late additions to core and not something that was included from the beginning which resulted in some issues with how plugins were implemented.

Plugins right now are a simple object that exports some information and has a register and deregister method for when the application starts or stops. The issue with this is that plugins have to perform all their registrations in a specific order to get full access to everything that other plugins expose. This can't be avoided 100% as plugins have dependencies on each other but more information can be made accessible earlier in the application life cycle and this is where the new service providers come in.

Service providers will be what is used for application bootstrapping. Your own plugins as well as all core services are bootstrapped via service providers which provide register and boot method.

The register method should be used to create container bindings and store other information without starting things like HTTP servers or any kind of long-running services so other plugins can have access to that information for their own usage.

The boot method will be called after all services providers have been and registered and had their register called. At this point you have access to all information other plugins exposed via their register method and you are ready to start long-running processes like HTTP servers or background services.

The dispose method will be called on application shutdown and should be used to terminate any instances of background services or HTTP servers your plugin started.

An initial PoC can be seen at https://github.com/ArkEcosystem/core/blob/3.0/packages/core-kernel/src/support/service-provider.ts.

[faustbrian]

Configuration

With the new container it will become a lot easier to load and alter configuration at runtime which also means we will finally provide configuration hot-reloading which means every time you edit one of your configuration files they will be read and rebound to the container which will trigger certain plugins to reload without having to restart the whole process.

Bootstrap

When the application is starting and running all bootstrappers it will load the configuration either via the local or remote adapter, depending on if the user specified a remote host for configuration retrieval.

An initial PoC can be seen at https://github.com/ArkEcosystem/core/blob/3.0/packages/core-kernel/src/bootstrap/load-configuration.ts.

Local Adapter

The local adapter will load configuration from a local configuration file like it has been done like the release of Core 2.0.

An initial PoC can be seen at https://github.com/ArkEcosystem/core/blob/3.0/packages/core-kernel/src/config/adapters/local.ts.

Remote Adapter

The remote adapter will load configuration via a remote host the user specified and create a local configuration based on that to allow next starts to work without having to specify a remote host. This will provide an easy way of setting up a new node with the same configuration as your other nodes without the need of manually copying files.

An initial PoC can be seen at https://github.com/ArkEcosystem/core/blob/3.0/packages/core-kernel/src/config/adapters/remote.ts.

Hot Reloading

Hot Reloading will watch all relevant configuration files for changes and any occur it will trigger either a full reboot or fire an event so plugins can reload.

An initial PoC can be seen at https://github.com/ArkEcosystem/core/blob/3.0/packages/core-kernel/src/watcher.ts.

[faustbrian]

Validation

TBD

[faustbrian]

Error Handling

TBD

[faustbrian]

Logging

A logger is always required to be available but at the moment core-container has to use console.log early on because no longer is available. To solve this issue core-kernel will ship with a default logger which is available very early on in the application life-cycle.

The majority of users will never replace the default logger and if it needs to be replaced it can be replaced by doing app.bind('logger', myLogger).

An initial PoC can be seen at https://github.com/ArkEcosystem/core/blob/3.0/packages/core-kernel/src/logger.ts.

[faustbrian]

Cache

In Core 2.0 various packages have their own implementation of a cache which means implementations might vary between packages instead of being able to rely on a single common interface to know how every cache works.

To resolve this issue a new class is introduced which is called CacheStore and provides an implementation of a cache that uses https://github.com/lukechilds/keyv under the hood to allow easy configuration and changing of the storage system.

An initial PoC can be seen at https://github.com/ArkEcosystem/core/blob/3.0/packages/core-kernel/src/cache.ts.

[faustbrian]

Events

When an event emitter was initially added to Core 2.0 it was late in development like the container which resulted in similar issues. At first it used https://github.com/primus/eventemitter3 for its performance but those gains weren't as big with node.js anymore and thus it moved to using the native event emitter that comes with node.js.

The issue with both of them is that they don't have proper support for async/await which means you can only fire and forget which is a problem if you rely on an internal event being completed before continuing code execution. As Core 3.0 will make stronger use of internal events to communicate state it will start to use https://github.com/sindresorhus/emittery as it's event emitter with a wrapper around it.

The advantage of this emitter is that we can await an emit and wrap them in try/catch to see if the listener of an event threw an error which will be important for internal events.

Naming Conventions

At the moment there is a lack of internal events which would solve a few architectural issues and events don't have any naming conventions other than being prefixed with the resource type like block, transaction or wallet.

To solve both of those issues there will be internal, state and action events.

1. Internal events will be prefixed with internal. to make it clear that they are not intended for use by third-party packages and should be used with caution as their data could change at any time or they get removed.
2. State events will be triggered when changes to the state of the application or plugins happen like onPreStart, onPostStart, onPreServiceRegistration, onPostServiceRegistration or onTearDown.
3. Action events will be triggered when core is about to perform an action that involves a resource like onPreBlockApply, onPostBlockApply, onPreRoundRevert or onPostRoundRevert.

All of those changes will allow plugins to listen for more granular events and get access to more data without having to hook into specific plugins, simply listen and process the incoming data.

An initial PoC can be seen at https://github.com/ArkEcosystem/core/blob/3.0/packages/core-kernel/src/event-dispatcher.ts.

[faustbrian]

Package Development

The development of packages and plugins for Core will mostly remain the same with a few breaking changes that require minimal changes unless you rely on deep burrowed internals.

The biggest change is the way your plugin is registered and exposed within the application through service providers that are run when the application is bootstrapped.

Old
https://github.com/ArkEcosystem/core/blob/develop/packages/core-api/src/plugin.ts

New
https://github.com/ArkEcosystem/core/blob/3.0/packages/core-api/src/service-provider.ts

As you can see in the above example the service provider extends an abstract service provider that is imported from core-kernel. That abstract provides access to the application object and the options for the plugin and specifies a few methods that are required to be implemented by the concrete.

The main change here is the structure, moving from a simple object to a class that allows a cleaner implementation and gets access to additional functionality through the abstract it extends.

A new feature that was implemented is to allow the definition of dependencies. For example you could specify that you require the core-database package to be installed with a version constraint of >=3.1.6 because this version contained a bugfix your package relies on.

[faustbrian]

Testing

As previously mentioned the container and event emitter were late additions to Core which resulted in several architectural and implementation issues. The result of those issues is that testing can be quite cumbersome unless you have a good understanding of the container and it's internals.

Over time several helpers like transactions factories, custom jest matchers, server mocks, etc. have been created but they are too far spread out between directories to be immediately visible to a developer and often aren't self-explanatory.

Ideally at some point there will be a small framework that is build around core internals to allow easy modification of the application state and internals to produce testing scenarios with as little instructions needed as possible and removing the need of having to run a full node during tests.

Work on this has recently already started but there is still a lot left to do and changes will occur gradually.

[faustbrian]

Helpers

Core has various packages that contain utilities and helpers which are required by the majority of packages but are spread out too much. All of them will be combined into a core-kernel/src/helpers directory which will be exposed via import { Helpers } from "@arkecosystem/core-kernel".

A few helpers might be removed or replaced with alternatives from https://github.com/ArkEcosystem/utils. Another goal of moving them into core-kernel is to implement alternatives that have no third-party dependencies if the scope of it is reasonable.

[faustbrian]

Closing as more detailed information is coming.