refactor[next]: Refactor workflow based executors and move the backend class.

[DropD]

Description

Changed

• OTFCompileExecutor and CachedOTFCompileExecutor merged into ModularExecutor

Moved

• OTFBackend moved and renamed to next.backend.Backend

Reasoning

The two *CompileExecutor classes were identical, except for type hints. Since they are now almost exclusively used inside OTFBackend, which does not retain typing information about which of them it contains, this distinction is no longer helpful for static type checking.

OTFBackend has always been more general than it's naming and location suggested. It is the de-facto definition of a backend within gt4py.next: a wrapper around an executor and an allocator. This change makes the status quo visible.

Requirements

• All fixes and/or new features come with corresponding tests.
• Important design decisions have been documented in the approriate ADR inside the docs/development/ADRs/ folder.

[tehrengruber]

I think it makes sense. From a quick look it seems that Backend.executor could directly become otf_workflow as the only task ModularExecutor has is to transform the args of its __call__ method into a form that is understood by otf_workflow. This could equally be done by Backend.__call__. Since this involves some minor refactoring in other parts of the codebase that rely on Backend.executor we can do it in another PR (maybe if there are some free cycles sometime).

[DropD]

I think it makes sense. From a quick look it seems that Backend.executor could directly become otf_workflow as the only task ModularExecutor has is to transform the args of its __call__ method into a form that is understood by otf_workflow. This could equally be done by Backend.__call__. Since this involves some minor refactoring in other parts of the codebase that rely on Backend.executor we can do it in another PR (maybe if there are some free cycles sometime).

I think it makes sense. From a quick look it seems that Backend.executor could directly become otf_workflow as the only task ModularExecutor has is to transform the args of its __call__ method into a form that is understood by otf_workflow. This could equally be done by Backend.__call__. Since this involves some minor refactoring in other parts of the codebase that rely on Backend.executor we can do it in another PR (maybe if there are some free cycles sometime).

My first attempt was to do just that. It turned out to be a much bigger refactoring because the assumption that Backend.executor exists is in many test utils. So I opted for the current PR as a first step.

The current cycle will take us a bit closer still.

[tehrengruber]

👍 Maybe a todo would be useful in the future, if not for yourself for another person stumbling upon it.