drun vNext Ideas

[brad-jones]

So the issue with this architecture is that it's difficult to provide the "Once" functionality & if we wanted to further extend this to generate CI pipeline config it's difficult to get the dependency graph without actually executing the tasks. We would have to resort to some sort of AST parsing.

void main(List<String> argv) {
  buildAll();
}

void buildAll() {
  buildFoo();
  buildBar();
  print('finished buildAll');
}

void buildFoo() {
  buildBaz('v2.0.0');
  print('finished buildFoo');
}

void buildBar() {
  buildBaz('v1.0.0');
  print('finished buildBar');
}

// this needs to be a map of the function args but I got lazy with the example
var buildBazExecuted = false;

void buildBaz(String version) {
  if (buildBazExecuted) return;
  print('finished buildBaz @ ${version}');
  buildBazExecuted = true;
}

So something like this makes it easy to build out the dependency graph & have the actual execution deferred thus allowing us to easily deduplicate tasks. However we can not specify task parameter via DependsOn.

void main(List<String> argv) {
  buildAll();
}

@DependsOn([buildFoo, buildBar])
void buildAll() {
  print('finished buildAll');
}

@DependsOn([buildBaz])
void buildFoo() {
  print('finished buildFoo');
}

@DependsOn([buildBaz])
void buildBar() {
  print('finished buildBar');
}

void buildBaz(String version) {
  print('finished buildBaz @ ${version}');
}

We could perhaps do something similar to this which is what PyInvoke does http://docs.pyinvoke.org/en/stable/concepts/invoking-tasks.html#parameterizing-pre-post-tasks However it's not typesafe and the values of the parameters can no longer be based on the input to the depender.

void main(List<String> argv) {
  buildAll();
}

@DependsOn(buildFoo)
@DependsOn(buildBar)
void buildAll() {
  print('finished buildAll');
}

@DependsOn(buildBaz, {'version': 'v1.0.0'})
void buildFoo() {
  print('finished buildFoo');
}

@DependsOn(buildBaz, {'version': 'v2.0.0'})
void buildBar() {
  print('finished buildBar');
}

void buildBaz(String version) {
  print('finished buildBaz @ ${version}');
}

At this point I have come to the conclusion that it is very difficult, I don't want to say impossible just in case someone comes up with a brilliant suggestion, to create a task runner with the following properties:

• 100% TypeSafe
• Simple function based API
• Declarative (or at least has deffered execution)

So then I started thinking outside the box, what if we used classes like this. The only thing I don't like about this is that it's just not as succinct as I would like.

The advantage of doing something like this is that we can easily parse the graph and generate say CI/CD config. Additional meta data can easily be added to the task classes via more fields/getters/setters/attributes to support different providers. eg: TeamCity vs Github Actions

The action method could in fact just refer to an actual Github Action. eg: actions/checkout@v2 or actions/cache@v1, etc...

void main(List<String> argv) {
  // Build the graph but don't execute anything.
  // The graph starting point could of course be different based on `argv`
  // just the same way that drun currently chooses which function to execute.
  var graph = BuildAll();

  // Then have some funtion / service that can execute the graph
  //run(graph);
}

abstract class Task {
  Deps get deps => Deps();

  Deps serial(List<Task> tasks) {
    return Deps(serialDeps: tasks);
  }

  Deps parallel(List<Task> tasks) {
    return Deps(parallelDeps: tasks);
  }

  void action();
}

class Deps extends Task {
  final List<Task> parallelDeps;
  final List<Task> serialDeps;
  Deps({this.parallelDeps, this.serialDeps});

  @override
  void action() {}
}

class HelloWorld extends Task {
  action() => print('Hello World');
}

class BuildAll extends Task {
  get deps => parallel([BuildFoo(), BuildBar('v2')]);

  action() {
    print('finished buildAll');
  }
}

class BuildFoo extends Task {
  get deps => parallel([BuildBaz('v1.0.0')]);

  action() {
    print('finished BuildFoo');
  }
}

class BuildBar extends Task {
  final String majorVersion;
  BuildBar(this.majorVersion);

  get deps => parallel([BuildBaz('${majorVersion}.0.0')]);

  action() {
    print('finished BuildBar');
  }
}

class BuildBaz extends Task {
  final String version;
  BuildBaz(this.version);

  action() {
    print('finished BuildBaz @ ${version}');
  }
}

[brad-jones]

TLDR: I am going to archive drun & gomake and create a totally band new never seen before (well I haven't seen it anyway) kind of task runner. If your looking for a traditional task runner written in either dart or go checkout https://github.com/google/grinder.dart & https://github.com/magefile/mage

Preface

You should read this in conjunction with brad-jones/gomake#14. Both these tickets tell a story, a broken up story but a story non the less. One day I might collect all my notes in these issues together and write a single cohesive blog post but until then you will have to put up with my ramblings.

The following examples I am going to use some pseudo yaml to describe the structure of some pseudo task runners / pipelines.

Task Runners vs Pipelines

Ok so I have had a complete change of mind. Yes the above "Class" based idea might actually work to build a type safe parametrized task runner that can de-duplicate the work correctly but I have a better idea, let me explain. A traditional task runner might look like this.

notify:
  run: acme-slack-client --channel #releases --message "Build 123 has started"

build:
  dependsOn:
    - build-foo
    - build-bar

build-foo:
  dependsOn:
    - build-baz
  run: acme-builder --project foo

build-bar:
  dependsOn:
    - build-baz
  run: acme-builder --project bar

build-baz:
  run: acme-builder --project baz

You would execute something like this acme-runner notify build. A traditional task runner would execute the notify & build tasks, it would de-duplicate the build-baz dependency, all pretty standard stuff. Where as if this were a CI/CD pipeline the structure might look something like the following.

notify:
  run: acme-slack-client --channel #releases --message "Build 123 has started"

build-baz:
  run: acme-builder --project baz

build-bar:
  dependsOn:
    - build-baz
  run: acme-builder --project bar

build-foo:
  dependsOn:
    - build-baz
  run: acme-builder --project foo

Notice how there is no build task. In the task runner example we added a task that does nothing it's self. It just an entry point into our build chain. The pipeline doesn't need that as it looks for all tasks that have their dependencies satisfied (ie: the first tasks that are run won't have any dependencies at all) and starts executing them all in parallel & then looks for the next set of tasks that it can run, repeating that process until all tasks are finished.

Ignoring the build-bar branch, in a traditional task runner the execution order looks like this.

• build
  • build-foo
    • build-baz

At least in an imperative task runner like drun this allows information to be passed from build through to build-baz before build has actually executed anything. Eg: a version number perhaps. In fact for drun and other similar task runners the execution order is really like this.

• pre-build
  • pre-build-foo
    • build-baz
  • post-build-foo
• post-build

Even in a truly declarative task runner like go-task it is still possible to pass information from the depender to the dependent. see: https://taskfile.dev/#/usage?id=calling-another-task Where as in a CI/CD pipeline the execution order looks like this.

• build-baz
  • build-foo

And so there is no way for build-foo to influence the input to build-baz as build-foo is yet to run but build-baz can of course influence build-foo. Task runners & pipelines are basically the inverse of one another.

Latest Idea

The solution I am conjuring up is to have a task runner that operates in a very similar model to a pipeline. Lets again look at an example.

dns1:
  run: ping 1.1.1.1

dns2:
  run: ping 1.0.0.1

dns3:
  run: ping 8.8.8.8

dns4:
  run: ping 8.8.4.4

Executing brads-new-runner without any other input would start pinging all 4 addresses concurrently. You wouldn't be able to say run brads-new-runner dns3 in isolation. The entire task runner definition is treated as a whole just like a pipeline would see it. This then allows us to easily generate CI/CD pipeline configuration for different providers (Github Actions / CircleCI / BuildKite / TeamCity / etc) from a single task runner or rather workflow definition. In fact what I am planning is to build essentially the opposite of https://github.com/nektos/act