v3 ideas, remove the ".gomake" folder #14
Comments
|
Just curious, but can you describe the differences between your tool and mage? |
|
Basically I started gomake to solve this issue magefile/mage#24 Essentially it generates a https://github.com/spf13/cobra app |
|
I have actually done a bit more dev on v3 in the last couple of days and after much deliberation I don't think this project has a viable future. It's basically a catch 22 situation, let me explain. While gomake has successfully solved magefile/mage#24 allowing targets to accept parameters from the command line. func Foo(bar string) error {
fmt.Println("bar: ", bar)
return nil
}
It can not easily support the func Foo(bar string) error {
fmt.Println("bar: ", bar)
return nil
}
func Baz() error {
mg.Deps(Foo("abc")) // this is invalid
}We could resolve this by using closures and it's the compromise v1 & v2 of gomake made. func Baz() error {
return run.Serial(
func() error { return Task1("abc") },
func() error { return Task2("xyz") },
run.Parallel(
func() error { return Task3("123") },
func() error { return Task3("456") },
),
func() error { return Task4("xyz") },
)()
}This works I guess but after using it in some production code bases for a while I still wasn't happy with it. You say ok, just don't use those APIs and call the functions directly. Which of course you can certainly do and for serial workflows it works just fine. But when you want to run things concurrently you create some "goroutines", then you need to wait from them to finish somehow, probably a wait group, maybe some channels. You need to handle errors as well and before you know it a simple target that calls 2 other targets asynchronously is full of go's bloated concurrency primitives.
I then went and created https://github.com/brad-jones/goasync and while I think this has merit in it's own right. It still didn't really help me define that nice and clear DSL I am looking for. |
|
v3 of gomake could look something like: type Task interface {
Run() error
}
type SerialTasks struct {
tasks []Task
}
func (st *SerialTasks) Run() error {
return nil
}
func Serial(tasks ...Task) Task {
return &SerialTasks{tasks}
}
type ParallelTasks struct {
tasks []Task
}
func (st *ParallelTasks) Run() error {
return nil
}
func Parallel(tasks ...Task) Task {
return &ParallelTasks{tasks}
}
// Build docs
//
// Structs with the "Run" method become cli sub commands just
// like functions currently do in the current version of gomake.
type Build struct {
// Foo docs
//
// Struct fields become cli options / flags just like
// function arguments currently do in the current
// version of gomake.
Foo string
}
// Run executes the actual task code
func (b *Build) Run() error {
fmt.Println(b.Foo)
return nil
}
type BuildAll struct{}
func (ba *BuildAll) Run() error {
// We can now construct pipelines of tasks like this
return Parallel(
&Build{Foo: "abc"},
&Build{Foo: "xyz"},
Serial(
&Build{Foo: "123"},
&Build{Foo: "456"},
),
).Run()
} |
|
But in all honesty I think I prefer https://github.com/brad-jones/drun |
|
Some more thoughts... Any task runner worth using should de-duplicate the tasks. So in this example All my task runners that I have built to date (this & For example in drun we can do something like: Future foo(String bar) => task((drun) => drun.once(() => drun.log(bar)));But if we call Other imperative based task runners like say https://magefile.org/ get away with this because their tasks don't allow input. They are strict targets like in a Makefile. But the whole reason we started building these task runners was because we ran into issues when we wanted "parametrize" a target/task. We would either have a collection of Anyway the real answer is we need to build a declarative task runner that is type safe (no magic strings) using an imperative language. AKA like the AWS CDK package main
import (
"fmt"
"github.com/brad-jones/goasync/v2/await"
"github.com/brad-jones/goasync/v2/task"
)
func deps(awaitables ...await.Awaitable) *task.Task {
return await.AllOrErrorAsync(awaitables...)
}
func build() *task.Task {
return deps(buildApi("Debug"), buildServiceHost("Debug"))
}
func buildDomain(configuration string) *task.Task {
return task.New(func(t *task.Internal) {
fmt.Println("building domain... config = ", configuration)
})
}
func buildApi(configuration string) *task.Task {
return deps(buildDomain(configuration)).Then(func(t *task.Internal) {
fmt.Println("building api... config = ", configuration)
})
}
func buildServiceHost(configuration string) *task.Task {
return deps(buildDomain(configuration)).Then(func(t *task.Internal) {
fmt.Println("building service host... config = ", configuration)
})
}
func main() {
await.All(build())
} |
|
Latest ideas, think I finally found the relatively clean yet powerful API/DSL I was looking for. You can probably see where I am going with this, the first task/job is modeled after Github Actions. The plan will be to support Github Actions and Teamcity out of the box with others to be provided by the community / as needed. The important thing is what we have here is TypeSafe and has deferred execution so the task deduplication can happen correctly. var SomeArtifact = gomake.Artifact("./foo/bar/**")
func BuildAll() *gomake.J {
return gomake.Job(func(j *gomake.J) {
j.RunsOn("ubuntu-latest")
j.Deps(BuildFoo(), BuildBar("v2"))
j.Steps(func(s *gomake.S) {
s.Action("actions/checkout@v2", nil)
s.Action("actions/cache@v2", map[string]string{
"path": "~/go/pkg/mod",
"key": "${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}",
"restore-keys": "${{ runner.os }}-go-",
})
s.GoAction(func() {})
s.Run("ping 1.1.1.1")
})
})
}
func BuildFoo() *gomake.J {
return gomake.Job(func(j *gomake.J) {
j.Deps(BuildBaz("v1.0.0"))
t.Action(func() {
fmt.Println("BuildFoo")
})
})
}
func BuildBar(baseVersion string) *gomake.J {
return gomake.Job(func(j *gomake.J) {
j.Deps(BuildBaz(baseVersion + ".0.0"))
t.Consumes(SomeArtifact)
t.Action(func() {
fmt.Println("BuildBar")
})
})
}
func BuildBaz(version string) *gomake.J {
return gomake.Job(func(j *gomake.J) {
t.Produces(SomeArtifact)
t.Action(func() {
fmt.Println("BuildBaz", version)
})
})
} |
|
An interesting observation, the way I have tended to build a task runner or rather the way I execute a task runner is from the top down whereas a pipeline tends to be from the bottom up. jobs:
buildBaz:
steps:
- run: gomake build-baz --version "???"
buildFoo:
needs:
- buildBaz
steps:
- run: gomake build-foo --no-deps
buildBar:
needs:
- buildBaz
steps:
- run: gomake build-bar --no-deps
buildAll:
needs:
- buildFoo
- buildBar
steps:
- run: gomake build-all --no-deps |
|
Locally if I execute |
|
Something more like this is what we would need to generate. jobs:
buildAllPre:
outputs:
baseVersion: ${{ steps.1.outputs.test }}
steps:
- run: echo "::set-output name=baseVersion::v2"
buildFoo:
outputs:
version: ${{ steps.1.outputs.test }}
steps:
- run: gomake build-foo --no-deps
- run: echo "::set-output name=version::v1.0.0"
buildBar:
needs:
- buildAllPre
outputs:
version: ${{ steps.0.outputs.test }}
steps:
- run: gomake build-bar --no-deps
- run: echo "::set-output name=version::${{needs.buildAll.outputs.baseVersion}}.0.0"
buildBazFoo:
needs:
- buildFoo
steps:
- run: gomake build-baz --version "${{needs.buildFoo.outputs.version}}"
buildBazBar:
needs:
- buildBar
steps:
- run: gomake build-baz --version "${{needs.buildBar.outputs.version}}"
buildAllPost:
needs:
- buildBazFoo
- buildBazBar
steps:
- run: gomake build-all --no-deps |
|
I guess the other thing is that there are 2 levels of concurrency. That provided by the pipeline running multiple jobs/agents and that provided by the task runner/operating system with goroutines/threads/processes. Perhaps we can better model this in our DSL design. |
|
Seriously I need sleep, I got all excited a few hours ago when I thought I had found my perfect DSL but it still suffers from not being able to deduplicate tasks because the task parameters are part of the closure not the struct returned from the function. We would have to something something like this to capture the arguments into the struct which just sucks. func Foo(bar string, baz int) *gomake.T {
return gomake.Task(func(t *gomake.T) {
t.Props(bar, baz)
t.Action(func() {
fmt.Println("Foo", bar, baz)
})
})
}Considering we are already doing AST parsing to generate the cobra app, I mean I guess we could do it through that way but thats fraught with issues too and would only work for static cases. The only way is the Class/struct way like earlier on in this issue or here brad-jones/drun#1 |
|
So the reason I built func Build() {
BuildGeneric("foo")
BuildGeneric("bar")
BuildGeneric("baz")
}
func BuildGeneric(someArg) {}Or even func Build() {
for _, v := range someDynamicThing() {
BuildGeneric(v)
}
}
func BuildGeneric(someArg) {}In the context of a monorepo I needed the ability to execute any of the tasks either individually or collectively as required. For example if I wanted to build the entire solution I would run The alternative (more traditional approach) being something like. func Build() {
BuildFoo()
BuildBar()
BuildBaz()
}
func BuildGeneric(someArg) {}
func BuildFoo() {
BuildGeneric("foo")
}
func BuildBar() {
BuildGeneric("bar")
}
func BuildBaz() {
BuildGeneric("baz")
}But then you can't generate tasks like in the above example with And now we are wanting to use a task runner definition to generate CI/CD pipelines, it seemed like a nice idea because that's what I have tended do in the past. Build a task runner and then build a pipeline that executes that task runner so I don't have to repeat myself, at least not as much as I would if those 2 things were completely separate. But I have discovered that the way a task runner (at least an imperative one like If we have a look at https://github.com/nektos/act you can see that the local runner looks nothing like a task runner, which came as a surprise after reading more than the first few lines of the readme. Eg: I can't call We could probably make This is exactly how https://nuke.build/ works, it's tasks/targets have no function parameters & they are able to generate CI/CD config. When you stop to think about it a pipeline essentially has static input to it's first job. There is no human there that's entering a dynamic value for some parameter. The only dynamic data is the git metadata, what branch, tags, what code changed, etc... so it makes total sense that The more I think about it the more I think that so long as we do all the nice things like build caching locally as we do in the pipeline then there should be no reason for a developer not to want to run the entire pipeline or at least "up to" a certain job/stage. It's also now so much more apparent as to why pretty much every single other task runner I have used, https://www.gnu.org/software/make/manual/make.html, https://github.com/microsoft/just, https://gulpjs.com, https://magefile.org, etc... don't have parameters for individual tasks. Some might say I have learnt that the hard way, best way to learn right :) |
While gomake by and large solves the issues I set out to solve there are some new ideas I have that I would like to explore.
.gomakefolder name in some cases (IDEs / gopls) tends to cause issues because of it's "hidden" namego.mod&makefile.goin the root of the project.mainpackage we have the user create their makefile with whatever package seems relevant. It could bemakefileorprojectfoo, ormicroservicea, etc... this has the added benefit of allowing any exported members to be used in other "makefiles" and no errors returned from IDEs whenmakefile_generated.godoes not exist..gomakefolder that does everything for an entire monorepo we have many smallermakefile.gofilesmakefile_generated.goand the built binary in the users home directory out of the way somewhereThe text was updated successfully, but these errors were encountered: