Services v2: new API, c2h and h2c networking, start/stop support, run…

… via gateway (#5557)

* h2c: reuse buf, set to 32k (same as io.Copy)

Signed-off-by: Alex Suraci <[email protected]>

* c2h: don't treat dial errors as fatal

Signed-off-by: Alex Suraci <[email protected]>

* c2h: set timeout for container release

matched 30 seconds to a timeout we set elsewhere on the same thing

Signed-off-by: Alex Suraci <[email protected]>

* Service.Clone: clone ports too

Signed-off-by: Alex Suraci <[email protected]>

* multierror -> errors.Join

Signed-off-by: Alex Suraci <[email protected]>

* h2c: cancel on server close

Signed-off-by: Alex Suraci <[email protected]>

* un-globalize AllServices

and get rid of WithServices because generic methods aren't supported and
wrappers are annoying anyway

Signed-off-by: Alex Suraci <[email protected]>

* fix premature cancel()

Signed-off-by: Alex Suraci <[email protected]>

* split Services out into a separate file, add godoc

lots of tricky behavior, worth documenting

Signed-off-by: Alex Suraci <[email protected]>

* elixir SDK: support INPUT_OBJECT

unclear why this is needed now, when we already have BuildArg

Signed-off-by: Alex Suraci <[email protected]>

* add changelog

Signed-off-by: Alex Suraci <[email protected]>

* make services no longer experimental

Couldn't find any instances on GitHub of anyone disabling this, and
there are no known open issues.

Code Search query:

    _EXPERIMENTAL_DAGGER_SERVICES_DNS AND (NOT "Currently experimental" OR "to running the Dagger Engine")

Signed-off-by: Alex Suraci <[email protected]>

* regen SDKs

Signed-off-by: Alex Suraci <[email protected]>

* disable engine-in-engine-dev for now

Signed-off-by: Alex Suraci <[email protected]>

* fix progrock deprecations/removals

Signed-off-by: Alex Suraci <[email protected]>

* testdata/nested -> testdata/nested-c2c

Signed-off-by: Alex Suraci <[email protected]>

* tests: client.Close cleanups

Signed-off-by: Alex Suraci <[email protected]>

* Zenith episode two: revenge of the function

Signed-off-by: Solomon Hykes <[email protected]>

* core: baseline core schema implementation for envs+checks.

Signed-off-by: Erik Sipsma <[email protected]>

* WIP module+function graphql schemas

Signed-off-by: Erik Sipsma <[email protected]>

* WIP modules+functions implementation

Signed-off-by: Erik Sipsma <[email protected]>

* WIP go sdk updates

Signed-off-by: Erik Sipsma <[email protected]>

* base working functions w/ go sdk

Signed-off-by: Erik Sipsma <[email protected]>

* update module test connect usage

Signed-off-by: Alex Suraci <[email protected]>

* disable depguard lint

not sure why we would want this, it fails locally on innocuous imports

Signed-off-by: Alex Suraci <[email protected]>

* disable unused param linters

These have mostly just been frustrating and unhelpful, usually resulting
in worse code than before.

Signed-off-by: Alex Suraci <[email protected]>

* fix up lints

Signed-off-by: Alex Suraci <[email protected]>

* support id-able types as inputs; extending core types

Signed-off-by: Erik Sipsma <[email protected]>

* fix support for chainable types

Signed-off-by: Erik Sipsma <[email protected]>

* ignore non-fatal compile errors

These we're causing annoying situations where you change a function
signature in user code, run `dagger mod sync` and then codegen gets a
compilation error because the old main func is incompatible with the new
signature and hasn't been updated yet.

Just ignoring these sorts of errors avoids this situation and doesn't
really cause any harm. We can still try parse the code and generate from
there. If the end result is still not compilable, then it just won't be
runnable.

Signed-off-by: Erik Sipsma <[email protected]>

* flesh out zenith intro instructions, add demo mod

zenith/ is now has an .envrc, making it suitable as a temporary
playground for writing Zenith modules

Signed-off-by: Alex Suraci <[email protected]>

* bump Go runtime to 1.21

otherwise fresh mods created with 1.21 fail on 'go 1.21.0' in go.mod

Signed-off-by: Alex Suraci <[email protected]>

* query: fix --progress=plain requirement

Signed-off-by: Alex Suraci <[email protected]>

* fix sdk/go unit tests

Signed-off-by: Alex Suraci <[email protected]>

* codegen: generate querybuilder/ package

fixes go mod replace issue

Signed-off-by: Alex Suraci <[email protected]>

* mod init: bootstrap Go module

This is a really delicate dance. For whatever reason Go doesn't like
these go.mod/go.sum files hoisted over from sdk/go/ and requires you to
run `go mod tidy`, otherwise it complains about entries missing from
go.sum, which is confusing because it works just fine for the SDK.

So the order of operations is:

* Detect if a main package exists.
* If it doesn't, write an initial main.go file before we even start
  doing codegen, otherwise the initial codegen won't have anything to
  process.
* Run code generation.
* Write initial go.mod/go.sum files with a sane default Go module name
  based on the outer module.
  * If no outer module is found, the bare Dagger module name is used and
    a warning is printed.
* Run go mod tidy.

The last step is critical; without it, dagger mod sync will error on the
same errors mentioned in the first paragraph.

Signed-off-by: Alex Suraci <[email protected]>

* support multiple passes for codegen

There's a chicken/egg problem with running codegen and generating stub
templates. If the template refers to a type like Container, the
introspection will have no idea what that refers to on the first pass.
So we need to do one pass to generate all the base types, and then
another pass to generate from the stub.

Signed-off-by: Alex Suraci <[email protected]>

* fix post-merge compilation errors

Signed-off-by: Alex Suraci <[email protected]>

* un-break cmd/client-gen

Signed-off-by: Alex Suraci <[email protected]>

* re-bump progrock for ioctl fix

Signed-off-by: Alex Suraci <[email protected]>

* respect existing .gitattributes

Signed-off-by: Alex Suraci <[email protected]>

* update core/integration for input pointer change

not completely sure if we want this, rolling with it for now

pros:

* everything is always a pointer, don't need to special-case input types
  vs regular types

cons:

* breaking change
* plain input types don't feel as intuitive as pointers

Signed-off-by: Alex Suraci <[email protected]>

* codegen: respect existing go.mod

Signed-off-by: Alex Suraci <[email protected]>

* listen: add --allow-cors

makes it easy to use off-the-shelf GraphQL inspectors

Signed-off-by: Alex Suraci <[email protected]>

* support non-local module dependencies

Signed-off-by: Alex Suraci <[email protected]>

* fix TypeDef { kind } resolving to 'null'

Signed-off-by: Alex Suraci <[email protected]>

* fix uninterruptible codegen

Signed-off-by: Alex Suraci <[email protected]>

* query: unfocus by default

Signed-off-by: Alex Suraci <[email protected]>

* bump progrock to avoid printing TUI to redirected stderr

Signed-off-by: Alex Suraci <[email protected]>

* use a single strcase package

Signed-off-by: Alex Suraci <[email protected]>

* fix module dependencies not loading

(this is thanks to a flying Erik)

Signed-off-by: Alex Suraci <[email protected]>

* Module: expose sdk, dependencies, source dir

To be used in Da Daggerverse.

Signed-off-by: Alex Suraci <[email protected]>

* dagger mod extend -> use

Signed-off-by: Alex Suraci <[email protected]>

* fix always getting description of first func

Signed-off-by: Alex Suraci <[email protected]>

* expose dependency configs

this helps daggerverse crawl a module recursively, since otherwise it
can't know how people should fetch the module

Signed-off-by: Alex Suraci <[email protected]>

* new module ref format, resolve at use time

The new format is Go module style:

  github.com/foo/bar/subpath@version

In dagger.json, the version will _always_ be a commit. The CLI accepts
any ref; it will be resolved by dagger mod use.

This format might change again, since there are gotchas that make this
hard to support in the general case (e.g. GitLab's arbitrarily-nested
repo URIs). but this is easier on the eyes. We probably also want to
stick with semver tags rather than commits. I'll figure that out next.

It would be nice if there was an easy-on-the-eyes scheme that was
totally unambiguous as to whether the provided ref is a branch, tag, or
commit, and which part is the repo and which part is the subpath.

Signed-off-by: Alex Suraci <[email protected]>

* fix uninterruptible package loading

Signed-off-by: Alex Suraci <[email protected]>

* thread ctx through to template packages.Load too

Signed-off-by: Alex Suraci <[email protected]>

* fix a couple of rogue filepath.Dirs

Signed-off-by: Alex Suraci <[email protected]>

* support distinguishing git commit from desired tag

in practice these are both just commits until semver tags are supported

Signed-off-by: Alex Suraci <[email protected]>

* fix -m with git ref

Signed-off-by: Alex Suraci <[email protected]>

* Added example to README

Signed-off-by: Vikram Vaswani <[email protected]>

* Fixed indentation in README example

Signed-off-by: Vikram Vaswani <[email protected]>

* tweak git docs

Signed-off-by: kpenfound <[email protected]>

* revise README

* avoid ./zenith/zenith
* avoid reusing Vito module name for 'create your own' flow
* add caveat for extending core types

Signed-off-by: Alex Suraci <[email protected]>

* Zenith README: chaining example

Signed-off-by: Solomon Hykes <[email protected]>

* Zenith README: fix errors

Signed-off-by: Solomon Hykes <[email protected]>

* fix: dagger mod use should override previous versions

Previously, dagger mod use would allow creating multiple versions of a
single module:

	{
	  "root": "",
	  "name": "foo",
	  "sdk": "go",
	  "dependencies": [
	    "github.com/<example>/daggerverse@aaa"
	    "github.com/<example>/daggerverse@bbb"
	  ]
	}

To prevent this, we can resolve all the dependencies, and then use the
dependency set based on only the path instead of the combination of path
and version.

Signed-off-by: Justin Chadwell <[email protected]>

* non-error returns, void returns, optional ctx

Signed-off-by: Alex Suraci <[email protected]>

* bump graphql for explicit scalar error handling

otherwise it's impossible to have a JSON scalar that evaluates to null
(which is how void return values are represented), since it gets treated
as an error

Signed-off-by: Alex Suraci <[email protected]>

* fix lints

Signed-off-by: Alex Suraci <[email protected]>

* skip non-working env tests

Signed-off-by: Alex Suraci <[email protected]>

* avoid panic if proxy env not set

likely an underlying bug here though; env should be set

Signed-off-by: Alex Suraci <[email protected]>

* fix services not having ProxyEnv

this used to be baked into the DAG as part of WithExec, but that moved
to Client.Solve, so we need to do the same thing here.

Signed-off-by: Alex Suraci <[email protected]>

* allow non-ctx first arg

Signed-off-by: Alex Suraci <[email protected]>

* sdk/go modules support Opts args

This is an alternative format to regular args that allows you to accept
optional args with default values.

There's a slight smell that defaults aren't applied for
function-to-function calls within the module. It might not be worth
having the default:"..." tag for that reason, but it's nice to have them
in the GraphQL schema.

Signed-off-by: Alex Suraci <[email protected]>

* refactor for gocyclo lint

Signed-off-by: Alex Suraci <[email protected]>

* add test for custom types

Signed-off-by: Alex Suraci <[email protected]>

* fix not including functions for custom types

this fix is a bit hacky (bool arg, whee); the long-term fix is to
refactor the schema so that modules can provide multiple Object types,
instead of relying on crawling the module's own functions

Signed-off-by: Alex Suraci <[email protected]>

* fix: prevent Directory.Without from failing on missing files

Other similar Without methods, such as `container.WithoutMount` do not
fail if the target is not present.

However, previously in BuildKit, the llb.WithAllowNotFound was never
explicitly required. Updating to the newer BuildKit correctly respects
this option, so we need to include llb.WithAllowNotFound directly.

Signed-off-by: Justin Chadwell <[email protected]>

* don't recursively load packages

attempting to peacefully coexist with tools like goenv and asdf, don't
think this ever needed to be recursive

Signed-off-by: Alex Suraci <[email protected]>

* fix: apply ToLowerCamel consistently

Signed-off-by: Justin Chadwell <[email protected]>

* fix + test using local modules

Signed-off-by: Alex Suraci <[email protected]>

* switch from input types to withFoo pattern

Previously functions and types were represented as input types, which
was a little inconvenient in two ways:

* You can't return an input type, so any type that needed to be returned
  had a duplicate 'regular' type definition.
* One of the input types (TypeDef) was self-referential, meaning we had
  to switch to pointers for input types, which was a breaking change for
  things like BuildArg (which still needs to be rolled back).

This commit changes TypeDef to use the more familiar withFoo style of
construction instead, and adds a TypeDefID so that it can be passed as
an input.

A possible downside here is that the module's entrypoint that registers
all the objects might involve multiple API calls to pass IDs around. I
haven't measured performance, but it's certainly hard to beat passing
all of the information in as one big precomputed type.

Signed-off-by: Alex Suraci <[email protected]>

* Make engine version overrideable

Signed-off-by: Solomon Hykes <[email protected]>

* dagger version also prints worker registry address

Signed-off-by: Solomon Hykes <[email protected]>

* debugging module failure

Signed-off-by: Alex Suraci <[email protected]>

* fix references to other objects

Signed-off-by: Alex Suraci <[email protected]>

* only unmarshal opts that are present

Signed-off-by: Alex Suraci <[email protected]>

* fix passing lists to functions

Signed-off-by: Alex Suraci <[email protected]>

* fix self-referential (chaining) types

Signed-off-by: Alex Suraci <[email protected]>

* un-weird container.Clone

this was causing panics to happen at a later time, not sure if/why it's
needed or leftover debugging

Signed-off-by: Alex Suraci <[email protected]>

* fixup remove logs

confirmed everything was ok

Signed-off-by: Alex Suraci <[email protected]>

* register multiple objects

Previously we would only register the main module, which only allowed
functions on types referenced and embedded in the module's own TypeDefs
to be registered.

Now we'll instead register each object separately, with shallow
references to other types embedded.

Along the way, fixed an issue that I was able to reproduce with
self-referential types:

  vito/dagger-gitutil@d919459

The function was being called with an empty ModuleID. I checked to see
if it was a regression and it seemed to be broken in a different way
before (circular reference in TypeDef somehow).

Signed-off-by: Alex Suraci <[email protected]>

* remove overly defensive guard

this was mostly for debugging

Signed-off-by: Alex Suraci <[email protected]>

* fix extending existing types

Signed-off-by: Alex Suraci <[email protected]>

* only stitch module constructor, not all types

This was never the intention, and seems to be breaking shykes/dagger.

I think we might just want to let users define explicit constructors or
something. Maybe by defining methods on *Client? (Ideally that'd be *DAG
or something instead.)

Signed-off-by: Alex Suraci <[email protected]>

* Add minimal Python support

Signed-off-by: Helder Correia <[email protected]>

* go codegen: preserve definition order, pretty-print

Signed-off-by: Alex Suraci <[email protected]>

* fix code that can't run anyway, explain why

Signed-off-by: Alex Suraci <[email protected]>

* roll back debugging code

Signed-off-by: Alex Suraci <[email protected]>

* Add git to the Python runtime

Signed-off-by: Helder Correia <[email protected]>

* mount dagger CLI to /bin/dagger when nesting

Motivated by the next commit, but seems like a worthwhile experiment in
its own right.

Signed-off-by: Alex Suraci <[email protected]>

* goruntime: run dagger mod sync before building

This way you don't need to commit all the generated code.

A better way to approach this would be to move this responsibility into
a pluggable runtime image, which technically we're doing, just in a
round-about way. Maybe runtimes can just be external images.

Signed-off-by: Alex Suraci <[email protected]>

* fix setting empty field description

Signed-off-by: Alex Suraci <[email protected]>

* remove now-superfluous mod sync commands

codegen now happens on-the-fly

Signed-off-by: Alex Suraci <[email protected]>

* add TESTFLAGS="-foo -bar ./baz/" engine:testcustom

Signed-off-by: Alex Suraci <[email protected]>

* test wrapping/unwrapping core types

Signed-off-by: Alex Suraci <[email protected]>

* only generate main.go if it doesn't already exist

Signed-off-by: Alex Suraci <[email protected]>

* fix a couple of module loading bugs

TestModuleGoUseLocal has been driving my crazy - it was failing in CI,
and in ./hack/make engine:test, but succeeding with ./hack/dev go test!

I had also previously seen it fail by allowing `dagger query` to hit
transitive dependencies, but it would sometimes to pass.

The latest flavor of failure is because (*Client).Dep is not defined in
the generated code for the Use module (which uses a dep called Dep). The
root cause turned out to be two bugs interacting:

* The schema stitching process was checking for pre-existing types by
  querying the currentSchemaView, which seemed to be the "" schema view,
  rather than checking the view that's being extended.

* The dagger CLI was calling .Serve on each dependency, which caused
  them to be stitched in to the "" schema view.

As a result, Dep's schema wasn't being fully generated into the view
because the stitching process saw that it was installed in the "" view.

The first fix is to simply stop calling Serve for dependencies in the
CLI. Dependencies are already served to the dependent module's schema
view at FromConfig time, so this isn't necessary for codegen. I may have
added this by mistake while figuring out how everything works. Not sure.

The second fix is to check against the destination schema view to
determine whether things are already defined, rather than checking
against the "" schema view. This just feels more correct in general. It
mostly worked before because the "" schema view and the destination
schema view always started from the schema of all core types, but
technically if someone loaded a module into the "" schema you'd get the
same breakage observed here.

Signed-off-by: Alex Suraci <[email protected]>

* fix lints

Signed-off-by: Alex Suraci <[email protected]>

* sdk/nodejs: handle reserve words, regen

now appends _ to any word that is reserved in JavaScript/TypeScript.
chiefly "function."

Signed-off-by: Alex Suraci <[email protected]>

* pythonruntime: fmt

Signed-off-by: Alex Suraci <[email protected]>

* draft some GraphQL API docs

Signed-off-by: Alex Suraci <[email protected]>

* regen rust

Signed-off-by: Alex Suraci <[email protected]>

* regen elixir sdk

had to manually revert the original changes first; SDK regen doesn't
remove no-longer-needed files (e.g. environment.ex)

Signed-off-by: Alex Suraci <[email protected]>

* sdk/nodejs: allow from and export without _

this was working before, so let's stick with it.

Signed-off-by: Alex Suraci <[email protected]>

* regen go, python, nodejs

Signed-off-by: Alex Suraci <[email protected]>

* regen java

Signed-off-by: Alex Suraci <[email protected]>

* Abandon black's preview style

Signed-off-by: Helder Correia <[email protected]>

* add basic test for extending core types

intentionally covers the smallest example: extending a core type, while
defining no additional types or functions of our own. previously this
did not work because only types discoverable by crawling the module's
functions would be added.

Signed-off-by: Alex Suraci <[email protected]>

* Add support for new scalars JSON, Void, ModuleID, FunctionID, TypeDefID in Java SDK

Signed-off-by: Jean-Christophe Sirot <[email protected]>

* refactor + test codegen

trying to move this into a higher-level API that SDK codegen, `dagger
mod sync`, and Runtimes can all use directly

Signed-off-by: Alex Suraci <[email protected]>

* dagger mod init: test and handle more situations

also generally clean up and consolidate logging/progrock usage

Signed-off-by: Alex Suraci <[email protected]>

* move querybuilder back under ./internal/

Signed-off-by: Alex Suraci <[email protected]>

* go codegen: .gitignore generated files

Signed-off-by: Alex Suraci <[email protected]>

* fix modules_test.go

Signed-off-by: Alex Suraci <[email protected]>

* refactor codegen order, .gitignore generated files

also remove previously-committed generated files along the way

Signed-off-by: Alex Suraci <[email protected]>

* fix loading not-generated package

Signed-off-by: Alex Suraci <[email protected]>

* move git tests to separate

Signed-off-by: Alex Suraci <[email protected]>

* clean up and regen modules

Signed-off-by: Alex Suraci <[email protected]>

* HACK: add cache buster

this codegen really needs to be put into the image

Signed-off-by: Alex Suraci <[email protected]>

* load non-module package, too

needed for sdk:go:generate

Signed-off-by: Alex Suraci <[email protected]>

* remove unused func

Signed-off-by: Alex Suraci <[email protected]>

* fix handling of root: .., support outer go.mod

Signed-off-by: Alex Suraci <[email protected]>

* generate base dagger.gen.go if it doesn't exist

otherwise you can end up stuck with code that refers to unknown types if
you deleted the file

Signed-off-by: Alex Suraci <[email protected]>

* fix tricky ordering, yay tests

Signed-off-by: Alex Suraci <[email protected]>

* disable crippling memoization

Signed-off-by: Solomon Hykes <[email protected]>

* module-ize codegen by moving it under sdk/go

Here's the plan: each SDK provides a module for running codegen against
user modules.

These SDK modules replace the runtimes that current live in core. Once
bootstrapped, they can be pushed to an image registry and invoked from
the CLI.

For the Go SDK this works out nicely because codegen already depended on
the SDK client, so now it can just live in the same module.

For now, the NodeJS codegen also made the trip into sdk/go/.
Theoretically this codegen could be split out into a separate module,
but right now the juice isn't worth the squeeze. The dagger.io/dagger
module is already oriented the right way dependency-wise.

Signed-off-by: Alex Suraci <[email protected]>

* use bootstrapped vito/dagger-sdk-go image

This was tricky: now when we're loading a module we _also_ need to load
its SDK module and call its function to build the runtime. Loading
modules and calling functions currently actually lives in core/schema/,
while constructing a module and setting up the runtime lives in core/,
so for now I've just hacked a callback function into
(core.Module).FromConfig, which lets me call back to core/schema/.

There's a specific interface that SDK modules must implement, currently
totally ad-hoc, worth bikeshedding:

  moduleRuntime(modSource: DirectoryID!, subPath: String): Container!

Signed-off-by: Alex Suraci <[email protected]>

* tidy up runtime bootstrapping

Previously this passed the _dependent_ module's source path + dir to the
SDK which was completely bogus. now it uses the SDK's. I'm starting to
suspect this didn't actually work before, and I was being fooled by
aggressive function caching, _or_ the fact that my 'with-dev' helper was
actually running the wrong CLI.

Now we use a special label to get the path to the module config. Not
totally sure about this. Committing mostly as a checkpoint.

Signed-off-by: Alex Suraci <[email protected]>

* avoid blank line at start of .gitignore

Signed-off-by: Alex Suraci <[email protected]>

* avoid generating bogus go.mod

Signed-off-by: Alex Suraci <[email protected]>

* Directory.asModule can be given a pre-built runtime

This is useful for bootstrapping SDKs (see next commit).

Signed-off-by: Alex Suraci <[email protected]>

* sdk/go: add bootstrapping for runtime image

We are now fully independent from the goruntime code currently living on
zenith-functions! See help.txt for details.

Signed-off-by: Alex Suraci <[email protected]>

* fully decouple dagger CLI from SDK codegen

* SDK runtime modules are now used for codegen, exposed as
  Module.generatedCode for ease of consumption
* Dagger CLI now hardcodes a mapping from SDK name ("go") to a pre-built
  SDK runtime image, currently pointing to a personal image ref. In
  practice we'll likely want to push these to tags that match the engine
  version.
* Dagger CLI now places the runtime image ref in dagger.json, which is
  the new source of truth for the module runtime.
* Merged core/moduleconfig/ and core/resolver/ into sdk/go/modules/.
  These are both critical for codegen, and make sense to expose in a
  stable library so that things like Daggerverse can use it.

Signed-off-by: Alex Suraci <[email protected]>

* sdk/go: fix generating empty object stub

this is actually just removing complexity that doesn't need to be around
anymore

Signed-off-by: Alex Suraci <[email protected]>

* sdk/go: build image on linux/amd64 and linux/arm64

Signed-off-by: Alex Suraci <[email protected]>

* fix constructing container from ID dropping platform

Signed-off-by: Alex Suraci <[email protected]>

* refactor sdk name => runtime flow, fix backwards compat

Signed-off-by: Alex Suraci <[email protected]>

* cache internal module calls

With SDKs as modules, this VERY quickly stacks up!

Caches the call to load the module, and the calls to the SDK runtime.
Everything else remains uncached. dagger mod sync is back to sub-0.5s.

Signed-off-by: Alex Suraci <[email protected]>

* silence that pesky version compatibility check

* Skip the entire call when connecting internally, to save time
* Log at Debug level for remaining cases where it makes sense to keep
  the check. If I'm already developing, I don't need to be warned about
  it all the time.

Signed-off-by: Alex Suraci <[email protected]>

* fix zero-ing out SDKRuntime when no SDK name set

Signed-off-by: Alex Suraci <[email protected]>

* show progress during codegen

kinda neat; directly uses progrock.sock for dagger mod sync, and just
uses local console output for on-the-fly codegen to avoid noise

Signed-off-by: Alex Suraci <[email protected]>

* add GeneratedCode type so codegen can give VCS hints

Now that codegen happens remotely, it doesn't make sense at all for it
to attempt to automate your local git repository. It also is hokey for
each SDK to concern itself with all the different kinds of VCSes. Git is
obviously dominant today, but there's no reason to make it the only
option at the SDK layer.

Instead SDKs now return a GeneratedCode type which lists file paths to
ignore and/or file paths to mark as generated. The CLI picks these up
and takes it from there.

Adding a whole new ID-able type took a bit of boilerplate, but it seems
worth it, and hopefully this pattern becomes cheaper with v2 IDs.

Signed-off-by: Alex Suraci <[email protected]>

* fix: don't use --vcs flag

not a thing anymore

Signed-off-by: Alex Suraci <[email protected]>

* Bust Function runtime cache once per-session.

Using a timestamp to bust function execution results in every call to a
function resulting in the full DAG of "internal" function calls (i.e.
calls to other functions from the implementation of one function) to be
walked along every possible path, which has exponential time complexity
and thus can cause genuine performance problems.

Now we instead use the server ID, which is randomly generated once per
session, to bust so that we still cache within the context of a single
session and thus avoid the poor time complexity.

Signed-off-by: Erik Sipsma <[email protected]>

* remove dead code

Signed-off-by: Alex Suraci <[email protected]>

* fix a couple of tests

Signed-off-by: Alex Suraci <[email protected]>

* fix gitignore path already there check

Signed-off-by: Alex Suraci <[email protected]>

* fix up go/nodejs codegen

Signed-off-by: Alex Suraci <[email protected]>

* bump sdk

Signed-off-by: Alex Suraci <[email protected]>

* regen python and java, fix python lint

Signed-off-by: Alex Suraci <[email protected]>

* Handle field parameters colliding with java keywords by prepending with a '_'.
Upgrade `TestWithDatabase` sample code with the new Service API

Signed-off-by: Jean-Christophe Sirot <[email protected]>

* fix: consistent errors for non-existent module directory

Previously, when a module was not provided for the dagger query command
using the `-m` flag, the error message did not indicate that the module
did not exist even when it was requested by the user.

This patch reworks the logic - dagger query will now always error if the
module provided by `-m` did not exist, but will keep the desired
behaviour that it should not error if the current directory is not a
module.

Signed-off-by: Justin Chadwell <[email protected]>

* fix: dagger mod init should require sdk and name args

We were previously calling the wrong function to do this.

Signed-off-by: Justin Chadwell <[email protected]>

* Add `dagger call` command

This is very WIP, just publishing so others can hack on it. Only supports "top-level" (from type that matches current module) functions without arguments and only tested functions that return a string.

Added `ParentName` to `core.Function` to fix a direct `dagger.Function.Call`. It used to work but is failing due to recent changes while unmarshaling input. However, since we plan to make a chained call, it's better to build a query instead of calling the functions directly.

See TODO list at the top for follow-ups.

Signed-off-by: Helder Correia <[email protected]>

* Small linting fix

Signed-off-by: Helder Correia <[email protected]>

* backport dagger shell

Signed-off-by: Erik Sipsma <[email protected]>

* chore: dagger shell branch: fix conflicts

Signed-off-by: grouville <[email protected]>

* fix: mount the correct dependency sub-folder

Without this, we can end up caching the wrong top-level directory.

Signed-off-by: Justin Chadwell <[email protected]>

* cleanup egregious stuff from dagger shell impl

Signed-off-by: Erik Sipsma <[email protected]>

* fix go sdk lints

Signed-off-by: Erik Sipsma <[email protected]>

* appease engine lints

Signed-off-by: Erik Sipsma <[email protected]>

* fix missed engine lint

Signed-off-by: Erik Sipsma <[email protected]>

* upgrade go+golangci to see if it fixes bizarre lint fail

Signed-off-by: Erik Sipsma <[email protected]>

* fix gitignore of sdk/go/runtime

Signed-off-by: Erik Sipsma <[email protected]>

* fix + test using multiple modules

two bug fixes:

1. dagger mod use incorrectly deduped on path alone, instead of also
   considering the subpath
1. a fun mutation bug in the schema was causing all sorts of fun module
   identity crises

Signed-off-by: Alex Suraci <[email protected]>

* fix go sdk tests

Signed-off-by: Erik Sipsma <[email protected]>

* fix elixer sdk lints

Signed-off-by: Erik Sipsma <[email protected]>

* appease residual lints

Signed-off-by: Erik Sipsma <[email protected]>

* upgrade sphinx-rtd-theme to see if it fixes read-the-docs build

Signed-off-by: Erik Sipsma <[email protected]>

* use session: prefix for blobsource

Signed-off-by: Erik Sipsma <[email protected]>

* add explicit requirements.txt for python docs

As of a few days ago, readthedocs requires you to explicitly set
dependencies in a requirements.txt:
https://blog.readthedocs.com/defaulting-latest-build-tools/

Signed-off-by: Erik Sipsma <[email protected]>

* stop pointerizing input types

this was needed to avoid dependency cycles in self-referential input
types, but we don't have those anymore, in favor of just always using
objects and passing them in by their ID.

Signed-off-by: Alex Suraci <[email protected]>

* use a hyper specific tag

just trying things on

Signed-off-by: Alex Suraci <[email protected]>

* regen go sdk

Signed-off-by: Alex Suraci <[email protected]>

* regen nodejs sdk

Signed-off-by: Alex Suraci <[email protected]>

* fix up tests (botched merge somewhere)

Signed-off-by: Alex Suraci <[email protected]>

* updated services docs for pr-5557

Signed-off-by: Jenn Allen <[email protected]>

* apply docs suggestions

thanks!

Co-authored-by: vikram-dagger <[email protected]>
Signed-off-by: Alex Suraci <[email protected]>

* Updated services guides

Signed-off-by: Vikram Vaswani <[email protected]>

* Lint fix

Signed-off-by: Vikram Vaswani <[email protected]>

* Update docs/current/guides/snippets/build-test-publish-php/dagger.php

Co-authored-by: Alex Suraci <[email protected]>
Signed-off-by: vikram-dagger <[email protected]>

* Update docs/current/guides/757394-use-service-containers.md

Co-authored-by: Alex Suraci <[email protected]>
Signed-off-by: vikram-dagger <[email protected]>

* services: adopt new IDable patterns

Signed-off-by: Alex Suraci <[email protected]>

* regen SDKs

Signed-off-by: Alex Suraci <[email protected]>

* bump go sdk

Signed-off-by: Alex Suraci <[email protected]>

* fix newly added test

Signed-off-by: Alex Suraci <[email protected]>

* Updated examples and content

Signed-off-by: Vikram Vaswani <[email protected]>

* Lint fix

Signed-off-by: Vikram Vaswani <[email protected]>

* Lint fix

Signed-off-by: Vikram Vaswani <[email protected]>

* Updated filename and paths

Signed-off-by: Vikram Vaswani <[email protected]>

* Copy edit

Signed-off-by: Vikram Vaswani <[email protected]>

* Code update

Signed-off-by: Vikram Vaswani <[email protected]>

* docs tweaks

* compromise on Foo.barBaz syntax for referring to APIs
* fix what I think was a sentence ordering glitch ('select first port'
  applies to Endpoint specifically)
* turn Host.unixSocket tip into a guidepost instead of pointing out
  internal implementation details

Signed-off-by: Alex Suraci <[email protected]>

* Container.service -> Container.asService

Signed-off-by: Alex Suraci <[email protected]>

* make python lint happy

Signed-off-by: Alex Suraci <[email protected]>

* fix awkward gofmt formatting

Signed-off-by: Alex Suraci <[email protected]>

* Added code samples

Signed-off-by: Vikram Vaswani <[email protected]>

* Lint fixes

Signed-off-by: Vikram Vaswani <[email protected]>

* comment out incorrect code in example

Signed-off-by: Vikram Vaswani <[email protected]>

* Lint fix

Signed-off-by: Vikram Vaswani <[email protected]>

* Lint fix

Signed-off-by: Vikram Vaswani <[email protected]>

* Create guide expose-host-services-to-container for the node SDK

Signed-off-by: jffarge <[email protected]>

* Create a Node.js SDK guide for start-stop-services

Signed-off-by: jffarge <[email protected]>

* Remove unused fetch library

Signed-off-by: jffarge <[email protected]>

* rename main.go to main_test.go for to test

Signed-off-by: Alex Suraci <[email protected]>

* Added Python snippets

Signed-off-by: Vikram Vaswani <[email protected]>

* decouple core from internal/mage

this doesn't work when there are breaking changes (noted in two
comments)

Signed-off-by: Alex Suraci <[email protected]>

* regen SDKs

Signed-off-by: Alex Suraci <[email protected]>

* Revert "decouple core from internal/mage"

This reverts commit 1807036ff2c4a71e0878eecaed2c6497b68de2b1.

Signed-off-by: Alex Suraci <[email protected]>

* internal/mage: move consts into separate package

Signed-off-by: Alex Suraci <[email protected]>

* sdk/go: fix go.mod

Signed-off-by: Alex Suraci <[email protected]>

* undo main_test.go rename

misinterpreted the failure, and this actually causes ci failure

Signed-off-by: Alex Suraci <[email protected]>

* tidy

Signed-off-by: Alex Suraci <[email protected]>

* fix docs link

Signed-off-by: Alex Suraci <[email protected]>

* add a couple of awaits

Signed-off-by: Alex Suraci <[email protected]>

* services docs: v0.8.8 -> v0.9.0

Signed-off-by: Alex Suraci <[email protected]>

---------

Signed-off-by: Alex Suraci <[email protected]>
Signed-off-by: Solomon Hykes <[email protected]>
Signed-off-by: Erik Sipsma <[email protected]>
Signed-off-by: Alex Suraci <[email protected]>
Signed-off-by: Vikram Vaswani <[email protected]>
Signed-off-by: kpenfound <[email protected]>
Signed-off-by: Justin Chadwell <[email protected]>
Signed-off-by: Helder Correia <[email protected]>
Signed-off-by: Jean-Christophe Sirot <[email protected]>
Signed-off-by: grouville <[email protected]>
Signed-off-by: Jenn Allen <[email protected]>
Signed-off-by: vikram-dagger <[email protected]>
Signed-off-by: jffarge <[email protected]>
Co-authored-by: Solomon Hykes <[email protected]>
Co-authored-by: Erik Sipsma <[email protected]>
Co-authored-by: Vikram Vaswani <[email protected]>
Co-authored-by: kpenfound <[email protected]>
Co-authored-by: Justin Chadwell <[email protected]>
Co-authored-by: Solomon Hykes <[email protected]>
Co-authored-by: Helder Correia <[email protected]>
Co-authored-by: Jean-Christophe Sirot <[email protected]>
Co-authored-by: grouville <[email protected]>
Co-authored-by: Jenn Allen <[email protected]>
Co-authored-by: vikram-dagger <[email protected]>
Co-authored-by: jffarge <[email protected]>