Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

-j2 option breaking builds #2042

Closed
matt2e opened this issue Jul 11, 2024 · 0 comments · Fixed by #2051
Closed

-j2 option breaking builds #2042

matt2e opened this issue Jul 11, 2024 · 0 comments · Fixed by #2051
Assignees
@wesbillman

Comments

@matt2e
Copy link
Collaborator

matt2e commented Jul 11, 2024 •
edited
Loading

From our partner project

Error message:

error: initial deploy failed: failed to build module "[mod3]": err: exit status 1: stderr: go: cannot load module ../../../.ftl/go/modules/[mod1] listed in go.work file: open ../../../.ftl/go/modules/[mod1]/go.mod: no such file or directory
go: cannot load module ../../../.ftl/go/modules/[mod2] listed in go.work file: open ../../../.ftl/go/modules/[mod2]/go.mod: no such file or directory

failed to build module "offerings": err: exit status 1: stderr: go: cannot load module ../../../.ftl/go/modules/[mod2] listed in go.work file: open ../../../.ftl/go/modules/[mod2]/go.mod: no such file or directory
go: cannot load module ../../../.ftl/go/modules/[mod1] listed in go.work file: open ../../../.ftl/go/modules/[mod1]/go.mod: no such file or directory

^Cerror:controller0: Failed to receive notification: context canceled
@github-actions github-actions bot added the triage Issue needs triaging label Jul 11, 2024
@ftl-robot ftl-robot mentioned this issue Jul 11, 2024
Open
@wesbillman wesbillman self-assigned this Jul 11, 2024
@github-actions github-actions bot removed the triage Issue needs triaging label Jul 11, 2024
@wesbillman wesbillman mentioned this issue Jul 11, 2024
Merged
wesbillman added a commit that referenced this issue Jul 11, 2024
@wesbillman
fix: building withing lower parallelism than dependency group size (#…
d16ef79 
…2051)

Fixes #2042

Before this change, we would get build errors because the `go.work` file
for a given module would include all the built modules including modules
that were in the same build `group` layer as a given module.

For example, if we had 4 modules all in the same group `[a, b, c, d]`
and we build with a `-j2` option (2 parallel builds), we would run into
an issue where `a` and `b` would build first, in parallel. `a` would
finish and we would start to build `c`. But, when we looked up the
`schema` to get shared modules for the `go.work` file for `c`, that list
would now include `a` even though they are in the same build group and
should never depend on each other.

This fix changes the build func to not re-lookup built modules and
instead just use the set of `builtModules` that were available at the
start of the build group.

With this change we should be able to safely support any `-j` option
higher or lower than the number of modules. This probably would have
eventually happened either way once we had a system with more modules
than the number of cpus on a given machine.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wesbillman wesbillman

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: building withing lower parallelism than dependency group size block/ftl
2 participants
@wesbillman @matt2e