Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make pipeline faster #104

Closed
jlebon opened this issue Jul 16, 2019 · 2 comments
Closed

Make pipeline faster #104

jlebon opened this issue Jul 16, 2019 · 2 comments

Comments

@jlebon
Copy link
Member

jlebon commented Jul 16, 2019

It takes about 1h right now for a full build. In the past few days, we've had to run and iterate the pipeline multiple times, and waiting for that 1h has been really painful at times. I think there are multiple ways we can speed this up:

  1. first and foremost, while not related to a single build, we need to unblock concurrent builds (Allow concurrent builds #68)
  2. by far the longest step is xz compression. Fortunately, xz supports -T for multi-threading. But doing this correctly for a k8s environment is tricky. See e.g. virt-install: do not assign vpu's when executed in Kubernetes coreos-assembler#284 (comment) and virt-install: do not assign vpu's when executed in Kubernetes coreos-assembler#284 (comment).
  3. parallelize buildextend commands -- this will likely require driving some changes into cosa. But at least for the ones that just do gf-platformid it shouldn't be too hard. Of course, this goes right into YAML/JSON interface for COSA coreos-assembler#531; cosa itself could potentially include the parallelization code.
@jlebon
Copy link
Member Author

jlebon commented Jul 18, 2019

by far the longest step is xz compression. Fortunately, xz supports -T for multi-threading.

coreos/coreos-assembler#632

@jlebon
Copy link
Member Author

jlebon commented Jun 20, 2022

1 and 2 and 3 are done now. And more parallelization work is happening.

@jlebon jlebon closed this as completed Jun 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant