Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor image pulling to use goroutines for concurrency #1868

Merged
merged 25 commits into from
Sep 21, 2023

Conversation

corang
Copy link
Contributor

@corang corang commented Jun 30, 2023

Description

Concurrently retrieves image metadata and layers during package create.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Other (security config, docs update, etc)

Checklist before merging

@netlify
Copy link

netlify bot commented Jun 30, 2023

Deploy Preview for zarf-docs canceled.

Name Link
🔨 Latest commit 093bd63
🔍 Latest deploy log https://app.netlify.com/sites/zarf-docs/deploys/650cae96a566950008f1d43d

@corang
Copy link
Contributor Author

corang commented Jun 30, 2023

bigbang example package generation
goroutine branch run times:

real    2m35.223s
user    6m15.280s
sys     0m59.717s
---
real    2m36.357s
user    6m20.161s
sys     0m59.530s
---
real    2m35.577s
user    6m24.044s
sys     0m56.640s

0.28.0 run times:

real    3m35.299s
user    9m21.381s
sys     0m47.760s
---
real    3m34.414s
user    9m33.166s
sys     0m46.836s
---
real    3m41.898s
user    8m47.642s
sys     0m44.218s

0.28.0 skip sbom run times:

real    2m28.487s
user    2m38.983s
sys     0m12.519s
---
real    2m22.387s
user    2m37.232s
sys     0m13.272s
---
real    2m22.329s
user    2m36.669s
sys     0m12.810s

@corang
Copy link
Contributor Author

corang commented Jun 30, 2023

tldr:

  • goroutine avg 2m35.719s
  • v28 normal avg 3m37.204s
  • v28 skip avg 2m24.401s

sbom creation time is now 11s vs 1m13s on my machine with the bigbang example package

src/internal/packager/sbom/catalog.go Outdated Show resolved Hide resolved
src/internal/packager/sbom/catalog.go Outdated Show resolved Hide resolved
src/internal/packager/sbom/catalog.go Outdated Show resolved Hide resolved
@corang corang changed the title Refactor Image SBOM creation to use goroutines Refactor Image SBOM creation and image pulling to use goroutines Jul 4, 2023
@corang
Copy link
Contributor Author

corang commented Jul 4, 2023

hijacked the PR to include image pull concurrency too @Racer159 Can move it to another PR if it's not wanted here.

@corang
Copy link
Contributor Author

corang commented Jul 4, 2023

Creating the BB package on my machine with cached images is now 0m56.267s vs 3m37.204s on v0.28.0

@corang corang requested a review from Racer159 July 4, 2023 12:37
@corang
Copy link
Contributor Author

corang commented Jul 4, 2023

Don't know why ci is failing, same commands work fine on my local machine

@corang corang requested a review from cmwylie19 as a code owner August 16, 2023 22:37
@corang corang changed the title Refactor Image SBOM creation and image pulling to use goroutines Refactor image pulling to use goroutines for concurrency Aug 17, 2023
src/internal/packager/images/pull.go Outdated Show resolved Hide resolved
src/internal/packager/images/pull.go Outdated Show resolved Hide resolved
src/internal/packager/images/pull.go Outdated Show resolved Hide resolved
src/internal/packager/images/pull.go Outdated Show resolved Hide resolved
src/internal/packager/images/pull.go Outdated Show resolved Hide resolved
src/pkg/utils/concurrency.go Outdated Show resolved Hide resolved
src/pkg/utils/concurrency.go Outdated Show resolved Hide resolved
src/pkg/utils/concurrency.go Outdated Show resolved Hide resolved
src/pkg/message/message.go Outdated Show resolved Hide resolved
src/internal/packager/images/pull.go Outdated Show resolved Hide resolved
@corang corang requested a review from Racer159 September 7, 2023 02:54
src/internal/packager/images/pull.go Outdated Show resolved Hide resolved
src/pkg/utils/concurrency.go Show resolved Hide resolved
src/internal/packager/images/pull.go Outdated Show resolved Hide resolved
@corang corang requested review from Racer159 and Noxsios September 21, 2023 20:59
@Racer159 Racer159 merged commit 3c6c74b into main Sep 21, 2023
@Racer159 Racer159 deleted the sbom-gotta-go-fast branch September 21, 2023 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants