How can we improve dependency compilation?

[sophiewigmore]

Context

The dependencies we ship in the dep-server are for the most part compiled from source. This is to provide a more thorough chain of custody for our dependencies, and also compiling against the stack they will ultimately run provides us with an extra guarantee of compatibility. The dependency compilation step of our automation reaches out to compilation code from the previous iteration of buildpacks (Cloud Foundry buildpacks) located at:

• https://github.com/cloudfoundry/buildpacks-ci/blob/master/tasks/build-binary-new/builder.rb
• https://github.com/cloudfoundry/binary-builder.

This code is fairly hard to maintain due to its complexity, being located in multiple repositories, and being written in Ruby (the Paketo codebase is largely written in Golang). Previous refactoring efforts were challenging, leading to the second repository (linked above) that contains some of the compilation code, but not all.

We should do a complete overhaul of our compilation code to make it more modular, maintainable, and well-documented.

Desired Outcome:

This issue is meant to surface the need for a new compilation codebase, as well as enumerate items a new implementation should include. The intended outcome of this issue has two parts:

1. Come to a consensus about an architecture that makes sense for a new dependency compilation system. It should be summarized in a comment on this issue, or in a in a new issue.
2. Create a batch of issues to track the work involved in the implementation.

The new system we decide on should have the following features:

• Minimal code duplication across dependencies (factor out common functionality where possible.)
• The code should be easily extendable to other operating systems (like ARM64 to enable Roadmap 2023: ARM64 Support feedback#23)
• The code should be easily extendable when adding new dependencies.
• Documentation in the form of comments (or something else) outlining decisions for compiling dependencies with certain flags

Open Questions:

• Are there other projects or tools that accomplish a similar task? If so, how do they manage dependency compilation?
• Should it live in the dep-server, or somewhere else?
• What parts of the previous implementation do we want to keep?