Support multiple architectures for extensions

[viniciusd]

tl;dr

trunk doesn't currently support multi-arch extensions

Context

As I was trying to run pgmq locally on my M1 MacBook pro, I noticed it wasn't working. By debugging, I found the culprit: pg_partman_bgw wasn't successfully installed.

By trying to manually install it using trunk (trunk install pg_partman), I got the error message:
This package is not compatible with your architecture: aarch64, it is compatible with x86_64

Then I started reverse-engineering trunk itself, and I noticed trunk itself doesn't support multi-arch, it assumes an extension artifact is single-arch. Indeed, architecture checking was added a few months ago via 1ce91d9.

Issue

Multi-arch support is paramount for production, but also for development, given the wider use of ARM with Apple in that direction.

I acknowledge #104 to support architecture as part of the artifacts. Similarly, there is #105 for postgres versions, but #106 discusses support for multiple versions. It doesn't look like multi-arch has been discussed.

Propposed fix

From what I gathered, the right way of doing so is changing the manifest so the manifest format itself supports multiple architectures.

Required changes

I acknowledge the architecture dependency of some files. For example, for pg_partman, there is the shared object pg_partman_bgw.so.

The manifest file should be rethought in an architecture-aware manner.

A few points that will need change:

1. The manifest description itself
2. The internal representation of a manifest, which current uses a String for holding the architecture
3. Manifest parsing for package installation should support multiple architectures

How others fix it

Enabling multi-arch artifacts isn't new, Docker manifests already do so. projectcalico/calico#4227 discusses changes required to get multi-arch on quay.io (we will also need this in future because of pgmq). According to the examples there, the Docker manifest allows a list of per-architecture manifests.

How we should fix it

Making architecture a list would be the first call to avoid repetition of dependencies, but some dependencies are architecture-dependent. Maybe including architecture-dependent files within files or a new section of, well, architecture-dependent files to the main json?

Overall, it brings the question of how manifest changes should be dealt with into the future, and I don't know how [or if] tembo.io has decided to do so. This type of change isn't backward-compatible, and introducing extra if manifest version blocks into the project doesn't sound ideal either.

[sjmiller609]

I had originally thought that building on ARM64 would suffice for making "trunk install" work on Mac chips. However, actually brew installed postgresql is different than linux arm64 postgresql, and I think it actually has to be built on Mac itself, not only Linux ARM64 then use that for Mac.

The way that trunk build works, it basically does "docker build" to build your extension, running on top of one of the trunk builder images, which are prepared postgresql extensions build environments. After doing "docker build", then it starts the container and performs the installation command. There is a docker feature "docker diff" that is then used to find files that have changed. Trunk build then finds the files that have changed and zips them into a tar ball. So, if we can find a way to build on macOS arm64-like docker container, then we can make a new builder image for supporting mac.

Here is what I tried so far:

FROM homebrew/brew:latest # this is an image I found on darwin/arm64

RUN brew install postgresql git

ARG RELEASE=v1.5.2

# Clone repository
RUN git clone https://github.com/citusdata/pg_cron.git

RUN cd pg_cron && \
  git fetch origin ${RELEASE} && \
  git checkout ${RELEASE} && \
  make

docker build -t test --platform darwin/arm64 .

but then I got this error, and I haven't yet looked further.

------
 > exporting to image:
------
operating system is not supported

Next, I would try to see if I could get the correct version of docker or maybe I need to turn on some setting inside docker? But this seems more complicated than I expected it to be. I expected compiled for linux/arm64 to just work on mac but that is incorrect.

[viniciusd]

Thanks for your input! Correct me if I am wrong, but I believe we are tackling it from different yet complementary ends.

You are trying to use trunk install directly from a Mac machine, right? I am trying to run it from within a container, it will still run on ARM, but it will run on Docker's Linux VM.

The steps I am doing are:

$ git clone [email protected]:tembo-io/pgmq.git
$ cd pgmq/pgmq-pg
$ docker build -t tembo-pgmq .
$ docker run --rm -it tembo-pgmq bash
$ trunk install pg_partman # From within docker

With muti-arch support, I would expect docker run -d --name postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 quay.io/tembo/pgmq-pg:latest to just work.

You, on the other hand, are building the .tar.gz artifact on a Linux ARM machine and trying to install it with trunk install -f. Is that correct? If so, I assume I can reproduce it with trunk build inside Docker (might need Docker-in-Docker then?), copying the artifact to my host OS X and running trunk install -f locally.

Running trunk directly on my machine, avoiding Docker altogether, would also be interesting, but tackling multi-arch already enables development/usage atop Docker. However, considering linux-built artifacts would not be compatible with Mac, we might want to consider the tuple architecture, system as the underlying supporting platform. I wonder if WebAssembly can help us here with getting platform-independent extensions.

[sjmiller609]

@viniciusd Yes, I think you have it exactly right.

You are trying to use trunk install directly from a Mac machine, right?

Yes exactly

I am trying to run it from within a container, it will still run on ARM, but it will run on Docker's Linux VM.

I think it should work to build then install with trunk install -f an extension on a normal linux ARM environment, if built with the trunk command line option --platform linux/arm64, then install the resulting trunk package with trunk install -f ...

With muti-arch support, I would expect docker run -d --name postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 quay.io/tembo/pgmq-pg:latest to just work.

I think Trunk is capable to support this use case today, but the extensions are not published to Trunk with ARM because Trunk has only built and uploaded AMD64 in CI. Trunk does have the "builder images" published on both architectures, but does not have any of the extensions published for ARM. For example, try this:

git clone https://github.com/tembo-io/pgmq.git
cd pgmq
trunk build --platform linux/arm64

[viniciusd]

Thanks! Then my initial discussion makes sense: I believe this issue is more about whether/how to support multiple architectures.

You mentioned ARM extensions aren't uploaded, but the manifest itself is single-arch, and trunk also considers artifacts as single-arch (as per install.rs).

Considering the repository page, I would assume architecture would become a breakdown of an extension. I.e., a single extension could support multiple architectures. This links to my Required changes section, but it would also require a 0. Upload multi-arch extensions

[viniciusd]

I think Trunk is capable to support this use case today, but the extensions are not published to Trunk with ARM because Trunk has only built and uploaded AMD64 in CI. Trunk does have the "builder images" published on both architectures, but does not have any of the extensions published for ARM. For example, try this:

git clone https://github.com/tembo-io/pgmq.git
cd pgmq
trunk build --platform linux/arm64

I tried it in tembo-io/pgmq#30, but I could not get it compiling without a further dive into the trunk build process