Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add common build environment #14

Merged
merged 1 commit into from
Nov 22, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/php
26 changes: 26 additions & 0 deletions .github/workflows/build-php.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: Build PHP
on:
push:
# By specifying branches explicitly, we avoid this workflow from
# running on tag push. We have a dedicated workflow to be ran when
# a tag is pushed.
branches:
- "*"
pull_request:
jobs:
build-php:
strategy:
fail-fast: false
matrix:
version: [7.3.33, 7.4.32]
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
- name: Build PHP
run: make php/php-${{ matrix.version }}
- name: Upload php-cgi-${{ matrix.version }}.wasm artifact
uses: actions/upload-artifact@v3
with:
name: php-cgi-${{ matrix.version }}.wasm
path: php/build-output/php/php-${{ matrix.version }}/bin/php-cgi
80 changes: 80 additions & 0 deletions .github/workflows/release-php.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
name: Release PHP
on:
push:
tags:
- php/*
jobs:
release-php:
strategy:
matrix:
version: [7.3.33, 7.4.32]
runs-on: ubuntu-latest
env:
BINARYEN_VERSION: 110
steps:
- name: Checkout repository
# Only run for the PHP version specified in the git tag.
#
# This if could be moved to the parent `job` section when it's
# supported by GitHub (https://github.com/community/community/discussions/37883)
if: github.event.ref == format('refs/tags/php/{0}', matrix.version)
uses: actions/checkout@v3
- name: Build PHP
# Only run for the PHP version specified in the git tag.
#
# This if could be moved to the parent `job` section when it's
# supported by GitHub (https://github.com/community/community/discussions/37883)
if: github.event.ref == format('refs/tags/php/{0}', matrix.version)
run: make php/php-${{ matrix.version }}
- name: Rename release artifacts
# Only run for the PHP version specified in the git tag.
#
# This if could be moved to the parent `job` section when it's
# supported by GitHub (https://github.com/community/community/discussions/37883)
if: github.event.ref == format('refs/tags/php/{0}', matrix.version)
shell: bash
run: |
sudo mv php/build-output/php/php-${{ matrix.version }}/bin/php-cgi{,.wasm}
- name: Setup binaryen
# Only run for the PHP version specified in the git tag.
#
# This if could be moved to the parent `job` section when it's
# supported by GitHub (https://github.com/community/community/discussions/37883)
if: github.event.ref == format('refs/tags/php/{0}', matrix.version)
shell: bash
run: |
wget https://github.com/WebAssembly/binaryen/releases/download/version_${{ env.BINARYEN_VERSION }}/binaryen-version_${{ env.BINARYEN_VERSION }}-x86_64-linux.tar.gz
tar -xf binaryen-version_${{ env.BINARYEN_VERSION }}-x86_64-linux.tar.gz --strip-components=1 -C /opt
rm binaryen-version_${{ env.BINARYEN_VERSION }}-x86_64-linux.tar.gz
- name: Optimize release artifacts
# Only run for the PHP version specified in the git tag.
#
# This if could be moved to the parent `job` section when it's
# supported by GitHub (https://github.com/community/community/discussions/37883)
if: github.event.ref == format('refs/tags/php/{0}', matrix.version)
shell: bash
run: |
sudo /opt/bin/wasm-opt -Os -o php/build-output/php/php-${{ matrix.version }}/bin/php-cgi.size-optimized.wasm php/build-output/php/php-${{ matrix.version }}/bin/php-cgi.wasm
sudo /opt/bin/wasm-opt -O -o php/build-output/php/php-${{ matrix.version }}/bin/php-cgi.speed-optimized.wasm php/build-output/php/php-${{ matrix.version }}/bin/php-cgi.wasm
- name: Append version to release artifacts
# Only run for the PHP version specified in the git tag.
#
# This if could be moved to the parent `job` section when it's
# supported by GitHub (https://github.com/community/community/discussions/37883)
if: github.event.ref == format('refs/tags/php/{0}', matrix.version)
run: |
sudo mv php/build-output/php/php-${{ matrix.version }}/bin/php-cgi{,-${{ matrix.version }}}.wasm
sudo mv php/build-output/php/php-${{ matrix.version }}/bin/php-cgi{,-${{ matrix.version }}}.size-optimized.wasm
sudo mv php/build-output/php/php-${{ matrix.version }}/bin/php-cgi{,-${{ matrix.version }}}.speed-optimized.wasm
- name: Release
# Only run for the PHP version specified in the git tag.
#
# This if could be moved to the parent `job` section when it's
# supported by GitHub (https://github.com/community/community/discussions/37883)
if: github.event.ref == format('refs/tags/php/{0}', matrix.version)
uses: softprops/action-gh-release@v1
with:
files: |
php/build-output/php/php-${{ matrix.version }}/bin/php-cgi-${{ matrix.version }}.wasm
php/build-output/php/php-${{ matrix.version }}/bin/php-cgi-${{ matrix.version }}.size-optimized.wasm
php/build-output/php/php-${{ matrix.version }}/bin/php-cgi-${{ matrix.version }}.speed-optimized.wasm
6 changes: 3 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Ignore everything in this directory
build-output/**
build-staging/**
# Ignore everything in these directories
build-output
build-staging
.vscode/**
*.log
# Except this file
Expand Down
19 changes: 19 additions & 0 deletions Dockerfile.wasi-builder
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
FROM ubuntu:latest@sha256:4b1d0c4a2d2aaf63b37111f34eb9fa89fa1bf53dd6e4ca954d47caebca4005c2
Angelmmiguel marked this conversation as resolved.
Show resolved Hide resolved
ARG WASI_SDK_VERSION=16
ENV WASI_SDK=wasi-sdk-${WASI_SDK_VERSION}
ENV WASI_SDK_ROOT=/wasi-sdk
RUN apt update && \
DEBIAN_FRONTEND=noninteractive apt install -y \
autoconf \
automake \
build-essential \
clang \
git \
pkg-config \
wget
RUN wget https://github.com/WebAssembly/wasi-sdk/releases/download/${WASI_SDK}/${WASI_SDK}.0-linux.tar.gz && \
mkdir /wasi-sdk && \
tar xf ${WASI_SDK}.0-linux.tar.gz --strip-components=1 -C ${WASI_SDK_ROOT} && \
rm ${WASI_SDK}.0-linux.tar.gz
ADD . /wlr
WORKDIR /wlr
9 changes: 9 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# This Makefile contains the main targets for all supported language runtimes

.PHONY: php/php-*
php/php-*:
make -C php $(subst php/php-,php-,$@)

.PHONY: clean
clean:
make -C php clean
5 changes: 5 additions & 0 deletions Makefile.builders
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
BUILDER_ROOT_DIR := $(dir $(realpath $(lastword $(MAKEFILE_LIST))))

.PHONY: wasi-builder-16
wasi-builder-16:
docker build --build-arg WASI_SDK_VERSION=16 -f ${BUILDER_ROOT_DIR}/Dockerfile.wasi-builder -t ghcr.io/vmware-labs/wasi-builder:16 ${BUILDER_ROOT_DIR}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be tagged in projects.registry.vmware.com/wasmlabs/containers instead of ghcr.io ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized with this work that we can publish on ghcr.io already (https://github.com/orgs/vmware-labs/packages/container/package/wasi-builder) and (https://github.com/orgs/vmware-labs/packages/container/package/php-builder).

I think we might want to discuss it. If we are going to work with GitHub workflows and automate stuff on GitHub I think it might be more convenient to deploy assets on the GitHub registry, given it makes authentication & authorization easier and more integrated.

48 changes: 27 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,38 +2,31 @@

## Overview

This repository contains patches provided for language runtimes to be
compiled for the wasm32-wasi target.
This repository contains patches provided for language runtimes to be compiled for the wasm32-wasi target.

## Getting started

To run this builds for WASM/WASI you will need to install [WebAssembly/wasi-sdk](https://github.com/WebAssembly/wasi-sdk) in advance. Then export the location as `WASI_SDK_ROOT`.
All you need in order to run these builds is to have `docker` or `podman` available in your system. You can execute the following `Makefile` targets:

First ensure you have the build tools required to build the respective language runtime or standalone library. These could be `autoconf`, `libtool`, `make`, etc.
- `php/php-7.3.33`
- Resulting binaries are placed in `php/build-output/php/php-7.3.33/bin`.

For the current version of our build orchestration scripts we use `bash`
- `php/php-7.4.32`
- Resulting binaries are placed in `php/build-output/php/php-7.4.32/bin`.

Just call the `wl-make.sh` script for the folder with the tagged version of whatever you want to build
### Build strategy

```console
./wl-make.sh php/php-7.4.32
```

or

```console
./wl-make.sh libs/sqlite/version-3.39.2
```
If you are interested in knowing more about the build system and how it produces the final binaries, keep reading.

## Code Organization
### Code Organization

All build orchestration scripts are written in bash in this initial version. The start with a `wl-` prefix (short for WasmLabs). Review the [build orchestration scripts](#build-orchestration-scripts) section for more info.

All intermediary source code checkouts and build objects get created within the `build-staging` folder. The final output gets written to the `build-output` folder.
All intermediary source code checkouts and build objects get created within the `build-staging` folder. The final output gets written to the `build-output` folder.

The patches and scripts to build different language runtimes are organized in a folder hierarchy that follows the tagged versions from the respective source code repositories. Several `wl-` scripts are added around that to facilitate setup of a local clone of the repository, application of respective patches and building with respective build configuration options.

For language runtimes we have something like this.
For language runtimes we have something like this.

```
$LANGUAGE_RUNTIME_NAME (e.g. php)
Expand Down Expand Up @@ -62,7 +55,7 @@ libs (common libraries, needed by different modules)
└── wl-env-repo.sh (script that sets up the source code repository for given langauge and tag)
```

## Build orchestration scripts
### Build orchestration scripts

1. The main script used to build something is `wl-make.sh` in the root folder. It gets called with a path to the folder for a respective tag of what we want to build

Expand All @@ -74,7 +67,7 @@ libs (common libraries, needed by different modules)

5. Before building this will call a `$LANG/$TAG/wl-build-deps.sh` if there is any to build required dependencies and setup CFLAGS or LDFLAGS for their artifacts. Then it will call the `$LANG/$TAG/wl-build.sh` script to build the actual target itself.

## Adding a new build target
### Adding a new build target

To add a build setup for a new version of something that is already configured:

Expand All @@ -92,7 +85,7 @@ touch php/php-7.3.33/wl-build.sh

3. Setup your build environment via `scripts/wl-env.sh` with the target path then query the respective environment variables, like this:

```console
```console
source scripts/wl-env.sh php/php-7.3.33
export | grep WASMLABS_
```
Expand Down Expand Up @@ -140,3 +133,16 @@ scripts/wl-update-patches.sh
git add php/php-7.3.33
git commit -m "Add support to build php version 7.3.33"
```

## Performing a release

In order to perform a release, push a tag of the following form
depending on the project artifacts you want to be built and published:

- `php/version`, where version can be any of:
- `7.3.33`
- `7.4.32`

When the tag is pushed to the repository, a GitHub release will be
created automatically, and relevant artifacts will be automatically
published to the release.
19 changes: 0 additions & 19 deletions libs/sqlite/README.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,3 @@
# About

Build scripts and patches for the sqlite3 library.
# Prerequisites

1. All build operations rely on WASISDK. You could get it from here - https://github.com/WebAssembly/wasi-sdk

2. The sqlite build uses autoconf make and libtool. On a ubuntu machine you may need to do

```console
sudo apt update && sudo apt install autoconf make libtool-bin -y
```

3. Before building define WASI_SDK_ROOT to point to a local installation of WasiSDK. For example

```console
export WASI_SDK_ROOT=/opt/wasi-sdk
```

# References

The patch and approach have been greatly influenced by the changes in [rcarmo/wasi-sqlite](https://github.com/rcarmo/wasi-sqlite)
2 changes: 2 additions & 0 deletions php/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
build-output
build-staging
8 changes: 8 additions & 0 deletions php/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
ARG WASI_SDK_VERSION=16
FROM ghcr.io/vmware-labs/wasi-builder:${WASI_SDK_VERSION}
RUN DEBIAN_FRONTEND=noninteractive apt install -y \
bison \
re2c \
libsqlite3-dev \
tcl
ADD . /wlr/php
18 changes: 18 additions & 0 deletions php/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
WASI_SDK_VERSION ?= 16

ROOT_DIR := $(dir $(realpath $(lastword $(MAKEFILE_LIST))))

include ../Makefile.builders

.PHONY: php-builder
php-builder: wasi-builder-16
docker build -f ${ROOT_DIR}/Dockerfile --build-arg WASI_SDK_VERSION=$(WASI_SDK_VERSION) -t ghcr.io/vmware-labs/php-builder:wasi-$(WASI_SDK_VERSION) ${ROOT_DIR}

.PHONY: php-*
php-*: php-builder
mkdir -p build-output build-staging
docker run --rm -v ${ROOT_DIR}/build-output:/wlr/build-output -v ${ROOT_DIR}/build-staging:/wlr/build-staging ghcr.io/vmware-labs/php-builder:wasi-${WASI_SDK_VERSION} ./wl-make.sh php/$@

.PHONY: clean
clean:
rm -rf build-output build-staging
43 changes: 5 additions & 38 deletions php/README.md
Original file line number Diff line number Diff line change
@@ -1,46 +1,13 @@
# About

Basic instructions on how to build different php versions.
## Building

# Prerequisites
You can build PHP by running the following Makefile targets:

1. All build operations rely on WASISDK. You could get it from here - https://github.com/WebAssembly/wasi-sdk
- `make php-7.3.33`
- `make php-7.4.32`

2. The php build uses autoconf make and libtool. On a ubuntu machine you may need to do

```console
sudo apt update && sudo apt install autoconf make libtool-bin -y
```

3. Before building define WASI_SDK_ROOT to point to a local installation of WasiSDK. For example

```console
export WASI_SDK_ROOT=/opt/wasi-sdk
```

# 7.3.33 - patch.v2.diff

This is work in progress and we're currently building only php-cgi.

**Note**: A build with this patch has some issues interpreting WP. We are working on it.

1. To build just run `php/patches/7.3.33/build.sh`

2. You can find `php-cgi` in `$WASMLABS_BUILD_OUTPUT/bin/php-cgi`

# 7.4.32 - patch.v1.diff

This is work in progress and we're currently building only php-cgi.

This build also downloads and builds sqlite3(3.39.2) and links it into the php binary

**Note**: A build with this patch has some issues interpreting WP. We are working on it.

1. To build just run `php/patches/7.4.32/build.sh`

2. You can find `php-cgi` in `$WASMLABS_BUILD_OUTPUT/bin/php-cgi`

# Running a script with php-cgi
## Running a script with php-cgi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though this PR simplifies build a lot, I would still keep some information in the README file about the commands to build PHP. Even it's a simple make php-X. In that way, I don't need to check all the files in the folder, look for a "build" system, notice the Makefile file and read it before knowing the command.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, you are right. I was a bit dubious about adding it because it's easy to forget updating the README when we update the versions.

However I think we can add automation to fix that problem. I have some ideas we can discuss :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that fine now?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 👏


Don't forget to map the folder that contains the php script, which you want to run. For example:

Expand Down
Empty file removed php/php-7.4.32/README.md
Empty file.