ARROW-11982: [Rust] Donate Ballista Distributed Compute Platform

This PR contains a donation of the Ballista Distributed Compute platform.

We need to meet the following conditions before merging this PR:

1. Pass a vote on the Arrow dev@ mailing list (Completed on 3/25/2021)
2. Pass the IP clearance process

This PR is based on the contents of Ballista main branch at commit hash [9373241314bb9b01c639aa51c92159611fa42d3e](https://github.com/ballista-compute/ballista/commit/9373241314bb9b01c639aa51c92159611fa42d3e) and contains work from the following contributors:

```
$ git shortlog -n -s -- rust

231  Andy Grove
 27  Ximo Guanter
  7  QP Hou
  6  Boaz Berman
  5  Daniël Heres
  4  Jack Chan
  4  Patrick More
  3  Ben Cohen
  3  Hendrik Makait
  3  Jorge Leitao
  2  Hamish Nicholson
  2  Tanapol Prucksakorn
  1  Andrew Fitzgerald
  1  Dyqer
  1  Enrico Risa
  1  Jeremy Mei(梅杰)
  1  K.I. (Dennis) Jung
  1  Kyle Prifogle
  1  Matthijs Brobbel
  1  Nikita Lapkov
  1  Sieu TSoi
  1  Son
  1  Viktor Gal
  1  Yeshwanth Kumar
  1  jschmitz28

$ git shortlog -n -s -- dev
    45  Andy Grove
     3  Ximo Guanter
     2  Jorge Leitao

$ git shortlog -n -s -- docker
    30  Andy Grove
     6  Ximo Guanter
     2  Daniël Heres
     1  Ben Sully
     1  Hamish Nicholson
     1  Jorge Leitao
     1  Rick Richardson

$ git shortlog -n -s -- docs
    37  Andy Grove
     2  K.I. (Dennis) Jung
     1  Hamish Nicholson
     1  Jorge Leitao
     1  Ximo Guanter
```

Closes #9723 from andygrove/donate-ballista

Authored-by: Andy Grove <[email protected]>
Signed-off-by: Andy Grove <[email protected]>

dev/release/rat_exclude_files.txt

@@ -195,3 +195,5 @@ julia/Arrow/docs/mkdocs.yml
 julia/Arrow/docs/src/index.md
 julia/Arrow/docs/src/manual.md
 julia/Arrow/docs/src/reference.md
+rust/ballista/rust/benchmarks/tpch/queries/q*.sql
+rust/ballista/rust/scheduler/testdata/*

rust/Cargo.toml

@@ -31,4 +31,4 @@ members = [
 # this package is excluded because it requires different compilation flags, thereby significantly changing
 # how it is compiled within the workspace, causing the whole workspace to be compiled from scratch
 # this way, this is a stand-alone package that compiles independently of the others.
-exclude = ["arrow-pyarrow-integration-testing"]
+exclude = ["arrow-pyarrow-integration-testing", "ballista"]

rust/ballista/dev/build-rust-base.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+BALLISTA_VERSION=0.4.2-SNAPSHOT
+set -e
+docker build -t ballistacompute/rust-base:$BALLISTA_VERSION -f docker/rust-base.dockerfile .

rust/ballista/dev/build-rust.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+BALLISTA_VERSION=0.4.2-SNAPSHOT
+
+set -e
+
+docker build -t ballistacompute/ballista-rust:$BALLISTA_VERSION -f docker/rust.dockerfile .

rust/ballista/dev/integration-tests.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -e
+./dev/build-rust.sh
+pushd rust/benchmarks/tpch
+./tpch-gen.sh
+
+docker-compose up -d
+docker-compose run ballista-client ./run.sh
+docker-compose down
+
+popd

rust/ballista/docker/README.md

@@ -0,0 +1,29 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Ballista Docker Images
+
+Pre-built docker images are available from [Docker Hub](https://hub.docker.com/orgs/ballistacompute/repositories) but here are the commands to build the images from source.
+
+Run these commands from the root directory of the project.
+
+```bash
+./dev/build-all.sh
+```
+

rust/ballista/docker/rust-base.dockerfile

@@ -0,0 +1,99 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Turn .dockerignore to .dockerallow by excluding everything and explicitly
+# allowing specific files and directories. This enables us to quickly add
+# dependency files to the docker content without scanning the whole directory.
+# This setup requires to all of our docker containers have arrow's source
+# as a mounted directory.
+
+
+# Base image extends debian:buster-slim
+FROM rust:1.49.0-buster AS builder
+
+RUN apt update && apt -y install musl musl-dev musl-tools libssl-dev openssl
+
+#NOTE: the following was copied from https://github.com/emk/rust-musl-builder/blob/master/Dockerfile under Apache 2.0 license
+
+# The OpenSSL version to use. We parameterize this because many Rust
+# projects will fail to build with 1.1.
+#ARG OPENSSL_VERSION=1.0.2r
+ARG OPENSSL_VERSION=1.1.1b
+
+# Build a static library version of OpenSSL using musl-libc.  This is needed by
+# the popular Rust `hyper` crate.
+#
+# We point /usr/local/musl/include/linux at some Linux kernel headers (not
+# necessarily the right ones) in an effort to compile OpenSSL 1.1's "engine"
+# component. It's possible that this will cause bizarre and terrible things to
+# happen. There may be "sanitized" header
+RUN echo "Building OpenSSL" && \
+    ls /usr/include/linux && \
+    mkdir -p /usr/local/musl/include && \
+    ln -s /usr/include/linux /usr/local/musl/include/linux && \
+    ln -s /usr/include/x86_64-linux-gnu/asm /usr/local/musl/include/asm && \
+    ln -s /usr/include/asm-generic /usr/local/musl/include/asm-generic && \
+    cd /tmp && \
+    curl -LO "https://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz" && \
+    tar xvzf "openssl-$OPENSSL_VERSION.tar.gz" && cd "openssl-$OPENSSL_VERSION" && \
+    env CC=musl-gcc ./Configure no-shared no-zlib -fPIC --prefix=/usr/local/musl -DOPENSSL_NO_SECURE_MEMORY linux-x86_64 && \
+    env C_INCLUDE_PATH=/usr/local/musl/include/ make depend && \
+    env C_INCLUDE_PATH=/usr/local/musl/include/ make && \
+    make install && \
+    rm /usr/local/musl/include/linux /usr/local/musl/include/asm /usr/local/musl/include/asm-generic && \
+    rm -r /tmp/*
+
+RUN echo "Building zlib" && \
+    cd /tmp && \
+    ZLIB_VERSION=1.2.11 && \
+    curl -LO "http://zlib.net/zlib-$ZLIB_VERSION.tar.gz" && \
+    tar xzf "zlib-$ZLIB_VERSION.tar.gz" && cd "zlib-$ZLIB_VERSION" && \
+    CC=musl-gcc ./configure --static --prefix=/usr/local/musl && \
+    make && make install && \
+    rm -r /tmp/*
+
+RUN echo "Building libpq" && \
+    cd /tmp && \
+    POSTGRESQL_VERSION=11.2 && \
+    curl -LO "https://ftp.postgresql.org/pub/source/v$POSTGRESQL_VERSION/postgresql-$POSTGRESQL_VERSION.tar.gz" && \
+    tar xzf "postgresql-$POSTGRESQL_VERSION.tar.gz" && cd "postgresql-$POSTGRESQL_VERSION" && \
+    CC=musl-gcc CPPFLAGS=-I/usr/local/musl/include LDFLAGS=-L/usr/local/musl/lib ./configure --with-openssl --without-readline --prefix=/usr/local/musl && \
+    cd src/interfaces/libpq && make all-static-lib && make install-lib-static && \
+    cd ../../bin/pg_config && make && make install && \
+    rm -r /tmp/*
+
+ENV OPENSSL_DIR=/usr/local/musl/ \
+    OPENSSL_INCLUDE_DIR=/usr/local/musl/include/ \
+    DEP_OPENSSL_INCLUDE=/usr/local/musl/include/ \
+    OPENSSL_LIB_DIR=/usr/local/musl/lib/ \
+    OPENSSL_STATIC=1 \
+    PQ_LIB_STATIC_X86_64_UNKNOWN_LINUX_MUSL=1 \
+    PG_CONFIG_X86_64_UNKNOWN_LINUX_GNU=/usr/bin/pg_config \
+    PKG_CONFIG_ALLOW_CROSS=true \
+    PKG_CONFIG_ALL_STATIC=true \
+    LIBZ_SYS_STATIC=1 \
+    TARGET=musl
+
+# The content copied mentioned in the NOTE above ends here.
+
+## Download the target for static linking.
+RUN rustup target add x86_64-unknown-linux-musl
+RUN cargo install cargo-build-deps
+
+# prepare toolchain
+RUN rustup update && \
+    rustup component add rustfmt

rust/ballista/docker/rust.dockerfile

@@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Turn .dockerignore to .dockerallow by excluding everything and explicitly
+# allowing specific files and directories. This enables us to quickly add
+# dependency files to the docker content without scanning the whole directory.
+# This setup requires to all of our docker containers have arrow's source
+# as a mounted directory.
+
+ARG RELEASE_FLAG=--release
+FROM ballistacompute/rust-base:0.4.0-20210213 AS base
+WORKDIR /tmp/ballista
+RUN apt-get -y install cmake
+RUN cargo install cargo-chef 
+
+FROM base as planner
+COPY rust .
+RUN cargo chef prepare --recipe-path recipe.json
+
+FROM base as cacher
+COPY --from=planner /tmp/ballista/recipe.json recipe.json
+RUN cargo chef cook $RELEASE_FLAG --recipe-path recipe.json
+
+FROM base as builder
+COPY rust .
+COPY --from=cacher /tmp/ballista/target target
+ARG RELEASE_FLAG=--release
+
+# force build.rs to run to generate configure_me code.
+ENV FORCE_REBUILD='true'
+RUN cargo build $RELEASE_FLAG
+
+# put the executor on /executor (need to be copied from different places depending on FLAG)
+ENV RELEASE_FLAG=${RELEASE_FLAG}
+RUN if [ -z "$RELEASE_FLAG" ]; then mv /tmp/ballista/target/debug/ballista-executor /executor; else mv /tmp/ballista/target/release/ballista-executor /executor; fi
+
+# put the scheduler on /scheduler (need to be copied from different places depending on FLAG)
+ENV RELEASE_FLAG=${RELEASE_FLAG}
+RUN if [ -z "$RELEASE_FLAG" ]; then mv /tmp/ballista/target/debug/ballista-scheduler /scheduler; else mv /tmp/ballista/target/release/ballista-scheduler /scheduler; fi
+
+# put the tpch on /tpch (need to be copied from different places depending on FLAG)
+ENV RELEASE_FLAG=${RELEASE_FLAG}
+RUN if [ -z "$RELEASE_FLAG" ]; then mv /tmp/ballista/target/debug/tpch /tpch; else mv /tmp/ballista/target/release/tpch /tpch; fi
+
+# Copy the binary into a new container for a smaller docker image
+FROM ballistacompute/rust-base:0.4.0-20210213
+
+COPY --from=builder /executor /
+
+COPY --from=builder /scheduler /
+
+COPY --from=builder /tpch /
+
+ENV RUST_LOG=info
+ENV RUST_BACKTRACE=full
+
+CMD ["/executor", "--local"]

rust/ballista/docs/README.md

@@ -0,0 +1,37 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+# Ballista Developer Documentation
+
+This directory contains documentation for developers that are contributing to Ballista. If you are looking for 
+end-user documentation for a published release, please start with the 
+[Ballista User Guide](https://ballistacompute.org/docs/) instead.
+
+## Architecture & Design
+
+- Read the [Architecture Overview](architecture.md) to get an understanding of the scheduler and executor 
+  processes and how distributed query execution works.
+
+## Build, Test, Release
+
+- Setting up a [Rust development environment](dev-env-rust.md).
+- Setting up a [Java development environment](dev-env-jvm.md).
+- Notes on building [Rust docker images](rust-docker.md)  
+- [Integration Testing](integration-testing.md)
+- [Release process](release-process.md)
+