Complete installation script for Ubuntu 20.04

[danijar]

Not an issue, just thought I'd share a full installation script for the DMLab Python package because it took a while to put all the pieces together and might be helpful for others. Tested on Ubuntu 20.04, might also work on newer versions:

#!/bin/sh
set -eu

# Dependencies
apt-get update && apt-get install -y \
    build-essential curl freeglut3 gettext git libffi-dev libglu1-mesa \
    libglu1-mesa-dev libjpeg-dev liblua5.1-0-dev libosmesa6-dev \
    libsdl2-dev lua5.1 pkg-config python-setuptools python3-dev \
    software-properties-common unzip zip zlib1g-dev g++
pip3 install numpy

# Bazel
apt-get install -y apt-transport-https curl gnupg
curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg
mv bazel.gpg /etc/apt/trusted.gpg.d/
echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | tee /etc/apt/sources.list.d/bazel.list
apt-get update && apt-get install -y bazel

# Build
git clone https://github.com/deepmind/lab.git
cd lab
echo 'build --cxxopt=-std=c++17' > .bazelrc
bazel build -c opt //python/pip_package:build_pip_package
./bazel-bin/python/pip_package/build_pip_package /tmp/dmlab_pkg
pip3 install --force-reinstall /tmp/dmlab_pkg/deepmind_lab-*.whl
cd ..
rm -rf lab

# Stimuli
mkdir dmlab_data
cd dmlab_data
pip3 install Pillow
curl https://bradylab.ucsd.edu/stimuli/ObjectsAll.zip -o ObjectsAll.zip
unzip ObjectsAll.zip
cd OBJECTSALL
python3 << EOM
import os
from PIL import Image
files = [f for f in os.listdir('.') if f.lower().endswith('jpg')]
for i, file in enumerate(sorted(files)):
  print(file)
  im = Image.open(file)
  im.save('../%04d.png' % (i+1))
EOM
cd ..
rm -rf __MACOSX OBJECTSALL ObjectsAll.zip

apt-get clean

[tkoeppe]

You can reuse the .precommit.bazelrc file instead of making your own (--bazelrc=...).

[danijar]

Good to know about --bazelrc=.... The .precommit.bazelrc specifies --cxxopt=-std=c++11 though, wouldn't that be an issue? Without any .bazelrc I got an error that the build requires at least C++ 14.

[tkoeppe]

Oh, I think that's a pending issue: Abseil has dropped support for C++11, so I think we need to do the same here.

[cirosantilli]

We need to add a Dockerfile to this project.

[elliottower]

We need to add a Dockerfile to this project.

Would love an official dockerfile but you can pretty easily adapt this installation script to work with docker:

# Install DM lab requirements
RUN apt-get -y update \
    && apt-get install --no-install-recommends -y \
    build-essential curl freeglut3 gettext git libffi-dev libglu1-mesa \
    libglu1-mesa-dev libjpeg-dev liblua5.1-0-dev libosmesa6-dev \
    libsdl2-dev lua5.1 pkg-config python-setuptools python3-dev \
    software-properties-common unzip zip zlib1g-dev g++

# Install Bazel
RUN apt-get install -y apt-transport-https curl gnupg  \
    && curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg  \
    && mv bazel.gpg /etc/apt/trusted.gpg.d/  \
    && echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | tee /etc/apt/sources.list.d/bazel.list \
    && apt-get update && apt-get install -y bazel

# Build DM lab
RUN git clone https://github.com/deepmind/lab.git \
    && cd lab \
    && echo 'build --cxxopt=-std=c++17' > .bazelrc \
    && bazel build -c opt //python/pip_package:build_pip_package  \
    && ./bazel-bin/python/pip_package/build_pip_package /tmp/dmlab_pkg \
    && pip3 install --force-reinstall /tmp/dmlab_pkg/deepmind_lab-*.whl \
    && cd .. \
    && rm -rf lab

[danijar]

Updated instal script for Ubuntu 22 (it assumes to be inside a Python 3 venv):

https://gist.github.com/danijar/ca6ab917188d2e081a8253b3ca5c36d3

Usage in Dockerfile:

RUN wget -O - https://gist.github.com/danijar/ca6ab917188d2e081a8253b3ca5c36d3/raw/install-dmlab.sh | sh

Unfortunately, I think DMLab is not compatible with Numpy 2, so you'll also need numpy<2 in your requirements.txt.

[tkoeppe]

(What does it take to make DMLab compatible with Numpy 2?)

[danijar]

My understanding is that Numpy's C API has changed a bit, so if that's used inside DMLab it would have to be update to the new API. I'm guessing Numpy is only used for the interface to Python so it shouldn't be that much work?

[ptonso]

I encountered issues with deprecated Python 2 (PY2) support and .so files being installed in the wrong location.
Here is my solution for Ubuntu 20.04: it consists of functions for setup and for building the virtual environment.
This script explicitly exclude PY2 support and ensure .so files are moved to the correct location. It should work completely standalone in Ubuntu based system.

If others face similar problems, it may be worth raising this as a standalone issue.

#!/bin/bash
set -e  # Exit on any error

SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
VENV_PATH="${SCRIPT_DIR}/venv"
LAB_REPO="https://github.com/google-deepmind/lab.git"
BAZELISK_URL="https://github.com/bazelbuild/bazelisk/releases/download/v1.17.0/bazelisk-linux-amd64"

function install_dependencies {
    echo "Installing dependencies..."
    sudo apt update
    sudo apt install -y build-essential libosmesa6-dev libgl1-mesa-dev \
        libxi-dev libxcursor-dev libxinerama-dev libxrandr-dev libopenal-dev \
        mesa-utils python3-dev python3-venv git wget unzip zlib1g-dev g++
}

function install_bazelisk {
    echo "Installing Bazelisk..."
    wget -q $BAZELISK_URL -O bazelisk
    chmod +x bazelisk
    sudo mv bazelisk /usr/local/bin/bazel
    
    echo "Verifying Bazel installation..."
    bazel --version
}

function setup_virtualenv {
    if [ -d "${VENV_PATH}" ]; then
        echo "Virtual environment already exists at $VENV_PATH. Activating..."
    else
        echo "Creating new virtual environment at $VENV_PATH..."
        python3 -m venv "${VENV_PATH}"
    fi

    source "${VENV_PATH}/bin/activate"
    echo "Upgrading pip, setuptools, and wheel..."
    pip install --upgrade pip setuptools wheel
    pip install numpy dm_env opencv-python
}

function clone_and_configure_lab {
    if [ ! -d "lab" ]; then
        echo "Cloning DeepMind Lab repository..."
        git clone $LAB_REPO
    else
        echo "DeepMind Lab repository already exists. Pulling latest changes..."
        cd lab
        git pull
        cd ..
    fi

    cd lab

    echo "Configuring .bazelrc..."
    cat > .bazelrc << EOL
build --enable_workspace
build --python_version=PY3
build --action_env=PYTHON_BIN_PATH=$(which python3)
EOL

    echo "Modifying BUILD file..."
    sed -i '/\[py_binary(/,/\]]/c\
py_binary(\
    name = "python_game_py3",\
    srcs = ["examples/game_main.py"],\
    data = ["//:deepmind_lab.so"],\
    main = "examples/game_main.py",\
    python_version = "PY3",\
    srcs_version = "PY3",\
    tags = ["manual"],\
    deps = ["@six_archive//:six"],\
)' BUILD

    echo "Replacing python_system.bzl content..."
    cat > python_system.bzl << 'EOL'
# Copyright 2021 DeepMind Technologies Limited.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
# ============================================================================

"""Generates a local repository that points at the system's Python installation."""

_BUILD_FILE = '''# Description:
#   Build rule for Python

exports_files(["defs.bzl"])


cc_library(
    name = "python_headers",
    hdrs = glob(["python3/**/*.h", "numpy3/**/*.h"]),
    includes = ["python3", "numpy3"],
    visibility = ["//visibility:public"],
)
'''

_GET_PYTHON_INCLUDE_DIR = """
import sys
from distutils.sysconfig import get_python_inc
from numpy import get_include
sys.stdout.write("{}\\n{}\\n".format(get_python_inc(), get_include()))
""".strip()

def _python_repo_impl(repository_ctx):
    """Creates external/<reponame>/BUILD, a python3 symlink, and other files."""

    repository_ctx.file("BUILD", _BUILD_FILE)

    result = repository_ctx.execute(["python3", "-c", _GET_PYTHON_INCLUDE_DIR])
    if result.return_code:
        fail("Failed to run local Python3 interpreter: %s" % result.stderr)
    pypath, nppath = result.stdout.splitlines()
    repository_ctx.symlink(pypath, "python3")
    repository_ctx.symlink(nppath, "numpy3")


python_repo = repository_rule(
    implementation = _python_repo_impl,
    configure = True,
    local = True,
    attrs = {"py_version": attr.string(default = "PY3", values = ["PY3"])},
)
EOL

    cd ..
}

function build_lab {
    echo "Building the project with Bazel..."
    cd lab
    bazel clean --expunge
    bazel build -c opt //python/pip_package:build_pip_package --verbose_failures

    echo "Generating wheel file..."
    ./bazel-bin/python/pip_package/build_pip_package /tmp/dmlab_pkg

    echo "Installing package..."
    pip install --force-reinstall /tmp/dmlab_pkg/deepmind_lab-*.whl

    echo "Cleaning up..."
    SITE_PACKAGES="${VENV_PATH}/lib/python3.10/site-packages/deepmind_lab"
    mv "${SITE_PACKAGES}/_main/"* "${SITE_PACKAGES}/"
    rmdir "${SITE_PACKAGES}/_main"
    cd ..
}

# Main Script
install_dependencies
install_bazelisk
setup_virtualenv
clone_and_configure_lab
build_lab

echo "DeepMind Lab setup and installation complete!"

[danijar]

@ptonso Thanks, that's super useful! I've updated your script for Ubuntu 24 and added the stimuli dataset (needed for some of the tasks to render correctly): #253