Skip to content

Fishtest server setup

Jump to bottom Edit New page
ppigazzini edited this page Sep 12, 2021 · 106 revisions
  1. Use a clean install of Ubuntu 18.04 (bionic) or 20.04 (focal) on a virtual machine, cloud instance etc.
  2. Copy the script setup-fishtest.sh:
    1. write your password in the variable usr_pwd
    2. (optional to use https) write your fully qualified domain name in the variable server_name
  3. Run the setup script with:
sudo bash setup-fishtest.sh 2>&1 | tee setup-fishtest.sh.log
  1. Setup the nets workflow:
    1. download the nets required by the tests (e.g. the default one) from the official fishtest server NN Stats page
    2. open a web browser using the ip_address of the fishtest server (http://ip_address/login), login as user01 (with password user01), select the "NN Upload page" (http://ip_address/upload) and upload the net. The net is written in /home/fishtest/post-server/nn
    3. (optional) use the script /home/fishtest/post-server/set_netserver.sh to set the server (the official server or the local development server) from which to download the nets during the tests
  2. Create some tests with these users:
    • user00 (with password user00), to approve test
    • user01 (with password user01), to create test
  3. Connect a worker using the ip_address of the fishtest server, to have multiple workers make some copies of the worker folder.
python3 worker.py <username> <password> --protocol <http/https> --host <ip_address> --port <80/443/custom> --concurrency <n_cores>
  1. Stop/start the fishtest services with:
# start the services
sudo systemctl start fishtest@{6543..6544}.service

# stop the services
sudo systemctl stop fishtest@{6543..6544}.service
  1. To debug the server with the Pyramid Debug Toolbar, login on Ubuntu, use the following commands to start/stop the fishtest_dbg.service, and open a browser using the port 6542 (http://ip_address:6542).
# start the debug session
sudo systemctl start fishtest_dbg.service

# stop the debug session
sudo systemctl stop fishtest_dbg.service

setup script for Ubuntu 18.04 (bionic) or 20.04 (focal)

#!/bin/bash
# 210912
# to setup a fishtest server on Ubuntu 18.04 (bionic) or 20.04 (focal), simply run:
# sudo bash setup_fishtest.sh 2>&1 | tee setup_fishtest.sh.log
#
# to use fishtest connect a browser to:
# http://<ip_address> or http://<fully_qualified_domain_name>

user_name='fishtest'
user_pwd='<your_password>'
# try to find the ip address
server_name=$(hostname --all-ip-addresses)
server_name="${server_name#"${server_name%%[![:space:]]*}"}"
server_name="${server_name%"${server_name##*[![:space:]]}"}"
# use a fully qualified domain names (http/https)
# server_name='<fully_qualified_domain_name>'

git_user_name='your_name'
git_user_email='[email protected]'

# create user for fishtest
useradd -m -s /bin/bash ${user_name}
echo ${user_name}:${user_pwd} | chpasswd
usermod -aG sudo ${user_name}
sudo -i -u ${user_name} << EOF
mkdir .ssh
chmod 700 .ssh
touch .ssh/authorized_keys
chmod 600 .ssh/authorized_keys
EOF

# get the user $HOME
user_home=$(sudo -i -u ${user_name} << 'EOF'
echo ${HOME}
EOF
)

# add some bash variables
sudo -i -u ${user_name} << 'EOF'
cat << 'EOF0' >> .profile

export FISHTEST_HOST=127.0.0.1
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export VENV="$HOME/fishtest/server/env"
export PSVENV="$HOME/post-server/env"
EOF0
EOF

# set secrets
sudo -i -u ${user_name} << EOF
echo '' > fishtest.secret
echo '' > fishtest.captcha.secret
echo 'http://127.0.0.1/upload_net' > fishtest.upload

cat << EOF0 > .netrc
# GitHub authentication to raise API rate limit
# create a <personal-access-token> https://github.com/settings/tokens
#machine api.github.com
#login <personal-access-token>
#password x-oauth-basic
EOF0
chmod 600 .netrc
EOF

# install required packages
apt update && apt full-upgrade -y && apt autoremove -y && apt clean
apt purge -y apache2 apache2-data apache2-doc apache2-utils apache2-bin
apt install -y ufw git bash-completion nginx mutt curl procps

# configure ufw
ufw allow ssh
ufw allow http
ufw allow https
ufw allow 6542
ufw --force enable
ufw status verbose

# configure nginx
cat << EOF > /etc/nginx/sites-available/fishtest.conf

upstream backend_tests {
    server 127.0.0.1:6543;
}

upstream backend_all {
    server 127.0.0.1:6544;
}

upstream upload_server {
    server 127.0.0.1:8080;
}

server {
    listen 80;
    listen [::]:80;

    server_name ${server_name};

    location ~ ^/(css|html|img|js|favicon.ico|robots.txt) {
        root        ${user_home}/fishtest/server/fishtest/static;
        expires     1y;
        add_header  Cache-Control public;
        access_log  off;
    }

	location /nn/ {
        root         ${user_home}/post-server;
        gzip_static  always;
        gunzip       on;
    }

    location / {
        proxy_pass http://backend_all;

        proxy_set_header X-Forwarded-Proto  \$scheme;
        proxy_set_header X-Forwarded-For    \$proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host   \$host:\$server_port;
        proxy_set_header X-Forwarded-Port   \$server_port;

        client_max_body_size        120m;
        client_body_buffer_size     128k;
        proxy_connect_timeout       60s;
        proxy_send_timeout          90s;
        proxy_read_timeout          90s;
        proxy_buffering             off;
        proxy_temp_file_write_size  64k;
        proxy_redirect              off;

        location ~ ^/api/(active_runs|download_pgn|download_pgn_100|request_version|upload_pgn) {
            proxy_pass http://backend_all;
        }

        location /api/ {
            proxy_pass http://backend_tests;
        }

        location ~ ^/tests/(finished|user/) {
            proxy_pass http://backend_all;
        }

        location /tests {
            proxy_pass http://backend_tests;
        }

        location /upload_net/ {
            proxy_pass http://upload_server;
        }
    }
}
EOF

unlink /etc/nginx/sites-enabled/default
ln -sf /etc/nginx/sites-available/fishtest.conf /etc/nginx/sites-enabled/fishtest.conf
systemctl enable nginx.service
systemctl restart nginx.service

# setup pyenv and install the latest python version
# https://github.com/pyenv/pyenv
apt update
apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

sudo -i -u ${user_name} << 'EOF'
cp .profile .profile.bkp
sed -i 's/^#umask 022/#umask 022\n\n# pyenv: keep before sourcing .bashrc\nexport PYENV_ROOT="$HOME\/.pyenv"\nexport PATH="$PYENV_ROOT\/bin:$PATH"/' .profile

cat << 'EOF0' >> .profile

# pyenv: keep at the end of the file
if command -v pyenv &>/dev/null; then
  eval "$(pyenv init --path)"
fi
EOF0
EOF

sudo -i -u ${user_name} << 'EOF'
python_ver="3.8.12"
git clone https://github.com/pyenv/pyenv.git "${PYENV_ROOT}"
pyenv install ${python_ver}
pyenv global ${python_ver}
EOF

# install mongodb community edition for Ubuntu 18.04 (bionic) or 20.04 (focal)
wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
ubuntu_release=$(lsb_release -c | awk '{print $2}')
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu ${ubuntu_release}/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
apt update
apt install -y mongodb-org

# set the cache size in /etc/mongod.conf
#  wiredTiger:
#    engineConfig:
#      cacheSizeGB: 1.75
cp /etc/mongod.conf mongod.conf.bkp
sed -i 's/^#  wiredTiger:/  wiredTiger:\n    engineConfig:\n      cacheSizeGB: 1.75/' /etc/mongod.conf

# setup logrotate for mongodb
sed -i '/^  logAppend: true/a\  logRotate: reopen' /etc/mongod.conf

cat << 'EOF' > /etc/logrotate.d/mongod
/var/log/mongodb/mongod.log
{
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0600 mongodb mongodb
    sharedscripts
    postrotate
        /bin/kill -SIGUSR1 $(pgrep mongod 2>/dev/null) 2>/dev/null || true
    endscript
}
EOF

# download fishtest
sudo -i -u ${user_name} << EOF
git clone --single-branch --branch master https://github.com/glinscott/fishtest.git
cd fishtest
git config user.email "${git_user_email}"
git config user.name "${git_user_name}"
EOF

# setup fishtest
sudo -i -u ${user_name} << 'EOF'
python3 -m venv ${VENV}
${VENV}/bin/python3 -m pip install --upgrade pip setuptools wheel
cd ${HOME}/fishtest/server
${VENV}/bin/python3 -m pip install -e .
EOF

# install fishtest as systemd service
cat << EOF > /etc/systemd/system/[email protected]
[Unit]
Description=Fishtest Server port %i
After=network.target mongod.service

[Service]
Type=simple
ExecStart=${user_home}/fishtest/server/env/bin/pserve production.ini http_port=%i
Restart=on-failure
RestartSec=3
User=${user_name}
WorkingDirectory=${user_home}/fishtest/server

[Install]
WantedBy=multi-user.target
EOF

# install also fishtest debug as systemd service
cat << EOF > /etc/systemd/system/fishtest_dbg.service
[Unit]
Description=Fishtest Server Debug port 6542
After=network.target mongod.service

[Service]
Type=simple
ExecStart=${user_home}/fishtest/server/env/bin/pserve development.ini --reload
User=${user_name}
WorkingDirectory=${user_home}/fishtest/server

[Install]
WantedBy=multi-user.target
EOF

# enable the autostart for mongod.service and [email protected]
# check the log with: sudo journalctl -u [email protected] --since "2 days ago"
systemctl daemon-reload
systemctl enable mongod.service
systemctl enable fishtest@{6543..6544}.service

# start fishtest server
systemctl start mongod.service
systemctl start fishtest@{6543..6544}.service

# add mongodb indexes
sudo -i -u ${user_name} << 'EOF'
${VENV}/bin/python3 ${HOME}/fishtest/server/utils/create_indexes.py actions flag_cache pgns runs users
EOF

# add some default users:
# "user00" (with password "user00"), as approver
# "user01" (with password "user01"), as normal user
sudo -i -u ${user_name} << 'EOF'
${VENV}/bin/python3 << EOF0
from fishtest.rundb import RunDb
rdb = RunDb()
rdb.userdb.create_user("user00", "user00", "[email protected]")
rdb.userdb.add_user_group("user00", "group:approvers")
user = rdb.userdb.get_user("user00")
user["blocked"] = False
user["machine_limit"] = 100
rdb.userdb.save_user(user)
rdb.userdb.create_user("user01", "user01", "[email protected]")
user = rdb.userdb.get_user("user01")
user["blocked"] = False
user["machine_limit"] = 100
rdb.userdb.save_user(user)
EOF0
EOF

sudo -i -u ${user_name} << 'EOF'
(crontab -l; cat << EOF0
VENV=${HOME}/fishtest/server/env
UPATH=${HOME}/fishtest/server/utils

# Backup mongodb database and upload to s3
# keep disabled on dev server
# 3 */6 * * * /usr/bin/cpulimit -l 50 -f -m -- sh \${UPATH}/backup.sh

# Update the users table
1,16,31,46 * * * * \${VENV}/bin/python3 \${UPATH}/delta_update_users.py

# Purge old pgn files
33 3 * * * \${VENV}/bin/python3 \${UPATH}/purge_pgn.py

# Clean up old mail (more than 9 days old)
33 5 * * * screen -D -m mutt -e 'push D~d>9d<enter>qy<enter>'

EOF0
) | crontab -
EOF

# setup post-server
sudo -i -u ${user_name} << 'EOF'
mkdir -p ${HOME}/post-server/nn
python3 -m venv ${PSVENV}
${PSVENV}/bin/python3 -m pip install --upgrade pip setuptools wheel

cat << EOF0 > ${HOME}/post-server/server.py
#!/usr/bin/env python3

# From https://gist.github.com/mdonkers/63e115cc0c79b4f6b8b3a6b797e485c7

import gzip
import logging
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path

class S(BaseHTTPRequestHandler):
    def _set_response(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()

    def do_GET(self):
        logging.info(
            "GET request,\nPath: {}\nHeaders:\n{}\n".format(
                str(self.path), str(self.headers)
            )
        )
        self._set_response()
        self.wfile.write("GET request for {}".format(self.path).encode("utf-8"))

    def do_POST(self):
        # Gets the size of data
        content_length = int(self.headers["Content-Length"])
        # Gets the data itself
        post_data = self.rfile.read(content_length)
        logging.info(
            "POST request,\nPath: {}\nHeaders:\n{}\n\nBody length:\n{}\n".format(
                str(self.path), str(self.headers), len(post_data)
            )
        )
        self._set_response()
        self.wfile.write("POST request for {}".format(self.path).encode("utf-8"))

        file_name = Path("${HOME}/post-server/nn/") / Path(self.path).name
        file_name = file_name.with_suffix(file_name.suffix + ".gz")
        try:
            with gzip.open(file_name, "wb") as f:
                f.write(post_data)
        except Exception as e:
            self.wfile.write(str(e))

def run(server_class=HTTPServer, handler_class=S, port=8080):
    logging.basicConfig(level=logging.INFO)
    server_address = ("", port)
    httpd = server_class(server_address, handler_class)
    logging.info("Starting httpd...\n")
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    httpd.server_close()
    logging.info("Stopping httpd...\n")

if __name__ == "__main__":
    from sys import argv

    if len(argv) == 2:
        run(port=int(argv[1]))
    else:
        run()
EOF0
EOF

# script to set the server to download the nets from
sudo -i -u ${user_name} << EOF
cat << EOF0 > \${HOME}/post-server/set_netserver.sh
#!/bin/bash

_usage () {
cat << EOF1
usage: bash \\\${0} <o|l>
  set the server to download the nets from:
  o: the "official" server used in fishtest
  l: this development "local" server
EOF1
exit
}

if [[ \\\${#} == '0' ]] ; then
  _usage
fi

if [[ \\\${1} == 'l' ]]; then
  sed -i 's/"https:\/\/data.stockfishchess.org\/nn\/"/"http:\/\/${server_name}\/nn\/"/' \\\${HOME}/fishtest/server/fishtest/api.py
elif [[ \\\${1} == 'o' ]]; then
  sed -i 's/"http:\/\/${server_name}\/nn\/"/"https:\/\/data.stockfishchess.org\/nn\/"/' \\\${HOME}/fishtest/server/fishtest/api.py
else
  _usage
fi

echo 'fishtest restart to apply the new setting:'
sudo systemctl restart fishtest@{6543..6544}
EOF0
EOF

### install post-server as systemd service
cat << EOF > /etc/systemd/system/post-server.service
[Unit]
Description=simple HTTP server in python
After=network.target

[Service]
Type=simple
ExecStart=${user_home}/post-server/env/bin/python3 ${user_home}/post-server/server.py
Restart=on-failure
RestartSec=3
User=${user_name}
WorkingDirectory=${user_home}/post-server

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable post-server.service
systemctl start post-server.service

cat << EOF
connect a browser to:
http://${server_name}
EOF

Let's Encrypt setup script for Ubuntu 18.04 (bionic) or 20.04 (focal)

# sudo bash setup-certbot.sh 2>&1 | tee setup-certbot.sh.log
# install certbot to setup let's encrypt
# https://certbot.eff.org/
# requires a DNS and a fully qualified domain name as servername

apt update && apt install -y software-properties-common
add-apt-repository -y universe
add-apt-repository -y ppa:certbot/certbot
apt update && apt install -y certbot python-certbot-nginx
cat << EOF
to configure let's encrypt run:
sudo certbot --nginx
EOF

fishtest server script to test a PR/branch

Use this script to test a PR/branch

#!/bin/bash
# to update a fishtest server simply run:
# sudo bash update_fishtest.sh 2>&1 | tee update_fishtest.sh.log
#
# to use fishtest connect a browser to:
# http://<ip_address>

user_name='fishtest'

# backup
sudo -i -u ${user_name} << EOF
tar -czvf fishtest.$(date +%Y%m%d%H%M%S --utc).tgz fishtest
EOF

systemctl stop cron
systemctl stop fishtest@{6543..6544}
sleep 5

# download and prepare fishtest
sudo -i -u ${user_name} << EOF
rm -rf fishtest
git clone --single-branch --branch master https://github.com/glinscott/fishtest.git
cd fishtest
git config user.email '[email protected]'
git config user.name 'your_name'

# add here the upstream branch to be tested
#git remote add <your_upstream> https://github.com/<your_username>/fishtest
#git pull --no-edit <your_upstream> <your_branch>

# add here the PR/PRs to be tested
#git pull --no-edit origin pull/<PR_number>/head
#git pull --no-edit origin pull/<PR_number>/head

# setup fishtest
sudo -i -u ${user_name} << 'EOF'
python3 -m venv ${VENV}
${VENV}/bin/python3 -m pip install --upgrade pip setuptools wheel
cd ${HOME}/fishtest/server
${VENV}/bin/python3 -m pip install -e .
EOF

# start fishtest
systemctl start cron
systemctl start fishtest@{6543..6544}

cat << EOF
connect a browser to:
http://$(hostname --all-ip-addresses)
EOF

fishtest worker script to test a PR/branch

#!/bin/bash
# requirements:
# sudo apt update && sudo apt install -y python3 python3-pip python3-venv git build-essential libnuma-dev

test_folder=/<full_path>/__test_folder
virtual_env=${test_folder}/env

rm -rf ${test_folder}
mkdir -p ${test_folder}
cd ${test_folder}
git clone --single-branch --branch master https://github.com/glinscott/fishtest.git
cd fishtest
git config user.email "[email protected]"
git config user.name "your_name"

# add here the upstream branches to be tested
#git remote add <upstream_0> https://github.com/<username_0>/fishtest
#git pull --no-edit <upstream_0> <branch_0>
#git remote add <upstream_1> https://github.com/<username_1>/fishtest
#git pull --no-edit <upstream_1> <branch_1>

# add here the PRs to be tested
#git pull --no-edit origin pull/<PR_number_0>/head
#git pull --no-edit origin pull/<PR_number_1>/head

cd worker
python3 -m venv ${virtual_env}
${virtual_env}/bin/python3 -m pip install --upgrade pip setuptools wheel
${virtual_env}/bin/python3 -m pip install requests

${virtual_env}/bin/python3 worker.py user00 user00 --protocol http --host <ip-address> --port 80 --concurrency 3

fishtest mongodb: backup and restore

Use the mongodb tools in a temporary folder, e.g.

  • backup: mongodump --gzip && tar -cv dump | gzip -1 > dump.tar.gz && rm -rf dump
  • restore: tar -xzvf dump.tgz && mongorestore --gzip --drop && rm -rf dump

Stop fishtest and cron services before a mongodb restore:

sudo systemctl stop fishtest@{6543..6544}
sudo systemctl stop cron

FAQ (server maintenance, etc.)

How to react if there is a bad worker?

Sometime a badly configured worker client may post lots of losses on time during tests, or cause some tests to stop. The best policy to follow in these cases would be:

  1. open a thread in the fishcooking forum, reporting about the worker anomalous behavior.
  2. write an email to the user asking to control the worker and to write some info in the fishcooking thread.
  3. in case of failure of previous steps: block the user.

How to block a malicious user?

Approvers (and only approvers) can now block malicious users on fishtest:

  1. login with your own approver username/password
  2. click on a run to see the test page
  3. all the workers' names are now a hyperlink for the approvers
  4. click on a worker name (field "username")
  5. you view some info about the user: e.g. the email
  6. you can block/unblock the user. If blocked, the user cannot login in fishtest (e.g. submitting a test) and the workers cannot join the framework
Add a custom footer
Add a custom sidebar
Clone this wiki locally