-
Notifications
You must be signed in to change notification settings - Fork 0
Building Stockfish on Windows
MSYS2 is a software distribution and building platform for Windows. It provides a Unix-like environment, a command line interface, and a software repository, making it easy to install software on Windows or build software on Windows with the GCC compiler.
MSYS2 consists of three subsystems, msys2
, mingw32
, and mingw64
:
- The
mingw32
andmingw64
subsystems are native Windows applications that use Win32 API. - The
msys2
subsystem provides an emulated mostly-POSIX-compliant environment based on Cygwin.
Each subsystem has an associated "terminal/shell", which is essentially a set of environment variables that allow the subsystems to co-operate properly:
-
MSYS2 MinGW 64-bit
, used to build Windows-native 64-bit applications. -
MSYS2 MinGW 32-bit
, used to build Windows-native 32-bit applications. -
MSYS2 MSYS
, used to build POSIX applications.
Refer to the MSYS2 homepage for more detailed information on the MSYS2 subsystems and terminals/shells.
Run Control Panel
and look under System
to check whether you have a 32-bit or a 64-bit version of Windows and chose your preferred way to install MSYS2.
Chocolatey is a command line package manager for Windows, always run Chocolatey commands in a powershell/cmd with administrator rights (right click on Start
menu, select Windows Powershell (Admin)
or Command Prompt (Admin)
):
- Open a powershell (admin) (not a cmd) and copy the official Chocolatey install command to install Chocolatey
- In a powershell/cmd (admin) execute the command
choco install msys2 -y
As alternative write this text file install_choco_msys2.cmd
, right click and select Run as administrator
:
@echo off
::https://chocolatey.org/install
::https://chocolatey.org/courses/installation/installing?method=installing-chocolatey?quiz=true
::download and run install.ps1
"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"
::install msys2
choco install msys2 -y
- Download and start the one-click installer for MSYS2. It's suggested to choose
C:\tools\msys64
as installation folder to be compatible with fishtest framework. MSYS2 no longer support an installer for Windows 32-bit, the latest provided is not able to install packages. - The installer runs a
MSYS2 MSYS
shell as a last step. Update the core packages by typing and executingpacman -Syuu
. When finished, close theMSYS2 MSYS
shell.
With MSYS2 installed to C:\tools\msys64
your home directory will be C:\tools\msys64\home\<your_username>
. Note that within the MSYS2 shell, paths are written in Unix-like way:
- Windows path:
C:\tools\msys64
- Unix-like path:
/c/tools/msys64
- Windows path:
C:\tools\msys64\home
- Unix-like path:
/home
or/c/tools/msys64/home
Note that you can also use ls
to list the files and folders in a directory, similar to how you would use dir
in Windows.
- Using your favorite text editor, copy and paste the following bash script, calling it
makefish.sh
:
- 64-bit Windows
#!/bin/bash
# makefish.sh
# install packages if not already installed
pacman -S --noconfirm --needed unzip make mingw-w64-x86_64-gcc
# download the Stockfish source code
wget -O master.zip https://github.com/official-stockfish/Stockfish/archive/master.zip
unzip -o master.zip
cd Stockfish-master/src
file_nnue=$(grep 'define.*EvalFileDefaultName' evaluate.h | grep -Ewo 'nn-[a-z0-9]{12}.nnue')
ls *.nnue | grep -v ${file_nnue} | xargs -d '\n' -r rm --
# find the CPU architecture
gcc_enabled=$(g++ -Q -march=native --help=target | grep "\[enabled\]")
gcc_arch=$(g++ -Q -march=native --help=target | grep "march")
if [[ "${gcc_enabled}" =~ "-mavx512vnni " && "${gcc_enabled}" =~ "-mavx512dq " && "${gcc_enabled}" =~ "-mavx512f " && "${gcc_enabled}" =~ "-mavx512bw " && "${gcc_enabled}" =~ "-mavx512vl " ]] ; then
arch_cpu="x86-64-vnni256"
elif [[ "${gcc_enabled}" =~ "-mavx512f " && "${gcc_enabled}" =~ "-mavx512bw " ]] ; then
arch_cpu="x86-64-avx512"
elif [[ "${gcc_enabled}" =~ "-mbmi2 " && ! "${gcc_arch}" =~ "znver1" && ! "${gcc_arch}" =~ "znver2" ]] ; then
arch_cpu="x86-64-bmi2"
elif [[ "${gcc_enabled}" =~ "-mavx2 " ]] ; then
arch_cpu="x86-64-avx2"
elif [[ "${gcc_enabled}" =~ "-mpopcnt " && "${gcc_enabled}" =~ "-msse4.1 " ]] ; then
arch_cpu="x86-64-modern"
elif [[ "${gcc_enabled}" =~ "-mssse3 " ]] ; then
arch_cpu="x86-64-ssse3"
elif [[ "${gcc_enabled}" =~ "-mpopcnt " && "${gcc_enabled}" =~ "-msse3 " ]] ; then
arch_cpu="x86-64-sse3-popcnt"
else
arch_cpu="x86-64"
fi
# build the fastest Stockfish executable
make -j profile-build ARCH=${arch_cpu} COMP=mingw
strip stockfish.exe
mv stockfish.exe ../../stockfish_${arch_cpu}.exe
make clean
cd
- 32-bit Windows
#!/bin/bash
# makefish.sh
# install packages if not already installed
pacman -S --noconfirm --needed unzip make mingw-w64-i686-gcc
# download the Stockfish source code
wget -O master.zip https://github.com/official-stockfish/Stockfish/archive/master.zip
unzip -o master.zip
cd Stockfish-master/src
file_nnue=$(grep 'define.*EvalFileDefaultName' evaluate.h | grep -Ewo 'nn-[a-z0-9]{12}.nnue')
ls *.nnue | grep -v ${file_nnue} | xargs -d '\n' -r rm --
# find the CPU architecture
gcc_enabled=$(g++ -Q -march=native --help=target | grep "\[enabled\]")
gcc_arch=$(g++ -Q -march=native --help=target | grep "march")
if [[ "${gcc_enabled}" =~ "-mpopcnt " && "${gcc_enabled}" =~ "-msse4.1 " ]] ; then
arch_cpu="x86-32-sse41-popcnt"
elif [[ "${gcc_enabled}" =~ "--msse2 " ]] ; then
arch_cpu="x86-32-sse2"
else
arch_cpu="x86-32"
fi
# build the fastest Stockfish executable
make -j profile-build ARCH=${arch_cpu} COMP=mingw
strip stockfish.exe
mv stockfish.exe ../../stockfish_${arch_cpu}.exe
make clean
cd
- Start a
MSYS2 MinGW 64-bit
shell (not aMSYS2 MSYS
one),C:\tools\msys64\mingw64.exe
, or start aMSYS2 MinGW 32-bit
shell ,C:\tools\msys64\mingw32.exe
, to build a 32 bit application. - Navigate to wherever you saved the script (e.g. type and execute
cd '/d/Program Files/Stockfish'
to navigate toD:\Program Files\Stockfish
.) - Run the script by typing and executing
bash makefish.sh
.
If this tutorial will not work on your pc, you may try to change the Windows Security
settings in via Windows Security
>> App & Browser Control
>> Exploit Protection Settings
:
- Try to turn off "Force randomization for images (Mandatory ASLR)", if this not solve the problem then,
- Try to turn off also "Randomize memory allocations (Bottom-up ASLR)" .
To use with MSYS2 a MinGW-w64 built by other projects, simply follow these instructions (Windows 64 bit):
-
download another version of MinGW-w64, e.g. MinGW-w64 (64-bit) GCC 8.1.0, extract the mingw64 folder renaming it to mingw64-810, copy the folder into C:\msys64, check to have the directory C:\msys64\mingw64-810\bin
-
build Stockfish writing and executing this bash script
#!/bin/bash
# makefish.sh
# set PATH to use GCC 8.1.0
if [ -d "/mingw64-810/bin" ] ; then
PATH="/mingw64-810/bin:${PATH}"
else
echo "folder error"
exit 1
fi
# download the Stockfish source code
wget -O master.zip https://github.com/official-stockfish/Stockfish/archive/master.zip
unzip master.zip
cd Stockfish-master/src
# find the CPU architecture
# CPU without popcnt and bmi2 instructions (e.g. older than Intel Sandy Bridge)
arch_cpu=x86-64
# CPU with bmi2 instruction (e.g. Intel Haswell or newer)
if [ "$(g++ -Q -march=native --help=target | grep mbmi2 | grep enabled)" ] ; then
# CPU AMD zen
if [ "$(g++ -Q -march=native --help=target | grep march | grep 'znver[12]')" ] ; then
arch_cpu=x86-64-avx2
else
arch_cpu=x86-64-bmi2
fi
# CPU with popcnt instruction (e.g. Intel Sandy Bridge)
elif [ "$(g++ -Q -march=native --help=target | grep mpopcnt | grep enabled)" ] ; then
arch_cpu=x86-64-modern
fi
# build the Stockfish executable
make profile-build ARCH=${arch_cpu} COMP=mingw
strip stockfish.exe
mv stockfish.exe ../../stockfish_${arch_cpu}.exe
make clean
cd
To use the compiler in the CLI write and run the script use_gcc810.sh
in the user home folder
# set PATH to use GCC 8.1.0
# use this command: source use_gcc810.sh
if [ -d "/mingw64-810/bin" ] ; then
PATH="/mingw64-810/bin:${PATH}"
else
echo "folder error"
fi
To measure the speedup of several builds of Stockfish, use one of these applications:
- FishBench: download the latest release Fishbench v6.0
- Buildtester: download the latest release Buildtester 1.4.7.0
- pyshbench: download the latest release pyshbench
Note:
- Don't run other applications when measuring the speedup of Stockfish (i.e. stop fishtest)
- Run at least 20 benches for each build of Stockfish to have accurate measures
- A speedup of 0.3% could be meaningless (i.e. within the measurement noise)
You can use also this bash script bench_parallel.sh
#!/bin/bash
_bench () {
${1} << EOF > /dev/null 2>> ${2}
bench 16 1 ${depth} default depth
EOF
}
# _bench function customization example
# setoption name SyzygyPath value C:\table_bases\wdl345;C:\table_bases\dtz345
# bench 128 4 ${depth} default depth
if [[ ${#} -ne 4 ]]; then
cat << EOF
usage: ${0} sf_base sf_test depth n_runs
fast sf_test:
${0} ./stockfish_base ./stockfish_test 13 10
slow sf_test:
${0} ./stockfish_base ./stockfish_test 20 10
EOF
exit 1
fi
sf_base=${1}
sf_test=${2}
depth=${3}
n_runs=${4}
# preload of CPU/cache/memory
(_bench ${sf_base} sf_base0.txt)&
(_bench ${sf_test} sf_test0.txt)&
wait
# temporary files initialization
: > sf_base0.txt
: > sf_test0.txt
: > sf_temp0.txt
# bench loop: SMP bench with background subshells
for ((k=1; k<=${n_runs}; k++)); do
printf "run %3d /%3d\n" ${k} ${n_runs}
# swap the execution order to avoid bias
if [ $((k%2)) -eq 0 ]; then
(_bench ${sf_base} sf_base0.txt)&
(_bench ${sf_test} sf_test0.txt)&
wait
else
(_bench ${sf_test} sf_test0.txt)&
(_bench ${sf_base} sf_base0.txt)&
wait
fi
done
# text processing to extract nps values
cat sf_base0.txt | grep second | grep -Eo '[0-9]{1,}' > sf_base1.txt
cat sf_test0.txt | grep second | grep -Eo '[0-9]{1,}' > sf_test1.txt
for ((k=1; k<=${n_runs}; k++)); do
echo ${k} >> sf_temp0.txt
done
printf "\nrun\tsf_base\tsf_test\tdiff\n"
paste sf_temp0.txt sf_base1.txt sf_test1.txt | awk '{printf "%3d %d %d %+d\n", $1, $2, $3, $3-$2}'
paste sf_base1.txt sf_test1.txt | awk '{printf "%d\t%d\t%d\n", $1, $2, $2-$1}' > sf_temp0.txt
# compute: sample mean, 1.96 * std of sample mean (95% of samples), speedup
# std of sample mean = sqrt(NR/(NR-1)) * (std population) / sqrt(NR)
cat sf_temp0.txt | awk '{sum1 += $1 ; sumq1 += $1**2 ;sum2 += $2 ; sumq2 += $2**2 ;sum3 += $3 ; sumq3 += $3**2 } END {printf "\nsf_base = %10d +/- %d\nsf_test = %10d +/- %d\ndiff = %10d +/- %d\nspeedup = %.6f\n\n", sum1/NR , 1.96 * sqrt(sumq1/NR - (sum1/NR)**2)/sqrt(NR-1) , sum2/NR , 1.96 * sqrt(sumq2/NR - (sum2/NR)**2)/sqrt(NR-1) , sum3/NR , 1.96 * sqrt(sumq3/NR - (sum3/NR)**2)/sqrt(NR-1) , (sum2 - sum1)/sum1 }'
# remove temporary files
rm -f sf_base0.txt sf_test0.txt sf_temp0.txt sf_base1.txt sf_test1.txt
It is possible to build Stockfish with Link Time Optimization: on Haswell and newer CPU this gave a speedup of 2%, on Ivy Bridge and older CPU at the moment the speedup is very little (0.3% on Ivy Bridge). Without editing the Makefile, the simple way is to use the Linux compile directive COMP=gcc, e.g. for Haswell CPU and newer:
make profile-build ARCH=x86-64-bmi2 COMP=gcc
Due to a bug in MinGW for Windows, it not possible to have at the moment a static LTO build, so copy, along stockfish.exe, these three dll:
- libgcc_s_seh-1.dll
- libstdc++-6.dll
- libwinpthread-1.dll
You find them in:
- for MSYS: C:\MinGW\mingw64\x86_64-w64-mingw32\lib
- for MSYS2: C:\msys64\mingw64\bin
It is possible to lower the compile time on cpu multi core using make with the flag -j <n_jobs>, where <n_jobs> is the number of jobs (commands) to run simultaneously. The flag -j enables one job for each logical CPU core.
make -j <n_jobs> profile-build ARCH=x86-64-modern COMP=mingw
To get the max speedup for your CPU (1.5% on Ivy Bridge) simply prepend the shell variable CXXFLAGS='-march=native'
to the make
command. At example, for a CPU Sandy/Ivy Bridge use this command:
CXXFLAGS='-march=native' make -j profile-build ARCH=x86-64-modern COMP=mingw
To view the compiler flags added for your cpu, use this command:
gcc -Q -march=native --help=target | grep -v "\[disabled\]"
-march=native implies -mtune=native, below a high level explanation of the compiler flags -march and -mtune, view the gcc manual for more technically sound details:
-
-march: determines what instruction sets are used in the binary. An instruction set is the list of commands implemented by the cpu. The generated code may not run at all on processors other than the one indicated.
-
-mtune: determines the cost model that is used when generating code. The cost model describes how long it takes the cpu to do operations. This information is used by the scheduler to decide what operations to use and in what order.
Write and run this script.
#!/bin/bash
# functions to build Stockfish
_build_sf () {
make build ARCH=x86-64$1 COMP=mingw -j
strip stockfish.exe
mv stockfish.exe ../../stockfish-x64${1}.exe
make clean
}
_build_sf_pgo () {
make profile-build ARCH=x86-64$1 COMP=mingw PGOBENCH="wine ./stockfish.exe bench" -j
strip stockfish.exe
mv stockfish.exe ../../stockfish-x64${1}-pgo.exe
make clean
}
# full-upgrade and install required packages
sudo apt update && sudo apt full-upgrade -y && sudo apt autoremove -y && sudo apt clean
sudo apt install -y \
make \
mingw-w64 \
git \
wine64 \
binutils
# clone Stockfish source code
git clone --single-branch --branch master https://github.com/official-stockfish/Stockfish.git
cd Stockfish/src
# build Stockfish executables
# to speedup the building process you can keep only the section fitting your CPU architecture
# build the binary for CPUs without popcnt and bmi2 instructions (e.g. older than Intel Sandy Bridge)
_build_sf_pgo
# build the binary for CPU with popcnt instruction (e.g. Intel Sandy Bridge)
if [ "$(x86_64-w64-mingw32-c++-posix -Q -march=native --help=target | grep mpopcnt | grep enabled)" ] ; then
_build_sf_pgo -modern
else
_build_sf -modern
fi
# build the binary for CPU with bmi2 instruction (e.g. Intel Haswell or newer)
if [ "$(x86_64-w64-mingw32-c++-posix -Q -march=native --help=target | grep mbmi2 | grep enabled)" ] ; then
_build_sf_pgo -bmi2
else
_build_sf -bmi2
fi
Yes, but it is required to explicitly set the stack reserve to avoid crashes. See point 5. below.
If you want to use MSVC to get a "optimized" build, you can change these settings in the IDE:
- Add "NDEBUG;USE_POPCNT;USE_PEXT" to preprocessor definitions.
- Optimization flags: /O2, /Oi, /Ot, /Oy, /GL
- Static link with runtime: /MT.
- Disable stack cookies: /GS-.
- Set stack reserve to 8388608 in under Linker -> System or use the linker option /STACK:reserve=8388608.
- Disable debugging information in compiler/linker.
- Make a PGO instrument build(set under General), it should depend on "pgort140.dll" and it probably won't start.
- Copy pgort140.dll from "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\Hostx64\x64" to the output folder.
- Run bench with the instrument build(very slow) and quit, it should generate "Stockfish.pgd" and "Stockfish!1.pgc".
- Make a PGO optimized build(set under General), should show something like:
1>0 of 0 ( 0.0%) original invalid call sites were matched.
1>0 new call sites were added.
1>54 of 4076 ( 1.32%) profiled functions will be compiled for speed, and the rest of the functions will be compiled for size
1>18615 of 46620 inline instances were from dead/cold paths
1>4076 of 4076 functions (100.0%) were optimized using profile data
1>14499840744 of 14499840744 instructions (100.0%) were optimized using profile data
- Enjoy, local tests show comparable speed to GCC builds.