Skip to content

Building Stockfish on Windows

ppigazzini edited this page Jul 26, 2021 · 5 revisions

About MSYS2 & MinGW-w64

MSYS2 is a software distribution and building platform for Windows. It provides a Unix-like environment, a command line interface, and a software repository, making it easy to install software on Windows or build software on Windows with the GCC compiler.

MSYS2 consists of three subsystems, msys2, mingw32, and mingw64:

  • The mingw32 and mingw64 subsystems are native Windows applications that use Win32 API.
  • The msys2 subsystem provides an emulated mostly-POSIX-compliant environment based on Cygwin.

Each subsystem has an associated "terminal/shell", which is essentially a set of environment variables that allow the subsystems to co-operate properly:

  • MSYS2 MinGW 64-bit, used to build Windows-native 64-bit applications.
  • MSYS2 MinGW 32-bit, used to build Windows-native 32-bit applications.
  • MSYS2 MSYS, used to build POSIX applications.

Refer to the MSYS2 homepage for more detailed information on the MSYS2 subsystems and terminals/shells.

Installing MSYS2

Run Control Panel and look under Systemto check whether you have a 32-bit or a 64-bit version of Windows and chose your preferred way to install MSYS2.

Install MSYS2 with Chocolatey

Chocolatey is a command line package manager for Windows, always run Chocolatey commands in a powershell/cmd with administrator rights (right click on Start menu, select Windows Powershell (Admin) or Command Prompt (Admin)):

  1. Open a powershell (admin) (not a cmd) and copy the official Chocolatey install command to install Chocolatey
  2. In a powershell/cmd (admin) execute the command choco install msys2 -y

As alternative write this text file install_choco_msys2.cmd, right click and select Run as administrator:

@echo off
::https://chocolatey.org/install
::https://chocolatey.org/courses/installation/installing?method=installing-chocolatey?quiz=true

::download and run install.ps1
"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"
::install msys2
choco install msys2 -y

Install MSYS2 with the official installer

  1. Download and start the one-click installer for MSYS2. It's suggested to choose C:\tools\msys64 as installation folder to be compatible with fishtest framework. MSYS2 no longer support an installer for Windows 32-bit, the latest provided is not able to install packages.
  2. The installer runs a MSYS2 MSYS shell as a last step. Update the core packages by typing and executing pacman -Syuu. When finished, close the MSYS2 MSYS shell.

With MSYS2 installed to C:\tools\msys64 your home directory will be C:\tools\msys64\home\<your_username>. Note that within the MSYS2 shell, paths are written in Unix-like way:

  • Windows path: C:\tools\msys64
  • Unix-like path: /c/tools/msys64
  • Windows path: C:\tools\msys64\home
  • Unix-like path: /home or /c/tools/msys64/home

Note that you can also use ls to list the files and folders in a directory, similar to how you would use dir in Windows.

Building Stockfish

  1. Using your favorite text editor, copy and paste the following bash script, calling it makefish.sh:
  • 64-bit Windows
#!/bin/bash
# makefish.sh

# install packages if not already installed
pacman -S --noconfirm --needed unzip make mingw-w64-x86_64-gcc

# download the Stockfish source code
wget -O master.zip https://github.com/official-stockfish/Stockfish/archive/master.zip
unzip -o master.zip
cd Stockfish-master/src
file_nnue=$(grep 'define.*EvalFileDefaultName' evaluate.h | grep -Ewo 'nn-[a-z0-9]{12}.nnue')
ls *.nnue | grep -v ${file_nnue} | xargs -d '\n' -r rm --

# find the CPU architecture
gcc_enabled=$(g++ -Q -march=native --help=target | grep "\[enabled\]")
gcc_arch=$(g++ -Q -march=native --help=target | grep "march")

if [[ "${gcc_enabled}" =~ "-mavx512vnni " && "${gcc_enabled}" =~ "-mavx512dq " && "${gcc_enabled}" =~ "-mavx512f " && "${gcc_enabled}" =~ "-mavx512bw " && "${gcc_enabled}" =~ "-mavx512vl " ]] ; then
  arch_cpu="x86-64-vnni256"
elif [[ "${gcc_enabled}" =~ "-mavx512f " && "${gcc_enabled}" =~ "-mavx512bw " ]] ; then
  arch_cpu="x86-64-avx512"
elif [[ "${gcc_enabled}" =~ "-mbmi2 " && ! "${gcc_arch}" =~ "znver1" && ! "${gcc_arch}" =~ "znver2" ]] ; then
  arch_cpu="x86-64-bmi2"
elif [[ "${gcc_enabled}" =~ "-mavx2 " ]] ; then
  arch_cpu="x86-64-avx2"
elif [[ "${gcc_enabled}" =~ "-mpopcnt " && "${gcc_enabled}" =~ "-msse4.1 " ]] ; then
  arch_cpu="x86-64-modern"
elif [[ "${gcc_enabled}" =~ "-mssse3 " ]] ; then
  arch_cpu="x86-64-ssse3"
elif [[ "${gcc_enabled}" =~ "-mpopcnt " && "${gcc_enabled}" =~ "-msse3 " ]] ; then
  arch_cpu="x86-64-sse3-popcnt"
else
  arch_cpu="x86-64"
fi

# build the fastest Stockfish executable
make -j profile-build ARCH=${arch_cpu} COMP=mingw
strip stockfish.exe
mv stockfish.exe ../../stockfish_${arch_cpu}.exe
make clean
cd
  • 32-bit Windows
#!/bin/bash
# makefish.sh

# install packages if not already installed
pacman -S --noconfirm --needed unzip make mingw-w64-i686-gcc

# download the Stockfish source code
wget -O master.zip https://github.com/official-stockfish/Stockfish/archive/master.zip
unzip -o master.zip
cd Stockfish-master/src
file_nnue=$(grep 'define.*EvalFileDefaultName' evaluate.h | grep -Ewo 'nn-[a-z0-9]{12}.nnue')
ls *.nnue | grep -v ${file_nnue} | xargs -d '\n' -r rm --

# find the CPU architecture
gcc_enabled=$(g++ -Q -march=native --help=target | grep "\[enabled\]")
gcc_arch=$(g++ -Q -march=native --help=target | grep "march")

if [[ "${gcc_enabled}" =~ "-mpopcnt " && "${gcc_enabled}" =~ "-msse4.1 " ]] ; then
  arch_cpu="x86-32-sse41-popcnt"
elif [[ "${gcc_enabled}" =~ "--msse2 " ]] ; then
  arch_cpu="x86-32-sse2"
else
  arch_cpu="x86-32"
fi

# build the fastest Stockfish executable
make -j profile-build ARCH=${arch_cpu} COMP=mingw
strip stockfish.exe
mv stockfish.exe ../../stockfish_${arch_cpu}.exe
make clean
cd
  1. Start a MSYS2 MinGW 64-bit shell (not a MSYS2 MSYS one), C:\tools\msys64\mingw64.exe, or start a MSYS2 MinGW 32-bit shell , C:\tools\msys64\mingw32.exe, to build a 32 bit application.
  2. Navigate to wherever you saved the script (e.g. type and execute cd '/d/Program Files/Stockfish' to navigate to D:\Program Files\Stockfish.)
  3. Run the script by typing and executing bash makefish.sh.

Troubleshooting

If this tutorial will not work on your pc, you may try to change the Windows Security settings in via Windows Security >> App & Browser Control >> Exploit Protection Settings:

  1. Try to turn off "Force randomization for images (Mandatory ASLR)", if this not solve the problem then,
  2. Try to turn off also "Randomize memory allocations (Bottom-up ASLR)" .

Using other MinGW-w64 with MSYS2

To use with MSYS2 a MinGW-w64 built by other projects, simply follow these instructions (Windows 64 bit):

  1. download another version of MinGW-w64, e.g. MinGW-w64 (64-bit) GCC 8.1.0, extract the mingw64 folder renaming it to mingw64-810, copy the folder into C:\msys64, check to have the directory C:\msys64\mingw64-810\bin

  2. build Stockfish writing and executing this bash script

#!/bin/bash
# makefish.sh

# set PATH to use GCC 8.1.0
if [ -d "/mingw64-810/bin" ] ; then
  PATH="/mingw64-810/bin:${PATH}"
else
  echo "folder error"
  exit 1
fi

# download the Stockfish source code
wget -O master.zip https://github.com/official-stockfish/Stockfish/archive/master.zip
unzip master.zip
cd Stockfish-master/src

# find the CPU architecture
# CPU without popcnt and bmi2 instructions (e.g. older than Intel Sandy Bridge)
arch_cpu=x86-64
# CPU with bmi2 instruction (e.g. Intel Haswell or newer)
if [ "$(g++ -Q -march=native --help=target | grep mbmi2 | grep enabled)" ] ; then
  # CPU AMD zen
  if [ "$(g++ -Q -march=native --help=target | grep march | grep 'znver[12]')" ] ; then
    arch_cpu=x86-64-avx2
  else
    arch_cpu=x86-64-bmi2
  fi
# CPU with popcnt instruction (e.g. Intel Sandy Bridge)
elif [ "$(g++ -Q -march=native --help=target | grep mpopcnt | grep enabled)" ] ; then
  arch_cpu=x86-64-modern
fi

# build the Stockfish executable
make profile-build ARCH=${arch_cpu} COMP=mingw
strip stockfish.exe
mv stockfish.exe ../../stockfish_${arch_cpu}.exe
make clean 
cd

To use the compiler in the CLI write and run the script use_gcc810.sh in the user home folder

# set PATH to use GCC 8.1.0
# use this command: source use_gcc810.sh
if [ -d "/mingw64-810/bin" ] ; then
  PATH="/mingw64-810/bin:${PATH}"
else
  echo "folder error"
fi

Advanced topics

How to measure the speedup of Stockfish

To measure the speedup of several builds of Stockfish, use one of these applications:

Note:

  • Don't run other applications when measuring the speedup of Stockfish (i.e. stop fishtest)
  • Run at least 20 benches for each build of Stockfish to have accurate measures
  • A speedup of 0.3% could be meaningless (i.e. within the measurement noise)

You can use also this bash script bench_parallel.sh

#!/bin/bash
_bench () {
${1} << EOF > /dev/null 2>> ${2}
bench 16 1 ${depth} default depth
EOF
}
# _bench function customization example
# setoption name SyzygyPath value C:\table_bases\wdl345;C:\table_bases\dtz345
# bench 128 4 ${depth} default depth

if [[ ${#} -ne 4 ]]; then
cat << EOF
usage: ${0} sf_base sf_test depth n_runs
fast sf_test:
${0} ./stockfish_base ./stockfish_test 13 10
slow sf_test:
${0} ./stockfish_base ./stockfish_test 20 10
EOF
exit 1
fi

sf_base=${1}
sf_test=${2}
depth=${3}
n_runs=${4}

# preload of CPU/cache/memory
(_bench ${sf_base} sf_base0.txt)&
(_bench ${sf_test} sf_test0.txt)& 
wait

# temporary files initialization
: > sf_base0.txt
: > sf_test0.txt
: > sf_temp0.txt

# bench loop: SMP bench with background subshells
for ((k=1; k<=${n_runs}; k++)); do
    printf "run %3d /%3d\n" ${k} ${n_runs}

    # swap the execution order to avoid bias
    if [ $((k%2)) -eq 0 ]; then
        (_bench ${sf_base} sf_base0.txt)&
        (_bench ${sf_test} sf_test0.txt)&
        wait
    else
        (_bench ${sf_test} sf_test0.txt)&
        (_bench ${sf_base} sf_base0.txt)&
        wait
    fi
  done

# text processing to extract nps values
cat sf_base0.txt | grep second | grep -Eo '[0-9]{1,}' > sf_base1.txt
cat sf_test0.txt | grep second | grep -Eo '[0-9]{1,}' > sf_test1.txt

for ((k=1; k<=${n_runs}; k++)); do
    echo ${k} >> sf_temp0.txt
done

printf "\nrun\tsf_base\tsf_test\tdiff\n"
paste sf_temp0.txt sf_base1.txt sf_test1.txt | awk '{printf "%3d  %d  %d  %+d\n", $1, $2, $3, $3-$2}'
paste sf_base1.txt sf_test1.txt | awk '{printf "%d\t%d\t%d\n", $1, $2, $2-$1}' > sf_temp0.txt

# compute: sample mean, 1.96 * std of sample mean (95% of samples), speedup
# std of sample mean = sqrt(NR/(NR-1)) * (std population) / sqrt(NR)
cat sf_temp0.txt | awk '{sum1 += $1 ; sumq1 += $1**2 ;sum2 += $2 ; sumq2 += $2**2 ;sum3 += $3 ; sumq3 += $3**2 } END {printf "\nsf_base = %10d +/- %d\nsf_test = %10d +/- %d\ndiff = %10d +/- %d\nspeedup = %.6f\n\n", sum1/NR , 1.96 * sqrt(sumq1/NR - (sum1/NR)**2)/sqrt(NR-1) , sum2/NR , 1.96 * sqrt(sumq2/NR - (sum2/NR)**2)/sqrt(NR-1) , sum3/NR  , 1.96 * sqrt(sumq3/NR - (sum3/NR)**2)/sqrt(NR-1) , (sum2 - sum1)/sum1 }'

# remove temporary files
rm -f sf_base0.txt sf_test0.txt sf_temp0.txt sf_base1.txt sf_test1.txt

Building Stockfish with Link Time Optimization (LTO)

It is possible to build Stockfish with Link Time Optimization: on Haswell and newer CPU this gave a speedup of 2%, on Ivy Bridge and older CPU at the moment the speedup is very little (0.3% on Ivy Bridge). Without editing the Makefile, the simple way is to use the Linux compile directive COMP=gcc, e.g. for Haswell CPU and newer:

make profile-build ARCH=x86-64-bmi2 COMP=gcc

Due to a bug in MinGW for Windows, it not possible to have at the moment a static LTO build, so copy, along stockfish.exe, these three dll:

  • libgcc_s_seh-1.dll
  • libstdc++-6.dll
  • libwinpthread-1.dll

You find them in:

  • for MSYS: C:\MinGW\mingw64\x86_64-w64-mingw32\lib
  • for MSYS2: C:\msys64\mingw64\bin

Lower the compile time using make with the flag -j <n_jobs>

It is possible to lower the compile time on cpu multi core using make with the flag -j <n_jobs>, where <n_jobs> is the number of jobs (commands) to run simultaneously. The flag -j enables one job for each logical CPU core.

make -j <n_jobs> profile-build ARCH=x86-64-modern COMP=mingw

Building Stockfish optimized for your cpu with -march=native

To get the max speedup for your CPU (1.5% on Ivy Bridge) simply prepend the shell variable CXXFLAGS='-march=native' to the make command. At example, for a CPU Sandy/Ivy Bridge use this command:

CXXFLAGS='-march=native' make -j profile-build ARCH=x86-64-modern COMP=mingw

To view the compiler flags added for your cpu, use this command:

gcc -Q -march=native --help=target | grep -v "\[disabled\]" 

-march=native implies -mtune=native, below a high level explanation of the compiler flags -march and -mtune, view the gcc manual for more technically sound details:

  • -march: determines what instruction sets are used in the binary. An instruction set is the list of commands implemented by the cpu. The generated code may not run at all on processors other than the one indicated.

  • -mtune: determines the cost model that is used when generating code. The cost model describes how long it takes the cpu to do operations. This information is used by the scheduler to decide what operations to use and in what order.

Cross compilation with Ubuntu 18.04

Write and run this script.

#!/bin/bash
# functions to build Stockfish
_build_sf () {
make build ARCH=x86-64$1 COMP=mingw -j                                      
strip stockfish.exe                                                                                                     
mv stockfish.exe ../../stockfish-x64${1}.exe                                                                                
make clean                                                                                                              
}

_build_sf_pgo () {
make profile-build ARCH=x86-64$1 COMP=mingw PGOBENCH="wine ./stockfish.exe bench" -j                                      
strip stockfish.exe                                                                                                     
mv stockfish.exe ../../stockfish-x64${1}-pgo.exe                                                                                
make clean                                                                                                              
}

# full-upgrade and install required packages
sudo apt update && sudo apt full-upgrade -y && sudo apt autoremove -y && sudo apt clean
sudo apt install -y \
  make \
  mingw-w64 \
  git \
  wine64 \
  binutils

# clone Stockfish source code
git clone --single-branch --branch master https://github.com/official-stockfish/Stockfish.git
cd Stockfish/src

# build Stockfish executables
# to speedup the building process you can keep only the section fitting your CPU architecture

# build the binary for CPUs without popcnt and bmi2 instructions (e.g. older than Intel Sandy Bridge)
_build_sf_pgo
  
# build the binary for CPU with popcnt instruction (e.g. Intel Sandy Bridge)
if [ "$(x86_64-w64-mingw32-c++-posix -Q -march=native --help=target | grep mpopcnt | grep enabled)" ] ; then
  _build_sf_pgo -modern
else
  _build_sf -modern
fi
  
# build the binary for CPU with bmi2 instruction (e.g. Intel Haswell or newer)
if [ "$(x86_64-w64-mingw32-c++-posix -Q -march=native --help=target | grep mbmi2 | grep enabled)" ] ; then
  _build_sf_pgo -bmi2
else
  _build_sf -bmi2
fi

Can I use Microsoft Visual Studio to build Stockfish?

Yes, but it is required to explicitly set the stack reserve to avoid crashes. See point 5. below.

If you want to use MSVC to get a "optimized" build, you can change these settings in the IDE:

  1. Add "NDEBUG;USE_POPCNT;USE_PEXT" to preprocessor definitions.
  2. Optimization flags: /O2, /Oi, /Ot, /Oy, /GL
  3. Static link with runtime: /MT.
  4. Disable stack cookies: /GS-.
  5. Set stack reserve to 8388608 in under Linker -> System or use the linker option /STACK:reserve=8388608.
  6. Disable debugging information in compiler/linker.
  7. Make a PGO instrument build(set under General), it should depend on "pgort140.dll" and it probably won't start.
  8. Copy pgort140.dll from "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\Hostx64\x64" to the output folder.
  9. Run bench with the instrument build(very slow) and quit, it should generate "Stockfish.pgd" and "Stockfish!1.pgc".
  10. Make a PGO optimized build(set under General), should show something like:
1>0 of 0 ( 0.0%) original invalid call sites were matched.
1>0 new call sites were added.
1>54 of 4076 (  1.32%) profiled functions will be compiled for speed, and the rest of the functions will be compiled for size
1>18615 of 46620 inline instances were from dead/cold paths
1>4076 of 4076 functions (100.0%) were optimized using profile data
1>14499840744 of 14499840744 instructions (100.0%) were optimized using profile data
  1. Enjoy, local tests show comparable speed to GCC builds.