Making a Windows Standalone Executable

[answerquest]

Collecting notes here from the trials to make a standalone windows executable.

[answerquest]

Got it 👍
It was a long-winded exploration, but condensing the process:

• Installed Miniconda from https://conda.io/miniconda.html (python3, win 64-bit)
• Opened Start Menu > Anaconda prompt and navigated to program folder
• Started and activated a virtual environment:

conda create -n exe python=3
activate exe

• Installed pyinstaller and associated packages, and put in a fix for pandas

pip install pandas pyinstaller pypiwin32
echo hiddenimports = ['pandas._libs.tslibs.timedeltas'] > %CONDA_PREFIX%\Lib\site-packages\PyInstaller\hooks\hook-pandas.py

• Installed remaining packages using pip:
  pip install tornado xmltodict tinydb pycryptodomex

• Ran the program to check if its all working properly:
  python GTFSManager.py

• Initiated PyInstaller to build windows standalone binary of the program
  pyinstaller GTFSManager.py --upx-dir=C:\DESIGN\upx394w --clean --icon=favicon.ico

• Once build was completed, went to ./dist/GTFSManager/ where all the binaries are kept. Right-click-dragged GTFSManager.exe to create its shortcut two levels up, at the original program folder.

• Opened up the properties of the shortcut. Removed the "Start in" field and replaced "Target" field with:
  %windir%\system32\cmd.exe /c start dist\GTFSmanager\GTFSmanager.exe

• This makes the shortcut have relative path.

• Renamed the shortcut to "GTFSManager".

• If the icon changed, open its properties again and set the icon to the favicon.ico file in the program folder.

• Cleanup: deleted the __pycache__ and build folders as they're no longer required after the binaries have been built.

Notes

• Explanation for --upx-dir=C:\DESIGN\upx394w : UPX is a compression library used to compress executables. To make pyinstaller use this compression, we have to download UPX from https://github.com/upx/upx/releases to a local folder (C:\DESIGN\upx394w in my case) and supply the path for upx-dir in the command.

• Explanation for shortcut : The .exe created by PyInstaller is in the dist\GTFSManager\ folder under the main program folder. It is accompanied by several dlls, binaries etc which make up the standalone python distribution and the required modules. When we run it, like the python script from which it was made, it assumes all the supporting data, html, js, DB files and folders are right next to it in the same folder as itself. So, directly double-clicking on GTFSManager.exe in the dist/GTFSManager folder FAILS. To solve this, we must either move all the files down to this binaries folder (and that creates a lot of confusion as we'll be mixing our data and frontend files in with the binaries), OR we run the program from outside, from the original program folder that is two levels up. I chose to do the latter.

• Note: PyInstaller has a --onefile option that combines all the binaries into one single large .exe, but it doesn't work in all Windows OS's, and some other drivers need to be installed. That takes away the "standalone" status. And anyways one big exe that actually stores several files inside is more susceptible to corruption by viruses, damage etc and any problems arising become harder to trace to root cause. So I opted out of that option. Instead I just made a shortcut to the .exe in the main folder. So, letting all the multiple binaries, dlls etc be there but reside out of sight in the dist folder.

• This process needed a small change in the main program : pycryptodomex package had to be used instead of pycryptodome. This import statement had to be changed: from Crypto.PublicKey import RSA changed to from Cryptodome.PublicKey import RSA.

• There were a lot of other failed trials and initially the distribution size was over 600MB, and the cryptography feature for password wasn't working. After all the optimization the binaries size is a more manageable 30MB - a 20x reduction.

Reference links

• https://stackoverflow.com/a/48846546/4355695 (optimizing binary by making virtual env, fixing pandas issue in pyinstaller and using pip only to install dependencies keep the size under control)
• https://github.com/upx/upx/releases (UPX compresssion)
• https://stackoverflow.com/a/50009769/4355695 (about Cryptography)

[answerquest]

Addendum: To place a shortcut on the desktop, we need to set absolute paths of where the program folder is, in both Target and Start-in fields.

The Target of course needs to point to the exe at [folder]/dist/GTFSManager/GTFSManager.exe,
But Start-in needs to point to [folder] so that the associated files, web pages etc can be picked up by the program.

This might help in automating things: https://superuser.com/questions/631243/set-start-in-directory-for-a-shortcut-using-batch-commands

Re-opening this issue to keep it open for more ideas.

[answerquest]

A little note on why the distribution size was initially 20 times bigger (600 MB): I had installed pandas module using conda install pandas. Lesson learnt after 3 days of headbanging: Whatever you do, do NOT use conda for installing pandas, just use pip install pandas. (Now most places you'll find a reverse recco but bear with me, and note, this recco is valid only for current conda version.)

It loops in a package named mkl as a dependency. But in reality it is just an optional module rarely used by numpy, which is a pandas dependency, and I don't know if any pandas command ever directly uses mkl. It's over a 100 MB when you download the package. By the time pyinstaller is done with it, it ends up including a battalion of DLL files that weigh in at over 600MB. All for a module that your program will most likely (seriously, internet, let me know if you find an actual use case) never use, which piggybacked in as a dependency of a dependency.

Here's the link to the stackoverflow answer that saved the windows version plan: https://stackoverflow.com/a/48846546/4355695

Update: Issue filed with conda after digging a bit more into mkl: conda-forge/numpy-feedstock#84

Update: Got a recommendation from conda devs to still use conda to install numpy/pandas as it solves some problems not covered by pip, but use this alternative command that avoids mkl:
conda install -c conda-forge numpy
Will give it a shot in next release. For now the windows binary is working fine.

[answerquest]

Some explanation for the point in the notes about pycryptodome, pycyptodomex, pycrypto : https://stackoverflow.com/a/50009769/4355695

[answerquest]

Update, Oct 2018:

• Created a single-exe executable version on a win7 32bit Virtual OS (virtualbox).
• pandas hiddenimport hook wasn't done successfully by the previous method, but by specifying it in the command itself it worked (see commands below).
• The HDF5 read/write through pandas needs another module, tables. While on an older windows system it needed VC++ runtime installed, in other regular systems this wasn't needed. This module is added to the list in requirements.txt.
• pyinstaller's --onefile option worked after installing a Universal C Runtime driver. With this, the executables came in one GTFSManager.exe that could be located on top folder. No more need for placing shortcut to dist/.. path and other complications. So now the end user too has a straightforward .exe to double-click on, and they can right-click and pin it to start menu, add a shortcut to desktop etc like other normal windows software.
• the output size was ok (~40mb) even without UPX compression, and after zipping the folder it became 30mb, so decided to go without compression so that the Standalone failing on Windows 10 64-bit #78 issue of vcruntime140.dll gets avoided.
• after the build process it's safe to delete the three folders created by pyinstaller: build, dist, __pycache__

Commands for creating, on anaconda/miniconda prompt, with current folder being the program folder

conda create -n exe python=3.6 --yes
activate exe
pip install pypiwin32 pyinstaller
pip install -r requirements.txt
pyinstaller --hidden-import pandas._libs.tslibs.timedeltas --clean --icon=favicon.ico --onefile GTFSManager.py
move "dist\GTFSManager.exe" .\
rmdir /s /q "dist" "build" "__pycache__"

last 2 lines: generated exe is move to top, and extra folders created by pyinstaller in build process are removed.

Notes for users:

Universal C Runtime driver requirement

This program requires a small driver called "Universal C Runtime". It is usually covered in Windows Update and is already present in win10, but in case your system doesn't have it, please install it from here: https://support.microsoft.com/en-gb/help/2999226/update-for-universal-c-runtime-in-windows
and then run the program. You may need to restart your system for the driver to be fully installed.

Without this, on running the exe you may get a popup error sayng: Entry point not found: The procedure entry point ucrtbase.abort could not be located in the dynamic link library api-ms-win-crt-runtime-l1-1-0.dll

Browser

On starting, the program will open a dos box for console output, and spawn a browser tab to load the static GTFS Manager interface. We recommend using Chrome or Chromium browser, else Firefox. If the tool opens up in Internet Explorer (because that's still set as your defaul browser), then please copy the URL (like localhost:5000) and open it in Chrome / Chromium / Firefox.