The documentation uses a special build of Sandcastle Help File Builder (SHFB).
To build the documentation from the command line, no install is required. Just run the following:
build.cmd Documentation
If you want to edit the documentation using Visual Studio, you must first install the custom build of Sandcastle Help File Builder and then install the custom Sandcastle Help File Builder Visual Studio extension.