Os/cpp browser example

.github/workflows/browser-cpp-example.yml

@@ -0,0 +1,41 @@
+name: Browser Cpp Example
+
+on:
+  push:
+    branches:
+      - os/cpp_BrowserExample
+
+jobs: 
+
+  build: 
+    name: Build the blueprint browser package, create a docker image from it and run some tests
+    runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        node-version: [12.x]
+
+    steps:
+    - name: Checkout Repository code 
+      uses: actions/checkout@v2
+    - name: Install Node ${{ matrix.node-version }}
+      uses: actions/setup-node@v1
+      with:
+          node-version: ${{ matrix.node-version }}
+    - name: Install Yarn
+      run : npm install yarn
+    - name: Run Yarn
+      run : yarn
+    - name: Bundle Blueprint Browser
+      run : yarn browser bundle
+    - name: Package Blueprint Browser
+      run : yarn browser package
+    - name: Create image
+      run : |
+       docker build --no-cache -t cpp_image -f ./applications/browser/Dockerfile .
+       docker images
+    - name: Run image
+      run : docker run -d -p 3000:3000 cpp_image
+    - name: Set a timeout
+      run : node -e "setTimeout(process.exit, 10000, 0)"
+    - name: Run headless test
+      run : yarn browser test

.gitignore

@@ -1,11 +1,10 @@
 .DS_Store
-**/node_modules
-**/.browser_modules
-**/dist
-**/lib
-**/src-gen
-**/webpack.config.js
-**/gen-webpack.config.js
-**/plugins
-**/tsconfig.tsbuildinfo
-*.log
+node_modules
+.browser_modules
+dist
+lib
+src-gen
+gen-webpack.config.js
+plugins
+tsconfig.tsbuildinfo
+*.log

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "submodules/pkg"]
+	path = submodules/pkg
+	url = https://github.com/marechal-p/pkg.git
+	branch = exec-argv

applications/browser/.gitignore

@@ -0,0 +1,2 @@
+/bundled
+/builtins

applications/browser/Dockerfile

@@ -0,0 +1,5 @@
+FROM ubuntu:20.04
+ARG DEBIAN_FRONTEND=noninteractive
+COPY ./applications/browser/dist /home/blueprint_cpp
+RUN apt-get update && apt-get -y install build-essential clangd-12 git
+ENTRYPOINT ["/home/blueprint_cpp/blueprint.exe", "-h", "0.0.0.0"]

applications/browser/README.md

@@ -0,0 +1,99 @@
+# Theia Blueprint Web
+## Setup
+
+This package defines a Browser Theia Application based on Blueprint.
+
+In order to bundle it and package it as a distributable executable, a few
+transformations are performed:
+
+1. The `gen-webpack.config.js` script was copied to `web-webpack.config.js` and
+   converted to work with Webpack v5.
+2. A new `node-webpack.config.js` script was added to bundle all of the Node
+   application bits together under `bundled/`. Quite a lot of magic is happening
+   in that config, which points to modifications to do to the framework in order
+   to better support packaging of the node version. See `TODO` below.
+3. We are using `pkg` on the bundle to make it into an executable. Since
+   everything cannot be packed into the final exe, folders like `lib`,
+   `builtins` and `node_modules` must be copied on the side.
+
+We still have to deal with a residual `node_modules` because of 2 packages:
+
+1. `drivelist` uses the `bindings` package which doesn't seem to work when
+   bundled or packaged, so we leave it out of the executable.
+2. `vscode-ripgrep` pulls a compiled version of `ripgrep` and it has to be
+   accessible on the disk for the system to spawn it.
+
+## Submodules
+
+`pkg` has a bug that prevents scripts from forking with some `execArgv` options.
+I implemented a workaround on a fork and opened a PR for it as well, see
+https://github.com/vercel/pkg/pull/1157.
+
+Until this is merged, I added my fork of `pkg` as a submodule to make use of it.
+
+You need to make sure that the submodule is initiliazed when working on Theia
+Blueprint Web:
+
+```sh
+git submodule update --init
+```
+
+# TODO
+
+## Blueprint
+
+### Remove `@theia/git` (optional)
+
+If I understand correctly, there is a Git VS Code extension out there that can
+replace Theia's git extension. It would simplify the Webpack config ever so
+slightly to use the VS Code extension instead.
+
+## Theia Framework
+
+### Refactor the way the frontend is served so that it can be customized (recommended)
+
+We expect to run the backend from `<root>/src-gen/backend/main.js` and we refer
+to the frontend files as `'../../lib'` relative to that script. When packaged,
+the backend must serve files located at `'./lib'` instead. Since our generators
+are currently hard-coded this is problematic.
+
+### Provide a module loader script to avoid direct calls to `require` (mandatory)
+
+This is especially an issue with dynamic requires. Webpack tries its best to
+statically bundle everything together, and we need a special configuration to
+handle dynamic `require` calls (`require(someVar)`). Extracting and factorizing
+this function into its own module will ease packaging since we will be able to
+provide a module replacement when bundling the application.
+
+I considered using DI instead and have some `ModuleLoader` component injected in
+places that do dynamic imports, but it appears that we need to cover places
+where DI is not available... Webpack provides a way to replace imported modules,
+so it would make sense to use that strategy here where we would import a dynamic
+import function from a separate file, and replace that file with the proper
+implementation when packaging.
+
+### Expose/declare entrypoints (optional)
+
+The backend spawns various child processes, and we need to know precisely what
+is spawned so that we can tell Webpack to create the right entrypoints for it.
+Backend modules should declare those to make it easier to package.
+
+### Use Webpack v5 (recommended)
+
+In order to split and re-use chunks between the various entrypoints we need
+to move from Webpack v4 to Webpack v5.
+
+See https://github.com/webpack/webpack/pull/8575/commits/f446bf8a9b6b718977d24ed402b8a1884d663622
+
+### Remove references to `process.env` in browser code (recommended)
+
+It surprised me to find this. Moving to Webpack v5 exposed this oddity in
+`@theia/terminal/src/browser/terminal-preferences.ts`...
+
+### Provide a better way to detect the application's installation (recommended)
+
+We currently rely on `process.cwd()` which is a pretty bad way of guessing where
+an application lives. "Current working directory" represents in fact the place
+where the program was invoked from, it has nothing to do with the application's
+installation location. We should look into a better heuristic, or at least a
+customizable one.

applications/browser/example_cpp/README.md

@@ -0,0 +1,18 @@
+# Small C++ project for the Blueprint Browser Image cpp example
+## Description
+
+This c++ file can be used after building the Blueprint Browser Docker Image to verify that the development environment works correctly. 
+
+## How to use it
+
+After building the Blueprint Browser Docker Image, you can follow these instructions :
+
+1. Assuming that you have just finished to build the image from the Dockerfile in the `browser` directory, you can go to the example_cpp directory (the one that contains this README) using `cd example_cpp`.
+2. Run the following command : `docker run -it -p 3000:3000 -v "$(pwd)":/home/hello_c [image-name]`
+You should see the Blueprint IDE on your `localhost:3000`. 
+3. Create a workspace in the Blueprint browser target by opening the `example_cpp` folder containing the 
+c++ file.
+4. You can execute the program by entering in the terminal (must be opened at the same location) these
+following commands : 
+    4.1. `g++ hello.cpp -o hello`
+    4.2. `./hello`

applications/browser/example_cpp/hello.cpp

@@ -0,0 +1,14 @@
+#include <iostream>
+#include <string>
+
+using namespace std;
+
+int main()
+{   
+    string name = "";
+    cout << "What is your name" << endl;
+    cin >> name;
+    cout << "Hello" << " " << name << " !" << endl;
+
+    return 0;
+}

applications/browser/node-webpack.config.js

@@ -0,0 +1,174 @@
+const path = require('path');
+const webpack = require('webpack');
+const CopyPlugin = require('copy-webpack-plugin');
+const { ModifySourcePlugin } = require('modify-source-webpack-plugin');
+const TerserPlugin = require('terser-webpack-plugin');
+
+/** @type {import('webpack').Configuration['mode']} */
+const mode = 'production';
+
+/** @type {import('webpack').Configuration} */
+module.exports = {
+    mode,
+    target: 'node',
+    devtool: mode === 'development' ? 'source-map' : false,
+    // Since theia starts child processes we need to make sure that each
+    // entry point is individually reachable by the system.
+    // This list has to be updated based on what the included Theia extensions
+    // are trying to spawn, this is has to be hand made for now:
+    entry: {
+        // Main entry point of the Theia application backend:
+        'blueprint': require.resolve('./src-gen/backend/main'),
+        // Theia's IPC mechanism:
+        'ipc-bootstrap': require.resolve('@theia/core/lib/node/messaging/ipc-bootstrap'),
+        // VS Code extension support:
+        'plugin-host': require.resolve('@theia/plugin-ext/lib/hosted/node/plugin-host'),
+        'backend-init-theia': {
+            import: require.resolve('@theia/plugin-ext/lib/hosted/node/scanners/backend-init-theia'),
+            library: {
+                type: 'commonjs2'
+            }
+        },
+        'plugin-vscode-init': {
+            import: require.resolve('@theia/plugin-ext-vscode/lib/node/plugin-vscode-init'),
+            library: {
+                type: 'commonjs2'
+            }
+        },
+        // NSFW service:
+        'nsfw-watcher': {
+            import: require.resolve('@theia/filesystem/lib/node/nsfw-watcher'),
+            library: {
+                type: 'commonjs2'
+            }
+        },
+        // Git service:
+        'git-locator-host': {
+            import: require.resolve('@theia/git/lib/node/git-locator/git-locator-host'),
+            library: {
+                type: 'commonjs2'
+            }
+        }
+    },
+    output: {
+        filename: '[name].js',
+        path: path.resolve(__dirname, 'bundled')
+    },
+    optimization: {
+        // Split and reuse code across the various entry points:
+        splitChunks: {
+            chunks: 'all'
+        },
+        minimize: true,
+        minimizer: [
+            new TerserPlugin({
+                exclude: /^(lib|builtins)\//
+            })
+        ]
+    },
+    resolve: {
+        // Default order causes an issue with deepmerge that exposes both an esm
+        // and cjs implementation. We expect to import cjs but webpack is linking
+        // the esm instead, putting the implementation behind a "default" field.
+        mainFields: ['main', 'module']
+    },
+    node: {
+        global: false,
+        __filename: false,
+        __dirname: false
+    },
+    externals: {
+        // Don't bundle drivelist and instead try to require it from disk as cjs.
+        drivelist: 'commonjs2 drivelist',
+        child_process: 'commonjs2 child_process'
+    },
+    plugins: [
+        new webpack.NormalModuleReplacementPlugin(/^vscode-ripgrep$/, path.resolve('replacements/vscode-ripgrep.js')),
+        new webpack.DefinePlugin({
+            '_THEIA_BLUEPRINT_PACKAGE': JSON.stringify(require('./package.json'))
+        }),
+        // This is where madness begins. We have this ipc-bootstrap mechanism that is spawned
+        // as a child process and proceeds to require a script as passed from environment variables.
+        // The issue is that Webpack replaces every require call with its own __webpack_require__,
+        // which in turn always throws when a dynamic import is done.
+        // Since it is bad taste to taylor our sources to include __non_webpack_require__ which is
+        // an escape hatch from Webpack's parsing, we'll use this plugin to replace the require
+        // call by that Webpack magic method, on the fly. Erk.
+        new ModifySourcePlugin({
+            rules: [
+                {
+                    test: /(ipc-bootstrap|vscode-debug-adapter-contribution|hosted-instance-manager|hosted-plugins-manager|plugin-host-rpc|grammars-reader|plugin-theia-directory-handler|plugin-vscode-directory-handler)\.js$/,
+                    modify(src) {
+                        return src.replace(/ require\((?!["'])/g, ' __non_webpack_require__(');
+                    }
+                },
+                {
+                    test: /plugin-vscode-init\.js$/,
+                    modify(src) {
+                        return src.replace(' module = require(', ' module = __non_webpack_require__(');
+                    }
+                },
+                {
+                    test: /plugin-host-rpc\.js$/,
+                    modify(src) {
+                        return src.replace(/require\.cache/g, '__non_webpack_require__.cache');
+                    }
+                }
+            ]
+        }),
+        // Webpack trips on the places where those modules are required.
+        // Since we'll never reach the code paths where they actually are required at runtime,
+        // it is safe to completely ignore them. Webpack will throw an error if they are required.
+        new webpack.IgnorePlugin({
+            checkResource: (resource) => [
+                'node-ssh',
+                'vertx'
+            ].includes(resource)
+        }),
+        new CopyPlugin({
+            patterns: [
+                // Copy over ripgrep's binaries
+                {
+                    from: path.resolve(require.resolve('vscode-ripgrep/package.json'), '../bin/*').replace(/\\/g, '/'),
+                    to: 'node_modules/vscode-ripgrep'
+                },
+                // #region drivelist copy
+                // drivelist relies on the bindings package to find its own native module (.node),
+                // which confuses webpack and the final output js. The fix implemented in this config
+                // is to leave drivelist out of the bundle, and copy only the relevant files from it.
+                {
+                    from: path.resolve(require.resolve('drivelist/package.json'), '../js/**/*.js').replace(/\\/g, '/'),
+                    to: 'node_modules/drivelist/'
+                },
+                ...[
+                    'drivelist/build/Release/drivelist.node',
+                    'drivelist/package.json'
+                ].map(file => ({
+                    from: require.resolve(file).replace(/\\/g, '/'),
+                    to: 'node_modules/' + file,
+                })),
+                // #endregion drivelist copy
+            ]
+        })
+    ],
+    module: {
+        rules: [
+            // Make sure we can still find and load our native addons.
+            // Since drivelist is specifically handled before, we'll skip it here:
+            {
+                test: /(?!drivelist)\.node$/,
+                // test: /\.node$/,
+                loader: 'node-loader',
+                options: {
+                    name: 'native/[name].[ext]'
+                }
+            },
+            // jsonc-parser exposes its UMD implementation by default, which
+            // confuses Webpack leading to missing js in the bundles.
+            {
+                test: /node_modules[\\/](jsonc-parser)/,
+                loader: 'umd-compat-loader'
+            }
+        ]
+    }
+};