Merge branch 'release-v0.3prtest' into pr-239

deploy/README.md

@@ -1,6 +1,6 @@
 # DTaaS on Linux Operating System
 
-This directory contains code for running DTaaS application
+These are installation instructions for running DTaaS application
 on a Ubuntu Server 22.04 Operating System.
 The setup requires a machine which can spare 16GB
 RAM, 8 vCPUs and 50GB Hard Disk space.
@@ -9,9 +9,16 @@ A dummy **foo.com** URL has been used for illustration.
 Please change this to your unique website URL.
 It is assumed that you are going to serve the application in only HTTPS mode.
 
+A successful installation will create a setup
+similar to the one shown in the figure.
+
+![Single host install](../docs/admin/single-host.png)
+
 Please follow these steps to make this work in your local environment.
 Download the codebase as zip file into your computer and unzip the same
-into a directory named **DTaaS**. The rest of the instructions assume
+into a directory named **DTaaS**.
+Alternatively, clone this git repository into your computer.
+The rest of the instructions assume
 that your working directory is **DTaaS**.
 
 ## Configuration
@@ -22,20 +29,28 @@ The first step is to decide on the number of users and their usenames.
 The traefik gateway configuration has a template for two users.
 You can modify the usernames in the template to the usernames chosen by you.
 
-### The traefik gateway server
+### Traefik gateway server
 
-You can run the Run the Traefik gateway server in both and
+You can run the Traefik gateway server in both and
 HTTPS and HTTPS mode to experience the DTaaS application.
 The installation guide assumes that you can run the application in HTTPS mode.
 
-The Traefik gateway configuration is at [fileConfig](../config/gateway/fileConfig.yml).
-Change `localhost` to `foo.com` and user1/user2 to the usernames chosen by you.
+The Traefik gateway configuration is at _[fileConfig](config/gateway/fileConfig.yml)_.
+Change `foo.com` to your local hostname and user1/user2 to the usernames chosen by you.
 
 **NOTE**: Do not use `http://` or `https://`
-in [fileConfig](../config/gateway/fileConfig.yml).
+in [fileConfig](config/gateway/fileConfig.yml).
 
 #### Authentication
 
+This step requires `htpasswd` commandline utility. If
+it is not available on your system, please install the same by using
+
+```bash
+sudo apt-get install -y apache2-utils
+```
+
+You can now proceed with update of the gateway authentication setup.
 The dummy username is `foo` and the password is `bar`.
 Please change this before starting the gateway.
 
@@ -46,24 +61,25 @@ htpasswd deploy/config/gateway/auth <first_username>
 password: <your password>
 ```
 
-The user credentials added in [auth](../config/gateway/auth) should match
-the usernames in [fileConfig](../config/gateway/fileConfig.yml).
+The user credentials added in _[config/gateway/auth](config/gateway/auth)_
+should match the usernames in
+_[config/gateway/fileConfig](config/gateway/fileConfig.yml)_.
 
-## Configure lib microservice
+## Lib microservice
 
 The library microservice requires configuration.
-A template of this configuration file is given in _config/lib_ file.
+A template of this configuration file is given in _[config/lib](config/lib)_ file.
 Please modify this file as per your needs.
 
 The first step in this configuration is to prepare the a filesystem for users.
 An example file system in `files/` directory.
 You can rename the top-level user1/user2 to the usernames chosen by you.
 
-Add an environment file named .env in lib for the library microservice.
+Add an environment file named `.env` in lib for the library microservice.
 An example `.env` file is given below.
 The simplest possibility is to use `local` mode with the following example.
 The filepath is the absolute filepath to `files/` directory.
-You can copy this configuration into _config/lib_ file to get started.
+You can copy this configuration into _[config/lib](config/lib)_ file to get started.
 
 ```env
 PORT='4001'
@@ -74,7 +90,7 @@ APOLLO_PATH='/lib'
 GRAPHQL_PLAYGROUND='true'
 ```
 
-## Configure React Client Website
+## React Client Website
 
 ### Gitlab OAuth application
 
@@ -101,11 +117,11 @@ the [Authentication page](client/auth.md).
 
 ### Update Client Config
 
-Change the React website configuration in _deploy/config/client/env.js_.
+Change the React website configuration in _[config/client/env.js](config/client/env.js)_.
 
 ```js
 window.env = {
-  REACT_APP_ENVIRONMENT: 'dev',
+  REACT_APP_ENVIRONMENT: 'prod',
   REACT_APP_URL: 'https://foo.com/',
   REACT_APP_URL_BASENAME: 'dtaas',
   REACT_APP_URL_DTLINK: '/lab',
@@ -141,3 +157,12 @@ You can run this script multiple times until the installation is successful.
 ## Access the application
 
 Now you should be able to access the DTaaS application at: _<http:>https://foo.com</http:>_
+
+## References
+
+Image sources: [Ubuntu logo](https://logodix.com/linux-ubuntu),
+[Traefik logo](https://www.laub-home.de/wiki/Traefik_SSL_Reverse_Proxy_f%C3%BCr_Docker_Container),
+[ml-workspace](https://github.com/ml-tooling/ml-workspace),
+[nodejs](https://www.metachris.com/2017/01/how-to-install-nodejs-7-on-ubuntu-and-centos/),
+[reactjs](https://krify.co/about-reactjs/),
+[nestjs](https://camunda.com/blog/2019/10/nestjs-tx-email/)

deploy/config/gateway/fileConfig.yml

@@ -24,11 +24,11 @@ http:
         - basic-auth
       service: user2
 
-    vis:
+    libms:
       entryPoints:
         - http
-      rule: 'Host(`foo.com`) && PathPrefix(`/vis`)'
-      service: grafana
+      rule: 'Host(`foo.com`) && PathPrefix(`/lib`)'
+      service: libms
 
 
   # Middleware: Basic authentication
@@ -55,8 +55,7 @@ http:
         servers:
           - url: "http://localhost:8091"
 
-    grafana:
+    libms:
       loadBalancer:
         servers:
-          - url: "http://localhost:3000"
-
+          - url: "http://localhost:4001"

deploy/services/README.md

@@ -0,0 +1,42 @@
+# Platform Services
+
+It is recommended to install certain third-party software for use
+by digital twins running inside the DTaaS software.
+
+The installation scripts in this directory install:
+
+* **Influx** time-series database and dashboard service
+* **Grafana** visualization and dashboard service
+* **RabbitMQ** AMQP broker and its' management interface
+* Eclipse Mosquitto **MQTT** broker
+
+## Configure and Install
+
+The first step in installation is to specify the config of the services.
+There are two configuration files. The __services.yml__ contains most
+of configuration settings. The __mqtt-default.conf__ file contains
+the MQTT listening port. Update these two config files before proceeding
+with the installation of the services.
+
+```bash
+yarn install
+node services.js
+```
+
+## Use
+
+After the installation is complete, you can see the following services active
+at the following ports / URLs.
+
+| service | external url |
+|:---|:---|
+| Influx | services.foo.com |
+| Grafana | services.foo.com:3000 |
+| RabbitMQ Broker | services.foo.com:5672 |
+| RabbitMQ Broker Management Website | services.foo.com:15672 |
+| MQTT Broker | services.foo.com:1883 |
+||
+
+The firewall and network access settings of corporate / cloud network need to be
+configured to allow external access to the services. Otherwise the users of DTaaS
+will not be able to utilize these services from their user workspaces.

deploy/services/mqtt-default.conf

@@ -0,0 +1,2 @@
+listener 1883
+password_file /etc/mosquitto/passwd

deploy/services/package.json

@@ -0,0 +1,13 @@
+{
+  "name": "@into-cps-association/dtaas-install-services",
+  "version": "0.0.1",
+  "description": "Install platform services for the DTaaS software",
+  "author": "Prasad Talasila",
+  "private": false,
+  "type": "module",
+  "dependencies": {
+    "chalk": "^5.3.0",
+    "execa": "^8.0.1",
+    "js-yaml": "^4.1.0"
+  }
+}

deploy/vagrant/two-machine/services.js → deploy/services/services.js

@@ -15,7 +15,7 @@ const sleep = (ms) =>
   });
 
   try {
-  console.log(chalk.blue("Load services configuration"));
+  log(chalk.blue("Load services configuration"));
   config = await yaml.load(fs.readFileSync('services.yml', 'utf8'));
   log(chalk.green("configuration loading is successful and config is a valid yaml file"));
 } catch (e) {
@@ -84,7 +84,7 @@ await $$`docker run -d \
   grafana/grafana:10.1.4`;
 log(chalk.green("Grafana server docker container started successfully"));
 
-console.log(chalk.blue("Wait one minute for Grafana server to bootstrap"));
+log(chalk.blue("Wait one minute for Grafana server to bootstrap"));
 await sleep(60000);  //60 seconds
 
 await $$`docker exec grafana grafana-cli admin reset-admin-password ${grafanaConfig.password}`;
@@ -102,17 +102,37 @@ try {
   await $$`docker rm rabbitmq-server`;  
 } catch (e) {
 }
-//await $$`docker run -d --name rabbitmq-server -p 5672:5672  -p 15672:15672 rabbitmq:3-management`;
+
 log(chalk.green("Start RabbitMQ server docker container"));
 await $$`docker run -d --name rabbitmq-server \
   -p ${rabbitmqConfig.ports.main}:5672 \
   -p ${rabbitmqConfig.ports.management}:15672 rabbitmq:3-management`;
 log(chalk.green("RabbitMQ server docker container started successfully\n"));
 
-console.log(chalk.blue("Wait 2 minutes for RabbitMQ server to bootstrap"));
+log(chalk.blue("Wait 2 minutes for RabbitMQ server to bootstrap"));
 await sleep(120000);  //120 seconds
 
 let args = [rabbitmqConfig.username, rabbitmqConfig.password];
-//console.log(chalk.blue("Add ${rabbitmqConfig.username} user and give permission to ${rabbitmqConfig.vhost} vhost"));
+log(chalk.blue(
+  "Add %s user and give permission to %s vhost"),
+  rabbitmqConfig.username, rabbitmqConfig.vhost);
 await $$`docker exec rabbitmq-server rabbitmqctl add_user ${args}`;
 await $$`docker exec rabbitmq-server rabbitmqctl set_permissions -p ${rabbitmqConfig.vhost} ${rabbitmqConfig.username} ".*" ".*" ".*"`;
+
+//---------------
+log(chalk.blue("Install and start MQTT server"));
+const mqttConfig = config.services.mqtt;
+
+log(chalk.blue("Attempt to install mosquitto MQTT server using apt-get package manager"));
+await $$`sudo apt-get install -y mosquitto mosquitto-clients`;
+log(chalk.blue("Create user account for %s in MQTT server"), mqttConfig.username);
+await $$`sudo sudo mosquitto_passwd -c -b /etc/mosquitto/passwd ${mqttConfig.username} ${mqttConfig.password}`;
+await $$`sudo chown root:mosquitto /etc/mosquitto/passwd`;
+await $$`sudo chmod 660 /etc/mosquitto/passwd`;
+
+log(chalk.blue("Set MQTT listening port configuration"));
+await $$`sudo cp mqtt-default.conf /etc/mosquitto/conf.d/default.conf`;
+await $$`sudo chmod 664 /etc/mosquitto/conf.d/default.conf`;
+await $$`sudo chown root:mosquitto /etc/mosquitto/conf.d/default.conf`;
+await $$`sudo systemctl restart mosquitto`;
+await $$`sudo systemctl status mosquitto`;

deploy/vagrant/two-machine/services.yml → deploy/services/services.yml

@@ -17,3 +17,6 @@ services:
         password: "dtaas1357"   # need to have letters and numbers
         datapath: "/home/prasad/git/prasadtalasila/DTaaS/deploy/vagrant/two-machine/influx" #no spaces in the path
         port: 80
+    mqtt:  # usernames and passwords are in deploy/config/services/mqtt/config/password
+        username: "dtaas"
+        password: "dtaas"

deploy/single-script-install.sh

@@ -116,7 +116,7 @@ else
   git clone https://github.com/INTO-CPS-Association/DTaaS.git DTaaS
   cd DTaaS || exit
   git fetch --all
-  git checkout feature/distributed-demo
+  git checkout release-v0.3
 fi
 
 TOP_DIR=$(pwd)

deploy/vagrant/two-machine/README.md

@@ -15,10 +15,11 @@ server1. The _workspaces.sh_ contains installation commands for provisioning
 user workspaces. If you desire to have more users, you need to modify this
 shell script.
 
-The default installation setup also installs InfluxDB, Grafana and RabbitMQ
-services on server2. If you would like to install more services, you can create
-shell scripts to install the same on server2. If you have these scripts ready,
-you can place them in this directory and invoke them from _services.sh_ script.
+The default installation setup also installs InfluxDB, Grafana, RabbitMQ
+and MQTT services on server2. If you would like to install more services,
+you can create
+scripts to install the same on server2. If you have these scripts ready,
+you can place them in this directory and invoke them from _services.js_ script.
 
 ## Create Base Vagrant Box
 
@@ -60,37 +61,19 @@ The fields to update are:
 
 ## Launch platform default services
 
-RabbitMQ, Grafana and InfluxDB services are provisioned on this server.
-InfluxDB webUI will be available at: _services.foo.com_.
-The RabbitMQ service and its management interface shall be available
-at 5672 and 15672 TCP ports respectively.
-The Grafana service shall be available at TCP port 3000.
-
-The firewall and network access settings of corporate / cloud network need to be
-configured to allow external access to the services. Otherwise the users of DTaaS
-will not be able to utilize these services from their user workspaces.
+RabbitMQ, Grafana, InfluxDB and MQTT services are provisioned on this server.
 
 Execute the following commands from terminal to start the machine.
 
 ```bash
 vagrant up --provision services
 vagrant ssh services
-wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/two-machine/services.sh
-bash services.sh
 wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/route.sh
 sudo bash route.sh
 ```
 
-After the server is up and running, you can see the following services active
-within server2.
-
-| service | external url |
-|:---|:---|
-| Influx visualization service | services.foo.com |
-| Grafana visualization service | services.foo.com:3000 |
-| RabbitMQ communication service | services.foo.com:5672 |
-| RabbitMQ management service | services.foo.com:15672 |
-||
+Follow the instructions in [services](../../services/README.md) to install
+the platform default services on this vagrant machine.
 
 ## Launch DTaaS application
 
@@ -99,6 +82,11 @@ Execute the following commands from terminal
 ```bash
 vagrant up --provision dtaas
 vagrant ssh dtaas
+```
+
+Once inside the vagrant box, execute
+
+```bash
 wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/route.sh
 sudo bash route.sh
 ```