diff --git a/deploy/README.md b/deploy/README.md index 0a688d8f2..06f7dbf5e 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,6 +1,6 @@ # DTaaS on Linux Operating System -This directory contains code for running DTaaS application +These are installation instructions for running DTaaS application on a Ubuntu Server 22.04 Operating System. The setup requires a machine which can spare 16GB RAM, 8 vCPUs and 50GB Hard Disk space. @@ -9,9 +9,16 @@ A dummy **foo.com** URL has been used for illustration. Please change this to your unique website URL. It is assumed that you are going to serve the application in only HTTPS mode. +A successful installation will create a setup +similar to the one shown in the figure. + +![Single host install](../docs/admin/single-host.png) + Please follow these steps to make this work in your local environment. Download the codebase as zip file into your computer and unzip the same -into a directory named **DTaaS**. The rest of the instructions assume +into a directory named **DTaaS**. +Alternatively, clone this git repository into your computer. +The rest of the instructions assume that your working directory is **DTaaS**. ## Configuration @@ -22,20 +29,28 @@ The first step is to decide on the number of users and their usenames. The traefik gateway configuration has a template for two users. You can modify the usernames in the template to the usernames chosen by you. -### The traefik gateway server +### Traefik gateway server -You can run the Run the Traefik gateway server in both and +You can run the Traefik gateway server in both and HTTPS and HTTPS mode to experience the DTaaS application. The installation guide assumes that you can run the application in HTTPS mode. -The Traefik gateway configuration is at [fileConfig](../config/gateway/fileConfig.yml). -Change `localhost` to `foo.com` and user1/user2 to the usernames chosen by you. +The Traefik gateway configuration is at _[fileConfig](config/gateway/fileConfig.yml)_. +Change `foo.com` to your local hostname and user1/user2 to the usernames chosen by you. **NOTE**: Do not use `http://` or `https://` -in [fileConfig](../config/gateway/fileConfig.yml). +in [fileConfig](config/gateway/fileConfig.yml). #### Authentication +This step requires `htpasswd` commandline utility. If +it is not available on your system, please install the same by using + +```bash +sudo apt-get install -y apache2-utils +``` + +You can now proceed with update of the gateway authentication setup. The dummy username is `foo` and the password is `bar`. Please change this before starting the gateway. @@ -46,24 +61,25 @@ htpasswd deploy/config/gateway/auth password: ``` -The user credentials added in [auth](../config/gateway/auth) should match -the usernames in [fileConfig](../config/gateway/fileConfig.yml). +The user credentials added in _[config/gateway/auth](config/gateway/auth)_ +should match the usernames in +_[config/gateway/fileConfig](config/gateway/fileConfig.yml)_. -## Configure lib microservice +## Lib microservice The library microservice requires configuration. -A template of this configuration file is given in _config/lib_ file. +A template of this configuration file is given in _[config/lib](config/lib)_ file. Please modify this file as per your needs. The first step in this configuration is to prepare the a filesystem for users. An example file system in `files/` directory. You can rename the top-level user1/user2 to the usernames chosen by you. -Add an environment file named .env in lib for the library microservice. +Add an environment file named `.env` in lib for the library microservice. An example `.env` file is given below. The simplest possibility is to use `local` mode with the following example. The filepath is the absolute filepath to `files/` directory. -You can copy this configuration into _config/lib_ file to get started. +You can copy this configuration into _[config/lib](config/lib)_ file to get started. ```env PORT='4001' @@ -74,7 +90,7 @@ APOLLO_PATH='/lib' GRAPHQL_PLAYGROUND='true' ``` -## Configure React Client Website +## React Client Website ### Gitlab OAuth application @@ -101,11 +117,11 @@ the [Authentication page](client/auth.md). ### Update Client Config -Change the React website configuration in _deploy/config/client/env.js_. +Change the React website configuration in _[config/client/env.js](config/client/env.js)_. ```js window.env = { - REACT_APP_ENVIRONMENT: 'dev', + REACT_APP_ENVIRONMENT: 'prod', REACT_APP_URL: 'https://foo.com/', REACT_APP_URL_BASENAME: 'dtaas', REACT_APP_URL_DTLINK: '/lab', @@ -141,3 +157,12 @@ You can run this script multiple times until the installation is successful. ## Access the application Now you should be able to access the DTaaS application at: _https://foo.com_ + +## References + +Image sources: [Ubuntu logo](https://logodix.com/linux-ubuntu), +[Traefik logo](https://www.laub-home.de/wiki/Traefik_SSL_Reverse_Proxy_f%C3%BCr_Docker_Container), +[ml-workspace](https://github.com/ml-tooling/ml-workspace), +[nodejs](https://www.metachris.com/2017/01/how-to-install-nodejs-7-on-ubuntu-and-centos/), +[reactjs](https://krify.co/about-reactjs/), +[nestjs](https://camunda.com/blog/2019/10/nestjs-tx-email/) diff --git a/deploy/config/gateway/fileConfig.yml b/deploy/config/gateway/fileConfig.yml index 9190f365b..efd35bb28 100644 --- a/deploy/config/gateway/fileConfig.yml +++ b/deploy/config/gateway/fileConfig.yml @@ -24,11 +24,11 @@ http: - basic-auth service: user2 - vis: + libms: entryPoints: - http - rule: 'Host(`foo.com`) && PathPrefix(`/vis`)' - service: grafana + rule: 'Host(`foo.com`) && PathPrefix(`/lib`)' + service: libms # Middleware: Basic authentication @@ -55,8 +55,7 @@ http: servers: - url: "http://localhost:8091" - grafana: + libms: loadBalancer: servers: - - url: "http://localhost:3000" - + - url: "http://localhost:4001" \ No newline at end of file diff --git a/deploy/services/README.md b/deploy/services/README.md new file mode 100644 index 000000000..62ce64329 --- /dev/null +++ b/deploy/services/README.md @@ -0,0 +1,42 @@ +# Platform Services + +It is recommended to install certain third-party software for use +by digital twins running inside the DTaaS software. + +The installation scripts in this directory install: + +* **Influx** time-series database and dashboard service +* **Grafana** visualization and dashboard service +* **RabbitMQ** AMQP broker and its' management interface +* Eclipse Mosquitto **MQTT** broker + +## Configure and Install + +The first step in installation is to specify the config of the services. +There are two configuration files. The __services.yml__ contains most +of configuration settings. The __mqtt-default.conf__ file contains +the MQTT listening port. Update these two config files before proceeding +with the installation of the services. + +```bash +yarn install +node services.js +``` + +## Use + +After the installation is complete, you can see the following services active +at the following ports / URLs. + +| service | external url | +|:---|:---| +| Influx | services.foo.com | +| Grafana | services.foo.com:3000 | +| RabbitMQ Broker | services.foo.com:5672 | +| RabbitMQ Broker Management Website | services.foo.com:15672 | +| MQTT Broker | services.foo.com:1883 | +|| + +The firewall and network access settings of corporate / cloud network need to be +configured to allow external access to the services. Otherwise the users of DTaaS +will not be able to utilize these services from their user workspaces. diff --git a/deploy/services/mqtt-default.conf b/deploy/services/mqtt-default.conf new file mode 100644 index 000000000..d1e2a74d3 --- /dev/null +++ b/deploy/services/mqtt-default.conf @@ -0,0 +1,2 @@ +listener 1883 +password_file /etc/mosquitto/passwd \ No newline at end of file diff --git a/deploy/services/package.json b/deploy/services/package.json new file mode 100644 index 000000000..7378a6aad --- /dev/null +++ b/deploy/services/package.json @@ -0,0 +1,13 @@ +{ + "name": "@into-cps-association/dtaas-install-services", + "version": "0.0.1", + "description": "Install platform services for the DTaaS software", + "author": "Prasad Talasila", + "private": false, + "type": "module", + "dependencies": { + "chalk": "^5.3.0", + "execa": "^8.0.1", + "js-yaml": "^4.1.0" + } +} diff --git a/deploy/vagrant/two-machine/services.js b/deploy/services/services.js similarity index 76% rename from deploy/vagrant/two-machine/services.js rename to deploy/services/services.js index a4284bcd4..4fc960b55 100755 --- a/deploy/vagrant/two-machine/services.js +++ b/deploy/services/services.js @@ -15,7 +15,7 @@ const sleep = (ms) => }); try { - console.log(chalk.blue("Load services configuration")); + log(chalk.blue("Load services configuration")); config = await yaml.load(fs.readFileSync('services.yml', 'utf8')); log(chalk.green("configuration loading is successful and config is a valid yaml file")); } catch (e) { @@ -84,7 +84,7 @@ await $$`docker run -d \ grafana/grafana:10.1.4`; log(chalk.green("Grafana server docker container started successfully")); -console.log(chalk.blue("Wait one minute for Grafana server to bootstrap")); +log(chalk.blue("Wait one minute for Grafana server to bootstrap")); await sleep(60000); //60 seconds await $$`docker exec grafana grafana-cli admin reset-admin-password ${grafanaConfig.password}`; @@ -102,17 +102,37 @@ try { await $$`docker rm rabbitmq-server`; } catch (e) { } -//await $$`docker run -d --name rabbitmq-server -p 5672:5672 -p 15672:15672 rabbitmq:3-management`; + log(chalk.green("Start RabbitMQ server docker container")); await $$`docker run -d --name rabbitmq-server \ -p ${rabbitmqConfig.ports.main}:5672 \ -p ${rabbitmqConfig.ports.management}:15672 rabbitmq:3-management`; log(chalk.green("RabbitMQ server docker container started successfully\n")); -console.log(chalk.blue("Wait 2 minutes for RabbitMQ server to bootstrap")); +log(chalk.blue("Wait 2 minutes for RabbitMQ server to bootstrap")); await sleep(120000); //120 seconds let args = [rabbitmqConfig.username, rabbitmqConfig.password]; -//console.log(chalk.blue("Add ${rabbitmqConfig.username} user and give permission to ${rabbitmqConfig.vhost} vhost")); +log(chalk.blue( + "Add %s user and give permission to %s vhost"), + rabbitmqConfig.username, rabbitmqConfig.vhost); await $$`docker exec rabbitmq-server rabbitmqctl add_user ${args}`; await $$`docker exec rabbitmq-server rabbitmqctl set_permissions -p ${rabbitmqConfig.vhost} ${rabbitmqConfig.username} ".*" ".*" ".*"`; + +//--------------- +log(chalk.blue("Install and start MQTT server")); +const mqttConfig = config.services.mqtt; + +log(chalk.blue("Attempt to install mosquitto MQTT server using apt-get package manager")); +await $$`sudo apt-get install -y mosquitto mosquitto-clients`; +log(chalk.blue("Create user account for %s in MQTT server"), mqttConfig.username); +await $$`sudo sudo mosquitto_passwd -c -b /etc/mosquitto/passwd ${mqttConfig.username} ${mqttConfig.password}`; +await $$`sudo chown root:mosquitto /etc/mosquitto/passwd`; +await $$`sudo chmod 660 /etc/mosquitto/passwd`; + +log(chalk.blue("Set MQTT listening port configuration")); +await $$`sudo cp mqtt-default.conf /etc/mosquitto/conf.d/default.conf`; +await $$`sudo chmod 664 /etc/mosquitto/conf.d/default.conf`; +await $$`sudo chown root:mosquitto /etc/mosquitto/conf.d/default.conf`; +await $$`sudo systemctl restart mosquitto`; +await $$`sudo systemctl status mosquitto`; diff --git a/deploy/vagrant/two-machine/services.yml b/deploy/services/services.yml similarity index 79% rename from deploy/vagrant/two-machine/services.yml rename to deploy/services/services.yml index 89fa3d3d6..1ce7191f9 100644 --- a/deploy/vagrant/two-machine/services.yml +++ b/deploy/services/services.yml @@ -17,3 +17,6 @@ services: password: "dtaas1357" # need to have letters and numbers datapath: "/home/prasad/git/prasadtalasila/DTaaS/deploy/vagrant/two-machine/influx" #no spaces in the path port: 80 + mqtt: # usernames and passwords are in deploy/config/services/mqtt/config/password + username: "dtaas" + password: "dtaas" diff --git a/deploy/single-script-install.sh b/deploy/single-script-install.sh index 6cf781007..4ff80e6be 100755 --- a/deploy/single-script-install.sh +++ b/deploy/single-script-install.sh @@ -116,7 +116,7 @@ else git clone https://github.com/INTO-CPS-Association/DTaaS.git DTaaS cd DTaaS || exit git fetch --all - git checkout feature/distributed-demo + git checkout release-v0.3 fi TOP_DIR=$(pwd) diff --git a/deploy/vagrant/two-machine/README.md b/deploy/vagrant/two-machine/README.md index 2bfb4cb32..393189e04 100644 --- a/deploy/vagrant/two-machine/README.md +++ b/deploy/vagrant/two-machine/README.md @@ -15,10 +15,11 @@ server1. The _workspaces.sh_ contains installation commands for provisioning user workspaces. If you desire to have more users, you need to modify this shell script. -The default installation setup also installs InfluxDB, Grafana and RabbitMQ -services on server2. If you would like to install more services, you can create -shell scripts to install the same on server2. If you have these scripts ready, -you can place them in this directory and invoke them from _services.sh_ script. +The default installation setup also installs InfluxDB, Grafana, RabbitMQ +and MQTT services on server2. If you would like to install more services, +you can create +scripts to install the same on server2. If you have these scripts ready, +you can place them in this directory and invoke them from _services.js_ script. ## Create Base Vagrant Box @@ -60,37 +61,19 @@ The fields to update are: ## Launch platform default services -RabbitMQ, Grafana and InfluxDB services are provisioned on this server. -InfluxDB webUI will be available at: _services.foo.com_. -The RabbitMQ service and its management interface shall be available -at 5672 and 15672 TCP ports respectively. -The Grafana service shall be available at TCP port 3000. - -The firewall and network access settings of corporate / cloud network need to be -configured to allow external access to the services. Otherwise the users of DTaaS -will not be able to utilize these services from their user workspaces. +RabbitMQ, Grafana, InfluxDB and MQTT services are provisioned on this server. Execute the following commands from terminal to start the machine. ```bash vagrant up --provision services vagrant ssh services -wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/two-machine/services.sh -bash services.sh wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/route.sh sudo bash route.sh ``` -After the server is up and running, you can see the following services active -within server2. - -| service | external url | -|:---|:---| -| Influx visualization service | services.foo.com | -| Grafana visualization service | services.foo.com:3000 | -| RabbitMQ communication service | services.foo.com:5672 | -| RabbitMQ management service | services.foo.com:15672 | -|| +Follow the instructions in [services](../../services/README.md) to install +the platform default services on this vagrant machine. ## Launch DTaaS application @@ -99,6 +82,11 @@ Execute the following commands from terminal ```bash vagrant up --provision dtaas vagrant ssh dtaas +``` + +Once inside the vagrant box, execute + +```bash wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/route.sh sudo bash route.sh ``` diff --git a/deploy/vagrant/two-machine/package.json b/deploy/vagrant/two-machine/package.json deleted file mode 100644 index 67281a69c..000000000 --- a/deploy/vagrant/two-machine/package.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "type": "module", - "dependencies": { - "chalk": "^5.3.0", - "execa": "^8.0.1", - "js-yaml": "^4.1.0" - } -} diff --git a/docs/FAQ.md b/docs/FAQ.md index b22d56fa1..94c45cc87 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -10,7 +10,7 @@ ## General Questions - + ??? Question "What is DTaaS?" @@ -216,9 +216,7 @@ The core feature of DTaaS software is to help users create DTs from assets already available in the library. - ![Create Library Assets](./user/servers/lib/author.png) - However, it is possible for users to take advantage of services available in their workspace to install asset authoring tools in their own workspace. @@ -227,4 +225,45 @@ Thus any licensed software tools installed in their workspace is only available to them. - +## GDPR Concerns + +??? Question "Does your platform adhere to GDPR compliance standards? If so, how?" + + The DTaaS software platform does not store any personal information + of users. It only stores username to identify users and these + usernames do not contain enough information to deduce the true + identify of users. + +??? Question "Which security measures are deployed? How is data encrypted (if exists)?" + + The default installation requires a HTTPS terminating reverse proxy server + from user to the DTaaS software installation. The administrators of DTaaS + software can also install HTTPS certificates into the application. + The codebase can generate HTTPS application and the users also have + the option of installing their own certificates obtained from + certification agencies such as LetsEncrypt. + +??? Question "What security measures does your cloud provider offer?" + + The current installation of DTaaS software runs on Aarhus University + servers. The university network offers firewall access control to servers + so that only permitted user groups have access to the network and + physical access to the server. + +??? Question "How is user access controlled and authenticated?" + + There is a two-level authentication mechanism in place in each default + installation of DTaaS. The first-level is HTTP basic authentication + over secure HTTPS connection. The second-level is the OAuth PKCE + authentication flow for each user. The OAuth authentication is provider + by a Gitlab instance. The DTaaS does not store the account and + authentication information of users. + +??? Question "Does you platform manage personal data? How is data classified and tagged based on the sensitivity? Who has access to the critical data?" + + The platform does not store personal data of users. + +??? Question "How are identities and roles managed within the platform?" + + There are two roles for users on the platform. One is the administrator + and the other one is user. The user roles are managed by the administrator. diff --git a/docs/admin/client/CLIENT.md b/docs/admin/client/CLIENT.md index a96a5d0bd..dbc2e0437 100644 --- a/docs/admin/client/CLIENT.md +++ b/docs/admin/client/CLIENT.md @@ -99,10 +99,10 @@ one [ML Workspace](https://github.com/ml-tooling/ml-workspace) serving the following routes. ```js -https:foo.com//lab -https:foo.com//terminals/main -https:foo.com//tools/vnc/?password=vncpassword -https:foo.com//tools/vscode/ +https://foo.com//lab +https://foo.com//terminals/main +https://foo.com//tools/vnc/?password=vncpassword +https://foo.com//tools/vscode/ ``` The `username` is the user workspace created using ML Workspace docker container. diff --git a/docs/admin/host.md b/docs/admin/host.md index 71a90a23d..4ec9ec66c 100644 --- a/docs/admin/host.md +++ b/docs/admin/host.md @@ -38,9 +38,9 @@ The first step is to decide on the number of users and their usenames. The traefik gateway configuration has a template for two users. You can modify the usernames in the template to the usernames chosen by you. -### The traefik gateway server +### Traefik gateway server -You can run the Run the Traefik gateway server in both +You can run the Traefik gateway server in both HTTP and HTTPS mode to experience the DTaaS application. The installation guide assumes that you can run the application in HTTPS mode. @@ -80,7 +80,7 @@ password: The user credentials added in _deploy/config/gateway/auth_ should match the usernames in _deploy/config/gateway/fileConfig.yml_. -## Configure lib microservice +## Lib microservice The library microservice requires configuration. A template of this configuration file is given in _deploy/config/lib_ file. @@ -90,6 +90,7 @@ The first step in this configuration is to prepare the a filesystem for users. An example file system in `files/` directory. You can rename the top-level user1/user2 to the usernames chosen by you. +Add an environment file named `.env` in lib for the library microservice. An example `.env` file is given below. The simplest possibility is to use `local` mode with the following example. The filepath is the absolute filepath to `files/` directory. @@ -104,7 +105,7 @@ APOLLO_PATH='/lib' GRAPHQL_PLAYGROUND='true' ``` -## Configure React Client Website +## React Client Website ### Gitlab OAuth application @@ -131,7 +132,7 @@ Change the React website configuration in _deploy/config/client/env.js_. ```js window.env = { - REACT_APP_ENVIRONMENT: 'dev', + REACT_APP_ENVIRONMENT: 'prod', REACT_APP_URL: 'https://foo.com/', REACT_APP_URL_BASENAME: 'dtaas', REACT_APP_URL_DTLINK: '/lab', diff --git a/docs/admin/servers/lib/LIB-MS.md b/docs/admin/servers/lib/LIB-MS.md index 8cee47ae5..49974732c 100644 --- a/docs/admin/servers/lib/LIB-MS.md +++ b/docs/admin/servers/lib/LIB-MS.md @@ -96,6 +96,16 @@ yarn build # build the application yarn start # start the application ``` +#### Config flag + +If the environment file is named something other than `.env`, +the filename must be specifed with the command `-c, --config `, +when starting the application. For instance, + +```sh +yarn start -c ".env.development" +``` + You can press `Ctl+C` to halt the application. If you wish to run the microservice in the background, use diff --git a/docs/admin/services.md b/docs/admin/services.md index 96362b054..676a799ba 100644 --- a/docs/admin/services.md +++ b/docs/admin/services.md @@ -3,125 +3,67 @@ The DTaaS software platform uses third-party software services to provide enhanced value to users. -InfluxDB, RabbitMQ and Grafana are default services +InfluxDB, Grafana, RabbitMQ and Mosquitto are default services integrated into the DTaaS software platform. -_The InfluxDB service requires a dedicated hostname. The management -interface of RabbitMQ service requires a dedicated hostname as well._ - -Thus successful installation of these services -is dependent on your ability to use -multiple hostnames for different services. You can download the required -services using the docker commands. - -```sh -docker pull grafana/grafana:10.1.4 -docker pull influxdb:2.7 -docker pull telegraf:1.28.2 -docker pull rabbitmq:3-management -docker pull eclipse-mosquitto:2 -``` - -The two-machine vagrant deployment scenario installs the RabbitMQ, Grafana, and -InfluxDB services on the second vagrant machine. - -If you would like to install some of these services for native OS -installation or single vagrant machine, you can do this as well. - -## RabbitMQ - -Start the RabbitMQ service with - -```bash -docker run -d \ - --name rabbitmq-server \ - -p 15672:15672 -p 5672:5672 \ - rabbitmq:3-management -``` +## Pre-requisites -Users and the vhosts need to be setup on the server. Sample commands to do so are: +All these services run on raw TCP/UDP ports. Thus a direct network +access to these services is required for both the DTs running inside +the DTaaS software and the PT located outside the DTaaS software. -```bash -docker exec rabbitmq-server rabbitmqctl add_user -docker exec rabbitmq-server rabbitmqctl set_permissions -p "/" ".*" ".*" ".*" -``` - -The RabbitMQ service requires raw TCP/UDP protocol access to network. -The default Traefik configuration of DTaaS does not permit -TCP/UDP traffic. There are two possible choices here: +There are two possible choices here: * Configure Traefik gateway to permit TCP/UDP traffic -* Bypass Traefik altogether for RabbitMQ service +* Bypass Traefik altogether Unless you are an informed user of Traefik, we recommend bypassing traefik -for RabbitMQ service. +and provide raw TCP/UDP access to these services from the Internet. -## Grafana - -Grafana service can run well behind Traefik gateway. Here is a sample docker -command to run Grafana service at port 3000: - -```bash -docker run -d \ - -p 3000:3000 \ - --name=grafana \ - -e "GF_SERVER_SERVE_FROM_SUB_PATH=true" \ - -e "GF_SERVER_DOMAIN=localhost" \ - -e "GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s" \ - -e "GF_AUTH_BASIC_ENABLED=false" \ - -e "GF_AUTH_PROXY_ENABLED=false" \ - -e "GF_SECURITY_ADMIN_PASSWORD=DTaaSGrafana" \ - -e "GF_SECURITY_ALLOW_EMBEDDING=true" \ - -e "GF_SECURITY_ALLOW_EMBEDDING=true" \ - -e "GF_AUTH_ANONYMOUS_ENABLED=true" \ - -e "GF_AUTH_ANONYMOUS_ORG_NAME=Main" \ - -e "GF_AUTH_ANONYMOUS_ORG_ROLE=Editor" \ - -e "GF_USERS_ALLOW_SIGN_UP=false" \ - -e "GF_FEATURE_TOGGLES_ENABLE=publicDashboards" \ - -e "GF_PATHS_CONFIG=/etc/grafana/grafana.ini" \ - -e "GF_PATHS_DATA=/var/lib/grafana" \ - -e "GF_PATHS_HOME=/usr/share/grafana" \ - -e "GF_PATHS_LOGS=/var/log/grafana" \ - -e "GF_PATHS_PLUGINS=/var/lib/grafana/plugins" \ - -e "GF_PATHS_PROVISIONING=/etc/grafana/provisioning" \ - -e "HOME=/home/grafana" \ - grafana/grafana -printf "Complete the setup from GUI" -``` +_The InfluxDB service requires a dedicated hostname. The management +interface of RabbitMQ service requires a dedicated hostname as well._ -The user credentials have also been set in the command as: +Grafana service can run well behind Traefik gateway. The default Traefik +configuration makes permits access to Grafana at URL: http(s): _foo.com/vis_. -**username**: admin +## Configure and Install -**password**: DTaaSGrafana +If you have not cloned the DTaaS git repository, cloning would be +the first step. +In case you already have the codebase, you can skip the cloning step. +To clone, do: -Remember to change these credentials before starting the docker container. +```bash +git clone https://github.com/into-cps-association/DTaaS.git +cd DTaaS/deploy/services +``` -## InfluxDB +The next step in installation is to specify the config of the services. +There are two configuration files. The __services.yml__ contains most +of configuration settings. The __mqtt-default.conf__ file contains +the MQTT listening port. Update these two config files before proceeding +with the installation of the services. -The barebones InfluxDB service can be installed using: +Now continue with the installation of services. ```bash -INFLUXDB_DATA="${PWD}/data/influxdb2" -mkdir -p "$INFLUXDB_DATA" - -# Remember to change the settings -docker run -d -p 80:8086 \ - --name influxdb24 \ - -v "$INFLUXDB_DATA/data":/var/lib/influxdb2 \ - -v "$INFLUXDB_DATA/config":/etc/influxdb2 \ - -e DOCKER_INFLUXDB_INIT_MODE=setup \ - -e DOCKER_INFLUXDB_INIT_USERNAME=dtaas \ - -e DOCKER_INFLUXDB_INIT_PASSWORD=dtaas1357 \ - -e DOCKER_INFLUXDB_INIT_ORG=dtaas \ - -e DOCKER_INFLUXDB_INIT_BUCKET=dtaas \ - influxdb:2.4 +yarn install +node services.js ``` -The user credentials have also been set in the command as: +## Use -**username**: dtaas +After the installation is complete, you can see the following services active +at the following ports / URLs. -**password**: dtaas1357 +| service | external url | +|:---|:---| +| Influx | services.foo.com | +| Grafana | services.foo.com:3000 | +| RabbitMQ Broker | services.foo.com:5672 | +| RabbitMQ Broker Management Website | services.foo.com:15672 | +| MQTT Broker | services.foo.com:1883 | -Remember to change these credentials before starting the docker container. +The firewall and network access settings of corporate / cloud network need to be +configured to allow external access to the services. Otherwise the users of DTaaS +will not be able to utilize these services from their user workspaces. diff --git a/docs/admin/trial.md b/docs/admin/trial.md index 39dc168be..15da97711 100644 --- a/docs/admin/trial.md +++ b/docs/admin/trial.md @@ -43,7 +43,7 @@ for getting the Gitlab OAuth application details. ## Install ```bash -wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/single-script-install.sh +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/release-v0.3/deploy/single-script-install.sh bash single-script-install.sh ``` diff --git a/docs/admin/vagrant/single-machine.md b/docs/admin/vagrant/single-machine.md index 9d068ad1e..876017509 100644 --- a/docs/admin/vagrant/single-machine.md +++ b/docs/admin/vagrant/single-machine.md @@ -69,7 +69,7 @@ Set a cronjob inside the vagrant virtual machine to remote the conflicting default route. ```bash -wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/route.sh +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/release-v0.3/deploy/vagrant/route.sh sudo bash route.sh ``` diff --git a/docs/admin/vagrant/two-machines.md b/docs/admin/vagrant/two-machines.md index f9332b89d..37ab817e0 100644 --- a/docs/admin/vagrant/two-machines.md +++ b/docs/admin/vagrant/two-machines.md @@ -82,33 +82,15 @@ The fields to update are: ## Installation Steps -### Launch DTaaS Platform Default Services - -RabbitMQ, Grafana, InfluxDB and MQTT services are provisioned on this server. -InfluxDB and visualization service will be available at: _services.foo.com_. -The Grafana service shall be available at TCP port 3000. -The MQTT service shall be available at TCP port 1833. -The RabbitMQ service and its management interface shall be available at -5672 and 15672 TCP ports respectively. +The installation instructions are given separately for each vagrant machine. -The firewall and network access settings of corporate / cloud network -need to be configured to allow external access to the services. -Otherwise the users of DTaaS will not be able to utilize these -services from their user workspaces. - -Execute the following commands from terminal to start the machine. +### Launch DTaaS Platform Default Services -```bash -vagrant up --provision services -vagrant ssh services -wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/two-machine/services.sh -bash services.sh -wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/route.sh -sudo bash route.sh -``` +Follow the installation guide for [services](../services.md) +to install the DTaaS platform services. -After the server is up and running, -you can see the following services active within server2. +After the services are up and running, +you can see the following services active within server2 (_services.foo.com_). | service | external url | |:---|:---| @@ -125,7 +107,7 @@ Execute the following commands from terminal ```bash vagrant up --provision dtaas vagrant ssh dtaas -wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/feature/distributed-demo/deploy/vagrant/route.sh +wget https://raw.githubusercontent.com/INTO-CPS-Association/DTaaS/release-v0.3/deploy/vagrant/route.sh sudo bash route.sh ``` diff --git a/docs/index.md b/docs/index.md index 52daafde2..8eb4f20cc 100644 --- a/docs/index.md +++ b/docs/index.md @@ -17,7 +17,7 @@ It is also possible to share the services offered by one DT with other users. There is an overview of the software available in the form of [slides](https://odin.cps.digit.au.dk/into-cps/dtaas/assets/DTaaS-short-intro.pdf), [video](https://odin.cps.digit.au.dk/into-cps/dtaas/assets/videos/DTaaS-short-intro.mp4), -and [feature walkthrough](https://odin.cps.digit.au.dk/into-cps/dtaas/assets/videos/dtaas-v0.2.0-demo.mp4). +and [feature walkthrough](https://odin.cps.digit.au.dk/into-cps/dtaas/assets/videos/dtaas-v0.3-demo.mp4). ## License diff --git a/docs/redirect-page.html b/docs/redirect-page.html index b7b4bd918..d53a7ed1d 100644 --- a/docs/redirect-page.html +++ b/docs/redirect-page.html @@ -60,6 +60,11 @@ Online PDF + + Version 0.3.0 + Online + PDF + Version 0.2.0 Online diff --git a/docs/thanks.md b/docs/thanks.md index f6872380b..0ca23f27d 100644 --- a/docs/thanks.md +++ b/docs/thanks.md @@ -15,6 +15,6 @@ Henrik Ejersbo, Tanusree Roy, Farshid Naseri Digital Twin as a Service (DTaaS): A Platform for Digital Twin Developers and Users [arXiv preprint arXiv:2305.07244](https://arxiv.org/abs/2305.07244). -1. Astitva Sehgal -1. Tanusree Roy -1. Farshid Naseri +1. Astitva Sehgal for developer and example documentation. +1. Tanusree Roy and Farshid Naseri for asking interesting questions + that ended up in FAQs. diff --git a/servers/config/gateway/dynamic/fileConfig.yml b/servers/config/gateway/dynamic/fileConfig.yml index dbb7319af..86c2d0ba7 100644 --- a/servers/config/gateway/dynamic/fileConfig.yml +++ b/servers/config/gateway/dynamic/fileConfig.yml @@ -24,47 +24,38 @@ http: - basic-auth service: user2 - vis: - entryPoints: - - http - rule: 'Host(`localhost`) && PathPrefix(`/vis`)' - service: grafana - - lib: + libms: entryPoints: - http rule: 'Host(`localhost`) && PathPrefix(`/lib`)' - service: lib + service: libms + # Middleware: Basic authentication middlewares: basic-auth: basicAuth: - usersFile: '/etc/traefik/auth' + usersFile: "/etc/traefik/auth" removeHeader: true + services: dtaas: loadBalancer: servers: - - url: 'http://localhost:4000' + - url: "http://localhost:4000" user1: loadBalancer: servers: - - url: 'http://localhost:8090' + - url: "http://localhost:8090" user2: loadBalancer: servers: - - url: 'http://localhost:8091' - - grafana: - loadBalancer: - servers: - - url: 'http://localhost:3000' + - url: "http://localhost:8091" - lib: + libms: loadBalancer: servers: - - url: 'http://localhost:4001' + - url: "http://localhost:4001" \ No newline at end of file diff --git a/servers/lib/README.md b/servers/lib/README.md index 7bd339f49..d1f2fa520 100644 --- a/servers/lib/README.md +++ b/servers/lib/README.md @@ -48,6 +48,12 @@ APOLLO_PATH='/lib' or '' GRAPHQL_PLAYGROUND='false' or 'true' ``` +The `LOCAL_PATH` variable is the absolute filepath to the +location of the local directory which will be served to users +by the Library microservice. + +The `GITLAB_URL`, `GITLAB_GROUP` and `TOKEN` are only relevant for `gitlab` mode. + The `TOKEN` should be set to your GitLab Group access API token. For more information on how to create and use your access token, [gitlab page](https://docs.gitlab.com/ee/user/group/settings/group_access_tokens.html). @@ -55,12 +61,31 @@ For more information on how to create and use your access token, Once you've generated a token, copy it and replace the value of `TOKEN` with your token for the gitlab group, -You can adjust other config values as per your local setup. + +Replace the default values the appropriate values for your setup. + +**NOTE**: + +1. When \__MODE=local_, only _LOCAL_PATH_ is used. + Other environment variables are unused. +1. When _MODE=gitlab_, _GITLAB_URL, TOKEN_, + and _GITLAB_GROUP_ are used; _LOCAL_PATH_ is unused. + +## User Commands This config is saved `.env` file by convention. The __libms__ looks for `.env` file in the working directory from which it is run. If the environment file is named something other than `.env`, -the filename must be explicitly provided to __libms__ at the execution time. + +the filename must be specifed with the command `-c, --config `, +when starting the application. For instance, + +```sh +yarn start -c ".env.development" +``` + +You can press `Ctl+C` to halt the application. +If you wish to run the microservice in the background, use ## :rocket: Use