Merge pull request #1 from tejones/develop-pl

Training, Prediction Service, Recommendation Service, Store UI and more

.gitignore

@@ -21,3 +21,23 @@
 
 # virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
 hs_err_pid*
+
+*.ipynb_checkpoints
+*__pycache__
+**/.git/
+**/.idea/
+**/.vscode/
+*.iws
+
+
+**/__pycache__/
+*.py[cod]
+*$py.class
+
+*.so
+
+**/.venv*
+**/venv/
+
+*.priv.*
+*.priv

README.md

@@ -1 +1,39 @@
-# retailstoreofthefuture
+# Retail Store of the Future
+
+Disclaimer! This solution was created for demo purposes only. It contains simplifications and must not be used for production purposes!
+
+## Solution diagram and components description
+
+![Solution basic diagram](documentation/images/basic_diagram.png)
+
+### Training
+
+The training part is made in the jupyter notebooks using Intel DAAL libraries.
+
+### Prediction service
+
+Prediction service handles the model.
+
+It provides a REST API that can be called to retrieve predictions for coupons.
+
+Check [Prediction service README](prediction-service/README.md) for details.
+
+### Recommendation service
+
+Recommendation service listens for MQTT requests.
+
+When the entry event occurs it calls (TODO) the central resource for client data. The component stores the data in a cache - database (TODO).
+
+When the focus event occurs the component gets client and coupon data from the cache (TODO) and calls prediction service.
+The result of the prediction is pushed to MQTT.
+
+Check [Recommendation service README](recommendation-service/README.md) for details.
+MQTT topics description and schema can also be found there.
+
+### Visualization app
+
+The application was made for demo purposes. [More details int app's README](visualization-app/README.md).
+
+### Customers simulator
+
+This simulator was made for demo purposes. See more details in the [README file](scenario-player/README.md)

amq-broker/README.md

@@ -0,0 +1,47 @@
+# Deployment of AMQ Broker
+
+
+## Red Hat Integration - AMQ Broker
+Make sure the operator is installed in your namespace
+
+## Generate certificates
+
+
+Generate certificates with the following command and fill all required certs information:
+
+IMPORTANT: For all passwords use `password`
+
+For `"Trust this certificate? [no]` prompt type `yes`
+```
+$ ./gen-certs.sh
+```
+
+Create secret with the following command:
+```
+$ oc create secret generic ex-aao-amqp-secret \
+--from-file=broker.ks=./certs/broker.ks \
+--from-file=client.ts=./certs/broker.ts \
+--from-literal=keyStorePassword=password \
+--from-literal=truststorePassword=password
+```
+
+In this case we'll use the same certificates for MQTT as for AMQP
+```
+$ oc create secret generic ex-aao-mqtt-secret \
+--from-file=broker.ks=./certs/broker.ks \
+--from-file=client.ts=./certs/broker.ts \
+--from-literal=keyStorePassword=password \
+--from-literal=truststorePassword=password
+```
+
+## Deploy brokers 
+Apply broker and address:
+```
+$ oc apply -f cr/
+```
+
+Verify the pods are up and running
+```
+$ oc get pods
+```
+

amq-broker/cr/address.yaml

@@ -0,0 +1,8 @@
+apiVersion: broker.amq.io/v2alpha1
+kind: ActiveMQArtemisAddress
+metadata:
+  name: ex-aao-address-foo
+spec:
+  addressName: sample.address
+  queueName: sample.address
+  routingType: anycast

amq-broker/cr/broker.yaml

@@ -0,0 +1,29 @@
+apiVersion: broker.amq.io/v2alpha2
+kind: ActiveMQArtemis
+metadata:
+ name: ex-aao
+spec:
+  deploymentPlan:
+    size: 3
+    image: registry.redhat.io/amq7/amq-broker:7.6
+    requireLogin: false
+  adminUser: admin
+  adminPassword: admin
+  console:
+    expose: true
+  acceptors:
+  - name: amqp
+    protocols: amqp
+    port: 5672
+    sslEnabled: true
+    sslSecret: ex-aao-amqp-secret
+    verifyHost: false
+    expose: true
+  - name: mqtt
+    protocols: mqtt
+    port: 8883
+    sslEnabled: true
+    sslSecret: ex-aao-mqtt-secret
+    verifyHost: false
+    expose: true
+

amq-broker/gen-certs.sh

@@ -0,0 +1,20 @@
+#! /bin/bash
+
+WORKDIR="./certs"
+mkdir -p $WORKDIR
+
+echo "=== Generating Broker keystore ==="
+keytool -genkey -alias broker -keyalg RSA -keystore $WORKDIR/broker.ks
+echo "=== Exporting Broker certificate ==="
+keytool -export -alias broker -keystore $WORKDIR/broker.ks -file $WORKDIR/broker_cert
+echo "=== Generating Client keystore ==="
+keytool -genkey -alias client -keyalg RSA -keystore $WORKDIR/client.ks
+echo "=== Importing Broker certificate to Client Truststore ==="
+keytool -import -alias broker -keystore $WORKDIR/client.ts -file $WORKDIR/broker_cert
+echo "=== Exporting Client certificate ==="
+keytool -export -alias client -keystore $WORKDIR/client.ks -file $WORKDIR/client_cert
+echo "=== Importing Client certificate to Broker Truststore ==="
+keytool -import -alias client -keystore $WORKDIR/broker.ts -file $WORKDIR/client_cert
+
+openssl x509 -inform DER -in $WORKDIR/client_cert -out $WORKDIR/client.pem -outform PEM
+openssl x509 -inform DER -in $WORKDIR/broker_cert -out $WORKDIR/broker.pem -outform PEM

amq-broker/mqtt-client/publisher.py

@@ -0,0 +1,22 @@
+import paho.mqtt.client as mqtt
+
+TOPIC_NAME = "TestTopic"
+
+def on_connect(client, userdata, rc):
+    print("Connected with result code "+str(rc))
+
+def on_publish(client, userdata, mid):
+    print("Message Sent!")
+
+client = mqtt.Client()
+client.username_pw_set("admin", "admin")
+client.tls_set("broker_cert.pem")
+client.tls_insecure_set(True) # disable hostname verification
+client.on_connect = on_connect
+client.on_publish = on_publish
+
+client.connect("ex-aao-mqtt-0-svc-rte-pkolakow.apps.red.ocp.public", 443, 60)
+
+message = "Test Message"
+print(f"Sending mesasge to {TOPIC_NAME}: message")
+client.publish(TOPIC_NAME,message)

amq-broker/mqtt-client/subscriber.py

@@ -0,0 +1,21 @@
+import paho.mqtt.client as mqtt
+
+TOPIC_NAME = "TestTopic"
+
+def on_connect(client, userdata, flags, rc):
+    print("Connected with result code {0}".format(str(rc)))
+    client.subscribe(TOPIC_NAME)  
+
+def on_message(client, userdata, msg):
+    print(msg.topic + ": "+ str(msg.payload))
+
+client = mqtt.Client()
+client.username_pw_set("admin", "admin")
+client.tls_set("certs/broker.pem")
+client.tls_insecure_set(True) # disable hostname verification
+
+client.on_connect=on_connect
+client.on_message = on_message
+
+client.connect("ex-aao-mqtt-2-svc-rte-pkolakow.apps.red.ocp.public", 443, 60)
+client.loop_forever()

artificial-data-generator/.vscode/settings.json

@@ -0,0 +1,3 @@
+{
+    "python.pythonPath": "/home/pgrabusz/Documents/SAP/generator/retailstoreofthefuture/customerdatagen/.venv/bin/python3"
+}

artificial-data-generator/README.md

@@ -0,0 +1,70 @@
+# Data generator
+
+This generator can generate transaction data for predicting the probability of coupon usage by the given customer.
+
+It makes a few assumptions for that purpose:
+1. Men use coupons with the probability of p=0.8 and women with p=0.9
+1. Women buy products from the end of the products list* more often
+1. Men buy products from the beginning of the product list* more often
+1. People in the age up to 35 buys products from the beginning to 1/3 of the list* more often,
+1. People in the age 36-60 buys products from 1/3 to 2/3 of the list* more often,
+1. People in the age 61 and above buys products from the 2/3 to the end of the list* more often,
+1. The customer has personal preferences which can change the probability of buying products by up to 40% (20% on average) in 20% of cases.
+
+
+> *There are actually 3 lists - vendors, departments, and categories. The order in all lists is random but always the same for all generated customers. Every customer has his/her  own list of preferences for all 3 types. All lists are generated the same way and using the same assumptions described above.
+
+All numbers and functions described above can be changed using the config.py file.
+
+## The algorithm
+
+Here's an approximate algorithm of the generator. For more details please check the code.
+
+1. Generate customers
+1. Generate products
+1. Generate inventory (number of items)
+1. Generate customer preferences. For each list: departments, vendors, categories create a preferences list:
+   1. create a list of random numbers (length = number of products). ~80% of them will be equal to 0.8. The rest should be random with the even distribution of numbers from 0.6 to 1.0 (personal preferences),
+   1. map elements: multiply every element by the gender function result for the element's position in the list. If X is the number on the list and I is its index on the list, then the final number is X*gender_function(I)
+   1. map new elements: multiply each number by age function: X*age_function(I)
+   1. normalize the list - scale the list so the sum of all elements is 1, and the proportions between numbers are the same. In other words, divide all elements by the sum.
+   1. the new list describes the client's probability of buying a product from a particular department/vendor/category
+1. Generate coupons
+   * The algorithm makes sure that there are 3 (configurable) coupons/day for each of the departments
+1. Generate orders
+   1. For dates in a given range, for every hour when the shop is opened, for each customer get the probability of her/him entering the shop,
+   1. Pick a number <0.0, 1.0> - if this number is < then the probability, the customers enters the store,
+   1. Create a transaction: get X products (where X is random) but consider customer's preferences. Select a vendor, then the department, and then the category.
+   1. If there's a coupon for the product, there's a high chance of using it (95%).
+   1. If there's no coupon, the customer checks if there's a similar product with the coupon. If there is, it buys a similar product.
+
+## Usage
+
+```
+python3 generate.py 
+```
+
+optional arguments:
+
+```
+-h,             --help
+                      show this help message and exit
+-c CUSTOMERS,   --customers CUSTOMERS
+                      number of customers
+-d DEPARTMENTS, --departments DEPARTMENTS
+                      number of departments
+-p PRODUCTS,    --products PRODUCTS
+                      number of products
+-C COUPONS,     --coupons COUPONS
+                      number of coupons
+-S START,       --start START
+                      start date
+-E END,         --end END
+                      end date
+-D DAYS,        --days DAYS 
+                      number of days if end not set
+-P PATH,        --path PATH
+                      path where to generate a files
+-v,             --verbose    
+                      verbose
+```