Merge pull request #156 from ibpsa/issue155_v070

Issue155 v070

.github/workflows/github-actions.yml

@@ -16,8 +16,6 @@ jobs:
         uses: actions/checkout@v3
       - name: Pull boptestgym image from registry
         run: make pull-boptestgym
-      - name: Pull boptest_base image from registry
-        run: make pull-boptestbase
       - name: Install Docker Compose
         run: |
           sudo apt-get update
@@ -34,8 +32,6 @@ jobs:
         uses: actions/checkout@v3
       - name: Pull boptestgym image from registry
         run: make pull-boptestgym
-      - name: Pull boptest_base image from registry
-        run: make pull-boptestbase
       - name: Install Docker Compose
         run: |
           sudo apt-get update
@@ -52,15 +48,13 @@ jobs:
         uses: actions/checkout@v3
       - name: Pull boptestgym image from registry
         run: make pull-boptestgym
-      - name: Pull boptest_base image from registry
-        run: make pull-boptestbase
       - name: Install Docker Compose
         run: |
           sudo apt-get update
           sudo apt-get install -y docker-compose
       - name: Test vectorized environment
         run: make test-vectorized-in-container
-  test-service:
+  test-tutorial:
     runs-on: ubuntu-latest
     defaults:
       run:
@@ -70,8 +64,6 @@ jobs:
         uses: actions/checkout@v3
       - name: Pull boptestgym image from registry
         run: make pull-boptestgym
-      - name: Pull boptest_base image from registry
-        run: make pull-boptestbase
-      - name: Test service version
-        run: make test-service-in-container
+      - name: Test tutorial
+        run: make test-tutorial-in-container
 

README.md

@@ -1,20 +1,19 @@
 # BOPTEST-Gym
 
-BOPTESTS-Gym is the [OpenAI-Gym](https://gym.openai.com/) environment for the [BOPTEST](https://github.com/ibpsa/project1-boptest) framework. This repository accommodates the BOPTEST API to the OpenAI-Gym convention in order to facilitate the implementation, assessment and benchmarking of reinforcement learning (RL) algorithms for their application in building energy management. RL algorithms from the [Stable-Baselines 3](https://github.com/DLR-RM/stable-baselines3) repository are used to exemplify and test this framework. 
+BOPTESTS-Gym is the [Gymnasium](https://gymnasium.farama.org/index.html) environment of the [BOPTEST](https://github.com/ibpsa/project1-boptest) framework. This repository accommodates the BOPTEST API to the Gymnasium standard in order to facilitate the implementation, assessment and benchmarking of reinforcement learning (RL) algorithms for their application in building energy management. RL algorithms from the [Stable-Baselines 3](https://github.com/DLR-RM/stable-baselines3) repository are used to exemplify and test this framework. 
 
 The environment is described in [this paper](https://www.researchgate.net/publication/354386346_An_OpenAI-Gym_environment_for_the_Building_Optimization_Testing_BOPTEST_framework). 
 
 ## Structure
-- `boptestGymEnv.py` contains the core functionality of this Gym environment.
+- `boptestGymEnv.py` contains the core functionality of this Gymnasium environment.
 - `environment.yml` contains the dependencies required to run this software. 
 - `/examples` contains prototype code for the interaction of RL algorithms with an emulator building model from BOPTEST. 
-- `/testing` contains code for unit testing of this software. 
+- `/testing` contains code for testing this software. 
 
-## Quick-Start (using BOPTEST-Service)
-BOPTEST-Service allows to directly access BOPTEST test cases in the cloud, without the need to run it locally. Interacting with BOPTEST-Service requires less configuration effort but is considerably slower because of the communication overhead between the agent and the test case running in the cloud. Use this approach when you want to quickly check out the functionality of this repository. 
+## Quick-Start
 
-1) Create a conda environment from the `environment.yml` file provided (instructions [here](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file)). 
-2) Check out the `boptest-gym-service` branch and run the example below that uses the [Bestest hydronic case with a heat-pump](https://github.com/ibpsa/project1-boptest/tree/master/testcases/bestest_hydronic_heat_pump) and the [DQN algorithm](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) from Stable-Baselines: 
+1) Create an environment from the `environment.yml` file provided (instructions [here](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file)). You can also see our Dockerfile in [testing/Dockerfile](testing/Dockerfile) that we use to define our testing environment. 
+2) Run the example below that uses the [Bestest hydronic case with a heat-pump](https://github.com/ibpsa/project1-boptest/tree/master/testcases/bestest_hydronic_heat_pump) and the [DQN algorithm](https://stable-baselines3.readthedocs.io/en/master/modules/dqn.html) from Stable-Baselines: 
 
 ```python
 from boptestGymEnv import BoptestGymEnv, NormalizedObservationWrapper, DiscretizedActionWrapper
@@ -68,33 +67,38 @@ env.get_kpis()
 
 ```
 
-## Quick-Start (running BOPTEST locally)
-Running BOPTEST locally is substantially faster
+In [this tutorial](https://github.com/ibpsa/project1-boptest-gym/blob/master/docs/tutorials/CCAI_Summer_School_2022/Building_Control_with_RL_using_BOPTEST.ipynb) you can find more details on how to use BOPTEST-Gym and on RL applied to buildings in general. 
 
-1) Create a conda environment from the `environment.yml` file provided (instructions [here](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file)). 
-2) Run a BOPTEST case with the building emulator model to be controlled (instructions [here](https://github.com/ibpsa/project1-boptest/blob/master/README.md)).  
-3) Check out the `master` branch of this repository and run the example above replacing the url to be `url = 'http://127.0.0.1:5000'` and avoiding the `testcase` argument to the `BoptestGymEnv` class. 
+### Note 1: on running BOPTEST in the server vs. locally
+The previous example interacts with BOPTEST in a server at `https://api.boptest.net` which is readily available anytime. Interacting with BOPTEST from this server requires less configuration effort but is slower because of the communication overhead between the agent and the test case running in the cloud. Use this approach when you want to quickly check out the functionality of this repository. 
 
-## Quick-Start (running BOPTEST locally in a vectorized environment)
+If you prioritize speed (which is usually the case when training RL agents), running BOPTEST locally is substantially faster. 
+You can do so by downloading the BOPTEST repository and running:
+```bash
+docker compose up web worker provision
+
+```
+
+Further details in the [BOPTEST GitHub page](https://github.com/ibpsa/project1-boptest/blob/master/README.md#quick-start-to-deploy-and-use-boptest-on-a-local-computer). 
 
-To facilitate the training and testing process, we provide scripts that automate the deployment of multiple BOPTEST instances using Docker Compose and then train an RL agent with a vectorized BOPTEST-gym environment. The deployment dynamically checks for available ports, generates a Docker Compose YAML file, and takes care of naming conflicts to ensure smooth deployment.
-Running a vectorized environment allows you to deploy as many BoptestGymEnv instances as cores you have available for the agent to learn from all of them in parallel (see [here](https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html) for more information, we specifically use [`SubprocVecEnv`](https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html#subprocvecenv)). This substantially speeds up the training process. 
+Then you only need to change the `url` to point to your local BOPTEST service deployment instead of the remote server (`url = 'http://127.0.0.1').
 
-### Usage
+### Note 2: on running BOPTEST locally in a vectorized environment
 
-1. Specify the BOPTEST root directory either by passing it as a command-line argument or by defining the `boptest_root` variable at the beginning of the script `generateDockerComposeYml.py`. The script prioritizes the command-line argument if provided. Users are allowed to change the Start Port number and Total Services as needed.
+BOPTEST allows the deployment of multiple test case instances using Docker Compose. 
+Running a vectorized environment enables the deployment of as many BoptestGymEnv instances as cores you have available for the agent to learn from all of them in parallel. See [here](https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html) for more information, we specifically use [`SubprocVecEnv`](https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html#subprocvecenv). This can substantially speed up the training process. 
 
-Example using command-line argument:
+To do so, deploy BOPTEST with multiple workers to spin multiple test cases. See the example below that prepares BOPTEST to spin two test cases.
 
 ```bash
-python generateDockerComposeYml.py absolute_boptest_root_dir
+docker compose up --scale worker=2 web worker provision
 ```
 
-2. Train an RL agent with parallel learning with the vectorized BOPTEST-gym environment. See `/examples/run_vectorized.py` for an example on how to do so. 
+Then you can train an RL agent with parallel learning with the vectorized BOPTEST-gym environment. See [`/examples/run_vectorized.py`](https://github.com/ibpsa/project1-boptest-gym/blob/master/examples/run_vectorized.py) for an example on how to do so. 
 
 ## Versioning and main dependencies
 
-Current BOPTEST-Gym version is `v0.6.0` which is compatible with BOPTEST `v0.6.0` 
+Current BOPTEST-Gym version is `v0.7.0` which is compatible with BOPTEST `v0.7.0` 
 (BOPTEST-Gym version should always be even with the BOPTEST version used). 
 The framework has been tested with `gymnasium==0.28.1` and `stable-baselines3==2.0.0`.
 You can see [testing/Dockerfile](testing/Dockerfile) for a full description of the testing environment. 

boptestGymEnv.py

@@ -36,7 +36,8 @@ class BoptestGymEnv(gym.Env):
     metadata = {'render.modes': ['console']}
 
     def __init__(self, 
-                 url                = 'http://127.0.0.1:5000',
+                 url                = 'http://127.0.0.1',
+                 testcase           = 'bestest_hydronic_heat_pump',
                  actions            = ['oveHeaPumY_u'],
                  observations       = {'reaTZon_y':(280.,310.)}, 
                  reward             = ['reward'],
@@ -56,6 +57,8 @@ def __init__(self,
         ----------
         url: string
             Rest API url for communication with the BOPTEST interface
+        testcase: string
+            The string identifier of the testcase
         actions: list
             List of strings indicating the action space. The bounds of 
             each variable from the action space the are retrieved from 
@@ -134,6 +137,7 @@ def __init__(self,
         super(BoptestGymEnv, self).__init__()
 
         self.url                = url
+        self.testcase           = testcase
         self.actions            = actions
         self.observations       = list(observations.keys())
         self.max_episode_length = max_episode_length
@@ -160,18 +164,26 @@ def __init__(self,
         #=============================================================
         # Get test information
         #=============================================================
+        # Get testid for the particular testcase
+        # Check if already started a test case and stop it if so before starting another
+        try:
+            requests.put('{0}/stop/{1}'.format(url, self.testid))
+        except:
+            pass
+        # Select and start a new test case
+        self.testid = requests.post('{0}/testcases/{1}/select'.format(url, testcase)).json()['testid']
         # Test case name
-        self.name = requests.get('{0}/name'.format(url)).json()['payload']
+        self.name = requests.get('{0}/name/{1}'.format(url, self.testid)).json()['payload']
         # Measurements available
-        self.all_measurement_vars = requests.get('{0}/measurements'.format(url)).json()['payload']
+        self.all_measurement_vars = requests.get('{0}/measurements/{1}'.format(url, self.testid)).json()['payload']
         # Predictive variables available
-        self.all_predictive_vars = requests.get('{0}/forecast_points'.format(url)).json()['payload']
+        self.all_predictive_vars = requests.get('{0}/forecast_points/{1}'.format(url, self.testid)).json()['payload']
         # Inputs available
-        self.all_input_vars = requests.get('{0}/inputs'.format(url)).json()['payload']
+        self.all_input_vars = requests.get('{0}/inputs/{1}'.format(url, self.testid)).json()['payload']
         # Default simulation step
-        self.step_def = requests.get('{0}/step'.format(url)).json()['payload']
+        self.step_def = requests.get('{0}/step/{1}'.format(url, self.testid)).json()['payload']
         # Default scenario
-        self.scenario_def = requests.get('{0}/scenario'.format(url)).json()['payload']
+        self.scenario_def = requests.get('{0}/scenario/{1}'.format(url, self.testid)).json()['payload']
 
         #=============================================================
         # Define observation space
@@ -470,15 +482,15 @@ def find_start_time():
             self.start_time = find_start_time()
 
         # Initialize the building simulation
-        res = requests.put('{0}/initialize'.format(self.url), 
+        res = requests.put('{0}/initialize/{1}'.format(self.url,self.testid),
                            json={'start_time':int(self.start_time),
                                  'warmup_period':int(self.warmup_period)}).json()['payload']
 
         # Set simulation step
-        requests.put('{0}/step'.format(self.url), json={'step':int(self.step_period)})
+        requests.put('{0}/step/{1}'.format(self.url,self.testid), json={'step':int(self.step_period)})
 
         # Set BOPTEST scenario
-        requests.put('{0}/scenario'.format(self.url), json=self.scenario)
+        requests.put('{0}/scenario/{1}'.format(self.url,self.testid), json=self.scenario)
 
         # Initialize objective integrand
         self.objective_integrand = 0.
@@ -493,6 +505,22 @@ def find_start_time():
 
         return observations, info
 
+    def stop(self):
+        '''
+        Stop the test case
+
+        '''
+
+        requests.put('{0}/stop/{1}'.format(self.url, self.testid))
+
+    def stop(self):
+        '''
+        Stop the test case
+
+        '''
+
+        requests.put('{0}/stop/{1}'.format(self.url, self.testid))
+
     def step(self, action):
         '''
         Advance the simulation one time step
@@ -543,7 +571,7 @@ def step(self, action):
             u[act.replace('_u','_activate')] = float(1)
 
         # Advance a BOPTEST simulation
-        res = requests.post('{0}/advance'.format(self.url), json=u).json()['payload']
+        res = requests.post('{0}/advance/{1}'.format(self.url,self.testid), json=u).json()['payload']
 
         # Compute reward of this (state-action-state') tuple
         reward = self.get_reward()
@@ -618,7 +646,7 @@ def get_reward(self):
         w = 1
 
         # Compute BOPTEST core kpis
-        kpis = requests.get('{0}/kpi'.format(self.url)).json()['payload']
+        kpis = requests.get('{0}/kpi/{1}'.format(self.url,self.testid)).json()['payload']
 
         # Calculate objective integrand function at this point
         objective_integrand = kpis['cost_tot'] + w*kpis['tdis_tot']
@@ -716,7 +744,7 @@ def get_observations(self, res):
         if self.is_regressive:
             regr_index = res['time']-self.step_period*np.arange(1,self.regr_n+1)
             for var in self.regressive_vars:
-                res_var = requests.put('{0}/results'.format(self.url), 
+                res_var = requests.put('{0}/results/{1}'.format(self.url, self.testid), 
                                        json={'point_names':[var],
                                              'start_time':int(regr_index[-1]), 
                                              'final_time':int(regr_index[0])}).json()['payload']
@@ -732,7 +760,7 @@ def get_observations(self, res):
 
         # Get predictions if this is a predictive agent. 
         if self.is_predictive:
-            predictions = requests.put('{0}/forecast'.format(self.url), 
+            predictions = requests.put('{0}/forecast/{1}'.format(self.url, self.testid), 
                                        json={'point_names': self.predictive_vars,
                                              'horizon':     int(self.predictive_period),
                                              'interval':    int(self.step_period)}).json()['payload']
@@ -753,7 +781,7 @@ def get_kpis(self):
         '''
 
         # Compute BOPTEST core kpis
-        kpis = requests.get('{0}/kpi'.format(self.url)).json()['payload']
+        kpis = requests.get('{0}/kpi/{1}'.format(self.url, self.testid)).json()['payload']
 
         return kpis
 
@@ -1246,7 +1274,7 @@ def get_reward(self):
         '''
 
         # Compute BOPTEST core kpis
-        kpis = requests.get('{0}/kpi'.format(self.url)).json()['payload']
+        kpis = requests.get('{0}/kpi/{1}'.format(self.url, self.testid)).json()['payload']
 
         # Calculate objective integrand function at this point
         objective_integrand = kpis['cost_tot'] + kpis['tdis_tot']
@@ -1283,7 +1311,7 @@ def get_reward(self):
         w = 0.1
 
         # Compute BOPTEST core kpis
-        kpis = requests.get('{0}/kpi'.format(self.url)).json()['payload']
+        kpis = requests.get('{0}/kpi/{1}'.format(self.url, self.testid)).json()['payload']
 
         # Calculate objective integrand function at this point
         objective_integrand = kpis['cost_tot'] + w*kpis['tdis_tot']
@@ -1317,7 +1345,7 @@ def get_reward(self):
         w = 10
 
         # Compute BOPTEST core kpis
-        kpis = requests.get('{0}/kpi'.format(self.url)).json()['payload']
+        kpis = requests.get('{0}/kpi/{1}'.format(self.url, self.testid)).json()['payload']
 
         # Calculate objective integrand function at this point
         objective_integrand = kpis['cost_tot'] + w*kpis['tdis_tot']