Interactive Track Analysis

.gitignore

@@ -0,0 +1,9 @@
+*.iml
+.idea
+venv
+log-analysis/venv
+reward
+!reward/reward_sample.py
+logs
+!logs/*-sample.log
+.ipynb_checkpoints

log-analysis/DeepRacer Log Analysis.ipynb

@@ -471,7 +471,9 @@
     }
    ],
    "source": [
-    "df = convert_to_pandas(data, None)\n",
+    "EPISODES_PER_ITERATION = 20\n",
+    "\n",
+    "df = convert_to_pandas(data, EPISODES_PER_ITERATION)\n",
     "df.head()"
    ]
   },
@@ -1793,7 +1795,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.5"
+   "version": "3.6.8"
   }
  },
  "nbformat": 4,

log-analysis/README.md

@@ -0,0 +1,56 @@
+# Log Analysis
+This folder contains tools making it simple to read information from training and evaluation logs.
+
+The main part of this project are [Jupyter](https://jupyter.org/) notebooks - think of it as a text editor document enriched with executable code.
+
+## What you need to know to use this
+
+If you just want to have a look, GitHub has a viewer for the notebooks, just click into them and enjoy.
+
+For working with the notebooks you need to be familiar with Python code, but the whole process is reasonably simple. Getting to know pandas and matplotlib will help you evolve from the solutions provided to your own bespoke analysis toolkit.
+
+Tinkering and trying things out is highly desirable. Please share your ideas
+
+## Notebooks
+
+There are currently following notebooks:
+* `DeepRacer Log Analysis.ipynb` - original notebook provided by the AWS folks (it has things not used in notebooks listed below)
+* `Training_analysis.ipynb` - built on top of the first one with some things removed and many added, prepared to monitor the training progress
+* `Evaluation_analysis.ipynb` - built on top of the first one, prepared to analyse evaluation data
+
+## Running the notebooks
+
+I recommend setting up a venv for this:
+```
+python3 -m venv venv
+```
+(I recommend folder venv as I have already added it to .gitignore)
+Then activate:
+```
+source venv/bin/activate
+```
+Then install dependencies:
+```
+pip install shapely matplotlib pandas sklearn boto3 awscli jupyter
+```
+Then run
+```
+jupyter notebook
+```
+From the opened page you can select a notebook to work with.
+
+## Useful hints
+* logs and reward folders have been configured to be ignored by git. This is so that you don't accidentally submit your reward functions or other useful info. Just make sure you secure it somehow yourself.
+* have a look at new_reward function usage in the notebooks. It lets you try and evaluate what the reward would look like for a different reward function.
+
+## What can I contribute?
+
+There is a number of opportunities for improvement:
+* Report issues/feature requests
+* Fix things
+* Improve descriptions
+* Provide more resources
+* Add analysis bits to notebooks
+* Complete the `logs_to_params` method in log_analysis to improve the logs replay for a different reward
+* Fill in track data used in breakdown in `Training_analysis.ipynb`
+* Make the notebooks work with more tracks

log-analysis/cw_utils.py

@@ -8,6 +8,7 @@
 import boto3
 import sys
 import dateutil.parser
+import os
 
 
 def get_log_events(log_group, stream_name=None, stream_prefix=None, start_time=None, end_time=None):
@@ -42,7 +43,11 @@ def get_log_events(log_group, stream_name=None, stream_prefix=None, start_time=N
 
 
 def download_log(fname, stream_name=None, stream_prefix=None,
-                 log_group=None, start_time=None, end_time=None):
+                 log_group=None, start_time=None, end_time=None, force=False):
+    if os.path.isfile(fname) and not force:
+        print('Log file exists, use force=True to download again')
+        return
+
     if start_time is None:
         start_time = 1451490400000  # 2018
     if end_time is None:
@@ -63,7 +68,7 @@ def download_log(fname, stream_name=None, stream_prefix=None,
             f.write("\n")
 
 
-def download_all_logs(pathprefix, log_group, not_older_than=None, older_than=None):
+def download_all_logs(pathprefix, log_group, not_older_than=None, older_than=None, force = False):
     client = boto3.client('logs')
 
     lower_timestamp = iso_to_timestamp(not_older_than)
@@ -84,7 +89,10 @@ def download_all_logs(pathprefix, log_group, not_older_than=None, older_than=Non
                 continue
             stream_prefix = stream['logStreamName'].split("/")[0]
             file_name = "%s%s.log" % (pathprefix, stream_prefix)
-            download_log(file_name, stream_prefix=stream_prefix, log_group=log_group)
+
+            if not os.path.isfile(file_name) or force:
+                download_log(file_name, stream_prefix=stream_prefix, log_group=log_group)
+
             fetched_files.append(
                 (file_name, stream_prefix, stream['firstEventTimestamp'], stream['lastEventTimestamp']))