ETH Gas Station preliminary refactor

.gitignore

@@ -102,3 +102,9 @@ venv.bak/
 
 # mypy
 .mypy_cache/
+
+# various dev stuff
+settings.dev.conf
+test.db
+.vscode
+json

Dockerfile

@@ -0,0 +1,12 @@
+FROM ubuntu:xenial
+RUN apt-get update
+RUN apt-get install -y python3 python3-pip
+
+ADD settings.docker.conf /etc/ethgasstation.conf
+ADD ethgasstation.py /opt/ethgasstation/ethgasstation.py
+ADD model_gas.py /opt/ethgasstation/model_gas.py
+ADD requirements.txt /opt/ethgasstation/requirements.txt
+ADD egs/ /opt/ethgasstation/egs/
+RUN pip3 install -r /opt/ethgasstation/requirements.txt
+
+CMD /usr/bin/python3 /opt/ethgasstation/ethgasstation.py

README.md

@@ -1,16 +1,67 @@
 # ethgasstation
-Adaptive Gas Price Oracle for Ethereum Blockchain
+#### an adaptive gas price oracle for the ethereum blockchain
 
-This python script is designed to monitor a local ethereum node (geth, does not work right now with Parity).  It will record data about pending and mined transactions, including the transactions in your nodes transaction pool.  Its main purpose is to generate adaptive gas price estimates that enable you to know what gas price to use depending on your confirmation time needs. It generates these estimates based on miner policy estimates as well as the number of transactions in the txpool and the gas offered by the transaction.
+This is the backend for [ethgasstation](https://ethgasstation.info), written in
+Python 3. This python script is designed to monitor a local Geth node. It will
+record data about pending and mined transactions, including the transactions in
+your node's transaction pool. Its main purpose is to generate adaptive gas price
+estimates that enable you to know what gas price to use depending on your
+confirmation time needs. It generates these estimates based on miner policy
+estimates as well as the number of transactions in the txpool and the gas
+offered by the transaction.
 
-It also stores transaction data in a mysql database.  Create a user named 'ethgas' password 'station' and add an empty database 'tx'.  This allows you to run the model_gas.py script to reestimate the regression model used to generate the predictions. 
+The basic strategy is to use statistical modelling to predict confirmation times
+at all gas prices from 0-100 gwei at the current state of the txpool and minimum
+gas prices accepted in blocks over the last 200 blocks.  Then, it selects the
+gas price that gives the desired confirmation time assuming standard gas offered
+(higher than 1m gas is slower).
 
-The basic strategy is to use statistical modelling to predict confirmation times at all gas prices from 0-100 gwei at the current state of the txpool and minimum gas prices accepted in blocks over the last 200 blocks.  Then, it selects the gas price that gives the desired confirmation time assuming standard gas offered (higher than 1m gas is slower). 
+### Installation and Prerequisites
 
-Note:  you need to create a folder 'json' in the parent directory and then a new json file will be written to the folder each block containing the gas price predictions and the prediction table.  The gas price in ethgasAPI is currently in 10gwei units, so divide them by 10 to get in gwei.
+ethgasstation requires **Python 3**, **MySQL/MariaDB**, and **Geth**. You will
+need to modify `settings.conf` for your specific environment; some (insecure)
+defaults are set to get you up and running.
 
-usage:  `python3 gasStationFull.py`
+The oracle outputs JSON files. These files are stored in the output
+directory specified by the `settings.conf` file. You may output these JSON
+strings to files by setting `json.output_type` to `file` and
+`json.output_location` to a filepath, such as:
 
-options: `-r` - generates full report for display on website
+```
+[json]
+    output_type = file
+    output_location = ./json
+```
 
-requirements: `pip3 install -r requirements.txt`
+or you may set `json.output_type` to Redis and give a redis connection string:
+
+```
+[json]
+    output_type = redis
+    output_location = http://localhost:6379
+```
+
+Redis password authentication is also supported by adding it to the output
+location string, e.g. `http://:password@localhost:6379/`.
+
+### Usage
+
+To run the script as is on bare metal or a VM, manually:
+
+0. Edit `settings.conf` and install to [an allowed directory](https://github.com/ethgasstation/ethgasstation-backend/pull/17/files#diff-bbda44d05044576b25a2c6cf4b0c3597R37).
+1. Install requirements using `pip3 install -r requirements.txt`
+2. Run `./ethgasstation.py` or `python3 ethgasstation.py`.
+
+If you are running a frontend to ETH Gas Station, use the `--generate-report`
+flag to generate detailed JSON reports for front-end or API consumption.
+
+It is also possible to run the oracle as a Docker container.
+
+1. Change the settings in settings.docker.conf.
+2. Run `docker build -t ethgasstation-backend .` from this directory.
+3. Run `docker run ethgasstation-backend:latest`.
+
+In the Docker service, the Python script will dump data to JSON on Redis.
+You will need to update your settings.conf to the internal hostnames
+available for MariaDB, Redis, and geth, respectively within your
+infrastructure.

egs.py → egs/egs_ref.py

@@ -1,3 +1,10 @@
+"""
+egs_ref.py
+
+Utility functions, MySQL Schemas, and other such architecture
+for the EthGasStation adaptive oracle.
+"""
+
 import pandas as pd
 import numpy as np
 import json

egs/exceptions.py

@@ -0,0 +1,6 @@
+"""
+Generic EGS Exceptions
+"""
+
+class EGSError(Exception):
+    pass

egs/jsonexporter.py

@@ -0,0 +1,118 @@
+"""
+JSON Exporter.
+
+Will export JSON to files or K/V store for microservices
+to pick up and use. Used to generate the JSON data used by the EGS
+v0 API, and similar legacy services.
+"""
+
+import json
+import os
+from urllib.parse import urlparse
+
+import redis
+from .settings import settings_file_loaded, get_setting
+
+class JSONExporter(object):
+    """JSON exporter main class. Allows for export of JSON strings to various
+    backing data stores."""
+
+    redis_key_prefix = ""
+    supported_types = ['file', 'redis']
+
+    redis = None
+
+    export_type = None
+    export_location = None
+
+
+    def __init__(self, export_type=None, export_location=None):
+        if export_type is None:
+            self._get_export_options_from_settings()
+        else:
+            self._check_export_type(export_type)
+            self.export_type = export_type
+            self.export_location = export_location
+
+    def write_json(self, key, object_or_str):
+        """Writes JSON to supported endpoint."""
+        if self.export_type == 'file':
+            self._write_json_file(key, object_or_str)
+        elif self.export_type == 'redis':
+            self._write_json_redis(key, object_or_str)
+
+    def _write_json_file(self, key, object_or_str):
+        """Writes JSON to filesystem."""
+        if not os.path.isdir(self.export_location):
+            raise JSONExporterException(
+                "Cannot write to output dir %s, doesn't exist." %
+                (self.export_location))
+
+        json_str = self._serialize(object_or_str)
+        output_path = os.path.join(self.export_location, "%s.json" % (key))
+        with open(output_path, 'w') as fd:
+            fd.write(json_str)
+
+    def _write_json_redis(self, key, object_or_str):
+        """Writes JSON to Redis store."""
+        # self.export_location should be parseable
+        conn = self._connect_redis()
+        key = "%s_%s" % (self.redis_key_prefix, key)
+        json_str = self._serialize(object_or_str)
+        conn.set(key, json_str)
+
+    def _check_export_type(self, export_type):
+        """Checks for a valid export type. Raises Error if not found."""
+        if not export_type in self.supported_types:
+            raise JSONExporterException("JSONExporter does not support type %s" % export_type)
+
+    def _serialize(self, mixed):
+        """Serializes mixed to JSON."""
+        if isinstance(mixed, str):
+            # serialize to validate is JSON
+            mixed = json.loads(mixed)
+        return json.dumps(mixed)
+
+    def _connect_redis(self, force_reconnect=False):
+        """Connect to redis. Saves connection as self.redis."""
+        if self.redis is None or force_reconnect is True:
+            loc = urlparse(self.export_location)
+            if loc.scheme == 'unix':
+                unix_socket_path = loc.path
+                conn = redis.Redis(unix_socket_path=unix_socket_path)
+            else:
+                hostname = loc.hostname
+                port = loc.port
+                if port is None:
+                    port = 6379  # default redis port
+                if loc.password is None:
+                    conn = redis.Redis(host=hostname, port=int(port))
+                else:
+                    conn = redis.Redis(host=hostname, port=int(port),
+                                       password=loc.password)
+            self.redis = conn
+        return self.redis
+
+    def _get_export_options_from_settings(self):
+        """Get export options from default settings."""
+        if settings_file_loaded() is False:
+            raise JSONExporterException("JSONExporter can't get settings.")
+        export_type = get_setting("json", "output_type")
+        self._check_export_type(export_type)
+        self.export_type = export_type
+        export_location = get_setting("json", "output_location")
+        if export_type == 'file':
+            export_location = os.path.abspath(export_location)
+            if not os.path.isdir(export_location):
+                # XXX should be on stderr
+                print("WARN: Could not find output directory %s" % export_location)
+                print("WARN: Making directory tree.")
+                # attempt to make dirs
+                os.makedirs(export_location, exist_ok=True)
+
+        self.export_location = export_location
+
+
+class JSONExporterException(Exception):
+    """Generic exception for JSON Exporter."""
+    pass