Merge pull request #86 from CounterpartyXCP/develop

1.0.0

.gitignore

@@ -1,8 +1,6 @@
 *.py[cod]
-
 # C extensions
 *.so
-
 # Packages
 *.egg
 *.egg-info
@@ -45,3 +43,4 @@ schema.py
 
 #Docs 
 docs/_build
+

README.md

@@ -1,4 +1,35 @@
 counterblockd
 ==============
 
-Provides extended API functionality over counterpartyd (such as market information, asset history, etc). Works alongside counterpartyd.
+Provides extended API functionality over counterpartyd (such as market information, asset history, etc).
+Works alongside `counterpartyd`.
+
+Automatic Installation
+------------------------
+
+Please see the [Setting up a Federated Node](http://counterparty.io/docs/build-system/federated-node/) guide
+(under the *"Node Setup"* section) for a script that will set counterblockd up for you, automatically. When prompted
+for the node type, choose "counterblockd basic".
+
+Manual Installation on Ubuntu
+---------------------------------
+
+Many of the requirements for `counterblockd` are similar to those for `counterpartyd`, so if you used the
+[counterpartyd installer](https://github.com/CounterpartyXCP/counterpartyd_build) to install, you already
+have many requirements installed.
+
+The first extra thing would be to install MongoDB:
+
+```bash
+sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
+echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/10gen.list
+sudo apt-get update
+sudo apt-get install mongodb-10gen
+```
+
+The last couple can be added by running (from the `counterblockd` directory):
+
+```bash
+sudo apt-get install python-pip cython libxml2-dev libxslt1-dev python-dev
+sudo pip install -r pip-requirements.txt 
+```

docs/API.rst

@@ -103,8 +103,8 @@ Asset Functions
   Retrieves the ordered balance history for a given address (or list of addresses) and asset pair, within the specified date range
 
   :param normalize: If set to True, return quantities that (if the asset is divisible) have been divided by 100M (satoshi).
-            :return: A list of tuples, with the first entry of each tuple being the block time (epoch TS), and the second being the new balance at that block time.
-            :rtype: [(<block time>, <balance>)]
+  :return: A list of tuples, with the first entry of each tuple being the block time (epoch TS), and the second being the new balance at that block time.
+  :rtype: [(<block time>, <balance>)]
 
 .. function:: get_base_quote_asset(asset1, asset2)
 
@@ -413,20 +413,6 @@ Betting Functions
       'target_value'
       }]
 
-RPS Betting Functions
-^^^^^^^^^^^^^^^^^^^^^
-
-.. function:: get_open_rps_count(possible_moves=3, exclude_addresses=[])
-
-  Get the open RPS bets matching the given parameters, except those from `exclude_addresses`
-
-  :rtype: [<total wager>, <game count>]
-
-.. function:: get_user_rps(addresses):
-
-  :rtype: [{'tx_hash', 'block_index', 'move', 'status', 'expiration', 'address', 'possible_moves', 'wager', 'counter_move'}]
-
-
 Debugging/Server Functions
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -474,14 +460,13 @@ Debugging/Server Functions
 Blockchain Functions
 ^^^^^^^^^^^^^^^^^^^^
 
-.. function:: get_chain_address_info(addresses, with_uxtos=True, with_last_txn_hashes=4, with_block_height=False)
+.. function:: get_chain_address_info(addresses, with_uxtos=True, with_last_txn_hashes=4)
 
   Get info for one or more addresses
 
   :parameter list addresses: Address to query
   :parameter boolean with_uxtos: Include Unspent
   :parameter int with_last_txn_hashes: Include n recent confirmed transactions
-  :param boolean with_block_height: Include block height
   :return: Address info
   :rtype: [{'addr', 'info',('uxto'),('last_txns'),('block_height')}]
 

docs/Modules.rst

@@ -0,0 +1,134 @@
+counterblockd Plugin Module Functionality
+=============================================
+
+``counterblockd`` allows the default set of processors and/or event handlers to be disabled or re-prioritized.
+Additional processors and/or event handlers can also be added. This allows developers to easily extend the
+capabilities of ``counterblockd``, as well as deactivate unused portions of the system as necessary.
+
+configuration file
+----------------------
+All configurations are contained in ``counterblockd_module.conf`` by default, which should be placed in to the
+`counterblockd data-dir`. To load a custom module specify the module path under ``[LoadModule]`` relative to
+the `counterblockd base-dir`. i.e.
+::
+    [LoadModule]
+    'lib/vendor' = True
+    
+To change the default behavior for ``counterblockd`` modules/events, change the corresponding processor config.
+(Note: function names must be exact.) 
+
+To disable a processor:
+::
+    #(must be bool)
+    [BlockProcessor]
+    generate_wallet_stats = False
+
+To change a processor's priority:
+::
+    #(must be int) 
+    [MessageProcessor]
+    parse_issuance = 5
+    
+To change priority and enable:
+::
+    #(tuple, order does not matter)
+    [MessageProcessor]
+    parse_issuance = 5, True 
+    parse_issuance = True, 5
+
+Command-line functions
+-----------------------------
+
+To enable a custom module in ``counterblockd``:
+::
+    python counterblockd.py enmod 'lib/vendor'
+    
+To disable a loaded module:
+::
+    python counterblockd.py dismod 'lib/vendor' 
+
+To list loaded modules and processors:
+:: 
+    python counterblockd.py listmod
+
+Adding Custom Methods
+-----------------------------------
+For Adding custom methods to built in ``counterblockd`` processors. The general syntax is:
+
+.. code-block:: python
+    from lib.processor import <processor_name> 
+        @<processor_name>.subscribe(enabled=<bool>, priority=<int>)
+
+If not specified, the defaults are ``enabled=true, priority=0``.
+When a processor is triggered methods are run in order of priority from the highest.
+*Please note that any priority less than ``0`` or greater than ``1000`` is reserved for internal ``counterblock``
+functionality, and custom plugins should only utilize priority settings under this number.*
+
+.. code-block:: python
+    @<Processor>.subscribe()
+
+``MessageProcessor`` runs once for each message as obtained from `counterpartyd`, ``msg`` will pass the message
+ in the same format as the ``get_messages`` counterpartyd api method, msg_data corresponds to ``json.loads(msg['bindings'])``. 
+
+.. code-block:: python
+    @MessageProcessor.subscribe(enabled=True, priority=90) 
+    def custom_received_xcp_alert(msg, msg_data):
+        if msg['category'] != 'sends': return
+        if message['status'] != 'valid': return
+        if not msg_data['destination'] in MY_ADDRESS_LIST: return
+        if not msg_data['asset'] == 'XCP': return 
+        print('Received %s XCP at My Address %s from %s' %(
+        	(float(msg_data['quantity'])/10**8), msg_data['destination'], msg_data['source']))
+        return
+
+Note that with ``MessageProcessor`` handlers, you can return ``'continue'`` to prevent the running of further MessageProcessors (i.e.
+of lesser priority than the current one) for the message being currently processed.
+
+``BlockProcessor`` run once per new block, after all ``MessageProcessor`` functions have completed. 
+
+.. code-block:: python
+    @BlockProcessor.subscribe(priority=0) 
+    def alertBlock(): 
+        print('Finished processing messages for this block') 
+
+A number of changing variables that a module may need to access are stored in ``config.state`` - For example if you
+want to run a process for every new block (but not when counterblockd is catching up). 
+
+.. code-block:: python
+    @BlockProcessor.subscribe() 
+    def my_custom_block_event(): 
+        if not (config.state['cpd_latest_block']['block_index'] - config.state['my_latest_block']['block_index']) == 1: 
+            return
+        #Do stuff here
+    
+``StartUpProcessor`` runs once on ``counterblockd`` startup. 
+
+.. code-block:: python
+    @StartUpProcessor.subscribe()
+    def my_db_config(): 
+        config.my_db = pymongo.Connection()['my_db'] 
+
+``CaughtUpProcessor`` runs once when ``counterblockd`` catches up to the latest Counterpartyd block. 
+
+.. code-block:: python
+    @CaughtUpProcessor.subscribe()
+    def caughtUpAlert(): 
+        print('counterblockd is now caught up to Counterpartyd!') 
+
+``RollbackProcessor`` runs whenever the ``counterblockd`` database is rolled back (either due to a blockchain
+reorg, or an explicit rollback command being specified to ``counterblockd`` via the command line). 
+
+.. code-block:: python
+    @RollbackProcessor.subscribe()
+    def rollbackAlert(max_block_index): 
+        print('counterblockd block database rolled back! Anything newer than block index %i removed!' % max_block_index) 
+
+To add a method from a module to the API dispatcher: 
+
+.. code-block:: python
+    from lib.processor import API
+    
+    #(note that the dispatcher add_method does not take arguments) 
+    @API.add_method
+    def my_foo_api_method(): 
+        return 'bar' 

docs/counterblockd_example_module.conf

@@ -0,0 +1,20 @@
+#Example Config file 
+[MessageProcessor]
+handle_exceptional = True
+handle_invalid = True
+parse_insert = True
+handle_reorg = True
+parse_issuance = 10, True
+parse_balance_change = True
+parse_trade_book = True
+parse_broadcast = True
+parse_for_socketio = False
+[StartUpProcessor]
+start_cpd_blockfeed = True
+check_blockchain_service = True
+expire_stale_prefs = True
+expire_stale_orders = True
+generate_wallet_stats = False
+start_api = True
+[LoadModule]
+lib/modules/reparse_timer = True

docs/index.rst

@@ -1,10 +1,8 @@
 counterblockd
 ==================================================
 
-``counterblockd`` features a full-fledged JSON RPC-based API, which services Counterwallet, as well as any
-3rd party services which wish to use it.
-
 ``counterblockd`` provides additional services to Counterwallet beyond those offered in the API provided by ``counterpartyd``.
+It features a full-fledged JSON RPC-based API, which services Counterwallet, as well as any 3rd party services which wish to use it.
 
 Such services include:
 
@@ -13,12 +11,17 @@ Such services include:
 - API includes functionality for retieving processed time-series data suitable for display and manipulation
   (useful for distributed exchange price data, and more)
 
+``counterblockd`` has an extensible architecture, and developers may write custom plugins for it, which are loaded
+dynamically and allow them to extend ``counterblockd`` with new parsing functionality, write gateways to other currencies
+or services, and much more.
+
 Contents:
 
 .. toctree::
    :maxdepth: 3
 
    API
+   Modules
 
 
 Indices and tables