Skip to content

Commit

Permalink
Merge pull request #286 from oceanprotocol/feature/update-dt-liquidity
Browse files Browse the repository at this point in the history 
update liquidity/price info
  • Loading branch information
@ssallam
ssallam authored Oct 12, 2020
2 parents 039cefc + decc7c4 commit 182f848
Show file tree
Hide file tree
Showing 17 changed files with 802 additions and 164 deletions.
29 changes: 12 additions & 17 deletions Dockerfile
View file
Original file line number Diff line number Diff line change
@@ -1,25 +1,21 @@
FROM python:3.6-alpine
FROM ubuntu:18.04
LABEL maintainer="Ocean Protocol <[email protected]>"

ARG VERSION

RUN apk add --no-cache --update\
build-base \
RUN apt-get update && \
apt-get install --no-install-recommends -y \
gcc \
gettext\
gmp \
gmp-dev \
libffi-dev \
openssl-dev \
py-pip \
python3 \
python3-dev \
&& pip install virtualenv
python3.8 \
python3-pip \
python3.8-dev \
gettext-base

COPY . /aquarius
WORKDIR /aquarius

# Only install install_requirements, not dev_ or test_requirements
RUN python3.8 -m pip install -U pip
RUN pip install setuptools
RUN pip install wheel
RUN pip install .

# config.ini configuration file variables
Expand All @@ -37,15 +33,14 @@ ENV DB_SCHEME='http'
ENV DB_NAMESPACE='namespace'
ENV AQUARIUS_URL='http://0.0.0.0:5000'
ENV ALLOW_FREE_ASSETS_ONLY='false'
ENV RATING_ALLOWED_UPDATER=''
# docker-entrypoint.sh configuration file variables
ENV AQUARIUS_WORKERS='1'
ENV EVENTS_ALLOW=''
ENV RUN_EVENTS_MONITOR=''
ENV EVENTS_RPC='http://127.0.0.1:8545'
ENV EVENTS_ECIES_PRIVATE_KEY='0xc6914ea1e5ac6a1cd2107240be714735bf799ce9ea4125016aeb479266720ff4'
ENV ARTIFACTS_PATH=''
ENV ADDRESS_FILE=''
#ENV ARTIFACTS_PATH=''
#ENV ADDRESS_FILE=''
#ENV ALLOWED_PUBLISHERS=['0x123','0x1234']
ENTRYPOINT ["/aquarius/docker-entrypoint.sh"]

Expand Down
98 changes: 84 additions & 14 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,13 @@

# Aquarius

> 🐋 Aquarius provides an off-chain database store for metadata about data assets.
> It's part of the [Ocean Protocol](https://oceanprotocol.com) software stack.
> 🐋 Aquarius provides an off-chain database cache for metadata that is published on-chain.
This enables faster query operations on datasets metadata.
The latest version of Aquarius consist of a Flask application to support fetching and
searching metadata and a blockchain events monitor that picks up new metadata published
on-chain and stores it in the database backend (elasticsearch).

**_"Aquarius is a constellation of the zodiac, situated between Capricornus and Pisces. Its name is Latin for "water-carrier" or "cup-carrier. Aquarius is one of the oldest of the recognized constellations along the zodiac (the Sun's apparent path)."_**
> It's part of the [Ocean Protocol](https://oceanprotocol.com) software stack.
[![Docker Build Status](https://img.shields.io/docker/build/oceanprotocol/aquarius.svg)](https://hub.docker.com/r/oceanprotocol/aquarius/) [![Travis (.com)](https://img.shields.io/travis/com/oceanprotocol/aquarius.svg)](https://travis-ci.com/oceanprotocol/aquarius) [![Codacy coverage](https://img.shields.io/codacy/coverage/10c8fddd5e8547c29de4906410a16ae7.svg)](https://app.codacy.com/project/ocean-protocol/aquarius/dashboard) [![PyPI](https://img.shields.io/pypi/v/ocean-aquarius.svg)](https://pypi.org/project/ocean-aquarius/) [![GitHub contributors](https://img.shields.io/github/contributors/oceanprotocol/aquarius.svg)](https://github.com/oceanprotocol/aquarius/graphs/contributors)

Expand All @@ -15,13 +18,76 @@

---

## What Aquarius does
* Aquarius runs a Flask RESTful server to support fetching and searching metadata of datasets that are published on-chain
* The metadata is published on-chain via the Metadata smartcontract:
* Metadata is first compressed then published on-chain
* The compressed metadata on-chain is not kept in storage, but rather is captured in an event log named `MetadataCreated`
* The id (DID) of the dataset asset is the Datatoken address, off-chain the did consist of `did:op:` prepended to the datatoken address
* Aquarius runs an events monitor that watches the:
* `MetadataCreated` event from the `Metadata` smartcontract
* Reads the events `data` argument, decompresses the metadata json object
then runs schema validation before saving it to the database
* `LOG_JOIN`, `LOG_EXIT` and `LOG_SWAP` events from the `BPool` smartcontracts
* Any of these events is an indication that liquidity and price have changed
* The watcher reads the liquidity of each token in the pool and updates the
corresponding metadata in the cache. This information is added to the metadata
to allow sorting and searching by price and/or liquidity volume

## Setup
The following environment variables are required for running Aquarius:

```
# URL of ethereum network.
# Recommendation: when connecting to an official network, create an Infura project id and set this
# to use the Infura url including the project id
EVENTS_RPC
examples:
"http://172.15.0.3:8545", "wss://rinkeby.infura.io/ws/v3/INFURA_ID"
# Use this to run the EventsMonitor in a thread from the main Flask app
EVENTS_ALLOW
accepted values:
"0" to disable
"1" to enable
# Run the EventsMonitor in a separate process, overrides `EVENTS_ALLOW`.
# This is only used when running in `docker` container
RUN_EVENTS_MONITOR
accepted values:
"0" to disable
"1" to enable
```
And these are optional
```bash
# Use this to decrypt metadata when read from the blockchain event log
EVENTS_ECIES_PRIVATE_KEY
# Path to abi files of the ocean contracts
ARTIFACTS_PATH
# Path to the `address.json` file or any json file that has the deployed contracts addresses
ADDRESS_FILE
# Specify the network name to use for reading the contracts addresses from the `ADDRESS_FILE`.
# If not set, the netwrok name is derived from current network id or from the `EVENTS_RPC` value
NETWORK_NAME
# Skip caching metadata of publishers that are not in this list
ALLOWED_PUBLISHERS
# The blockNumber of `BFactory` deployment
BFACTORY_BLOCK
# The blockNumber of `Metadata` contract deployment
METADATA_CONTRACT_BLOCK


```

## For Aquarius Operators

If you're developing a marketplace, you'll want to run Aquarius and several other components locally, and the easiest way to do that is to use Barge. See the instructions in [the Barge repository](https://github.com/oceanprotocol/barge).
If you're developing a marketplace, you'll want to run Aquarius and several other components locally,
and the easiest way to do that is to use Barge. See the instructions
in [the Barge repository](https://github.com/oceanprotocol/barge).

## For Aquarius API Users

[Here](https://docs.oceanprotocol.com/references/aquarius/) is API documentation. It shows docs for the version currently deployed with the Nile Testnet. For other versions, see "past versions" link at the top of the page.
[Here](https://docs.oceanprotocol.com/references/aquarius/) is API documentation.

If you have Aquarius running locally, you can find API documentation at
[http://localhost:5000/api/v1/docs](http://localhost:5000/api/v1/docs) or maybe
Expand Down Expand Up @@ -61,13 +127,15 @@ Run Docker image
docker run myaqua
```

To test with other ocean components in `barge` directory `compose-files/aquarius.yml` change `aquarius image` to `myaqua`
To test with other ocean components in `barge` set the `AQUARIUS_VERSION` environment variable to `myaqua`
Then

```bash
./start_ocean.sh
```

The setup for `Aquarius` and Alastic search in `barge` is in `compose-files/aquarius_elasticsearch.yml`

### Running Locally, for Dev and Test

First, clone this repository:
Expand All @@ -77,26 +145,28 @@ git clone [email protected]:oceanprotocol/aquarius.git
cd aquarius/
```

Then run mongodb database that is a requirement for Aquarius. MongoDB can be installed directly using instructions from [official documentation](https://docs.mongodb.com/manual/installation/). Or if you have `docker` installed, you can run:
Then run elasticsearch database that is a requirement for Aquarius.

```bash
docker run -d -p 27017:27017 mongo
export ES_VERSION=6.6.2
export ES_DOWNLOAD_URL=https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-${ES_VERSION}.tar.gz
wget ${ES_DOWNLOAD_URL}
tar -xzf elasticsearch-${ES_VERSION}.tar.gz
./elasticsearch-${ES_VERSION}/bin/elasticsearch &
```

Note that it runs MongoDB but the Aquarius can also work with BigchainDB or Elasticsearch. If you want to run ElasticSearch or BigchainDB, update the file `config.ini` and run the Database engine with your preferred method.

Then install Aquarius's OS-level requirements:

```bash
sudo apt update
sudo apt install python3-dev python3.7-dev libssl-dev
sudo apt install python3-dev python3.7-dev
```

(Note: At the time of writing, `python3-dev` was for Python 3.6. `python3.7-dev` is needed if you want to test against Python 3.7 locally. BigchainDB needs `libssl-dev`.)
(Note: At the time of writing, `python3-dev` was for Python 3.6. `python3.7-dev` is needed if you want to test against Python 3.7 locally.)

Before installing Aquarius's Python package requirements, you should create and activate a virtualenv (or equivalent).
Before installing Aquarius's Python package requirements, it's recommended to create and activate a virtualenv (or equivalent).

The most simple way to start is:
At this point, an Elasticsearch database must already be running, now you can start the Aquarius server:

```bash
pip install -r requirements.txt
Expand Down
61 changes: 4 additions & 57 deletions aquarius/app/auth_util.py
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
import os

from eth_account import Account
from eth_account.messages import encode_defunct
from ocean_lib.web3_internal.web3helper import Web3Helper
from web3 import Web3


Expand All @@ -14,14 +11,9 @@ def get_signer_address(message, signature, logger):
:return: Address or None in case of error
"""
try:
logger.debug('got %s as a message' % message)
signable_message = encode_defunct(text=message)
logger.debug(f'got {signable_message} as a message_hash')
address_recovered = Account.recover_message(
signable_message=signable_message,
signature=signature
)
logger.debug('got %s as address_recovered' % address_recovered)
logger.debug(f'got "{message}" as a message')
address_recovered = Web3Helper.personal_ec_recover(message, signature)
logger.debug(f'got "{address_recovered}" as address_recovered')
return address_recovered
except Exception as e:
logger.error(f'get_signer_address: {e}')
Expand Down Expand Up @@ -49,48 +41,3 @@ def compare_eth_addresses(address, checker, logger):
logger.debug("Checker is not web3 valid")
return False
return Web3.toChecksumAddress(address) == Web3.toChecksumAddress(checker)


def can_update_did(ddo, updated, signature, logger):
"""
Check if the signer is allowed to update the DDO
:param record ddo: DDO that has to be updated
:param str updated: Updated field passed by user
:param str signature: Signature of the updated field, using web3.eth.personal.sign
:param logger: instance of logging
:return: boolean TRUE if the signer is allowed to update the DDO
"""
if ddo['updated'] is None or updated is None or ddo['updated'] != updated:
return False
address = get_signer_address(updated, signature, logger)
if address is None:
return False
if compare_eth_addresses(address, ddo['publicKey'][0]['owner'], logger) is True:
return True
return False


def can_update_did_from_allowed_updaters(ddo, updated, signature, logger):
"""
Check if the signer is allowed to update the DDO. List of signers is taken from ENV variabile RATING_ALLOWED_UPDATER
:param record ddo: DDO that has to be updated
:param str updated: Updated field passed by user
:param str signature: Signature of the updated field, using web3.eth.personal.sign
:param logger: instance of logging
:return: boolean TRUE if the signer is allowed to update the DDO
"""
allowed_updater = os.environ.get("RATING_ALLOWED_UPDATER")
logger.debug('got RATING_ALLOWED_UPDATER: %s' % allowed_updater)
if ddo['updated'] is None or updated is None or ddo['updated'] != updated:
logger.debug("mismatch updated")
return False
address = get_signer_address(updated, signature, logger)
if address is None:
logger.debug("signer_address is none")
return False
if allowed_updater is None:
logger.debug("allowedUpdater is None")
return False
if compare_eth_addresses(address, allowed_updater, logger) is True:
return True
return False
8 changes: 4 additions & 4 deletions aquarius/app/dao.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@

class Dao(object):

def __init__(self, config_file=None):
self.oceandb = OceanDb(config_file).plugin
def __init__(self, oceandb=None, config_file=None):
self.oceandb = oceandb or OceanDb(config_file).plugin

def get_all_listed_assets(self):
assets = self.oceandb.list()
Expand All @@ -25,7 +25,7 @@ def get_all_listed_assets(self):
asset_with_id.append(self.oceandb.read(asset['id']))
except Exception as e:
logging.error(f'Dao.get_all_listed_assets: {str(e)}')
pass

return asset_with_id

def get_all_assets(self):
Expand All @@ -36,7 +36,7 @@ def get_all_assets(self):
asset_with_id.append(self.oceandb.read(asset['id']))
except Exception as e:
logging.error(f'Dao.get_all_assets: {str(e)}')
pass

return asset_with_id

def get(self, asset_id):
Expand Down
Loading

0 comments on commit 182f848

Please sign in to comment.