Skip to content

Commit

Permalink
Add advanced_usage and cleanup docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@iknite
iknite committed Dec 10, 2018
1 parent c445d70 commit 9c421e6
Show file tree
Hide file tree
Showing 4 changed files with 205 additions and 70 deletions.
1 change: 1 addition & 0 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ You can contribute in a few different ways:
* If you wish to make code changes, or contribute something new, please follow the
[GitHub Forks / Pull requests model](https://help.github.com/articles/fork-a-repo/):
fork the repo, make the change and propose it back by submitting a pull request.
* Useful notes used by the team are in [docs/development.md]() file
141 changes: 71 additions & 70 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,26 +3,51 @@
[![GoReport](https://goreportcard.com/badge/github.com/bbva/qed)](https://goreportcard.com/report/github.com/bbva/qed)
[![GoDoc](https://godoc.org/github.com/bbva/qed?status.svg)](https://godoc.org/github.com/bbva/qed)

<p align="center"><a href="https://en.wikipedia.org/wiki/Q.E.D."><img width="55%" src="./qed_logo.png" alt="Quod Erat Demonstrandum"/><br/>(quod erat demonstrandum)</a></p>
<p align="center"><img width="90%" src="./qed_whiteboard.png" alt="Whiteboard depicting a use case for qed"/></p>
<p align="center"><a href="https://en.wikipedia.org/wiki/Q.E.D."><img width="45%" src="./qed_logo.png" alt="Quod Erat Demonstrandum"/><br/>(quod erat demonstrandum)</a></p>
<p align="center"><img width="70%" src="./qed_whiteboard.png" alt="Whiteboard depicting a use case for qed"/></p>


## Overview

***qed*** is a software to test the scalability of authenticated data structures. Our mission is to design a system which, even when deployed into a non-trusted server, allows one to verify the integrity of a chain of events and detect modifications of single events or parts of its history.
***qed*** is a software to test the scalability of authenticated data structures.
Our mission is to design a system which, even when deployed into a non-trusted
server, allows one to verify the integrity of a chain of events and detect
modifications of single events or parts of its history.

This software is experimental and part of the research being done at BBVA Labs. We will eventually publish our research work, analysis and the experiments for anyone to reproduce.
This software is experimental and part of the research being done at BBVA Labs.
We will eventually publish our research work, analysis and the experiments for
anyone to reproduce.

## Motivation
The use of a technology that allows to verify the information it stores is quite broad. Audit logs are a common tool for forensic investigations and legal proceedings due to its utility for detecting database tampering. Malicious users, including insiders with high-level access, may perform unlogged activities or tamper with the recorded history. The evidence one seeks in these sorts of investigations often takes the form of statements of existence and order. But this kind of tamper-evident logs have also been used for other use cases: building versioned filesystems like version control systems, p2p protocols or as a mechanism to detect conflicts in distributing systems, like data inconsistencies between replicas.

All of these use cases share something in common: the proof of order and integrity is fulfilled building data structures based on the concept of hash chaining. This technique allows to establish a provable order between entries, and comes with the benefit of tamper-evidence, ensuring that any snapshot to a given state of the log is implicitly a snapshot to all prior states. Therefore, any subsequent attempt to remove or alter some log entries will invalidate the hash chain.

In order to prove that an entry has been included in the information storage, and that it has not been modified in an inconsistent way we need:

* **Proof of inclusion**, answering the question about if a given entry is in the log or not.
* **Proof of consistency**, answering the question about if a given entry is consistent with the prior ones. This ensures the recorded history has not been altered.
* **Proof of deletion**, so we are able to know when a log has been tampered with at its source location.
The use of a technology that allows to verify the information it stores is
quite broad. Audit logs are a common tool for forensic investigations and legal
proceedings due to its utility for detecting database tampering. Malicious
users, including insiders with high-level access, may perform unlogged
activities or tamper with the recorded history. The evidence one seeks in these
sorts of investigations often takes the form of statements of existence and
order. But this kind of tamper-evident logs have also been used for other use
cases: building versioned filesystems like version control systems, p2p
protocols or as a mechanism to detect conflicts in distributing systems, like
data inconsistencies between replicas.

All of these use cases share something in common: the proof of order and
integrity is fulfilled building data structures based on the concept of hash
chaining. This technique allows to establish a provable order between entries,
and comes with the benefit of tamper-evidence, ensuring that any snapshot to a
given state of the log is implicitly a snapshot to all prior states. Therefore,
any subsequent attempt to remove or alter some log entries will invalidate the
hash chain.

In order to prove that an entry has been included in the information storage,
and that it has not been modified in an inconsistent way we need:

* **Proof of inclusion**, answering the question about if a given entry is in
the log or not.
* **Proof of consistency**, answering the question about if a given entry is
consistent with the prior ones. This ensures the recorded history has not been
altered.
* **Proof of deletion**, so we are able to know when a log has been tampered
with at its source location.

Some of the systems that can be built upon these technologies are:

Expand All @@ -31,29 +56,32 @@ Some of the systems that can be built upon these technologies are:
* Verifiable Transaction Log - verifiable business activity
* Verifiable Data Blocks - verifiable HDFS blocks

A number of hash data structures have been proposed for storing data in a tamper-evident fashion (see references [below](#other-projects-papers-and-references)). All of them have at their core a Merkle tree or some variant.

Our work draws strongly from the **Balloon proposals**, with some modifications of our own that aim to improve scalability.
A number of hash data structures have been proposed for storing data in a
tamper-evident fashion (see references
[below](#other-projects-papers-and-references)). All of them have at their core
a Merkle tree or some variant.

## Environment

We use the [Go](https://golang.org) programming language and set up the environment as
described in its [documentation](https://golang.org/doc/code.html)
Our work draws strongly from the **Balloon proposals**, with some modifications
of our own that aim to improve scalability.

## Getting started

Below you will find the basic usage in order to test QED capabilites of storing
and verifing events. This example runs a standalone server. You can add any
source of ordered events like logs, ledgers, etc...

### Standalone example
- Download the software and its dependencies
```
go get -v -u -d github.com/bbva/qed/...
```
- Start the server
- Start the standalone server

```
cd "$GOPATH/src/github.com/bbva/qed"
rm -rf /var/tmp/qed
mkdir -p /var/tmp/qed/{data,raft}
ssh-keygen -t ed25519 -P '' -f ~/.ssh/id_ed25519-qed
go run main.go start --apikey key --keypath ~/.ssh/id_ed25519-qed --log info
go run main.go start --apikey my-key --keypath ~/.ssh/id_ed25519-qed
```

- Using the client
Expand All @@ -62,62 +90,35 @@ Our work draws strongly from the **Balloon proposals**, with some modifications

```
go run \
main.go \
--apikey my-key \
client \
--endpoint http://localhost:8080 \
add \
--key 'test event' \
--value 2 \
--log info
```
- membership event
```
go run \
main.go \
--apikey my-key \
client \
--endpoint http://localhost:8080 \
membership \
--hyperDigest 3ec11c37f0a53ff5c4cfc3cf2573c33a9721cd25d8e670a3b2be0fda5724bb5c \
--historyDigest 776b33eab8ed829ecffab3d579bf7ccbcc126b94bac1aaca7d5d8b0a2687bdec \
--version 0 \
--key 'test event' \
--log info
main.go \
--apikey my-key \
client \
--endpoint http://localhost:8080 \
add \
--key 'test event' \
--value 2 \
--log info
```
- verify event
```
go run \
main.go \
--apikey my-key \
client \
--endpoint http://localhost:8080 \
membership \
--hyperDigest 3ec11c37f0a53ff5c4cfc3cf2573c33a9721cd25d8e670a3b2be0fda5724bb5c \
--historyDigest 776b33eab8ed829ecffab3d579bf7ccbcc126b94bac1aaca7d5d8b0a2687bdec \
--version 0 \
--key 'test event' \
--log info \
--verify
main.go \
--apikey my-key \
client \
--endpoint http://localhost:8080 \
membership \
--hyperDigest 3ec11c37f0a53ff5c4cfc3cf2573c33a9721cd25d8e670a3b2be0fda5724bb5c \
--historyDigest 776b33eab8ed829ecffab3d579bf7ccbcc126b94bac1aaca7d5d8b0a2687bdec \
--version 0 \
--key 'test event' \
--log info \
--verify
```
## Useful commands
- Go [documentation server](http://localhost:6061/pkg/github.com/bbva/qed/)
```
godoc -http=:6061 # http://localhost:6061/pkg/github.com/bbva/qed/
```
- Test everything
For more elaborated examples please review the [Advanced Usage](docs/advanced_usage.md) documentation
```
go test -v "$GOPATH"/src/github.com/bbva/qed/...
```
## Other projects, papers and references
- github related projects
Expand Down
115 changes: 115 additions & 0 deletions docs/advanced_usage.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# Advanced Usage

## Overview

Besides the standalone example given in the README.md, QED are created to be a
production-ready cluster. Here you can find some detailed examples.

## QED cluster

In order to garantee reliability and High Availabity QED storage servers are
created around hashicorp's [raft](https://github.com/hashicorp/raft) implementation.

### Starting cluster mode

To have identified the leader beforehand, we launch a server and then some
disposable followers.

```
go run main.go start \
-k my-key \
-p $(mktemp -d /var/tmp/demo.XXX) \
--raftpath $(mktemp -d /var/tmp/demo.XXX) \
-y ~/.ssh/id_ed25519-qed \
--http-addr :8080 \
--raft-addr :9000 \
--mgmt-addr :8090 \
-l error
```

### Starting two followers
```
CLUSTER_SIZE=2
for i in $(seq 1 $CLUSTER_SIZE); do
go run main.go start \
-k my-key \
-p $(mktemp -d /var/tmp/demo.XXX) \
--raftpath $(mktemp -d /var/tmp/demo.XXX) \
-y ~/.ssh/id_ed25519-qed \
--http-addr :808$i \
--join-addr :8090 \
--raft-addr :900$i \
--mgmt-addr :809$i \
--node-id node$i \
-l error
done
```

Know events must be added **ONLY** in the leader, but events can be verified in
any follower (and it's the way to go).

A Quick example could be use the README standalone client example changing the
endpoint port `--endpoint http://localhost:8081` in the verify event command.

## Agents

In order to allow public `auditors`, we need to ensure a public storage in which
QED server sends the snapshot information. `publisher` agents are designed to do
it. Finally `monitors` agents are check internal consitency.

## test_service
To use a demo public log-storage QED provides a small helper wich uses a in-memory
structure to store signed snapshots.

```
go run tests/e2e/gossip/test_service.go
```

### Publisher

```
# this variables will be used in all the publiser, auditor and monitor examples.
export masterEndpoint="127.0.0.1:9100"
export publisherEndpoint="http://127.0.0.1:8888"
export alertsEndpoint="http://127.0.0.1:8888"
export qedEndpoint="http://127.0.0.1:8080"
```

```
go run main.go agent \
--alertsUrls $alertsEndpoint \
publisher \
-k my-key \
--bind 127.0.0.1:9300 \
--join $masterEndpoint \
--endpoints $publisherEndpoint \
--node publisher0 \
-l info
```

### auditor
```
go run main.go agent \
--alertsUrls $alertsEndpoint \
auditor \
-k my-key \
--bind 127.0.0.1:9100 \
--join $masterEndpoint \
--qedUrls $qedEndpoint \
--pubUrls $publisherEndpoint \
--node auditor0 \
-l info
```

### monitor
```
go run main.go agent \
--alertsUrls $alertsEndpoint \
monitor \
-k my-key \
--bind 127.0.0.1:920$i \
--join $masterEndpoint \
--endpoints $qedEndpoint \
--node monitor0 \
-l info
```
18 changes: 18 additions & 0 deletions docs/development.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
## Environment

We use the [Go](https://golang.org) programming language and set up the
environment as described in its [documentation](https://golang.org/doc/code.html)

## Useful commands

- Go [documentation server](http://localhost:6061/pkg/github.com/bbva/qed/)

```
godoc -http=:6061 # http://localhost:6061/pkg/github.com/bbva/qed/
```

- Test everything

```
go test -v "${GOPATH}"/src/github.com/bbva/qed/...
```

0 comments on commit 9c421e6

Please sign in to comment.