Doc: refactor documentation

README.md

@@ -20,6 +20,8 @@ This software is experimental and part of the research being done at BBVA Labs.
 We will eventually publish our research work, analysis and the experiments for
 anyone to reproduce.
 
+To dive into deep explanation, see [QED mission](docs/mission.md).
+
 ## Motivation
 The use of a technology that allows to verify the information it stores is
 quite broad. Audit logs are a common tool for forensic investigations and legal
@@ -146,7 +148,7 @@ For more elaborated examples please review the [Advanced Usage](docs/advanced_us
 
 ## Contributions
 
-Contributions are very welcome, see [CONTRIBUTING.md](https://github.com/BBVA/qed/blob/master/CONTRIBUTING.md)
+Contributions are very welcome, see [CONTRIBUTING.md](docs/contribute/contributing.md)
 or skim [existing tickets](https://github.com/BBVA/qed/issues) to see where you could help out.
 
 ## License

docs/advanced_usage.md

@@ -5,12 +5,12 @@
 Besides the standalone example given in the [README](../README.md), QED is also designed 
 to be a production-ready cluster. Here you can find some detailed examples.
 
-<p align="center"><img width="100%" src="./full_architecture.png" alt="Architecture overview"/></p>
+<p align="center"><img width="100%" src="./architecture/full_architecture.png" alt="Architecture overview"/></p>
 
 ## QED cluster
 
 In order to guarantee reliability and high availabity, QED servers include
-hashicorp's [raft](https://github.com/hashicorp/raft) consensus protocol implementation.
+[hashicorp's raft](https://github.com/hashicorp/raft) consensus protocol implementation.
 An architectural perspective can be found at [raft](architecture/raft.md) doc. file. 
 
 To have identified the leader beforehand (demo purpose), launch first a single 

CONTRIBUTING.md → docs/contribute/contributing.md

@@ -3,9 +3,9 @@
 You can contribute in a few different ways:
 
 * Submit issues through our [issue tracker](https://github.com/bbva/qed/issues) on Github.
-* If you wish to make code changes please check out [Our Guidelines](docs/pull_requests.md) about **Pull Requests** 
+* If you wish to make code changes please check out [Our Guidelines](pull_requests.md) about **Pull Requests** 
     and [GitHub Forks / Pull requests model](https://help.github.com/articles/fork-a-repo/):
 * If you wish to make code changes, or contribute something new, please follow the
 
 fork the repo, make the change and propose it back by submitting a pull request.
-* Useful notes used by the team are in [docs/development.md](docs/development.md) file 
+* Useful notes used by the team are in [development.md](development.md) file 

docs/mission.md

@@ -0,0 +1,115 @@
+# Introduction
+
+QED implements a forward-secure append-only persistent authenticated data structure.
+Each append operation produces as a result a cryptographic structure (a signed snapshot), which can verify:
+
+* whether or not a piece of data is on QED.
+* whether or not the appended data is consistent, in insertion order, to another entry.
+
+To verify both statements, we need the snapshot, the piece of data inserted and a QED proof.
+QED emits a snapshot on every event insertion, and they need to be accessible elsewhere.
+Also QED does not verify nor emit unsolicited proofs, it’s the user responsibility to know when and how to verify the data depending on their needs.
+
+Last, QED does not store the data itself, only a representation of it produced by a collision-resistant hash function.
+QED does not provide means to map a piece of data to a QED event, so the semantic of the appended data and the relation between each item appended is also a client responsibility.
+
+We define QED event as the result of applying a SHA256 hash function to a piece of data generated for its inclusion in QED. 
+Each event has a temporal relation between the next and the prior one.
+QED represents the temporal relation by an always increasing 64-bit integer called QED version.
+
+All the scripts and code described live in the branch ‘tampering’ in the main repository.
+
+## Why?
+
+There are multiple technologies to achieve a similar functionality as QED, such as signed data in a database, or block chain’s related structures.
+
+The advantages of the data structure QED implements are:
+
+* scalability to reach thousands of millions of events.
+* proof of membership or non-membership generation in logarithmic time.
+* proofs of logarithmic size.
+* proof of temporal consistency related to QED insertion time.
+
+# Architecture and components
+
+In every QED scenario there might be one or more components of this nature:
+
+* QED log: where the authentication data lives, normally a RAFT replicated cluster.
+* QED gossip agents: network on which QED emits snapshots to agents.
+* QED snapshot store: stores all the snapshots generated by QED.
+* Events source: something that stores the data needed to build QED events.
+* Application: a system/action that works with data from the event source to its job.
+* Third party services: a system/action used by application or event source to execute their job.
+
+The relation between these components will affect the deployment architecture as there are multiple valid alternatives which guarantee a correct QED operation and service.
+
+The relation between the QED log, gossip agents, and snapshot store depends on their connectivity requirements and APIs:
+
+* QED log receives events using an HTTP, and a protobuf based GRPC API.
+* QED log emits via a gossip UDP network messages processed by the QED agents.
+* QED publisher agent pushes snapshots to the store using a given HTTP API.
+
+The relationship between the event source, third party services, and the application does not affect the QED per se, but the application must be able to map an event to a QED event univocally, because QED Log stores no data but a representation in the form of a SHA256 hash.
+
+The application needs to talk to QED Log, either directly by its API or using one of the QED Log supported client libraries.
+
+Please refer to the QED manual to knows about how to deploy a production instance.
+
+# Tamperings
+
+### Event source
+
+* QED can emit a verifiable proof to check if an event is on QED.
+* QED can emit a verifiable proof to check if two events are consistent with each other in the order of insertion.
+* The user has the responsibility to ask for these proofs and verify them and can user the QED gossip network to build auditors and monitors adapted to its use case.
+* The user should use a secret unknown to QED for the QED event mapping function.
+* QED does not audit nor emits proof or verify proactively any event.
+* QED does not alert in real time about event source changes.
+
+### Application
+
+* We cannot guarantee an application will use QED.
+* We can use QED capabilities to build external tooling to check the application expected behaviour.
+
+### Third party
+
+* We can use QED to verify changes in third-party data source using a QED client which must implement a mapping function between the third-party data to QED events.
+* We can use QED to check the history of changes of a third party ordered data source. Also, the source of the order could be build from another means.
+
+### QED log
+
+QED is resistant to naïve attempts to tamper with its database. A modification of a single leaf of a tree, or path is detected easily. This kind of tampering tries to harm the credibility of the system by making it complain or to avoid the validation of a correct event. *Once the QED is tampered with, there is no rollback. Only a complete rebuild can ensure its coherence.*
+
+We can alter the events stored in QED in a way that the proofs will verify only if the QED version is reset to an old version and we insert events from that version again using the QED append algorithm to regenerate all the intermediate nodes of the trees:
+
+    v0————>v1————>v2————>v3 ————>v4 ————> v5              original history
+                         |                                version reset
+                         |—>v3’————>v4’————>v5’————>v6    forked history
+
+This a theoretical attack, in practice it is unfeasible to do such an attack without being detected, as it requires modifying a running program which replicates on real time, without being noticed.
+Also, even if the attack happens, it can be detected doing a full audit checking all events against the event source and the snapshot store.
+
+*QED will not know which component was tampered, only that an event being check has either its event source, its snapshot, or its QED event altered. We will not establish the source of truth unless we do a full audit which comprises the insertion of all QED events again to regenerate the source, the log and the snapshots to check the differences.*
+
+To further protect a QED deployment against such tampering, we recommend salting the QED events with a secret (which QED does not know) verifiable by the event stakeholders and recommends implementing a monitoring agent that check the snapshot store searching for duplicate QED versions.
+
+Another recommendation is to make QED clusters to block any arbitrary nonauthenticated joins, replications or from-disk recoveries.
+
+Last, the teams or companies in charge of the QED log, agents and snapshot store should be different to avoid collusion.
+
+### QED agents
+
+The agent's mission is to check the QED activities to identify anomalous behaviours and also publish the snapshots into the store.
+
+They can be tampered as any other application, making them publish altered snapshots or to omit any functionality.
+But the QED proofs verification process will detect modifications regarding the events being checked as long as the event source or QED log are untampered.
+
+The gossip agents can use certificates and/or user/password to authenticate against each other.
+
+### QED snapshot store
+
+The snapshot store can be compromised to change stored snapshots, but like in the QED agents case, the QED proofs verification process will fail as long as the event source or QED log are untampered.
+
+# Use cases
+
+See some identified [use cases](use_cases/use_cases.md).