Merge branch 'master' into javi-1147-rollback.wizard

modules/docs/src/main/tut/docs/Client.md

@@ -4,6 +4,9 @@ title: Client
 section: docs
 ---
 
-# Client
+This describes the different features of 9Cards Client:
 
-TODO
+* [Installation](client/Installation.html)
+* [Architecture](client/Architecture.html)
+* [Database](client/Database.html)
+* [Cloud Storage](client/CloudStorage.html)

modules/docs/src/main/tut/docs/Libraries.md

@@ -4,6 +4,24 @@ title: Libraries
 section: docs
 ---
 
-# Client
+# Client Stack
 
-TODO
+* [Scala](https://www.scala-lang.org/): we are using Scala instead of Java
+* [Cats](http://typelevel.org/cats/): a library which provides abstractions for functional programming in Scala
+* [Monix](https://monix.io/): a high-performance Scala library for composing asynchronous and event-based programs
+* [SBT Android Plugin](https://github.com/scala-android/sbt-android): SBT plugin for compiling Scala on Android
+* [Macroid](http://47deg.github.io/macroid/): DSL for Android in order to create user interfaces in functional way with Scala Macros
+* [Android SDK](https://developer.android.com/guide/index.html)
+
+# Server Stack
+
+* [Spray](http://spray.io/): an open-source toolkit for building REST/HTTP-based integration layers on top of Scala and Akka
+* [Akka](http://akka.io/): a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications
+* [Circe](https://circe.github.io/circe/): JSON library for Scala
+* [Doobie](http://github.com/tpolecat/doobie): a pure functional JDBC layer for Scala
+* [Http4s](http://http4s.org/): a typeful, purely functional HTTP library for client and server applications written in Scala.
+* [Cats](http://typelevel.org/cats/): a library which provides abstractions for functional programming in Scala
+* [Monix](https://monix.io/): a high-performance Scala library for composing asynchronous and event-based programs
+* [Flyway](https://flywaydb.org/): an open-source database migration tool
+* [Typesafe config](https://typesafehub.github.io/config/): a configuration library for JVM languages
+* [Shapeless](http://github.com/milessabin/shapeless) is used as a utility in several places in the code, and it is relied upon by the libraries `Circe`, `ScalaCheck`, or `Doobie`.

modules/docs/src/main/tut/docs/Server.md

@@ -4,6 +4,10 @@ title: Server
 section: docs
 ---
 
-# Server
+This describes the different features of 9Cards Server:
 
-TODO
+* [Installation](server/Installation.html)
+* [Architecture](server/Architecture.html)
+* [Endpoints](server/Endpoints.html)
+* [Authentication](server/Authentication.html)
+* [Cache](server/Cache.html)

modules/docs/src/main/tut/docs/client/Architecture.md

@@ -16,6 +16,8 @@ The architecture is divided in **3 layers**:
 - **Process**: This module contains the uses cases. We have connect to every service in the previous layer in order to create your process
 - **Android**: This module contains the Android SDK with Activities, Fragments and so on, used in your project. Every screen have jobs, with the actions in your UI and Ui Actions
 
+Our architecture is based in 2 libraries of Typelevel: Cats and Monix. This 2 libraries are the main libraries in order to create a functional architecture in our project. 9Cards aren’t using Android libraries for views, we only use the Android SDK and all view have been created in Scala
+
 ![architecture](/nine-cards-v2/img/9cards_architecture.png)
 
 In order to can compose the methods of the Ui and Services, all methods must return the same type. The type is define in commons module and it's the next:

modules/docs/src/main/tut/docs/index.md

@@ -6,4 +6,10 @@ section: docs
 
 # Introduction
 
-TODO
+In 2014 we created 9 Cards App 1.0 a new Launcher concept for Android applications. 9 Cards is a customizable Home Launcher to improve your Android experience organizing your apps and adapting to daily routine. 
+
+For 9 Cards 2.0 we have 3 main goals:
+
+ * Use Scala instead of Java in order to provide to the community a real application made in Scala with thousand of users
+ * Use a functional approach in a world where all people work in imperative way
+ * Use similar architecture in client and backend

modules/docs/src/main/tut/docs/server/Architecture.md

@@ -4,7 +4,7 @@ title: Architecture
 section: docs
 ---
 
-# 9Cards Backend Architecture
+# Architecture
 
 This file describes the architecture of the Nine Cards Backend.
 
@@ -48,36 +48,6 @@ for retrieving or storing information, or for sending notifications. The externa
 * **Firebase**. The [Firebase Cloud Messaging](http://firebase.google.com/docs/cloud-messaging/) API is used to notify
   the subscribers of any shared collection of changes to the list of apps included in the shared collection.
 
-### Libraries
-
-The backend depends for its functionality on several external libraries, apart from the `Scala` libraries.
-The main libraries and frameworks used in the backend are the following ones:
-
-* [Cats](http://typelevel.org/cats/) is a core library for the backend. The architecture follows the
-  [Data Types à la carte](http://dblp.org/rec/html/journals/jfp/Swierstra08) paper, and to implement
-  such architecture we make use of `Cats` implementations for [`Monad`](http://github.com/typelevel/cats/blob/master/core/src/main/scala/cats/Monad.scala),
-  for [free monads](http://github.com/typelevel/cats/blob/master/free/src/main/scala/cats/free/Free.scala),
-  [natural transformations]([`cats.arrow.FunctionK`](http://github.com/typelevel/cats/blob/master/core/src/main/scala/cats/arrow/FunctionK.scala),
-  and, in a few cases, [monad transformers](http://github.com/typelevel/cats/blob/master/core/src/main/scala/cats/data/EitherT.scala).
-
-* [Spray](http://spray.io/) is used to build the `HTTP-REST` API, that serves as the external
-  interface for the backend application. This `api` is used by each Nine Cards client to access
-  the functionality of the backend. The entities transmitted through this API are all encoded in
-  [Json](http://en.wikipedia.org/wiki/JSON), for which we use `spray-json`.
-* [Akka](http://akka.io/) is used, but just enough to support the `spray` api.
-* [Circe](http://travisbrown.github.io/circe/), a library for implementing JSON encoding and decoding for
-  data classes. Circe is used for a few classes in implementing the `api`, but it is mostly used for the
-  communication with the external HTTP-REST services.
-* [Doobie](http://github.com/tpolecat/doobie) is a purely-functional database access library for Scala, which
-  we use for performing the queries and insertions into the database.
-* [Http4s](http://http4s.org/) is an HTTP client that we use for implementing all the services that interact with the APIs
-  of external services, such as Google Authentication, Analytics, Firebase, or the Android Market.
-* [Enumeratum](http://github.com/lloydmeta/enumeratum/) is used to represent
-* Testing of the different modules is done with a combination of [Specs2](http://etorreborre.github.io/specs2/)
-  and [ScalaCheck](http://www.scalacheck.org/).
-* [Shapeless](http://github.com/milessabin/shapeless) is used as a utility in several places in the code,
-  and it is relied upon by the libraries `Circe`, `ScalaCheck`, or `Doobie`.
-
 ### Core Modules
 
 The backend source code is organised into four modules, or `sbt` projects, called

modules/docs/src/main/tut/docs/server/Authentication.md

@@ -3,7 +3,84 @@ layout: docs
 title: Authentication
 section: docs
 ---
+# Authentication and Authorization
 
-# Authentication
+The Nine Cards Back-End Application authenticates almost all of the endpoints.
 
-TODO
+# Client Signup in BackEnd
+
+We describe now the process of interactions performed for a client (user and device) to sign up in the NCBE.
+In essence, this process is just a third-party authentication (the third party being the NCBE) following the [OAuth 2.0 protocol](https://developers.google.com/identity/protocols/OAuth2).
+It consists of an interaction between the NCBE, the **Google Account Services** (GAC), and the **Client**, which is the Ninecards Android application running from the user device.
+
+1. **Grant Google Account Permissions.**
+   The *Client* sends a request to the GAC to open an authorization token. The request carries the user's `email` and the `deviceId`,
+   which uniquely identify the instance of the NineCardsClient issuing the request.
+   If the request is accepted, the GAC responds with success and includes in the response a `tokenId`, which identifies a short-lived OAuth session within the GAC.
+
+2. **Client signup in NCBE.**
+   The *Client* sends a HTTP request to the NCBE's endpoint `POST {apiRoot}/login`, to register in it.
+   This request carries (as a JSON object in the body entity) three fields:
+   * the `email` that serves as an external identifier for the user running the client,
+   * the `androidId`, that identifies the Android device in which the user is running the client application, and
+   * the `tokenId` that the client received from the GAC in Step 1.
+
+   If the request succeeds, the NCBE records the client's user and device and returns a response to the client app.
+   The response carries (in a JSON object in the request body) two fields: a `sessionToken` and an `apiKey`.
+   * The `sessionToken` is a unique identifier of the user within the NCBE instead of the user's email.
+   * The `apiKey` is the private cryptographic key that the client uses after signup to authenticate in the NCBE.
+
+3. **NCBE access to GAC.**
+   To carry out the process of endpoint `POST {apiRoot}/login`, the NCBE communicates to the GAC to validate the `tokenId` from the request body.
+   If the `tokenId` is validated, the GAC returns a successful response.
+
+# User Authentication
+
+All NCBE endpoints, except from the one to read the API documentation and the one to signup,
+carry out an authentication step, to check that the client app sending the requests is acting for a registered user.
+The information for user authentication is carried in the HTTP headers `X-Android-ID`, `X-Auth-Token` and `X-Session-Token`.
+
+* The `X-Android-ID` should give the `androidId` of the client's device. Note that, since the GAC process in the signup
+  involves a user and device, it is the device itself and not just the user that should be signed up beforehand.
+
+* The `X-Session-Token` should carry the user's `sessionToken` that the NCBE generated and gave to the client in Step 3 of the signup process.
+  This value acts as a way to identify the user within the NCBE.
+
+* The `X-Auth-Token` is a [Hash-based message authentication code](https://en.wikipedia.org/wiki/Hash-based_message_authentication_code), which is used for authenticating the request.
+  It ensures that the client which is using the `sessionToken` is the one that is acting for that user.
+
+The value of the `X-Auth-Token` header is computed as follows.
+The *message* to be signed is just the full [URI](https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.2.2) of the request, including protocol, host, port, path, and query.
+The *cryptographic key* used is the user's `apiKey`, which the NCBE generates and gives to the client (with the `sessionToken`) at the end of the signup process.
+The code is then calculated using the `sha512` hash function.
+To calculate the value, you can use one of these methods:
+
+* The command `openssl` can generate the HMAC digest of a message. For example, to digest the URI `http://localhost:8080/collections/a` with the key `foo`, you would run:
+
+        echo -n "http://localhost:8080/collections/a" | openssl dgst -sha512 -hmac "foo" | awk '{print $2}'
+
+* This [web page](http://www.freeformatter.com/hmac-generator.html) offers a graphic dialog to generate the digest.
+  The request URL would be into the `message` dialog, the user's `apiKey` would go into the `SecretKey` text box,
+  and the algorithm would be "SHA512". The result would be the digest.
+
+# Manually obtaining a TokenId
+
+Sometimes, you may want to carry out manually the first step of the signup, which is the communication between the
+Client and the Google Account Services.
+
+Google provides an [`OAuth 2.0 Playground`](https://developers.google.com/oauthplayground/), which can be used to generate a Google ID Token.
+We use it to generate an OAuth token that, within the Google API, allows us to read the information needed for the client.
+This information is only the user's email, so the scope used is `https://www.googleapis.com/auth/userinfo.email`.
+
+1. Open the [OAuth 2.0 Playground page](https://developers.google.com/oauthplayground/).
+   In this page, look in the lateral pane for the menu `Google+ API v1`, from this menu mark the scope `https://www.googleapis.com/auth/userinfo.email`,
+   and push the `Authorize APIs` button.
+2. If you have several Google accounts stored in the browser, you may be asked to select one. You will then
+   be presented with a usual Google permissions dialog, asking you to allow an application to _Read your email address_.
+    Press _Allow_.
+3. After pressing _Allow_, the playground page would change to a new view. The modified page shows you `Request/Response`.
+   In the left pane there is a text field labelled _Authorization Code_, and a button labelled _Exchange authorization code for tokens_.
+   Press this button. Google OAuth playground generates then a new token, which is shown in the API response in the right pane, in a field named as _id_token_.
+   This _id_token_ identifies a session within the _Google Account Services_, which is to expire within the hour.
+4. Copy the value of the _id_token_ generated. A request to login endpoint `POST {apiRoot}/login` should include in the
+   body a JSON object with the field `tokenId`. The value of this field should be the _id_token_.

modules/docs/src/main/tut/docs/server/Cache.md

@@ -6,4 +6,81 @@ section: docs
 
 # Cache
 
-TODO
+In order to reduce the number of request to Google Play API, we are storing the Google Play Info about the app in a intermediate cache using REDIS 
+
+##Redis cache management
+
+Redis cache will contain only one key per package and could store four type of values depending on the result of getting info from Google Play:
+- `resolved`: Those packages for which the process of getting Google Play info has been successfully completed will have this tag.
+- `pending`: If while the execution of a multi-package request, the Google Play API returns an error (`401 Unauthorized` or `429 Too many requests`), all the packages without info will be tag as pending. An internal process will try to get the info later.
+- `error`: If the request to get the Google Play information return a `404 Not Found` error, the package will be tagged as error.
+- `permanent`: All the packages with this tag are known applications that are not published in Google Play, but we can categorized them (for instance, the Samsung camera)
+
+The format of the key and value will be:
+- **Key:** `"com.package.name:TYPE"`
+- **Value:**
+  - Example of a `resolved` item
+
+```
+{
+    "packageName": "com.package.name",
+    "title": "Package title",
+    "free": true,
+    "icon": "http://lh3.googleusercontent.com/aYbdIM1abwyVSUZLDKoE0CDZGRhlkpsaPOg9tNnBktUQYsXflwknnOn2Ge1Yr7rImGk",
+    "stars": 4.5,
+    "downloads": "500,000,000 - 1,000,000,000",
+    "categories": ["SOCIAL"],
+    "screenshoots": ["url1", "url2"]
+}
+```
+  - Example of a `pending` item
+
+```
+{}
+```
+  - Example of a `error` item with the dates when the errors were produced
+
+```
+[
+    "date1",
+    "date2"
+] 
+```
+  - Example of a `permanent` item
+
+```
+{
+    "packageName": "com.package.name",
+    "categories": ["SOCIAL"]
+}
+```
+
+## Workflow
+
+The described workflow will be performed for each package of the list:
+
+1. Check if exists a key in Redis for the package and the type of the value is `resolved` or `permanent`
+   - If the key/value exists, the process will return the stored package info as resolved
+   - Otherwise the process will continue in step 2
+
+2. Try to get the package info by using Google Play API
+   - If the API returns a valid response:
+     - A new item of type `resolved` will be created in Redis
+     - If a previous `error` or `pending` item exists, this key will be removed
+     - The process will return the package info as resolved
+     - If an error is thrown (like `401 Unauthorized` or `429 Too many requests`), the process will continue in step 3
+
+3. Check if the package exists in Google Play by requesting the server headers (faster and small size of the response)
+   - If the package exists, the process will continue in step 4
+   - Otherwise the process will continue in step 5
+
+4. Check if exists a key in Redis for the package and the type of the value is `pending` or `error`
+   - If a value of type `pending` exists, the process will do nothing
+   - If a value of type `error` exists, the process will change the type of the item to `pending`
+   - Otherwise a new item of type `pending` will be created in Redis
+   - In all the cases, the package will be return as `pending`
+
+5. Check if exists a key in Redis for the package and the type of the value is `error`
+   - If the key exists, a new date will be appended to the value
+   - Otherwise a new item of type `error` will be created in Redis
+   - In all the cases, the package will be return as `error`