HMDA Platform

Introduction

For more information on HMDA, checkout the About HMDA page on the CFPB website.

The HMDA Platform

This repository contains the code for the entirety of the HMDA platform backend. This platform has been designed to accommodate the needs of the HMDA filing process by financial institutions, as well as the data management and publication needs of the HMDA data asset.

The HMDA Platform uses sbt's multi-project builds, each project representing a specific task. The platform is an Akka Cluster application that can be deployed on a single node or as a distributed application. For more information on how Akka Cluster is used, see the documentation here

The HMDA Platform is composed of the following modules:

Parser (JS/JVM)

Module responsible for reading incoming data and making sure that it conforms to the HMDA File Specification

Data Validation

Module responsible for validating incoming data by executing validation rules as per the Edit Checks documentation

Persistence

Module responsible for persisting information into the system. It becomes the system of record for HMDA data

Cluster

Module responsible for managing the various cluster roles, as well as starting the Hmda Platform

API

This module contains both public APIs for HMDA data for general use by third party clients and web applications, as well as endpoints for receiving data and providing information about the filing process for Financial Institutions

API Model

This module contains objects and JSON protocols for use by the API project

Query

This module is responsible for interacting with the back-end database, as well as conversion between model objects and database objects.

Panel

This module is responsible for parsing and persisting a CSV-format panel file

Model (JS/JVM)

This module is responsible for maintaining the objects used in our platform

Census

This module is responsible for geographic translation (e.g. state number -> state code)

Publication

This module generates Aggregate and Disclosure reports, as required by HMDA statute.

Dependencies

Java 8 SDK

The HMDA Platform runs on the Java Virtual Machine (JVM), and requires the Java 8 JDK to build and run the project. This project is currently being built and tested on Oracle JDK 8. See Oracle's JDK Install Overview for install instructions.

The HMDA Platform should also run on OpenJDK 8.

Scala

The HMDA Platform is written in Scala. To build it, you will need to download and install Scala 2.11.x

In addition, you'll need Scala's interactive build tool sbt. Please refer to sbt's installation instructions to get started.

Docker

Though Docker is not a dependency of the Scala project, it is very useful for running and smoke testing locally. Use the following steps to prepare a local environment for running the Platform with docker:

First, make sure that you have the Docker Toolbox installed.

If you don't have a Docker machine created, you can create one with the default parameters using the command below. This will be sufficient for running most docker containers (e.g. the dev dependencies for the API), but not for running the entire platform.

docker-machine create --driver virtualbox dev

If you wish to run the entire platform using Docker (currently the only way to run the entire platform), you'll need to dedicate more resources to the Docker machine. We've found that for the full stack to run efficiently, you need approximately:

4 CPUs
6 GB RAM
80 GB Disk space

Assuming you are using Docker Machine to provision your Docker environment, you can check you current settings with the following (ignore the second Memory):

    $ docker-machine inspect | grep 'CPU\|Memory\|DiskSize'
        "CPU": 4,
        "Memory": 6144,
        "DiskSize": 81920,
        "Memory": 0,

If your settings are below these suggestions, you should create a new Docker VM. The following will create a VM named hmda-platform with the appropriate resources:

    $ docker-machine create \
    --driver virtualbox \
    --virtualbox-disk-size 81920 \
    --virtualbox-cpu-count 4 \
    --virtualbox-memory 6144 \
    hmda-platform

After the machine is created, make sure that you connect your shell with the newly created machine

$ eval "(docker-machine env dev)"

Building and Running

Building the .jar

To build JVM artifacts (the default, includes all projects), from the sbt prompt:

> clean assembly

This task will create a fat jar, which can be executed directly on any JDK8 compliant JVM:

java -jar target/scala-2.11/hmda.jar

Running Interactively

Running the Dependencies

Assuming you have Docker-Compose installed (according to the Docker instructions above), the easiest way to get all of the platform's dependencies up and running with the provided docker-compose dev setup:

docker-compose -f docker-dev.yml up

When finished, use docker-compose down to gracefully stop the running containers.

In order to not get Postgres connection error upon signing a submission, you may want to start up the keycloak image as well.

docker-compose up keycloak

Not having the keycloak image running will produce an error in the console, but will not cause the platform to crash.

Running the API

Once the dependencies (above) are running, follow these steps in a separate terminal session to get the API running with sbt:

For smoke testing locally, add the following two environment variables:
- EDITS_DEMO_MODE: This will allow you to use the sample files in this repo for testing the app. Otherwise, edit S025 will trigger for all files.
- HMDA_IS_DEMO: This uses configuration files that allow running the app locally, instead of in a cluster.

export EDITS_DEMO_MODE=true
export HMDA_IS_DEMO=true

Start sbt

$ sbt

Select project to build and run.This will retrieve all necessary dependencies, compile Scala source, and start a local server. It also listens for changes to underlying source code, and auto-deploys to local server.

> project cluster
> clean
> ~re-start

Confirm that the platform is up and running by browsing to http://localhost:8080

When finished, press enter to get the sbt prompt, then stop the project by entering reStop.

Running the Project with Docker

To run only the API

First, ensure there's a compiled jar to create the Docker image with:

sbt clean assembly

Build the docker image

docker build -t hmda-api .

Then, run the docker image

docker run -d -p "8080:8080 -p 8082:8082" hmda-api

The Filing API will run on $(docker-machine ip):8080 The Public API will run on $(docker-machine ip):8082

By default, the HDMA Platform runs with a log level of INFO. This can be changed by establishing a different log level in the HMDA_LOGLEVEL environment variable. For the different logging options, see the reference.conf default configuration file for Akka.

API Load Testing

A load testing scenario for the API has been built using Gatling. The parameters for this test can be configured in the cluster/src/test/resources/application.conf file. To run the default scenario:

$ sbt
> project cluster
> gatling:test

After the test is run, some statistics will be presented on the screen. To further study the results of the test, a report can be opened on a browser from the sbt prompt:

> gatling:lastReport

** NOTE: The load test requires the sample panel file to be loaded before running it.

To run the entire platform

Ensure you have a Docker Machine with sufficient resources, as described in the Docker section above.

Clone hmda-platform-ui,

hmda-platform-auth, hmda-pub-ui, hmda-homepage, and hmda-platform-tools into the same directory as hmda-platform.

 ~/dev/hmda-project$ ls -l
 drwxr-xr-x  22 lortone  staff   748B Jul 27 16:28 hmda-platform/
 drwxr-xr-x  25 lortone  staff   850B Jul 25 17:13 hmda-platform-ui/
 drwxr-xr-x  23 lortone  staff   796B Jul 28 17:15 hmda-platform-auth/
 drwxr-xr-x  23 lortone  staff   796B Jul 28 17:15 hmda-pub-ui/
 drwxr-xr-x  23 lortone  staff   796B Jul 28 17:15 hmda-platform-tools/
 drwxr-xr-x  23 lortone  staff   796B Jul 28 17:15 hmda-homepage/

Build hmda-platform-ui
```
 cd hmda-platform-ui && \
 yarn && \
 cd ..
```
Note: This requires yarn to be installed.
Build hmda-pub-ui
```
 cd hmda-pub-ui && \
 yarn && \
 cd ..
```
Note: This requires yarn to be installed.
Build hmda-homepage
```
 cd hmda-homepage && \
 yarn && \
 cd ..
```
Note: This requires yarn to be installed.
Build hmda-platform-tools
```
 cd hmda-platform-tools && \
 yarn && \
 cd ..
```
Note: This requires yarn to be installed.

Build hmda-platform

 cd hmda-platform && \
 sbt clean assembly

Launch the stack with Docker Compose
```
 docker-compose up
```
This will bring up all the HMDA Platform services. The first run may take several minutes.
Discover your Docker host's IP
```
 echo $DOCKER_HOST
```
...or if using Docker Machine...
```
 docker-machine ip
```
Note: Docker Machine generally defaults to 192.168.99.100. We reference that IP throught this doc. If your Docker host IP differs, please adjust these instructions to match the Docker host IP provided by your system.
Use it! Below are steps representing a standard HMDA filing:
1. Browse to the homepage at http://192.168.99.100.
2. Select the "Get started filing your HMDA data" link. This will redirect your browser to the filing app.
3. Select the "Login" button. This will redirect your browser to the Keycloak login screen.
4. Select "create and account" on the login screen.
5. Enter you account information and select "Register".
  
  Note: You must register with an email address from our whitelist of email domains. For convenience, bank0.com and bank1.com address should be available automatically.
6. Browse to the mock email server at https://192.168.99.100:8443/mail/, and select the verification link in the email found there. This should take you back to the HMDA filing web app, now logged in.
  
  Note: This "MailDev" services is for development purposes only. In the case of an actual HMDA filing, you would receive a confirmation to your actual email account.
7. Submit a HMDA filing. Several sample files can be found here.
To view the publication UI (hmda-pub-ui) go to http://192.168.99.100/data-publication/.
To view the tools (hmda-platform-tools) go to http://192.168.99.100/tools/.

Updating an existing system

If you've updated any of the hmda-platform services, and would like to see those changes reflected in the Docker Compose setup, the simplest way to do this is to rebuild everything from scratch. The following command should be executed from within the hmda-platform directory.

docker-compose stop -t0 && \
docker-compose rm -vf && \
cd ../hmda-platform-ui && \
yarn && \
cd ../hmda-platform-tools && \
yarn && \
cd ../hmda-pub-ui && \
yarn && \
cd ../hmda-platform && \
sbt clean assembly && \
docker-compose build --no-cache && \
docker-compose up

Service URLs

When running the full stack via Docker Compose, the following services are available:

Service	URL
Filing UI	https://192.168.99.100
Filing API (Unsecured)	http://192.168.99.100:8080
Filing API (Secured)	https://192.168.99.100:4443/hmda/
Data Publication	https://192.168.99.100:8443/data-publication/
Tools	https://192.168.99.100:8443/tools/
Admin API	http://192.168.99.100:8081
Public API	https://192.168.99.100:4443/public/
Keycloak	https://192.168.99.100:8443
MailDev	https://192.168.99.100:8443/mail/

Development conveniences

Mounted volumes

For convenience when doing development on the UI, Auth setup, and API, the docker-compose file uses a volumes which mounts

the ui's dist/ directory into the hmda-platform-ui container,
the publication ui's data-publication/ directory into the hmda-pub-ui container,
the tools tools/ directory into the hmda-platform-tools container,
the hmda.jar into hmda-platform container,
and the hmda themes directory in the auth repo into the keycloak container.

# from the hmda-platform-ui, hmda-pub-ui, and/or hmda-platform-tools directory
npm run watch

Resources

API Documentation

Public API Documentation
Admin API Documentation
Filing API Documentation

Data Specifications

Filing Status
Submission Status
Panel
TS File Spec
LAR File Spec
Modified LAR File Spec
RID Spec
Loading 2016 Data

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

HMDA Platform

Introduction

The HMDA Platform

Parser (JS/JVM)

Data Validation

Persistence

Cluster

API

API Model

Query

Panel

Model (JS/JVM)

Census

Publication

Dependencies

Java 8 SDK

Scala

Docker

Building and Running

Building the .jar

Running Interactively

Running the Dependencies

Running the API

Running the Project with Docker

To run only the API

API Load Testing

To run the entire platform

Updating an existing system

Service URLs

Development conveniences

Mounted volumes

Resources

API Documentation

Data Specifications

Files

README.md

Latest commit

History

README.md

File metadata and controls

HMDA Platform

Introduction

The HMDA Platform

Parser (JS/JVM)

Data Validation

Persistence

Cluster

API

API Model

Query

Panel

Model (JS/JVM)

Census

Publication

Dependencies

Java 8 SDK

Scala

Docker

Building and Running

Building the .jar

Running Interactively

Running the Dependencies

Running the API

Running the Project with Docker

To run only the API

API Load Testing

To run the entire platform

Updating an existing system

Service URLs

Development conveniences

Mounted volumes

Resources

API Documentation

Data Specifications