SparkJava + RabbitMQ + Apache Storm + Apache Camel + Websocket

System requirements

• Java 8
• RabbitMQ installed and running
• Apache Maven 3.2.5

Python trader

• Install requirements

cd python_trader
pip install -r requirements/requirements.txt

• TODO: configure trader.cfg with host where Consumer is being launched

• Run the simluator

cd python_trader
python trader/simulate_trades.py

Start project parts

• Trades simulator:

cd sparkjava_storm_websocket/python
source .venv/bin/activate
python trader_simulator/simulate_trades.py

• RabbitMQ:

• SparkJava endpoint (message consumer):

• Apache Storm topology (message processor):

• Storm - RabbitMQ: https://github.com/ppat/storm-rabbitmq

Architecture schema

                                               
       +-----------+     +----------+     +--------------+    +----------+    +--------------+ Consumes RabbitMQ
+----> | Sparkjava +---> | RabbitMQ +---> | Apache Storm +--> | RabbitMQ +--> | Apache Camel | Publishes to     
       +-----------+     +----------+     +--------------+    +----------+    +-------+------+ Websocket        
                                                                                      |                         
         REST API                          RabbitMQ Spout.                            |                         
         endpoint                          Aggregators and                            |                         
                                           data processors                      +-----v-----+                   
                                                                                | WEBSOCKET |                   
                                                                                +-----^-----+                   
                                                                                      |                         
                                                                                      |                         
                                                                                      |                         
                                                                                +-----+------+                  
                                                                                | Javascript |                  
                                                                                | (socket.io)|                  
                                                                                +------------+                  

Technologies used

• Sparkjava for REST API endopint
• RabbitMQ for messaging
• Apache Storm for processing data
• Apache Camel for enrouting from RabbitMQ to websocket
• Websocket to publish real-time data
• Socket.io to generate the javascript frontend

Input data

JSON formatted:

{
	"userId":              "123456",
	"currencyFrom":        "EUR", 
	"currencyTo":          "GBP", 
	"amountSell":          1000, 
	"amountBuy":           747.10, 
	"rate":                0.7471, 
	"timePlaced" :         "24-JAN-15 10:27:44", 
	"originatingCountry" : "FR"
}

Data processed

TODO: fulfill this paragraph

Frontend design

Frontend will show a table with real-time data of the top X selled/bougth currencies during the last X minutes. The table could be something like the following example:

|--|--|--| |Last X minutes sold|Last X minutes bought| | |EUR: 1.250 (0.7564)|GBP: 820 (1.4322)| | | FR (3x), US (2x), UK (1x) | SP (20x), UK (13x), NZ (2x) | # This are the top 3 sold from and bought from countries (and times) |
|GBP: 823 (0.8235)|USD: 2.234 (0.2341)| | | FR (3x), US (2x), UK (1x) | SP (20x), UK (13x), NZ (2x) | # This are the top 3 sold from and bought from countries (and times) |
|...|...|...|

The json sent through websocket has the following format:

{
	"accumulated_since": "24-JAN-15 10:17:44",
	"sell": [
		{  
			"curr": "EUR", "amount": 1000, "avr_rate": 1.8224,
		   	"top": [{"country": "FR", "qtt": 3}, {"country": "US", "qtt": 2}, {"country": "UK", "qtt": 1}]
		},
		{ 
			"curr": "GBP", "amount":  500, "avr_rate": 1.2234,
		   	"top": [{"country": "FR", "qtt": 3}, {"country": "US", "qtt": 2}, {"country": "UK", "qtt": 1}]
		},
		...		
	],
	"buy": [
		{ 
		  	"curr": "EUR", "amount": 1000, "avr_rate": 1.8224,
			"top": [{"country": "FR", "qtt": 3}, {"country": "US", "qtt": 2}, {"country": "UK", "qtt": 1}]
		},
		{ 
			"curr": "GBP", "amount":  500, "avr_rate": 1.2234,
		   	"top": [{"country": "FR", "qtt": 3}, {"country": "US", "qtt": 2}, {"country": "UK", "qtt": 1}]
		},
		...		
	]
}

Message consuming

• TODO: validate received fields (existing currencies, existing original country, well formatted dated, etc.)
• TODO: better authentication, not hardcoded