jevo visualizer protocol

This document describes network protocol between jevo (server) and visualizer (client) applications. It helps developers write their own visualizer...

Seding requests from client to server and back

First, i have to mention that it's based on simple TCP protocol and uses Julia streams inside. Take a look on this simple example of client/server for details. Second thought is that this protocol is similar to WebSockets, because of two directional (server -> client, client -> server) requests. Generally speaking you may imagine one request() function, which is implemented on both sides (client and server). Sending request from, for example server to client, they both may bind their functions to events like before-response and after-request. before-response event is fired before client (or server) send a response to server (or client). after-request event is fired after client (or server) received and answer from server (or client). Let's create simple example of this behavior. In our example server sends a request to the client:

   server --------request------> client
after-request                before-response
   handler                      handler

after-request handler may be used for filling up arguments array of request. These arguments will be sent to the client in request using special Connection.Command type:

  type Command
    fn::Int             # unique request id
    args::Array{Any, 1} # custom request parameters
  end

before-response handler may be used for handling the request and creating the answer. If answer type will not be filled, then it will not be sent. Answering works with Connection.Answer type:

  type Answer
    id::Int   # unique request id
    data::Any # answer data
  end

Here you can find implementation of server and client, which do this.

Let's take a look on server. It should be created first, before any client try to connect. For this Server.create() method should be used. Let's try this in a julia console ran from jevo folder:

# server side
include("src\\global\\ImportFolders.jl")
using Server
server = Server.create(ip"127.0.0.1", 3030)

Server works in asynchronous way. It means, that you have to have some external code, which calls yield() or/and yieldto() functions to ask server check new requests, answer existing and so on. It also means that you may send requests in asynchronous way one by one without waiting of responses. Julia console already have this external code embedded. This is why our example works ina console from the scratch. In jevo we use Manager.run() as this "external code".

Important note here is that server creation doesn't mean it's running. For run Server.run() method should be used. But before we do this, we need to bind requests handler. This handler is used for catching requests sent from client to this server. This is how we may "answer" on such requests on the server side or just do something if request appears:

# server side
using Event
Event.on(
  server.observer,
  Connection.EVENT_BEFORE_RESPONSE,
  (sock::Base.TCPSocket, cmd::Connection.Command, ans::Connection.Answer) -> println(cmd)
)

EVENT_BEFORE_RESPONSE constant means reaction before server will answer to client. Connection.Command type contains request(command) unique id(Command.fn) and sent arguments(Command.args). As you may see from an example above we skip answer ans type. It means that answer will not be send to the client. Okay, our server is ready to handle requests. Let's run it:

# server side
Server.run(server)

Starting from this moment clients may connect to this server using 127.0.0.1 host and port 3030. Okay, we need to create a client using another julia console terminal:

# client side
include("src\\global\\ImportFolders.jl")
using Client
client = Client.create(ip"127.0.0.1", 3030)

We don't need to run the client, because it doesn't require some external code with yield()/yieldto() calls. After creation requests may be ran. Let's take a look on client's request() method declaration:

function request(con::Client.ClientConnection, fn::Int, args...)

fn means "unique remote request id". args - are custom request arguments. So, we are ready to send a request to server:

# client side
Client.request(client, 123, "Hello server!")

Here is server's console output:

julia> Connection.Command(123,Any["Hello server!"])

As you may see it's very simple to send request from client to server. Opposite sending is also simple. First, we have to add request handler on client side:

# client side
Event.on(
  client.observer,
  Connection.EVENT_BEFORE_RESPONSE,
  (sock::Base.TCPSocket, cmd::Connection.Command, ans::Connection.Answer) -> println(cmd)
)

After that server may send requests in a same way:

# server side
Server.request(server.socks[1], 124, "Hi, client!")

socks[1] is a socket of a client we want send a message to. Here is an output of client:

julia> Connection.Command(124,Any["Hi, client!"])

Working in fast and slow modes

Okay, now you know how to work with requests. Let's discuss their modes. There are two of them available: fast and slow. These modes mean transfer differences and simplicity level. fast works very fast, but it uses special predefined data structure (see FastApi.jl for details):

const API_UINT64        = UInt8(1)
const API_ASCIISTRING   = UInt8(3)
const API_UINT8         = UInt8(5)
const API_UINT16        = UInt8(7)
const API_DOT_COLOR     = UInt8(9)
const API_ORG_COLOR     = UInt8(13)
const API_FLOAT64       = UInt8(19)
...
const _api = DataType[
    UInt64, Void,                              # 1
    String, Void,                              # 3
    UInt8, Void,                               # 5
    UInt16, Void,                              # 7
    UInt16, UInt16, UInt16, Void,              # 9  x, y, color
    UInt16, UInt16, UInt16, UInt, UInt8, Void, # 13 x, y, dir|color, orgId, infoBits
    Float64, Void                              # 19
  ]

These constants define special data types or data type lists for sending through our client\server protocol. Memorize these constants we will back to them later... slow mode works in a slow mode, but it very simple to use. You may transfer any type of data, predefined or your own. By default jevo uses slow mode. If you need to swith it, you have to set additional parameter in Server.create() or Client.create() method and use another request() function. Let's create fast server in a separate julia terminal:

# server side 
include("src\\global\\ImportFolders.jl")
using Server
using Event
server = Server.create(ip"127.0.0.1", 3030, true) # true means "fast" mode
Event.on(
  server.observer,
  Connection.EVENT_BEFORE_RESPONSE,
  (sock::Base.TCPSocket, data::Array{Any, 1}, ans::Connection.Answer) -> println(data)
)
Server.run(server)

In example above we have created server in fast mode on localhost on port 3030. Let's create client:

# client side
include("src\\global\\ImportFolders.jl")
using Client
# true means "fast" mode
client = Client.create(ip"127.0.0.1", 3030, true)
# sends request to the server in "fast" mode. Put one UInt64
# number to the request
Client.request(client, FastApi.API_UINT64, UInt(123))

FastApi.API_UINT64 constant sets concrete type of sending data - one UInt64 type. If you need to send other type or list of types you have to use other constants. See FastApi.jl API_XXX constants for details.

Visualizer protocol

Vusualizer application and jevo server use both fast and slow modes for communication. slow mode is used for transferring region of the jevo world. fast mode is used for sending diffs between region and and latest world changes. First, client have to create two instances in different modes. Data flow is the following:

   jevo Server                               Visualizer client
1. "slow" and "fast" servers creation
2. servers run
3.                                           "slow" and "fast" clients creation
4.                                     <---  region request in "slow" mode
5. region answer in "slow" mode        --->  draw the region
6.                                     <---  streaming mode request in "fast" mode
7. requests to client in "fast" mode   --->  ...
   with world diffs

The region in transferred in special type RpcApi.Region:

  type Org
    offs::UInt8
    id::UInt32
    color::UInt16 # only 65536 colors are supported now
    age::UInt16
  end
  type Block
    energy::Array{UInt8, 1}
    orgs::Array{Org, 1}
  end
  type Region
    xBlocks::UInt8
    yBlocks::UInt8
    width::UInt16
    height::UInt16
    blocks::Array{Block, 1}
    ips::Float64
  end

The algorithm of how we collect data for region is a little bit complicated. Please see description for getRegion() function for details.

# visualizer side
local cfg::Config.ConfigData = Config.create()
client = Client.create(cfg.conServerIp, cfg.conServerPort)
fastClient = Client.create(cfg.conServerIp, cfg.conFastServerPort, true)

Next, we have to add all required event handlers:

# visualizer side
function _onDot(sock::Base.TCPSocket, data::Array{Any, 1}, ans::Connection.Answer)
  # draw the dot of organism or energy
end
function _onRegion(ans::Connection.Answer)
  # draw all dots of organisms or energy
  # ...
  # Asks server to start send diffs
  Client.request(rd.poolingCon, UInt8(FastApi.API_UINT8), UInt8(0))
end
Event.on(client.observer, Connection.EVENT_AFTER_REQUEST, _onRegion)
Event.on(fastClient.observer, Connection.EVENT_BEFORE_RESPONSE, _onDot)

The last thing is to run communication between visualizer and jeco server:

# visualizer side
# asks jevo for the region
Client.request(client, RpcApi.RPC_SET_WORLD_STREAMING)

See _onRegion() function for details of drawing obtained region.