first round of starting slicer/worker on startup resolves #519

[jsnoble]

simple first round, will need more work after we refactor cluster_master and know more of what connections we need or not

[godber]

This branch passes integration tests:

[godber]

It appears that this works. I have been able to run just a slicer and worker after setting up a job and an execution context and the slicer and worker successfully generate and process slices. This was done roughly using the following script (don't expect much from this script):

#!/bin/bash

export EX=$(cat job-43a39-ex.json)
export CLUSTER_MANAGER_TYPE='kubernetes'

# delete existing indices
curl -XDELETE -Ss localhost:9200/teracluster*
# run the master just to bootstrap index creation
node ./service.js -c examples/config/processor-master.yaml > master.log &
master_pid=$!
sleep 5
kill -9 $master_pid

# First the job must exist, so lets post that job to the job index

echo "Posting Job Definition"
curl -Ss -XPOST localhost:9200/teracluster__jobs/job/43a39480-a772-4e72-a35e-f56d3e00ad6e [email protected] | jq .result
echo "Posting Execution Context"
curl -Ss -XPOST localhost:9200/teracluster__jobs/job/5c010a6b-7e49-45cb-a0d5-b3597eb87403 [email protected] | jq .result

sleep 2

export NODE_TYPE='slicer'
node ./service.js -c examples/config/processor-worker.yaml | bunyan > ${NODE_TYPE}-out.log &

export NODE_TYPE='worker'

node ./service.js -c examples/config/processor-worker.yaml | bunyan > ${NODE_TYPE}-out.log &

The slicer log is

$ head slicer-out.log
[2017-09-14T15:58:36.306Z]  INFO: teraslice/8556 on stretch.local: Service starting
[2017-09-14T15:58:36.314Z]  INFO: teraslice/8556 on stretch.local: node stretch.local is attempting to connect to cluster_master: http://127.0.0.1:5678 (module=node_master)
[2017-09-14T15:58:36.403Z]  INFO: teraslice/8556 on stretch.local: Starting 1 slicer
[2017-09-14T15:58:37.267Z]  INFO: teraslice/8644 on stretch.local: Creating connection for elasticsearch
[2017-09-14T15:58:37.508Z]  INFO: teraslice/8644 on stretch.local: Using elasticsearch hosts: 127.0.0.1:9200
[2017-09-14T15:58:38.839Z]  INFO: teraslice/8644 on stretch.local: initializing (module=state_storage)
[2017-09-14T15:58:38.840Z]  INFO: teraslice/8644 on stretch.local: initializing (module=state_storage)
[2017-09-14T15:58:38.847Z]  INFO: teraslice/8644 on stretch.local: Initializing (module=ex_storage)
[2017-09-14T15:58:38.996Z]  INFO: teraslice/8644 on stretch.local: slicer: 5c010a6b-7e49-45cb-a0d5-b3597eb87403 has initialized and is running (ex_id=5c010a6b-7e49-45cb-a0d5-b3597eb87403, module=slicer)
[2017-09-14T15:58:40.250Z]  INFO: teraslice/8644 on stretch.local: worker: 172.29.0.181__1 has joined slicer: 5c010a6b-7e49-45cb-a0d5-b3597eb87403 (ex_id=5c010a6b-7e49-45cb-a0d5-b3597eb87403, module=slicer)

The worker log is ...

$ head worker-out.log
[2017-09-14T15:58:36.292Z]  INFO: teraslice/8558 on stretch.local: Service starting
[2017-09-14T15:58:36.298Z]  INFO: teraslice/8558 on stretch.local: node stretch.local is attempting to connect to cluster_master: http://127.0.0.1:5678 (module=node_master)
[2017-09-14T15:58:36.387Z]  INFO: teraslice/8558 on stretch.local: Starting 1 worker
[2017-09-14T15:58:37.222Z]  INFO: teraslice/8643 on stretch.local: worker 172.29.0.181__1 is online, communicating with host: http://172.29.0.181:45680 (ex_id=5c010a6b-7e49-45cb-a0d5-b3597eb87403, module=worker, worker_id=172.29.0.181__1)
[2017-09-14T15:58:37.235Z]  INFO: teraslice/8643 on stretch.local: Creating connection for elasticsearch
[2017-09-14T15:58:37.483Z]  INFO: teraslice/8643 on stretch.local: Using elasticsearch hosts: 127.0.0.1:9200
[2017-09-14T15:58:38.657Z]  INFO: teraslice/8643 on stretch.local: initializing (module=state_storage)
[2017-09-14T15:58:38.670Z]  INFO: teraslice/8643 on stretch.local: AnalyticsStorage: initializing (module=analytics_storage)
[2017-09-14T15:58:38.672Z]  INFO: teraslice/8643 on stretch.local: connecting to host: http://172.29.0.181:45680 (ex_id=5c010a6b-7e49-45cb-a0d5-b3597eb87403, module=worker, worker_id=172.29.0.181__1)
[2017-09-14T15:58:40.254Z]  INFO: teraslice/8643 on stretch.local: received slice: 36201ad2-4fff-43f4-bd64-6e97c66ea97f (ex_id=5c010a6b-7e49-45cb-a0d5-b3597eb87403, module=worker, worker_id=172.29.0.181__1)

The job is in the running state (it's persistant) and there are slices in the completed state:

      {
        "_index": "teracluster__state",
        "_type": "state",
        "_id": "20efdbf5-fb40-47d7-86e7-00a8c4f1ff26",
        "_score": 1,
        "_source": {
          "@timestamp": "2017-09-14T15:58:54.634Z",
          "slice_id": "20efdbf5-fb40-47d7-86e7-00a8c4f1ff26",
          "slicer_id": 0,
          "slicer_order": 52,
          "request": "1000",
          "state": "completed",
          "ex_id": "5c010a6b-7e49-45cb-a0d5-b3597eb87403"
        }
      }

• job-43a39-ex.json
• job-43a39.json