Endly introduction

Prerequisites

Docker service

Getting Started

Create test and credentials directories
```
mkdir -p ~/e2e
mkdir -p ~/.secret
```

Start endly docker image

docker run --name endly -v /var/run/docker.sock:/var/run/docker.sock -v ~/e2e:/e2e -v ~/.secret/:/root/.secret/ -p 7722:22  -d endly/endly:latest-ubuntu16.04

Logging to endly container

ssh [email protected] -p 7722 ## password is dev

Check endly version
```
 endly -v    
```
Create dev container credentials (use user:root, password:dev)
```
endly -c=dev    
```

Clone this repo

cd /e2e
git clone https://github.com/adrianwit/endly-introduction.git

Workflow

Hello world

endly workflow/hello @workflow/hello.yaml

 init:
   target:
     URL: ssh://127.0.0.1
     credentials: dev
 pipeline:
   task1:
     action: workflow.print
     message: Hello World
   task2:
     action: exec:run
     target: $target
     commands:
       - echo 'Hello World'

Basic workflow

Workflow definition @workflow/helloworld.yaml

init:
  name: $params.name
pipeline:
  task1:
    action: workflow.print
    message: Hello World $name
    description: print hello
  task2:
    description: test subtasks
    subTask1:
      action: print
      message: subTask 1 $params.key1
    subTask2:
      action: print
      message: subTask 2 $params.name
  task3:
    description: print bye
    action: print
    message: Bye $name

Workflow execution

with relative path and parameters

endly workflow/helloworld name=Endly key1=test

with URL

endly https://github.com/adrianwit/endly-introduction/blob/master/workflow/helloworld.yaml

Workflow task info
```
endly workflow/helloworld -t='?'
```

Workflow task selection

endly workflow/helloworld -t='task1,task3'

Services

Endly supported services:
```
endly -s='*'
```
Action attribute.
- defines what service method is used for an workflow action
- action attribute format: [serivce]:action
- default endly service is workflow
Service Contract
- specifies request type: optional and required action node attributes for an action
- specifies action response type
```
endly -s=workflow -a=print
#or
endly -s=workflow:print
```
Running adhoc service action from CLI
- With request attribute supplied as cli arguments
  - endly -run="print" message='hello'
  - endly -run="validator:assert" actual=1 expect=3
- With request delegated to JSON or YAML file
  - endly -run='validator:assert' request='@services/assert.yaml' @services/assert.yaml
```
actual:
    - 1
    - 2
    - 3
expect:
    - 1
    - 2
    - 3    
```

Exec SSH service usage example

Service action list: endly -s=exec
'exec:run' contract endly -s=exec -a=run
endly services/info.yaml
@services/info.yaml

init:
  target:
    url:  ssh://127.0.0.1/
    credentials: dev

defaults:
  target: $target

pipeline:
  init:
    action: exec:run
    commands:
      - whoami
      - echo 'hello ${cmd[0].stdout}'
  hostInfo:
    action: exec:run
    commands:
      - cat /etc/hosts
    extract:
      - key: aliases
        regExpr: (?sm)127.0.0.1[\s\t]+(.+)
  debug:
    action: print
    message: $AsJSON($init) $AsJSON($hostInfo)
  summary:
    action: print
    message: Extracted 127.0.0.1 aliases $hostInfo.aliases

Failing SSH workflow based on non 0 command exit code (endly 0.38+)
endly services/fail_on_exitcode.yaml
@services/info.yaml

init:
  target:
    url:  ssh://127.0.0.1/
    credentials: ${env.HOME}/.secret/dev.json
pipeline:
  package:
    action: exec:run
    target: $target
    checkError: true
    commands:
      - whoami
      - /bin/false

Workflow delegation:

@workflow/parent.yaml workflow

pipeline:
  task1:
    action: print
    message: hello from parent

  tassk2:
    action: run
    request: '@child'

  info:
    action: print
    message: "child response: $AsJSON($tassk2)"

@workflow/child.yaml workflow

init:
  message: hello from child

pipeline:
  task1:
    init:
      childVar1: $message
    action: print
    message: $childVar1

post:
  myOutput: $childVar1

execution: endly workflow/parent

Request delegation

@services/delegation.yaml workflow

init:
  target:
    url:  ssh://127.0.0.1/
    credentials: dev
defaults:
  target: $target
pipeline:
  task1:
    action: storage:copy
    request: @copy
  task2:
    action: exec:run
    commands:
      - ls -al /tmp/README.md

@services/copy.yaml request

source:
  URL: https://raw.githubusercontent.com/adrianwit/endly-introduction/master/
dest:
  URL: $target
assets:
  'README.md': /tmp/README.md

execution: endly services/delegation

Variables

Variables control workflow execution and data substitution
Variable are defined in init section at action, task or workflow node
Variable reference expression start with '$' sign
- any reference to variable is expanded/evaluated if variable is defined
- nesting variable uses {}
- variable expression supports basic arithmetic expression

Required variables usage

@variables/required.yaml

init:
  '!to': $params.to
  greeting: $params.greeting
  message: "$greeting:/$/? 'Hello $to' : '$greeting $to'"
pipeline:
    welome:
     action: print
     message: $message

endly variables/required.yaml
endly variables/required.yaml to=Endly
endly variables/required.yaml to=Endly greeting=Greetings

User defined function

listing loaded UDF endly -j
@variables/udf.yaml example

 init:
   fiveDaysAgo: $FormatTime('5daysAgoInUTC','yyyyMMdd')
   10DaysAhead: $FormatTime('5daysAheadInUTC','yyyy-MM-dd hh:mm:ss')
   encodedMsg: $Base64Encode("This is test")
   ID: $uuid.next
 pipeline:
   task1:
     action: print
     message: $ID $fiveDaysAgo $10DaysAhead $encodedMsg

endly variables/udf.yaml
Loading external data
@variables/external.yaml examle

init:
  userJ: $AsData($Cat('variables/user.json'))
  userY: $AsData($Cat('variables/user.yaml'))

pipeline:
  info:
    action: print
    message: '${userJ.name}: ${userY.email}, ${userY.phones[1]}'
  update:
    action: print
    init:
      user: $userJ
      'user.dob': 2001-01-01
    message: $AsJSON($user)

endly variables/external.yaml

Array/Collection type variable usage

@variables/collection.yaml

 init:
   i: 2
   array:
     - id: 1
       name: name 1
     - id: 2
       name: name 2
     - id: 3
       name: name 3
 
 pipeline:
   shift:
     action: print
     init:
       item: ${<-array}
     message: $AsJSON($item)
 
   push:
     action: nop
     init:
       - name: myItem
         value:
           id: 10,
           name: name 10
 
       - '->array': $myItem
   lastNameInfo:
     action: print
     message: ${array[$i].name}
   summary:
     action: print
     message: $AsJSON($array)

Basic arithmetic

@variables/arithmetic.yaml

init:
  i: 1
  j: 10
  k: 100
pipeline:
  task1:
    action: print
    message: ${(k/j)+i}

Action response variable

After each action execution response can be access via $ACTION_NAME
endly variables/inspect
@variables/inspect.yaml

pipeline:
  task1:
    inspect:
      action: docker:inspect
      '@name': endly
      logging: false
    ipInfo:
      action: print
      message: ${inspect.Info[0].NetworkSettings.IPAddress}
    networkInfo:
      action: print
      message: $AsJSON(${inspect.Info[0].NetworkSettings.Networks})
  task2:
    inspect:
      action: docker:inspect
      '@name': endly
      ':name': endlyInfo
      logging: false
    ipInfo:
      action: print
      message: ${endlyInfo.Info[0].NetworkSettings.IPAddress}
    networkInfo:
      action: print
      message: $AsJSON(${endlyInfo.Info[0].NetworkSettings.Networks})

Note that if name is both part of docker:inspect contract (endly -s='docker:inspect'), and workflow action node model (action has name attribute), '@' prefix is used for service contract, and ':' for workflow model respectively. In task1 name of action is inherited from node name ('inspect'), in task2 name is specified with ':' prefix.

Find more about Docker service

Workflow Actions Template

Action template repeats N time group of flattened actions defined under template node:

@template/basic.yaml example

pipeline:
  init:
    action: print
    message: init ...
  tasksX:
    tag: $pathMatch
    range: 1..002
    description: '@desc'
    subPath: basic/${index}_*
    template:
      action1:
        action: print
        message: "action 1 - idx: $index, tagId: $tagId, path: $path"
      action2:
        action: print
        request: '@print'
  done:
    action: print
    message: done.

template node
range
template variables: $index, $tagId, $path, $subPath
request location auto-discovery
description discovery

Tags selection
- endly template/basic.yaml -i=tasksX_basic_caseYY

Data loading

endly template/stress_test
@template/stress_test.yaml

pipeline:
  test:
    tag: LoadTest
    data:
      ${tagId}.[]Requests: '@data/Request*.json'
    range: '1..002'
    subPath: stess_test/${index}_*
    template:
      init:
        action: nop
        comments: set endpoint port and address
        init:
          idx: $AsInt($index)
          threadCont: ${idx * 4}
          testEndpointPort: 803$AsInt($index)
          testEndpoint: 127.0.0.1:${testEndpointPort}

      endpointUp:
        action: http/endpoint:listen
        comments: start test endpoint 127.0.0.1:${testEndpointPort}
        port: $testEndpointPort
        rotate: true
        baseDirectory: ${path}/endpoint/

      info:
        action: print
        message: starting load testing $testEndpont

      load:
        action: 'http/runner:load'
        comments: run load test (40k * 5)
        assertMod: 40
        threadCount: $threadCont
        '@repeat': 40000
        requests: ${data.${tagId}.Requests}

      load-info:
        action: print
        message: 'QPS: $load.QPS: Response: min: $load.MinResponseTimeInMs ms, avg: $load.AvgResponseTimeInMs ms max: $load.MaxResponseTimeInMs ms'

data node

Validation

Basic validation example

endly validation/test

@validation/test.yaml

pipeline:
  init:
    action: workflow:print
    message: init test ...
  test:
    workflow: assert
    actual:
      key1: value1
      key2: value20
      key3: value30
    expected:
      key1: value1
      key2: value2
      key3: value3
      key4: value4

Simple validation

# to get validator:assert contract:
endly -s=validator:assert
#run assets    
endly -run='validator:assert' actual A expect B
#or
endly validator:assert actual A expect B

Data structure validation

endly -run='validator:assert' actual '@validation/actual1.json' expected '@validation/expect1.json'

Data structure validation with @indexBy@ directive

endly -run='validator:assert' actual '@validation/actual2.json' expected '@validation/expect2.json'

Data structure alternative transformation validation (switch/case)

endly -run='validator:assert'  actual '@validation/actual3.json' expected '@validation/expect3.json'

Partial validation with @assertPath@ directive

endly -run='validator:assert'  actual '@validation/actual4.json' expected '@validation/expect4.json'

See more about validation expression

Validation expressions:

Assertly

Execution Control

Conditional action execution

@control/when.yaml

pipeline:
  task1:
    action1:
      when: $p = b1
      action: print
      message: hello from action 1
    action2:
      when: $p = b2
      action: print
      message: hello from action 2
    action3:
      when: $HasResource('control/hello.txt')
      action: print
      message: $Cat('control/hello.txt')

endly -r control/when
endly -r control/when p=b1

Parallel execution,

@control/parallel.yaml

pipeline:
  task1:
    multiAction: true
    action1:
      action: print
      message: hello from action 1
      sleepTimeMs: 3000
      async: true
    action2:
      action: print
      message: hello from action 2
      sleepTimeMs: 3000
    action3:
      action: print
      message: hello from action 3
      sleepTimeMs: 3000
      async: true

endly control/parallel
works on task level with flatten action:multiAction: true

Defer execution

@control/defer.yaml

pipeline:
  task1:
    action1:
      action: print
      message: hello action 1
    action2:
      when: $failNow=1
      action: fail
      message: execption in action 2
  defer:
    action: print
    message: allway run

endly control/defer.yaml
endly control/defer.yaml failNow=1

Handling error

@control/error.yaml

pipeline:
  task1:
    action1:
      action: print
      message: hello action 1
    action2:
      when: $p = failNow
      action: fail
      message: execption in action 2
    action3:
      action: print
      message: hello action 3
      
  task2:
    action1:
      action: print
      message: hello task 2 action 1

  catch:
    action: print
    message: caught $error.Error

endly control/error
endly control/error p=failNow

Loops

@control/loop.yaml

init:
  self.i: 1
pipeline:
  loop:
    info:
      action: print
      message: pushing message ${self.i}

#    trigger:
#      action: msg:push
#      dest:
#        URL: mye2eQueue
#        type: queue
#        vendor: aws
#      messages:
#        - data: 'Test: this is my ${self.i} message'
#          attributes:
#            id: $self.i

    inc:
      action: nop
      logging: false
      init:
        _: ${self.i++}
    until:
      action: goto
      logging: false
      task: loop
      when: $self.i <= 10
    done:
      action: print
      comments: done

endly control/loop

Debugging & Troubleshooting

Enabling logging endly workflow/helloworld -d
Inspecting workflow domain model endly workflow/helloworld -f=yaml -p
Diagnose endly execution with gops
- gops stack ENDLY_PID

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Endly introduction

Prerequisites

Getting Started

Workflow

Basic workflow

Services

Variables

Workflow Actions Template

Validation

Execution Control

Debugging & Troubleshooting

About

Releases

Packages

Contributors 2

Name		Name	Last commit message	Last commit date
Latest commit History 36 Commits
control		control
services		services
template		template
validation		validation
variables		variables
workflow		workflow
README.md		README.md

adrianwit/endly-introduction

Folders and files

Latest commit

History

Repository files navigation

Endly introduction

Prerequisites

Getting Started

Workflow

Basic workflow

Services

Variables

Workflow Actions Template

Validation

Execution Control

Debugging & Troubleshooting

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Contributors 2

Packages