Skip to content

Latest commit

 

History

History
115 lines (90 loc) · 4.42 KB

README.md

File metadata and controls

115 lines (90 loc) · 4.42 KB

DEPRECATED

This is no longer supported, it is moved to wix/supraworker

Supraworker - Pull Jobs from Anywhere

Build Status GitHub All Releases License Go Report Card Docker Pulls Docker Image Size (latest by date)

The abstraction layer around jobs, allows pull a job from any API, call-back your API, observe execution time and to control concurrent execution.

It's responsible for pulling the commands(jobs) from your API, running commands, and streaming the logs back to your API. It also sends state updates to remote API.

Getting started

Prerequisite:

  1. API service that serve jobs:
  2. Supraworker configuration:
  3. Running supraworker on a server

Installation

MacOs X

Binary installation

Homebrew is a free and open-source package management system for Mac OS X.

 brew tap weldpua2008/tap
 brew install weldpua2008/tap/supraworker

To update to the latest, run

brew upgrade weldpua2008/tap/supraworker

Installation from source code

  • Find the version you wish to install on the GitHub Releases page and download either the darwin-amd64 binary for MacOS or the linux-amd64 binary for Linux. No other operating systems or architectures have pre-built binaries at this time.

NOTE: Running releasses on MacOs: You need to download file, extract it and remove attributes with the following command (where ~/Downloads/supraworker_darwin_amd64/supraworker is Path to the file)

$ xattr -d com.apple.quarantine ~/Downloads/supraworker_darwin_amd64/supraworker
$ ~/Downloads/supraworker_darwin_amd64/supraworker

Linux

Download the latest release

curl --silent -L  "https://api.github.com/repos/weldpua2008/supraworker/releases/latest"  \
| jq --arg PLATFORM_ARCH "$(echo `uname -s`_amd| tr '[:upper:]' '[:lower:]')" -r '.assets[] | select(.name | contains($PLATFORM_ARCH)).browser_download_url' \
| xargs -I % curl -sSL  % \
| sudo tar --strip-components=1  -xzf  -

Installing from source

  1. install Go v1.13+
  2. clone this down into your $GOPATH
  • mkdir -p $GOPATH/src/github.com/weldpua2008
  • git clone https://github.com/weldpua2008/supraworker $GOPATH/src/github.com/weldpua2008/supraworker
  • cd $GOPATH/src/github.com/weldpua2008/supraworker
  1. install golangci-lint for linting + static analysis
  • Lint: docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint:v1.24.0 golangci-lint run -v

Configuration

Define config at $HOME/supraworker.yaml:

NOTE: Keys are not case sensitivity (spf13/viper#635 (comment)): Viper's default behaviour of lowercasing the keys for key insensitivity is incompatible with these standards and has the side effect of making it difficult for use cases such as case sensitive API credentials in configuration. For eg: MyApiKey=MySecret

# ClientId in case you need to identify the worker
clientId: "my_uniq_client_id"
# how ofen call your API
api_delay_sec: 10
# jobs related operations
jobs:
  get:
    url: "http://localhost:80/get/new/job"
    method: POST
    headers:
      "Content-type": "application/json"
      params:
        "clientid": "{{ .ClientId}}"

Use Cases

  • Airflow sends task for an execution on AWS EMR
  • Building your CI/CD system

Running tests

  • expires all test results
$ go clean -testcache
  • run all tests
$ go test -bench= -test.v  ./...