Skip to content

Commit

Permalink
Merge pull request #54 from timkpaine/rewerk
Browse files Browse the repository at this point in the history
Rewrite "Reactive" functionality as "Streaming"
  • Loading branch information
timkpaine authored Feb 17, 2020
2 parents 34f98d6 + c286fe3 commit f8bf080
Show file tree
Hide file tree
Showing 106 changed files with 3,250 additions and 2,747 deletions.
22 changes: 0 additions & 22 deletions .appveyor.yml

This file was deleted.

6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -169,4 +169,8 @@ package-lock.json

tmp.py
tmp2.py
docs/api/
docs/api/
tributary/tests/streaming/output/test_file_data.csv
python_junit.xml
tmps
docs/index.md
39 changes: 0 additions & 39 deletions .travis.yml

This file was deleted.

4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
build: ## Build the repository
python3 setup.py build
python3.7 setup.py build

buildpy2:
python2 setup.py build

tests: ## Clean and Make unit tests
python3 -m pytest -v tributary/tests --cov=tributary
python3.7 -m pytest -v tributary --cov=tributary --junitxml=python_junit.xml --cov-report=xml --cov-branch

notebooks: ## test execute the notebooks
./scripts/test_notebooks.sh
Expand Down
79 changes: 55 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# <a href="https://tributary.readthedocs.io"><img src="docs/img/icon.png" width="300"></a>
Python Data Streams

[![Build Status](https://travis-ci.org/timkpaine/tributary.svg?branch=master)](https://travis-ci.org/timkpaine/tributary)
[![Build Status](https://dev.azure.com/tpaine154/tributary/_apis/build/status/timkpaine.tributary?branchName=master)](https://dev.azure.com/tpaine154/tributary/_build/latest?definitionId=2&branchName=master)
[![GitHub issues](https://img.shields.io/github/issues/timkpaine/tributary.svg)]()
[![codecov](https://codecov.io/gh/timkpaine/tributary/branch/master/graph/badge.svg)](https://codecov.io/gh/timkpaine/tributary)
[![Coverage](https://img.shields.io/azure-devops/coverage/tpaine154/tributary/2)]()
[![BCH compliance](https://bettercodehub.com/edge/badge/timkpaine/tributary?branch=master)](https://bettercodehub.com/)
[![PyPI](https://img.shields.io/pypi/l/tributary.svg)](https://pypi.python.org/pypi/tributary)
[![PyPI](https://img.shields.io/pypi/v/tributary.svg)](https://pypi.python.org/pypi/tributary)
Expand All @@ -12,7 +12,6 @@ Python Data Streams
![](https://raw.githubusercontent.com/timkpaine/tributary/master/docs/img/example.gif)



# Installation
Install from pip:

Expand All @@ -22,42 +21,26 @@ or from source

`python setup.py install`


# Stream Types
Tributary offers several kinds of streams:

## Reactive
## Streaming
These are synchronous, reactive data streams, built using asynchronous python generators. They are designed to mimic complex event processors in terms of event ordering.

## Functional
These are functional streams, built by currying python functions (callbacks).

## Event Loop
TODO
These function as tornado based event-loop based streams similar to streamz.

## Lazy
These are lazily-evaluated python streams, where outputs are propogated only as inputs change.

# Examples
- [Reactive](docs/examples/reactive.md)
- [Streaming](docs/examples/streaming.md)
- [Lazy](docs/examples/lazy.md)

# Math
`(Work in progress)`

## Operations
- unary operators/comparators
- binary operators/comparators

## Rolling
- count
- sum

# Sources and Sinks
`(Work in progress)`

## Sources
- python function/generator/async function/async generator
- random
- file
- kafka
- websocket
Expand All @@ -68,5 +51,53 @@ These are lazily-evaluated python streams, where outputs are propogated only as
- file
- kafka
- http
- TODO websocket
- websocket
- TODO socket io

# Transforms
- Delay - Streaming wrapper to delay a stream
- Apply - Streaming wrapper to apply a function to an input stream
- Window - Streaming wrapper to collect a window of values
- Unroll - Streaming wrapper to unroll an iterable stream
- UnrollDataFrame - Streaming wrapper to unroll a dataframe into a stream
- Merge - Streaming wrapper to merge 2 inputs into a single output
- ListMerge - Streaming wrapper to merge 2 input lists into a single output list
- DictMerge - Streaming wrapper to merge 2 input dicts into a single output dict. Preference is given to the second input (e.g. if keys overlap)
- Reduce - Streaming wrapper to merge any number of inputs

# Calculations
- Noop
- Negate
- Invert
- Add
- Sub
- Mult
- Div
- RDiv
- Mod
- Pow
- Not
- And
- Or
- Equal
- NotEqual
- Less
- LessOrEqual
- Greater
- GreaterOrEqual
- Log
- Sin
- Cos
- Tan
- Arcsin
- Arccos
- Arctan
- Sqrt
- Abs
- Exp
- Erf
- Int
- Float
- Bool
- Str
- Len
155 changes: 125 additions & 30 deletions azure-pipelines.yml
Original file line number Diff line number Diff line change
@@ -1,33 +1,128 @@
# Python package
# Create and test a Python package on multiple Python versions.
# Add steps that analyze code, save the dist with the build record, publish to a PyPI-compatible index, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/python

trigger:
- master

pool:
vmImage: 'ubuntu-latest'

strategy:
matrix:
Python36:
python.version: '3.6'
Python37:
python.version: '3.7'

steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '$(python.version)'
displayName: 'Use Python $(python.version)'

- script: |
python -m pip install --upgrade pip
pip install -e .[dev]
displayName: 'Install dependencies'

- script: |
make lint
make tests
displayName: 'pytest'
jobs:
- job: 'Linux'
pool:
vmImage: 'ubuntu-latest'

strategy:
matrix:
Python37:
python.version: '3.7'

steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '$(python.version)'
displayName: 'Use Python $(python.version)'

- script: |
python -m pip install --upgrade pip
pip install -e .[dev]
displayName: 'Install dependencies'
- script: |
make lint
displayName: 'Lint'
- script:
make tests
displayName: 'Test'

- task: PublishTestResults@2
condition: succeededOrFailed()
inputs:
testResultsFiles: 'python_junit.xml'
testRunTitle: 'Publish test results for Python $(python.version) $(manylinux_flag)'

- task: PublishCodeCoverageResults@1
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(System.DefaultWorkingDirectory)/*coverage.xml'

- job: 'Mac'
pool:
vmImage: 'macos-10.14'

strategy:
matrix:
Python37:
python.version: '3.7'

steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '$(python.version)'
displayName: 'Use Python $(python.version)'

- script: |
python -m pip install --upgrade pip
pip install -e .[dev]
displayName: 'Install dependencies'
- script: |
make lint
displayName: 'Lint'
- script: |
make tests
displayName: 'Test'
- task: PublishTestResults@2
condition: succeededOrFailed()
inputs:
testResultsFiles: 'python_junit.xml'
testRunTitle: 'Publish test results for Python $(python.version) $(manylinux_flag)'

- task: PublishCodeCoverageResults@1
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(System.DefaultWorkingDirectory)/*coverage.xml'

- job: 'Windows'
pool:
vmImage: 'vs2017-win2016'

strategy:
matrix:
Python37:
python.version: '3.7'

steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '$(python.version)'
displayName: 'Use Python $(python.version)'

- script: |
which python > python.txt
set /p PYTHON=<python.txt
ln -s %PYTHON% %PYTHON%$(python.version)
python --version
which python$(python.version)
displayName: "Which python"
- script: |
python -m pip install --upgrade pip
pip install -e .[dev]
displayName: 'Install dependencies'
- script: |
make lint
displayName: 'Lint'
- script: |
make tests
displayName: 'Test'
- task: PublishTestResults@2
condition: succeededOrFailed()
inputs:
testResultsFiles: 'python_junit.xml'
testRunTitle: 'Publish test results for Python $(python.version) $(manylinux_flag)'

- task: PublishCodeCoverageResults@1
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(System.DefaultWorkingDirectory)/*coverage.xml'
Loading

0 comments on commit f8bf080

Please sign in to comment.