STREAMWORK -- SW-0.28.8

Copyright (C) 2019,2020,2021,2022 Thomas W. Young, [email protected]

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file or its derivitaves except in compliance with the License. You may obtain a copy of the License at

Apache License

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

All files in this repo are included in this license.

StreamWork

Go language code generator.

Contents

• Description
  • Synopsis
• Return code
• TestSWL program
• Network Definition
  • Network Definition Files
• QuickStart on Linux
  • Requirements
  • Installation
  • Post Install
  • Foo Bar test
• Building on Linux
• StreamWork Language(SWL) Notes
  • Stream types and component arguments
  • Variables
  • Semi-colon commentary
  • Left Arrow comments
  • Collate/Merge Example
• Bash Scripts
  • Hello World
  • swgo script
  • swgraph Script
  • gofbp script
    • gofbp Example
      • LOADBAL.SW
  • Generate a project
    • sw.sh Synopsis
• BUGS
• Home Page
• Release Notes
• Author

Description

StreamWork is a simple system for building complex programs.

StreamWork can generate and test a complete Golang project from a simple text file. It can also create a nice graph showing the components and their connections.

By default, StreamWork reads, parses, and analyzes; a simple SWL language StreamWork network definition(ND) file,
then generates a main Go program.

Options allow you to produce an abstract syntax tree, a GraphViz dot file, or a goFBP main prgram instead.

Synopsis

sw [-m MODE] [-cfg CONFIGURATION_FILE] [-d DEFAULT_PATH] [ SW_FILE ]

sw -v

sw --help

If SW_FILE is omitted, sw will read from stdin.

		MODE= 
			0-GOMODE		. Generate a main SW Go program[DEFAULT].
			1-ASTMODE		. Display the Abstract Syntax Tree 
			2-GENTREE		. Display the Linearized tree
			3-GRAPHMODE		. Generate Graphviz .dot file code
			4-JAVAFBP		. Incomplete and untested. Looking for help.
			5-PROJECT		. Generate **_swgen.sh_** code for a project tree.
			7-CMODE			. Incomplete and untested. Looking for help.
			8-GENGOFBP      . Generate main go code for J. P. Morrison's gofbp. 

Return code

sw returns SUCCESS unless it encounters an error, in which case sw returns FAILURE. See package documention for component return codes.

TestSWL program

TestSWL syntax checks stdin or a SteamWork network definition file or a SteamWork configuration file.

synopsis

    usage: Call with one of the following argument combinations:
	--help		Display this help message.
	(no arguments)	Parse stdin verbosely.
	(files)		Parse content of files verbosely.
	-s (files)	Silent mode. Parse content of files silently.

Network Definition

A StreamWork network definition describes a program as a list of data streams. Each data stream consists of a source process, a director, and a sink process. The network definition is a compact text-based representation of a data flow diagram.

StreamWork is concerned primarily with Go language FBP programs, however, the network definition described here is language agnostic. It is quite possible to generate corresponding code in other languages from a StreamWork ND.

The script, swgo, reads and executes a network definition employing sw to properly launch and connect the processes. By default, sw generates and builds a single main Go program which:

• imports sw compatible component package(s) as specified in the network definition,
• launches a goroutine for each process,
• connects these processes via Golang interface channels, and
• waits for all processes to finish.
• Once the processes launched, StreamWork has no subsequent interaction with them.

On option, sw will read, parse and interpret a network definition file then create either a:

• a StreamWork Go main program,
• a gofbp Go main program (see github.com/jpaulm/gofbp),
• a GraphViz .dot file(script,( swgraph will create and display a graph(s) directly)
• a bash script, (swproject generates a complete GO language project tree and tests it)
• an abstract syntax tree (a reverse polish notation rendition of the essential language components), OR
• a linearized tree (a network definition reconstruction).

Network Definition Files

A network definition file (in the SWL language[see below]) consists of a list of streams(aka dataflows); and may also contain subnet definitions, INCLUDE and PREFIX statements, and comments.

INCLUDEd files are also network definitions which may in turn contain INCLUDE statements. Exceeding 100 levels of includes will cause program termination.

A stream definition looks something like:

(a C "ARG..." ) -ipType> (b D);
     or [reversing direction and using default component names]
(E)IN <- OUT.1(F)0 <- (G); 

and consists of a list of streams. Each stream consists of a source process, a stream director, a sink process and a terminating semicolon(;). Each process consists of

• a processname
• optional component identifier
• optional string arguments, and
• optional process attributes [surrounded by brackets]. Attributes consist of KEY=STRING|NUMBER statements, used mainly to enhance graphic images.

all surrounded by parens, and followed and/or preceded by optional port identifiers consisting of a port number preceded by an optional port name and period(.). Omitted port identifiers default to zero(0).

The stream director points from the source process to the sink process. A director may include a type identifier and/or a stream buffer size: e.g. "<type_A 10-" or "-type_B 2>". If not included, buffer size defaults to zero.

Example with anonymous process names and default components:

  (_) -int 3> (_); #[integer stream with buffersize of three]; 

In this example, the default components [as specified in a config file, send their process names to the channel or to stdout:

    (Hello) <- (World); 

produces:

Hello World

A process is defined by its process name, its component identifier, component arguments, and process attributes; all surrounded by parens.
The component identifier, arguments, and attributes will default if omitted. Process names are not apparent to Go and so need not follow Go naming rules, just the StreamWork conventions.

A component identifier consists of its import module/package identifier, a slash, '/', and the component name. Go component names need to be capitalized. If the path is omitted, a configuration default path is assumed.
If the component name is also omitted, configuration default component names are assumed for sink, source, and filter(has both inputs and outputs) processes. Subnet components are identified by ' prefixing its identifier, as in (A ^SubnetName) ...

Stream directors consist of <, an optional, comma-separated list of type identifiers, an optional buffersize integer, and -. Example: <100- The reverse (Ex: -myDataType,yourType 100>) is also valid.

Information packets(IPs) are designed as nil(empty) interfaces. The data type is determined by the sending component. It is possible for properly coded sink components to handle multiple data types. A type mis-match will be reported by incompatible receiving components. Components can be coded to handle any type, including user-defined types and structures. Some components can process strings and integers; other components just a single type; on each receiving port.

sw versions beginning with v1.0.0 are backward compatible within the same major version(currently v0). There is no such guarantee with v0.N.N versions, but we will increase the minor version in such cases. The master github branch may have many commits without creating a new version.

sw builds a network model in memory, then optionally generates either

• an abstract syntax tree,
• a linearized tree(a network definition reconstruction),
• a GraphViz .dot file,
• input to swproject to build a complete working project tree,
• or Go source code from the network model.
• or Go main source code for JFM's gofbp.

C and other languages could also be generated, but this has not been implemented. JavaFBP is possible, for instance.

Comments and critiques are welcome. Contributors are encouraged.

Please do not submit code before contacting the project. Contact by e-mailing [email protected] or Discord/gofbp message is preferred to posting a request on Github.

QuickStart on Linux

Requirements

• Gcc, etc.
• Graphviz should be installed, but installation may proceed without it.
• The ...github/tyoung3/StreamWork backend is no longer required.
• sw is written in C.

Installation

• cd to any convenient workspace (/usr/local/src for example).
• Download the latest sw-0.28.0.tar.gz file from https://github.com/tyoung3/sw/build
• Run 'tar -xzf .../sw-0.28.0.tar.gz' to extract source files
• cd sw-0.28.0
• Run ./configure && make check
• Run sudo make install.
  sw and associated scripts will be installed in /usr/local/bin

Post Install

Foo Bar test

mkdir /tmp/x/

• echo "(Foo) <- (Bar);" | sw > /tmp/x/fb.go

cd /tmp/x/ go mod tidy

• Run: go run /tmp/fb.go

This is what the swgo script does: try swgo .../nds/fb.sw

OUTPUT: 

Foo Bar-1
Foo Bar-2
Foo Bar-3
Foo Bar-4
Foo Bar-5
Foo Bar-6
Foo Bar-7

Building on Linux

• Install autotools, ctags, libyaml-dev, bnfc, bison, and flex
• cd to your project workspace (like /usr/local/src )
• git clone https://github.com/tyoung3/sw.git
• bin/sw.sh build; # Runs autotools including .../sw/bin/configure.
• bin/sw.sh c; # Runs make check in .../sw/src
• bin/sw.sh bw; # Builds Windows/sw.exe. Requires mingw. Run make distclean first.

StreamWork Language(SWL) Notes

SWL is an unambiguous, context free grammar, making it directly interpretable, without preprocessing.

See the bnfc StreamWork language description file, SWL.pdf.

SWL has few reserved words, making it somewhat natural language agnostic.

Every effort is made to ensure that all currently valid SWL statements will remain valid. Should this prove impractible, the minor version will be incremented. Once v1.0.0 is reached, the major version will be incremented instead. Backward conpatibility is enforced for network definition files, sw, and published Go modules only.

Linux scripts and Go modules, are separately versioned.

Stream types and component arguments

Stream channels can be any Go language type: int, string, struct, or interface. '_' is shorthand for an interface channel type.
Components with channels of varying types, should group types to simplify their entry points.

Autojoin and autosplit work only with interface types.

While StreamWork packages a number of components, none of them are required. It is quite possible to define a network with no dependencies (other than Go packages) at all.
The only requirement for a component is an entry point with two or more parameters: wait group, argument string slice, and a channel for each port.

Variables

SWL accepts string($NAME), environment string($_NAME), symbol(&NAME), and numeric(%NAME) variables. See SWL.pdf for acceptable usage.

StreamWork uses the following environment variables:

• $BROWSER -- WWW browser
• $GOPATH -- Go language path
• $DEBUG -- 'Y' will cause some messages to print.

Semi-colons

Statements in SWL, are terminated with a semi-colon. Semi-colons in code are like periods at the end of English statements -- they tell the reader (and the SWL interpreter) when you have reached the end of a statement. This makes reading the statements easier. Imagine trying to read a book without any periods (or initial capitals). Additionally, without semi-colons line breaks become part of the language definition, leading to awkward, confusing syntax rules; difficult for the reader to interpret.

Left Arrow comments

The, <-, token is employed in order to be consistent with its usage in the Go language. The -> token is also available: "(A) -> (B)1 <- (C);" is valid SWL.

Collate/Merge Example

Run the following (in Linux):

	mkdir /tmp/collate
	sw <  sw/nds/collate.sw   > /tmp/collate.go
	go run /tmp/collate.go |grep Match0  
	
to produce: 

Match0 Int: 5
Match0 Int: 7
Match0 Int: 9
Match0 Int: 11
Match0 Int: 17
Match0 Int: 23

Hello World

echo "(HELLO)->(WORLD);" |sw > /tmp/hello.go
go run  /tmp/hello.go 
Produces:
WORLD HELLO-1
WORLD HELLO-2
WORLD HELLO-3
WORLD HELLO-4
WORLD HELLO-5
WORLD HELLO-6
WORLD HELLO-7

	sw -m 3 <  sw/nds/collate.sw | dot -Tsvg > /tmp/collate.svg

produces

an image of the collate network.

Bash Scripts

swgo script

The bash script, swgo, will generate and run a GO program from a network definition:

  .../nds/hw.sw
(Hello) <string- (World);

swgo .../nds/hw produces:

package main

/*
                  StreamWork Go code. 
	Generated by sw/swgo.c-0.22.0 Wed Mar 16 14:48:30 2022
	from Network file: /usr/local/src/sw/src/nds/hw.sw
	Configuration file: /usr/local/src/sw/src/nds/sw.cfg
            ***  Expanded Network Definition  ***
(Hello sw.Print ) 		<_-	0(World sw.Gens);	
1 stream, 2 processes, 2 components, 1 partition, 0 cycles.
*/

import "sync"
import "github.com/tyoung3/sw"


func main() {
var wg sync.WaitGroup

cs0 := make(chan interface{},0)	//World.0->Hello.0

	wg.Add(1)

var cp1 []chan interface{}
cp1=append(cp1,cs0)
go	sw.Gens(&wg,[]string{"World"},cp1) // World

var cp2 []chan interface{}
cp2=append(cp2,cs0)
go	sw.Print(&wg,[]string{"Hello"},cp2) // Hello

wg.Wait()
}


Which in turn produces:

Hello World-1
Hello World-2
Hello World-3
Hello World-4
Hello World-5
Hello World-6

swgraph Script

Another bash script, swgraph, will display a graphic image of a network definition:

    .../nds/mvc.sw
(m Model)1      -change>         (v View);
(c)2            -update>        1(v);
(v)2            -event>          (c Control); 
(c)1            -request>       (m);

swgraph .../nds/mvc produces:

for instance. Processes are colored such that all components in the same package have the same color. Not shown here, unfortunately, are the tooltips and html references. Arrows are colored according to stream buffersize: 0 - black; 1-green; 2 or more - orange. Coloring rules are not guaranteed to remain unchanged in future sw versions. WARNING: buffersize is related to deadlock potential in cyclic networks.

The resulting image is an annotated Data Flow Diagram.

gofbp script

gofbp produces and runs a gofbp program (see github.com/jpaulm/gofbp ) from a StreamWork net definition file.

gofbp Example (from sw/src/model/tests/gofbp/loadbal.sw)

Given: LOADBAL.SW

$core = "github.com/jpaulm/gofbp/core";
$trtn = "github.com/jpaulm/gofbp/components/testrtn";
(Sender $trtn/Sender "10 COUNT=10")OUT	-load> IN.0(LoadBalance $trtn/LoadBalance );
(LoadBalance)OUT.1 -load> IN(Receiver0 $trtn/DelayedReceiver);
(LoadBalance)OUT.2 -load> IN(Receiver1 $trtn/Receiver);
(LoadBalance)OUT.3 -load> IN(Receiver2 $trtn/Receiver);

gofbp -b loadbal

produces

and (on console)

Input to Receiver: Receiver1 > IP - # 1
Input to Receiver: Receiver1 > IP - # 4
Input to Receiver: Receiver1 > IP - # 6
Input to Receiver: Receiver1 > IP - # 8
Input to Receiver: Receiver2 > IP - # 0
Input to Receiver: Receiver2 > IP - # 3
Input to Receiver: Receiver2 > IP - # 7
Input to DelayedReceiver: Receiver0 > IP - # 2
Input to DelayedReceiver: Receiver0 > IP - # 5
Input to DelayedReceiver: Receiver0 > IP - # 9

Generate a project

swproject -yaml YAML -cfg ./src/nds/postage.cfg ./src/model/tests/postage.sw

produces the following source tree:

├── cmd
│   └── postage.go
├── ComputePostage.go
├── ComputePostage_test.go
├── config.go
├── GetLoc.go
├── GetLoc_test.go
├── GetWeight.go
├── GetWeight_test.go
├── go.mod
├── go.sum
├── internal
│   ├── gen.sh
│   ├── postage.sw
│   └── sw.cfg
├── ipT.go
├── PostageT.go
├── postage.yaml
├── PrintLabel.go
├── PrintLabel_test.go
├── Rates.go
├── Rates_test.go
├── RateT.go
├── testdata
│   └── README
└── WeighT.go

3 directories, 23 files

then produces this graph vets the code, and finally runs it. The result is a running program ready for your enhancements.

sw.sh Synopsis

sw.sh-1.0.1 USAGE: 
		b|build		. Build project with Autotools.
		c		. Make check
		cl       	. Show release check list. 
		d  [OPTs]	. Switch to docker container. 
		e		. Exit SW shell.
		d build [OPTs]	. Build SWdemo docker container. 
		doc		. Run and browse Doxygen 
		j		. Graph collate.sw with swgraph
		jl		. Generate and view locusts.png 
		p  [NAME..]	. Generate project(s) named NAME... 
		poc		. Build and run Proof of Concept 
		rm		. View README in /opt/google/chrome/chrome 
		rc		. Build and run Collate program 
		rl		. Run locusts program
		s		. Enter SW shell.  'e' to exit the shell. 
		v		. Display this script version
		x		. Edit this script
		--help	. Display this help

BUGS:

sw.Gen fails unless all three arguments are present.

Stream directors like => and <= are accepted, but treated the same as -> and <-, except for appearing differently in graphs.
The meaning is reserved for future use, which could break existing networks.

Fixed

[Fixed with v0.28.7] Subnet process names must be unique, but this is not guaranteed. Using the same process name in different subnet definitions will no longer cause unpredictable, negative consequences.

Home Page

(https://sw.twyoung.com)

Release Notes

0.18.0

• Reorganized repo tree structure
• Renamed package swbase to sw and moved code to repo root
• Changed package swutility to import package sw

0.18.4

• Fixes for restructure problems
• Fixes to bash scripts
• Fixes to githup builds

0.18.5

• More fixes
• Moved .../sw/c to .../sw/src

0.18.6

• Created gofbp script
• Added sw/testrtn package

0.18.7

• Moved build process to project tree root

0.18.8

• Fixed count in swbal.sw

0.18.9

• Fix testrtn/Sender

0.19.0

• Many fixes to swproject script

0.20.1

• Fix swgraph file search

0.20.6

• Incorporate component attributes in SWL for future graphing enhancements.
• Fix fan logic when connection is missing.

0.20.7

• Improve gofbp script
• Update README

0.20.8

• Created fileio package with GetFile and PutFile and FileT

0.20.10

• Provide process attributes.

0.21.0

• Can generate projects with or without YAML configurationswpro.
• Fixes to Wrap function.

0.21.1

• README updates
• Updated SWL.pdf

0.22.0

• Go code generation revamped
• Typed channels partly implemented, possibly breaking networks with types specified in a director.

0.26.6

• Fixed double arrow graph error

0.26.7

• Introduced mkdocs
• Attempt to fix go.in yaml Dependabot alert
• Fix build error in swgraph.cswpro

0.26.8

• Removed 'default:' error from swgraph.c

0.26.9

• Fixed bug with subnet attributes

0.26.10

• Fixed maximum subnet level checking
• Introducted mkdocs web site docs

0.26.11

• Changed type mismatches from FAILing to WARNING.

0.27.0

• Changed package name from sw/linux to sw/os
• Added sw/os/Debug function

0.28.0

• Implemented typing of component channels.
• Defined underscore '_' as shorthand for 'interface{}' type.
• Changed component argument from interface slice to interface in SW components with a single port.
• For components with multiple ports; adjacent channels of the same
  type are combined into a channel slice.
• Fixed numerous problems with swproject.

0.28.1

• Fixed IP type bugs
• Updated docs

0.28.2

* Fixed subnet expansion bugs   

0.28.3

* Fix project generation i/o bug	

0.28.4

* More project generation fixes	

0.28.6

* Fix cfg initialization
* Check Wextra warnings.
* Remove some dead code    

0.28.7

* Prefix subnet process names to make them unique.    

0.28.8

* Fix go.mod yaml.v3 dependency failure -- 'require stretch..v1.1
* Point package names to github.com.../sw
* Use src/model/sw not bin/sw(removed) in sw.sh/run script

Author

Tom Young, [email protected]

with thanks to J. Paul Morrison, Phillip W. Young, Sam Watkins, 
the contributors to [email protected], 
and to the developers/maintainers of Go, gcc, Linux/Ubuntu, 
BNFC, git, GraphViz, Github and others.