Multiple Workers per Test Case (#33)

* Varying Workers per Test Case

Signed-off-by: hfuss <[email protected]>

* fixing incorrect worker ids

Signed-off-by: hfuss <[email protected]>

* readme fix; not vendoring upstream chart dep

Signed-off-by: hfuss <[email protected]>

.gitignore

@@ -8,3 +8,4 @@ dist/
 .idea/
 
 **/local-values*.yaml
+charts/**/*.tgz

README.md

@@ -1,45 +1,58 @@
 # FireFly Performance CLI
 
-FireFly Performance CLI is a HTTP load testing tool that generates a constant request rate against a [FireFly](https://github.com/hyperledger/firefly) network and measure performance. This it to be confident [FireFly](https://github.com/hyperledger/firefly) can perform under normal conditions for an extended period of time.
+FireFly Performance CLI is a HTTP load testing tool that generates a constant request rate against a [FireFly](https://github.com/hyperledger/firefly)
+network and measure performance. This used to confirm confidence that [FireFly](https://github.com/hyperledger/firefly)
+can perform under normal conditions for an extended period of time.
 
 ## Items Subject to Testing
 
-- [x] Broadcasts (`POST /messages/broadcasts`)
-- [x] Private Messaging (`POST /messages/private`)
-- [x] Mint Tokens (`POST /tokens/mint`)
-- [x] Fungible vs. Non-Fungible Token Toggle
-- [x] Blobs
+- Broadcasts (`POST /messages/broadcasts`)
+- Private Messaging (`POST /messages/private`)
+- Mint Tokens (`POST /tokens/mint`)
+  - Fungible vs. Non-Fungible Token Toggle
+- Blobs
+- Contract Invocation (`POST /contracts/invoke`)
+  - Ethereum vs. Fabric
 
-## Run a test
+## Run
 
-`ffperf run-tests msg_broadcast -l 24h`
+The test configuration is structured around running `ffperf` as either a single process or in a distributed fashion as
+multiple processes.
 
-## Options
+In the test configuration you define one or more test _instances_ for a single `ffperf` process to run. An instance then
+describes running one or more test _cases_ with a dedicated number of goroutine _workers_ against a _sender_ org and
+a _recipient_ org. The test configuration consumes a file reference to the stack JSON configuration produced by the
+[`ff` CLI](https://github.com/firefly-cli) (or can be defined manually) to understand the network topology, so that
+sender's and recipient's just refer to indices within the stack.
+
+As a result, running the CLI consists of providing an `instances.yaml` file describe the test configuration
+and an instance index or name indicating which instance the process should run:
 
 ```shell
+ffperf run -c /path/to/instances.yaml -i 0
+```
+
+See [`example-instances.yaml`](config/example-instances.yaml) for examples of how to define multiple instances
+and multiple test cases per instance with all the various options.
+
+## Options
+
+```
+Executes a instance within a performance test suite to generate synthetic load across multiple FireFly nodes within a network
+
 Usage:
-  ffperf run-tests [flags]
+  ffperf run [flags]
 
 Flags:
-  -a, --address string            Address of custom contract
-      --chaincode string          Chaincode name for custom contract
-      --channel string            Fabric channel for custom contract
-  -h, --help                      help for run-tests
-  -l, --length duration           Length of entire performance test (default 1m0s)
-      --longMessage               Include long string in message
-  -r, --recipient string          Recipient for FireFly messages
-  -x, --recipientAddress string   Recipient address for FireFly transfers
-  -s, --stackJSON string          Path to stack.json file that describes the network to test
-      --tokenType string          [fungible nonfungible] (default "fungible")
-  -w, --workers int               Number of workers at a time (default 1)
+  -c, --config string          Path to performance config that describes the network and test instances
+  -d, --daemon                 Run in long-lived, daemon mode. Any provided test length is ignored.
+      --delinquent string      Action to take when delinquent messages are detected. Valid options: [exit log] (default "exit")
+  -h, --help                   help for run
+  -i, --instance-idx int       Index of the instance within performance config to run against the network (default -1)
+  -n, --instance-name string   Instance within performance config to run against the network
 )
 ```
 
-## Examples
-
-- 10 workers submit broadcast messages for 500 hours
-
-  - `ffperf run-tests msg_broadcast -l 500h -w 10`
+## Distributed Deployment
 
-- 75 workers submit broadcast messages, private messages, and token mints for 10 hours. 25 workers per item
-  - `ffperf run-tests msg_broadcast msg_private token_mint -l 10h -r "0x123" -w 75`
+See the [`ffperf` Helm chart](charts/ffperf) for running multiple instances of `ffperf` using Kubernetes.

charts/ffperf/values.yaml

@@ -43,11 +43,12 @@ wsConfig:
 
 instances: []
 #   - name: ff0-ff1-broadcast
-#     tests: ["msg_broadcast"]
+#     tests:
+#        - name: "msg_broadcast"
+#          workers: 10
 #     length: 5m
 #     sender: 0
 #     recipient: 1
-#     workers: 10
 #     messageOptions:
 #       longMessage: true
 

charts/firefly-perfnode/Chart.lock

@@ -1,6 +1,6 @@
 dependencies:
 - name: firefly
   repository: oci://ghcr.io/hyperledger/helm
-  version: 0.4.1-20220324-129
-digest: sha256:0d719aa388a17ef57b8cdbd7c3ecb01eba95b06943368bf91aa949ca66fba4db
-generated: "2022-03-25T12:56:06.588711-04:00"
+  version: 0.5.0-20220421-139
+digest: sha256:f53967f6d3dcb3bc924e6c13ddd7055e2fe13b599e8268450a1302bc7de7f0c3
+generated: "2022-04-25T11:10:45.964906-04:00"

charts/firefly-perfnode/Chart.yaml

@@ -19,8 +19,8 @@ name: firefly-perfnode
 description: |
   A Helm chart for deploying FireFly nodes configured for performance testing flows between members.
 type: application
-version: 0.1.0
-appVersion: "0.14.1"
+version: 0.1.1
+appVersion: "1.0.0"
 
 maintainers:
   - name: hfuss
@@ -34,5 +34,5 @@ maintainers:
 
 dependencies:
   - name: firefly
-    version: 0.4.1-20220324-129
+    version: 0.5.0-20220421-139
     repository: oci://ghcr.io/hyperledger/helm

cmd/run.go

@@ -138,8 +138,8 @@ func selectInstance(config *conf.PerformanceTestConfig) (*conf.InstanceConfig, e
 	return nil, errors.Errorf("please set either the \"instance-name\" or \"instance-index\" ")
 }
 
-func generateRunnerConfigFromInstance(instance *conf.InstanceConfig, perfConfig *conf.PerformanceTestConfig) (*conf.PerfRunnerConfig, error) {
-	runnerConfig := &conf.PerfRunnerConfig{
+func generateRunnerConfigFromInstance(instance *conf.InstanceConfig, perfConfig *conf.PerformanceTestConfig) (*conf.RunnerConfig, error) {
+	runnerConfig := &conf.RunnerConfig{
 		Tests: instance.Tests,
 	}
 
@@ -164,7 +164,6 @@ func generateRunnerConfigFromInstance(instance *conf.InstanceConfig, perfConfig
 	runnerConfig.MessageOptions = instance.MessageOptions
 	runnerConfig.TokenOptions = instance.TokenOptions
 	runnerConfig.ContractOptions = instance.ContractOptions
-	runnerConfig.Workers = instance.Workers
 	runnerConfig.Length = instance.Length
 	runnerConfig.WebSocket = perfConfig.WSConfig
 	runnerConfig.Daemon = perfConfig.Daemon
@@ -184,7 +183,7 @@ func generateRunnerConfigFromInstance(instance *conf.InstanceConfig, perfConfig
 	return runnerConfig, nil
 }
 
-func validateConfig(cfg conf.PerfRunnerConfig) error {
+func validateConfig(cfg conf.RunnerConfig) error {
 	if cfg.TokenOptions.TokenType != "" && cfg.TokenOptions.TokenType != fftypes.TokenTypeFungible.String() && cfg.TokenOptions.TokenType != fftypes.TokenTypeNonFungible.String() {
 		return fmt.Errorf("invalid token type. Choose from [%s %s]", fftypes.TokenTypeFungible.String(), fftypes.TokenTypeNonFungible.String())
 	}

config/example-instances.yaml

@@ -10,31 +10,33 @@ wsConfig:
   heartbeatInterval: 5s
 
 instances:
-  - name: ff0-broadcast
-    test: msg_broadcast
+  - name: ff0
+    tests:
+      - name: msg_broadcast
+        workers: 20
     length: 3m
     sender: 0
-    workers: 10
     messageOptions:
       longMessage: true
-  - name: ff0-ff0-private
-    test: msg_private
+  - name: ff0-ff1
+    tests:
+      - name: msg_private
+        workers: 10
+      - name: token_mint
+        workers: 5
     length: 3m
     sender: 0
     recipient: 1
-    workers: 10
     messageOptions:
       longMessage: true
-  - name: ff0-ff1-mint
-    test: token_mint
-    length: 3m
-    sender: 0
-    recipient: 1
-    workers: 10
     tokenOptions:
       tokenType: fungible
   - name: ff1
-    test: custom_ethereum_contract
+    tests:
+      - name: custom_ethereum_contract
+        workers: 10
+      - name: msg_broadcast
+        workers: 20
     length: 3m
     sender: 1
     workers: 10

internal/conf/conf.go

@@ -24,30 +24,15 @@ import (
 	"github.com/hyperledger/firefly/pkg/wsclient"
 )
 
-type MessageOptions struct {
-	LongMessage bool `json:"longMessage" yaml:"longMessage"`
-}
-
-type TokenOptions struct {
-	TokenType string `json:"tokenType" yaml:"tokenType"`
-}
-
-type ContractOptions struct {
-	Address   string `json:"address" yaml:"address"`
-	Channel   string `json:"channel" yaml:"channel"`
-	Chaincode string `json:"chaincode" yaml:"chaincode"`
-}
-
-type PerfRunnerConfig struct {
-	Tests            []fftypes.FFEnum
+type RunnerConfig struct {
+	Tests            []TestCaseConfig
 	Length           time.Duration
 	MessageOptions   MessageOptions
 	RecipientOrg     string
 	RecipientAddress string
 	TokenOptions     TokenOptions
 	ContractOptions  ContractOptions
-	WebSocket        FireFlyWsConf
-	Workers          int
+	WebSocket        FireFlyWsConfig
 	NodeURLs         []string
 	StackJSONPath    string
 	DelinquentAction string
@@ -58,23 +43,41 @@ type PerfRunnerConfig struct {
 type PerformanceTestConfig struct {
 	StackJSONPath string           `json:"stackJSONPath" yaml:"stackJSONPath"`
 	Instances     []InstanceConfig `json:"instances" yaml:"instances"`
-	WSConfig      FireFlyWsConf    `json:"wsConfig,omitempty" yaml:"wsConfig,omitempty"`
+	WSConfig      FireFlyWsConfig  `json:"wsConfig,omitempty" yaml:"wsConfig,omitempty"`
 	Daemon        bool             `json:"daemon,omitempty" yaml:"daemon,omitempty"`
 }
 
 type InstanceConfig struct {
 	Name            string           `yaml:"name" json:"name"`
-	Tests           []fftypes.FFEnum `yaml:"tests" json:"test"`
+	Tests           []TestCaseConfig `yaml:"tests" json:"tests"`
 	Length          time.Duration    `yaml:"length" json:"length"`
 	MessageOptions  MessageOptions   `json:"messageOptions,omitempty" yaml:"messageOptions,omitempty"`
 	Sender          int              `json:"sender" yaml:"sender"`
 	Recipient       *int             `json:"recipient,omitempty" yaml:"recipient,omitempty"`
 	TokenOptions    TokenOptions     `json:"tokenOptions,omitempty" yaml:"tokenOptions,omitempty"`
 	ContractOptions ContractOptions  `json:"contractOptions,omitempty" yaml:"contractOptions,omitempty"`
-	Workers         int              `json:"workers" yaml:"workers"`
 }
 
-type FireFlyWsConf struct {
+type TestCaseConfig struct {
+	Name    fftypes.FFEnum `json:"name" yaml:"name"`
+	Workers int            `json:"workers" yaml:"workers"`
+}
+
+type MessageOptions struct {
+	LongMessage bool `json:"longMessage" yaml:"longMessage"`
+}
+
+type TokenOptions struct {
+	TokenType string `json:"tokenType" yaml:"tokenType"`
+}
+
+type ContractOptions struct {
+	Address   string `json:"address" yaml:"address"`
+	Channel   string `json:"channel" yaml:"channel"`
+	Chaincode string `json:"chaincode" yaml:"chaincode"`
+}
+
+type FireFlyWsConfig struct {
 	APIEndpoint            string        `mapstructure:"apiEndpoint" json:"apiEndpoint" yaml:"apiEndpoint"`
 	WSPath                 string        `mapstructure:"wsPath" json:"wsPath" yaml:"wsPath"`
 	ReadBufferSize         int           `mapstructure:"readBufferSize" json:"readBufferSize" yaml:"readBufferSize"`
@@ -85,7 +88,7 @@ type FireFlyWsConf struct {
 	HeartbeatInterval      time.Duration `mapstructure:"heartbeatInterval" json:"heartbeatInterval" yaml:"heartbeatInterval"`
 }
 
-func GenerateWSConfig(nodeURL string, conf *FireFlyWsConf) *wsclient.WSConfig {
+func GenerateWSConfig(nodeURL string, conf *FireFlyWsConfig) *wsclient.WSConfig {
 	t, _ := url.QueryUnescape(conf.WSPath)
 
 	return &wsclient.WSConfig{