Adding different ramping methods and functionality (#49)

Updating documentation Adding testing Refactoring configuration Refactors context usage accordign to package doc Signed-off-by: Vicente Zepeda Mas <[email protected]>
cloud-bulldozer · Jan 10, 2022 · 65f059e · 65f059e
1 parent 1a8aacd
commit 65f059e
Show file tree

Hide file tree

Showing 22 changed files with 936 additions and 161 deletions.
diff --git a/README.md b/README.md
@@ -39,17 +39,27 @@ path as your binary, it will be autodetected and you could run by just calling i
 >> Default values don't count for precedence.
 
 ```
-      --config-file string     config file (default "config.yaml")
-      --duration int           Duration of each individual run in minutes. (default 1)
-      --cooldown int           Cooldown time between tests in seconds. (default 10 s)
-      --gateway-url string     Gateway url to perform the test against (default "https://api.integration.openshift.com")
-  -h, --help                   help for ocm-api-load
-      --ocm-token string       OCM Authorization token
-      --ocm-token-url string   Token URL (default "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token")
-      --output-path string     Output directory for result and report files (default "results")
-      --rate string            Rate of the attack. Format example 5/s. (Available units 'ns', 'us', 'ms', 's', 'm', 'h') (default "1/s")
-      --test-id string         Unique ID to identify the test run. UUID is recommended (default "dc049b1d-92b4-420c-9eb7-34f30229ef46")
-      --test-names strings     Names for the tests to be run. (default [all])
+      --aws-access-key string      AWS access key
+      --aws-access-secret string   AWS access secret
+      --aws-account-id string      AWS Account ID, is the 12-digit account number.
+      --aws-region string          AWS region (default "us-west-1")
+      --config-file string         config file (default "config.yaml")
+      --cooldown int               Cooldown time between tests in seconds. (default 10)
+      --duration int               Duration of each individual run in minutes. (default 1)
+      --end-rate int               Ending request per second rate. (E.g.: 5 would be 5 req/s)
+      --gateway-url string         Gateway url to perform the test against (default "https://api.integration.openshift.com")
+  -h, --help                       help for ocm-api-load
+      --ocm-token string           OCM Authorization token
+      --ocm-token-url string       Token URL (default "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token")
+      --output-path string         Output directory for result and report files (default "results")
+      --ramp-duration int          Duration of ramp in minutes, before normal execution. (default 0)
+      --ramp-steps int             Number of stepts to get from start rate to end rate. (Minimum 2 steps)
+      --ramp-type string           Type of ramp to use for all tests. (linear, exponential)
+      --rate string                Rate of the attack. Format example 5/s. (Available units 'ns', 'us', 'ms', 's', 'm', 'h') (default "1/s")
+      --start-rate int             Starting request per second rate. (E.g.: 5 would be 5 req/s)
+      --test-id string             Unique ID to identify the test run. UUID is recommended (default "c160dab1-7fa3-4965-9797-47da16e5c1b9")
+      --test-names strings         Names for the tests to be run.
+  -v, --verbose                    set this flag to activate verbose logging.
 ```
 
 ## Tests
@@ -60,12 +70,15 @@ path as your binary, it will be autodetected and you could run by just calling i
 | list-subscriptions | /api/accounts_mgmt/v1/subscriptions | GET |
 | access-review | /api/authorizations/v1/access_review | POST |
 | register-new-cluster | /api/accounts_mgmt/v1/cluster_registrations | POST |
+| register-existing-cluster | /api/accounts_mgmt/v1/cluster_registrations | POST |
 | create-cluster | /api/clusters_mgmt/v1/clusters | POST |
 | list-clusters | /api/clusters_mgmt/v1/clusters | GET |
 | get-current-account | /api/accounts_mgmt/v1/current_account | GET |
 | quota-cost | /api/accounts_mgmt/v1/organizations/{orgId}/quota_cost | GET |
 | resource-review | /api/authorizations/v1/resource_review | POST |
 | cluster-authorizations | /api/accounts_mgmt/v1/cluster_authorizations | POST |
+| self-terms-review | /api/authorizations/v1/self_terms_review | POST |
+| certificates | /api/accounts_mgmt/v1/certificates | POST |
 |--|--|--|
 
 ## Config file
@@ -83,6 +96,11 @@ path as your binary, it will be autodetected and you could run by just calling i
 - cooldown: Cooldown time between tests in seconds. (default 10 s)
 - rate: Rate of the attack. Format example 5/s. (Available units 'ns', 'us', 'ms', 's', 'm', 'h') (default "1/s")
 - test-id: Unique ID to identify the test run. UUID is recommended (default "dc049b1d-92b4-420c-9eb7-34f30229ef46")
+- ramp-type: Type of ramp to use for all tests. (linear, exponential)
+- ramp-duration: Duration of ramp in minutes, before normal execution. (default 0)
+- start-rate: Starting request per second rate. (E.g.: 5 would be 5 req/s)
+- end-rate: Ending request per second rate. (E.g.: 5 would be 5 req/s)
+- ramp-steps: Number of stepts to get from start rate to end rate. (Minimum 2 steps)
 - tests: List of the tests to run. Empty list means all.
 
 ### Test options
@@ -92,6 +110,31 @@ Each test can contain this options:
 - rate: Rate of the attack. Format example 5/s. (Available units 'ns', 'us', 'ms', 's', 'm', 'h') (default "1/s")
 - duration: Override duration for the test. (A positive integer accompanied of a valid unit)
 
+#### Ramping functionality
+
+Each test can have a specific configuration for ranmping up the rate, inthis case the following options must be provided.
+
+- duration: in minutes
+- ramp-type: Type of ramp to use for all tests. (linear, exponential)
+- ramp-duration: Duration of ramp in minutes, before normal execution. (default 0)
+- start-rate: Starting request per second rate. (E.g.: 5 would be 5 req/s)
+- end-rate: Ending request per second rate. (E.g.: 5 would be 5 req/s)
+- ramp-steps: Number of stepts to get from start rate to end rate. (Minimum 2 steps)
+
+> `rate` option is not needed for this.
+
+##### Example
+
+```yaml
+  cluster-authorizations:
+    duration: 30
+    ramp-type: exponential
+    ramp-duration: 10
+    start-rate: 1
+    end-rate: 50
+    ramp-steps: 6
+```
+
 ### Obligatory options
 
 - ocm-token
@@ -169,3 +212,47 @@ Steps:
 - Be sure you are in the latest version of `main` branch and have bumped the version
 
 - Now you are ready to run `make release` this will build the binary and generate the tarfiles that contain all the needed files
+
+## Ramping Up Theory
+
+The test will run a number <ramp-steps> with a running time of <duration>/<ramp-steps> rounded for each step, this can sometimes make the test last more or less than the expected duration, but we want to have a even distribution of times.
+
+As each step finishes it will increase the rate according to a delta that is calculated with the parameters:
+
+For both types of ramps we have common behaviour:
+
+- First rate: is always `start-rate`
+- Last rate: is always `end-rate`
+- Since we cannot use float values for rates, we round all the rates to it's closest integer.
+
+### For a linear ramp it will use this formula
+
+`delta = ( end-rate - start-rate ) / ( ramp-steps - 1 )`
+>ramp-steps, has always have to be greater than 1
+
+So the new rate will be:
+
+`newRate = oldRate + delta`
+
+### For an exponential distribution
+
+We are using the exponential formula `f(t)= x * <coeff> ^ t`
+
+the `coeff` is calculated with this formula
+
+`coeff = (end-rate / start-rate) ^ (1 / ramp-steps)`
+
+So the new rate will be:
+
+`newRate = start-rate * coeff ^ <# of step>`
+
+### `duration` vs `ramp-duration`
+
+The `duration` is the number of minutes the test is going to run. The `ramp-duration` is the number of minutes the ramp is going to last.
+
+- If `ramp-duration` is not set, the ramp will take the whole `duration`.
+- If `ramp-duration` is set, it will run the ramp for that long and then run the remaining of the `duration` at the `end-rate`.
+  - E.g.: if `duration` is 30 minutes and `ramp-duration` is 20 minutes. The test will run a ramp for 20 minutes and keep running at `end-rate` for the remaining 10 minutes. So it will run `end-rate` for `duration` - `ramp-duration` minutes.
+- If `ramp-duration` is greater than `duration` it will just perform a ramp for `ramp-duration` minutes.
+
+Overrides for the values work the same, localized test values take priority over global values.
diff --git a/automation.py b/automation.py
@@ -189,6 +189,127 @@ def show_graphs(directory, filename):
                  validate=False)
 
 
+def cma_graph(directory, filename):
+    regex = re.compile(r'(.*/)?[\w-]+_([\w-]+).(\w.+)')
+    matches = regex.match(filename)
+    if regex.match(filename) and matches.group(3) == 'json':
+        # Initializes database for current file in current directory
+        # Read by 20000 chunks
+        disk_engine = create_engine(
+            'sqlite:///{}.db'.format(matches.group(2)))
+
+        j = 0
+        index_start = 1
+        chunk = 20000
+        for df in pd.read_json(os.path.join(directory, filename),
+                               lines=True,
+                               chunksize=chunk):
+            df.index += index_start
+
+            columns = ['timestamp', 'latency']
+
+            for c in df.columns:
+                if c not in columns:
+                    df = df.drop(c, axis=1)
+
+            j += 1
+            logger.info(f'completed {j*chunk} rows')
+
+            df.to_sql('data', disk_engine, if_exists='append')
+            index_start = df.index[-1] + 1
+
+        df = pd.read_sql_query('SELECT * FROM data', disk_engine)
+        df_t = pd.DataFrame(df.iloc[:, -1])
+        df_t.index = df.timestamp
+
+        df_t['cma'] = df_t.expanding().mean()
+
+        data = [{
+            'type': 'line',
+            'x': df_t.index,
+            'y': df_t['cma']/1000000,
+        }]
+
+        layout = {
+            'title': '<b>Cumulative AVG Latency: {}</b>'.format(
+                matches.group(2)),
+            'xaxis': {'title': 'Time',
+                      'showgrid': 'true',
+                      'ticklabelmode': "period"},
+            'yaxis': {'title': 'Milliseconds (log)', 'type': 'linear'},
+        }
+
+        fig_dict = {'data': data, 'layout': layout}
+
+        os.remove('{}.db'.format(matches.group(2)))
+
+        pio.show(fig_dict,
+                 engine="kaleido",
+                 width=1600,
+                 height=900,
+                 validate=False)
+
+
+def count_graph(directory, filename):
+    regex = re.compile(r'(.*/)?[\w-]+_([\w-]+).(\w.+)')
+    matches = regex.match(filename)
+    if regex.match(filename) and matches.group(3) == 'json':
+        # Initializes database for current file in current directory
+        # Read by 20000 chunks
+        disk_engine = create_engine(
+            'sqlite:///{}.db'.format(matches.group(2)))
+
+        j = 0
+        index_start = 1
+        chunk = 20000
+        for df in pd.read_json(os.path.join(directory, filename),
+                               lines=True,
+                               chunksize=chunk):
+            df.index += index_start
+
+            columns = ['timestamp', 'latency']
+
+            for c in df.columns:
+                if c not in columns:
+                    df = df.drop(c, axis=1)
+
+            j += 1
+            logger.info(f'completed {j*chunk} rows')
+
+            df.to_sql('data', disk_engine, if_exists='append')
+            index_start = df.index[-1] + 1
+
+        df = pd.read_sql_query('SELECT * FROM data', disk_engine)
+        df_t = pd.DataFrame(df.iloc[:, -1])
+        df_t.index = df.timestamp
+
+        df_t['count'] = df_t.expanding().count()
+
+        data = [{
+            'type': 'line',
+            'x': df_t.index,
+            'y': df_t['count'],
+        }]
+
+        layout = {
+            'title': '<b>Request count : {}</b>'.format(matches.group(2)),
+            'xaxis': {'title': 'Time',
+                      'showgrid': 'true',
+                      'ticklabelmode': "period"},
+            'yaxis': {'title': 'Number of requests', 'type': 'linear'},
+        }
+
+        fig_dict = {'data': data, 'layout': layout}
+
+        os.remove('{}.db'.format(matches.group(2)))
+
+        pio.show(fig_dict,
+                 engine="kaleido",
+                 width=1600,
+                 height=900,
+                 validate=False)
+
+
 def generate_summaries(directory):
     try:
         os.stat('{}/summaries'.format(directory))
@@ -466,6 +587,26 @@ def main():
                               help='filename of a result to display the graph. \
                                 (Overrides generating all graphs.)')
 
+    cma_parser = action_subparsers.add_parser("cma",
+                                              help="generate cummulative average graph \
+                                               for the results file",
+                                              parents=[parent_parser])
+
+    cma_parser.add_argument('--filename',
+                            dest="filename",
+                            help='filename of a result to display the graph',
+                            required=True)
+
+    count_parser = action_subparsers.add_parser("count",
+                                                help="generate cummulative count of requests graph \
+                                                for the results file",
+                                                parents=[parent_parser])
+
+    count_parser.add_argument('--filename',
+                              dest="filename",
+                              help='filename of a result to display the graph',
+                              required=True)
+
     action_subparsers.add_parser("summary",
                                  help="generates vegeta \
                                   summary for results",
@@ -478,8 +619,7 @@ def main():
     report_parser.add_argument('--filename',
                                dest='filename',
                                default='report-{}.docx'.format(
-                                                date.strftime("%Y-%m-%d")
-                                                ),
+                                    date.strftime("%Y-%m-%d")),
                                help='name for the report file.')
 
     upload_parser = action_subparsers.add_parser("upload",
@@ -516,6 +656,10 @@ def main():
             show_graphs(args.directory, args.filename)
         else:
             generate_graphs(args.directory)
+    elif args.action_command == 'cma':
+        cma_graph(args.directory, args.filename)
+    elif args.action_command == 'count':
+        count_graph(args.directory, args.filename)
     elif args.action_command == 'summary':
         generate_summaries(args.directory)
     elif args.action_command == 'report':