Skip to content

Commit

Permalink
APIGOV-28475 - TA config optimization (#817)
Browse files Browse the repository at this point in the history 
* APIGOV-28475 - TA config optimization
- Deprecate USAGEREPORTING_INTERVAL
- New config for setting schedule for metric event, default to an hour
- Remove TRACEABILITY_SAMPLING_REPORTALLERRORS
- Added TRACEABILITY_SAMPLING_ONLYERRORS, default to false, when set only errors are sampled
- Updated sampling counting to include errors

* APIGOV-28475 - updated documentation

* APIGOV-28475 - usage reporting schedule config
- Deprecate CENTRAL_USAGEREPORTING_USAGESCHEDULE
- Added CENTRAL_USAGEREPORTING_SCHEDULE

* APIGOV-28475 - update QA usage config

* APIGOV-28475 - update

* APIGOV-28475 - fix to use overrides

* INT - update kin-openapi

---------

Co-authored-by: Jason Collins <[email protected]>
  • Loading branch information
@vivekschauhan @jcollins-axway
vivekschauhan and jcollins-axway authored Aug 8, 2024
1 parent ce6ec6c commit fde925d
Show file tree
Hide file tree
Showing 20 changed files with 620 additions and 295 deletions.
27 changes: 15 additions & 12 deletions docs/traceability/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,10 +74,9 @@ Below is the list of Central configuration properties in YAML and their correspo
| central.ssl.minVersion | CENTRAL_SSL_MINVERSION | String value for the minimum SSL/TLS version that is acceptable. If zero, empty TLS 1.0 is taken as the minimum. Allowed values are: TLS1.0, TLS1.1, TLS1.2, TLS1.3. |
| central.ssl.maxVersion | CENTRAL_SSL_MAXVERSION | String value for the maximum SSL/TLS version that is acceptable. If empty, then the maximum version supported by this package is used, which is currently TLS 1.3. Allowed values are: TLS1.0, TLS1.1, TLS1.2, TLS1.3. |
| central.proxyURL | CENTRAL_PROXYURL | The URL for the proxy for Amplify Central`<http://username:password@hostname:port>`. If empty, no proxy is defined. |
| central.usageReporting.url | CENTRAL_USAGEREPORTING_URL | The Lighthouse URL the agent publishes usage reports. See [Traceability usage reporting](#traceability-usage-reporting) |
| central.metricReporting.publish | CENTRAL_METRICREPORTING_PUBLISH | Enables/disables the sending of metric events to Amplify (default: `true`) |
| central.metricReporting.schedule | CENTRAL_METRICREPORTING_SCHEDULE | The frequency schedule in which the agent sends metric events to Amplify Central (default: `@hourly`) |
| central.usageReporting.publish | CENTRAL_USAGEREPORTING_PUBLISH | Enables/disables the sending of usage events to Amplify (default: `true`) |
| central.usageReporting.publishMetric | CENTRAL_USAGEREPORTING_PUBLISHMETRIC | Enables/disables the sending of metric events to Amplify (default: `true`) |
| central.usageReporting.interval | CENTRAL_USAGEREPORTING_INTERVAL | The frequency in which the agent published activity to Amplify Central (default: `15m`) |
| central.usageReporting.offline | CENTRAL_USAGEREPORTING_OFFLINE | Enables/disables the sending of usage events to lighthouse or saving to disk (default: `false`) |
| central.usageReporting.offlineschedule | CENTRAL_USAGEREPORTING_OFFLINESCHEDULE | The frequency schedule that the agent collects activity for the offline reports (default: `@hourly`) |
| central.grpc.enabled | CENTRAL_GRPC_ENABLED | Controls whether an agent uses a gRPC based stream connection to manage its internal cache. (Default value = false) |
Expand Down Expand Up @@ -979,27 +978,31 @@ By default all transaction data is sent to Amplify.

Below is the list of the sampling configuration properties in a YAML and their corresponding environment variables that can be set to override the config in YAML. All of these are children of output.traceability.sampling

| YAML property | Variable name | Description |
|-----------------|---------------------------------------|------------------------------------------------------------------------------------------------------------------------------|
| percentage | TRACEABILITY_SAMPLING_PERCENTAGE | Defines the percentage of events (0-10. Decimals allowed. Any value above will default it to 1) that are sent to Amplify |
| per_api | TRACEABILITY_SAMPLING_PER_API | Defines if the percentage above is applied to all events or separate based on API ID in the event |
| reportAllErrors | TRACEABILITY_SAMPLING_REPORTALLERRORS | Defines if all error transaction events are sent to Amplify |
| YAML property | Variable name | Description |
|-----------------|---------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|
| percentage | TRACEABILITY_SAMPLING_PERCENTAGE | Defines the percentage of events (0-10. Decimals allowed. Any value above will default it to 1) that are sent to Amplify |
| per_api | TRACEABILITY_SAMPLING_PER_API | Defines if the percentage above is applied to all events or separate based on API ID in the event |
| onlyErrors | TRACEABILITY_SAMPLING_ONLYERRORS | Defines if only error transaction events are sent to Amplify |


### Traceability usage reporting

The Amplify Agents SDK has the ability to track API usages and report them back to the Amplify platform.

By default usage reporting is on but can be configured using the following information.
By default metric and usage reporting is on but can be configured using the following information.

Below is the list of the metric reporting configuration properties, all of these properties are children of [[agent type]].central.metricReporting in the yaml.

| YAML property | Variable name | Default | Description |
|-----------------|----------------------------------------|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
| publish | CENTRAL_METRICREPORTING_PUBLISH | `true` | Defines if individual API Metrics will be published to Amplify |
| schedule | CENTRAL_METRICREPORTING_SCHEDULE | `@hourly` | Defines the schedule, in the default online mode, that metric events are sent to Amplify |

Below is the list of the usage reporting configuration properties, all of these properties are children of [[agent type]].central.usageReporting in the yaml.

| YAML property | Variable name | Default | Description |
|-----------------|----------------------------------------|------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|
| url | CENTRAL_USAGEREPORTING_URL | https://lighthouse.admin.axway.com | Defines the url on the Amplify platform that usage events are sent to |
| publish | CENTRAL_USAGEREPORTING_PUBLISH | `true` | Defines if API usage numbers will be published to Amplify |
| publishMetric | CENTRAL_USAGEREPORTING_PUBLISHMETRIC | `true` | Defines if individual API Metrics will be published to Amplify |
| interval | CENTRAL_USAGEREPORTING_INTERVAL | _15m_ | Defines the interval, in the default online mode, that usage data is sent to Amplify |
| offline | CENTRAL_USAGEREPORTING_OFFLINE | `false` | Defines if the agent is working in offline mode for generating usage reports, see [Offline usage reporting](#offline-usage-reporting) |
| offlineSchedule | CENTRAL_USAGEREPORTING_OFFLINESCHEDULE | `@hourly` | Defines the schedule in which the usage numbers are determined when in offline mode, see [Defining a schedule](../../pkg/jobs/README.md#defining-a-schedule) |

Expand Down
8 changes: 4 additions & 4 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ require (
github.com/emicklei/proto v1.9.2
github.com/fsnotify/fsnotify v1.5.4
github.com/gabriel-vasile/mimetype v1.4.0
github.com/getkin/kin-openapi v0.125.0
github.com/getkin/kin-openapi v0.127.0
github.com/golang-jwt/jwt v3.2.2+incompatible
github.com/google/uuid v1.6.0
github.com/gorhill/cronexpr v0.0.0-20180427100037-88b0669f7d75
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
github.com/invopop/yaml v0.2.0
github.com/invopop/yaml v0.3.1
github.com/lestrrat-go/jwx/v2 v2.0.21
github.com/opentracing/opentracing-go v1.2.0
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475
Expand Down Expand Up @@ -50,8 +50,8 @@ require (
github.com/elastic/go-ucfg v0.8.8 // indirect
github.com/elastic/go-windows v1.0.1 // indirect
github.com/fatih/color v1.16.0 // indirect
github.com/go-openapi/jsonpointer v0.20.2 // indirect
github.com/go-openapi/swag v0.22.8 // indirect
github.com/go-openapi/jsonpointer v0.21.0 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/gofrs/uuid v4.4.0+incompatible // indirect
github.com/golang/protobuf v1.5.4 // indirect
Expand Down
17 changes: 8 additions & 9 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -109,8 +109,8 @@ github.com/fsnotify/fsnotify v1.5.4 h1:jRbGcIw6P2Meqdwuo0H1p6JVLbL5DHKAKlYndzMwV
github.com/fsnotify/fsnotify v1.5.4/go.mod h1:OVB6XrOHzAwXMpEM7uPOzcehqUV2UqJxmVXmkdnm1bU=
github.com/gabriel-vasile/mimetype v1.4.0 h1:Cn9dkdYsMIu56tGho+fqzh7XmvY2YyGU0FnbhiOsEro=
github.com/gabriel-vasile/mimetype v1.4.0/go.mod h1:fA8fi6KUiG7MgQQ+mEWotXoEOvmxRtOJlERCzSmRvr8=
github.com/getkin/kin-openapi v0.125.0 h1:jyQCyf2qXS1qvs2U00xQzkGCqYPhEhZDmSmVt65fXno=
github.com/getkin/kin-openapi v0.125.0/go.mod h1:wb1aSZA/iWmorQP9KTAS/phLj/t17B5jT7+fS8ed9NM=
github.com/getkin/kin-openapi v0.127.0 h1:Mghqi3Dhryf3F8vR370nN67pAERW+3a95vomb3MAREY=
github.com/getkin/kin-openapi v0.127.0/go.mod h1:OZrfXzUfGrNbsKj+xmFBx6E5c6yH3At/tAKSc2UszXM=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
Expand All @@ -119,12 +119,12 @@ github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V
github.com/go-logr/logr v0.1.0/go.mod h1:ixOQHD9gLJUVQQ2ZOR7zLEifBX6tGkNJF4QyIY7sIas=
github.com/go-logr/logr v0.4.0/go.mod h1:z6/tIYblkpsD+a4lm/fGIIU9mZ+XfAiaFtq7xTgseGU=
github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
github.com/go-openapi/jsonpointer v0.20.2 h1:mQc3nmndL8ZBzStEo3JYF8wzmeWffDH4VbXz58sAx6Q=
github.com/go-openapi/jsonpointer v0.20.2/go.mod h1:bHen+N0u1KEO3YlmqOjTT9Adn1RfD91Ar825/PuiRVs=
github.com/go-openapi/jsonpointer v0.21.0 h1:YgdVicSA9vH5RiHs9TZW5oyafXZFc6+2Vc1rr/O9oNQ=
github.com/go-openapi/jsonpointer v0.21.0/go.mod h1:IUyH9l/+uyhIYQ/PXVA41Rexl+kOkAPDdXEYns6fzUY=
github.com/go-openapi/jsonreference v0.19.3/go.mod h1:rjx6GuL8TTa9VaixXglHmQmIL98+wF9xc8zWvFonSJ8=
github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
github.com/go-openapi/swag v0.22.8 h1:/9RjDSQ0vbFR+NyjGMkFTsA1IA0fmhKSThmfGZjicbw=
github.com/go-openapi/swag v0.22.8/go.mod h1:6QT22icPLEqAM/z/TChgb4WAveCHF92+2gF0CNjHpPI=
github.com/go-openapi/swag v0.23.0 h1:vsEVJDUo2hPJ2tu0/Xc+4noaxyEffXNIs3cOULZ+GrE=
github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM=
github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
Expand Down Expand Up @@ -227,8 +227,8 @@ github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:
github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/invopop/yaml v0.2.0 h1:7zky/qH+O0DwAyoobXUqvVBwgBFRxKoQ/3FjcVpjTMY=
github.com/invopop/yaml v0.2.0/go.mod h1:2XuRLgs/ouIrW3XNzuNj7J3Nvu/Dig5MXvbCEdiBN3Q=
github.com/invopop/yaml v0.3.1 h1:f0+ZpmhfBSS4MhG+4HYseMdJhoeeopbSKbq5Rpeelso=
github.com/invopop/yaml v0.3.1/go.mod h1:PMOp3nn4/12yEZUFfmOuNHJsZToEEOwoWsT+D81KkeA=
github.com/jcchavezs/porto v0.1.0/go.mod h1:fESH0gzDHiutHRdX2hv27ojnOVFco37hg1W6E9EZF4A=
github.com/jcchavezs/porto v0.6.0 h1:AgQLGwsXaxDkPj4Y+paFkVGLAR4n/1RRF0xV5UKinwg=
github.com/jcchavezs/porto v0.6.0/go.mod h1:fESH0gzDHiutHRdX2hv27ojnOVFco37hg1W6E9EZF4A=
Expand Down Expand Up @@ -778,7 +778,6 @@ gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
Expand Down
Loading

0 comments on commit fde925d

Please sign in to comment.