[kbn-ftr-common-functional-services] extend retry service #178660

dmlemeshko · 2024-03-13T16:49:19Z

Summary

I took the retry wrapper created by jpdjere in #173998 and extended retryForSuccess<T> with required capabilities to provide the same functionality.

This PR:

extends retry service with new function tryWithRetries<T> => : Promise<T> to retry block options.retryCount times within options.timeout period and return block result

  const response = await retry.tryWithRetries<SearchResponse>(
   'search request',
    async () => {
        const response = await supertest
          .post(`/internal/search/es`)
          .set(ELASTIC_HTTP_VERSION_HEADER, '1')
          .send({
            params: {
              body: {
                query: {
                  match_all: {},
                },
              },
            },
          })
          .expect(200);
        return response.body as SearchResponse;
    },
    {
      retryCount: 4,
      retryDelay: 100, // optional
      timeout: 30000, // optional
    }

removes utils/retry.ts wrapper and migrate tests to FTR Retry service
Adds descriptions to Retry service functions explaining the default use case

How the failures look like:

when reached timeout before retry count limit

 Error: retry.tryWithRetries reached timeout 11000 ms waiting for 'run console request'
 Error: expected '# Click the Variables button, above, to create your own variables.\nGET ${exampleVariable1} // _search\n{\n  "query": {\n    "${exampleVariable2}": {} // match_all\n  }\n}' to sort of equal '5'
     at Assertion.assert (expect.js:100:11)
     at Assertion.eql (expect.js:244:8)
     at _console.ts:47:41
     at processTicksAndRejections (node:internal/process/task_queues:95:5)
     at runAttempt (retry_with_retries.ts:29:15)
     at retryWithRetries (retry_with_retries.ts:97:21)
     at RetryService.tryForTime (retry.ts:38:12)
     at Context.<anonymous> (_console.ts:44:7)
     at Object.apply (wrap_function.js:73:16)

when reached retry count limit before timeout

 Error: retry.tryWithRetries reached the limit of attempts waiting for 'run console request': 2 out of 2
 Error: expected '# Click the Variables button, above, to create your own variables.\nGET ${exampleVariable1} // _search\n{\n  "query": {\n    "${exampleVariable2}": {} // match_all\n  }\n}' to sort of equal '5'
     at Assertion.assert (expect.js:100:11)
     at Assertion.eql (expect.js:244:8)
     at _console.ts:47:41
     at processTicksAndRejections (node:internal/process/task_queues:95:5)
     at runAttempt (retry_for_success.ts:29:15)
     at retryForSuccess (retry_for_success.ts:97:21)
     at RetryService.tryWithRetries (retry.ts:115:12)
     at Context.<anonymous> (_console.ts:44:7)
     at Object.apply (wrap_function.js:73:16)

…-retry

dmlemeshko · 2024-03-18T12:33:48Z

/ci

…ana into ftr-services/extend-retry

dmlemeshko · 2024-03-18T13:02:46Z

/ci

packages/kbn-ftr-common-functional-services/services/retry/retry_for_success.ts

packages/kbn-ftr-common-functional-services/services/retry/retry.ts

pheyos

LGTM
Also tried a few scenarios locally and everything worked fine ✔️

maryam-saeidi

Thanks for extending retryService functionality 🚀

maryam-saeidi · 2024-03-19T09:03:33Z

packages/kbn-ftr-common-functional-services/services/retry/retry_for_success.ts

+        onFailure(
+          lastError,
+          // optionally extend error message with description
+          `reached the limit of attempts${addText(description)}: ${


maryam-saeidi · 2024-03-19T09:05:30Z

...uites/detections_response/utils/rules/prebuilt_rules/install_prebuilt_rules_fleet_package.ts

-      log,
-    });
+      {
+        retryCount: MAX_RETRIES,


Looks much nicer by removing the log service and using the retryService function directly 💯

What would be the default values for these configs in case we don't provide it directly? Are they configurable at the test config level?

What would be the default values for these configs in case we don't provide it directly? Are they configurable at the test config level?

I believe hardcoding retryCount default value (e.g. to 2) will lead to misusage: similar to retry.waitForWithTimeout takes timeout value, tryWithRetries require options.retryCount for each call.

Few points:

I strongly believed that if retryCount is not defined explicitly, retry.try is the correct function to use.

Wrapping too many API calls with retry.tryWithRetries may hide instability of the product. By explicitly setting retryCount folks take more responsibility to evaluate individually case to case.

maximpn

@dmlemeshko thank you for extending retryForSuccess<T>() utility function 👍

The changes look good to me besides a couple of things I left comment about. In general I'm curious if you considered using p-retry for retrying functionality?

maximpn · 2024-03-19T08:49:36Z

packages/kbn-ftr-common-functional-services/services/retry/retry_for_success.ts


  while (true) {
+    if (retryCount) {
+      // Use retryCount as an optional condition
+      if (++attemptCounter > retryCount) {


nit: It's possible to get rid of nested ifs by merging the condition

if (retryCount && ++attemptCounter > retryCount) { // ... }

maximpn · 2024-03-19T08:55:23Z

packages/kbn-ftr-common-functional-services/services/retry/retry.ts

+  /**
+   * Use to retry block {options.retryCount} times within {options.timeout} period and return block result
+   * @param description block description
+   * @param block retrying operation


nit: If I got properly block here refers to a code block. I'd say it's quite a general term. Maybe rename it to something like action or retriableFn

maximpn · 2024-03-19T09:03:08Z

packages/kbn-ftr-common-functional-services/services/retry/retry_for_success.ts

    if (Date.now() - start > timeout) {
-      await onFailure(lastError);
+      onFailure(lastError, `reached timeout ${timeout} ms${addText(description)}`);
      throw new Error('expected onFailure() option to throw an error');
    } else if (lastError && criticalWebDriverErrors.includes(lastError.name)) {


nit: It could be refactored to omit else. Since previous if code block throws an error else is optional here.

maximpn · 2024-03-19T09:04:57Z

packages/kbn-ftr-common-functional-services/services/retry/retry_for_success.ts

+    if (retryCount) {
+      // Use retryCount as an optional condition
+      if (++attemptCounter > retryCount) {
+        onFailure(


What happens if onFailure doesn't break the while loop? The default implementation throws an error but what if a custom onFailure handler doesn't?

good catch! I will make a fix. Still you can't pass custom onFailure logic outside Retry service.

maximpn · 2024-03-19T09:06:31Z

packages/kbn-ftr-common-functional-services/services/retry/retry_for_success.ts

  retryDelay?: number;
+  retryCount?: number;
 }

 export async function retryForSuccess<T>(log: ToolingLog, options: Options<T>) {


Have you investigated if p-retry is applicable here? Additionally exponential back off timeout may work better.

We had a chat with @maryam-saeidi about p-retry and the issues she faced while using it. This PR is actually about to help migrate away from p-retry.
I think p-retry implementation is more complex vs FTR retry service, but since we never had stability issues with FTR retry (folks question logging & interface, understood :) ) I would keep logic as simple as possible.
As for logging and capabilities, I still believe we can collaborate with PRs like this one to achieve improvements.

@dmlemeshko Could you share found issues with p-retry?

My observation is p-retry feels like a quite popular library with 14M weekly downloads. A lot of bugs should've been fixed already. On top of that p-retry provides flexible configuration like exponential back off retry timeout while it still can be configured linearly. It allows to abort retries as well. Checking Kibana's codebase it's not hard to see it's used in multiple packages and plugins.

Having a comment for retryForSuccess with some explanation and improvements over p-retry will be really helpful for future maintenance.

@maximpn Here is the ticket that I explained what challenge we faced, and in our case, we didn't have any log about the retry attempts which made it hard for us to understand if it was one request that timeout or there were multiple requests and due to exponential back off, the overall attempt failed. (The way that we tested it locally was by throwing an error in the test function and setting the retry to 10 times)

In general, I think it would be better to only rely on one library/package for retry purposes, for easier maintenance and a better understanding of the tool.

@maryam-saeidi thanks for the details 👍

we didn't have any log about the retry attempts
Yes, it sounds like a bug. Or exponential timeout was too high so the test was interrupted due to a timeout. Btw, you could configure p-retry to have a constant retry interval.

jpdjere

Thanks @dmlemeshko for this work! 🚀

Glad I could contribute to improve the FTR testing experience

kibana-ci · 2024-03-19T13:45:28Z

💚 Build Succeeded

Buildkite Build
Commit: 2209cdf

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/ftr-common-functional-services`	31	21	-10

Unknown metric groups

API count

id	before	after	diff
`@kbn/ftr-common-functional-services`	31	36	+5

History

💛 Build #198163 was flaky 0d1093d
💔 Build #198062 failed 86e161f
💚 Build #197981 succeeded b0af5d3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…8660) ## Summary I took the retry wrapper created by [jpdjere](https://github.com/jpdjere) in elastic#173998 and extended `retryForSuccess<T>` with required capabilities to provide the same functionality. This PR: 1) extends retry service with new function `tryWithRetries<T> => : Promise<T>` to retry block `options.retryCount` times within `options.timeout` period and return block result ```ts const response = await retry.tryWithRetries<SearchResponse>( 'search request', async () => { const response = await supertest .post(`/internal/search/es`) .set(ELASTIC_HTTP_VERSION_HEADER, '1') .send({ params: { body: { query: { match_all: {}, }, }, }, }) .expect(200); return response.body as SearchResponse; }, { retryCount: 4, retryDelay: 100, // optional timeout: 30000, // optional } ``` 2) removes `utils/retry.ts` wrapper and migrate tests to FTR Retry service 3) Adds descriptions to Retry service functions explaining the default use case How the failures look like: - when reached timeout before retry count limit ``` Error: retry.tryWithRetries reached timeout 11000 ms waiting for 'run console request' Error: expected '# Click the Variables button, above, to create your own variables.\nGET ${exampleVariable1} // _search\n{\n "query": {\n "${exampleVariable2}": {} // match_all\n }\n}' to sort of equal '5' at Assertion.assert (expect.js:100:11) at Assertion.eql (expect.js:244:8) at _console.ts:47:41 at processTicksAndRejections (node:internal/process/task_queues:95:5) at runAttempt (retry_with_retries.ts:29:15) at retryWithRetries (retry_with_retries.ts:97:21) at RetryService.tryForTime (retry.ts:38:12) at Context.<anonymous> (_console.ts:44:7) at Object.apply (wrap_function.js:73:16) ``` - when reached retry count limit before timeout ``` Error: retry.tryWithRetries reached the limit of attempts waiting for 'run console request': 2 out of 2 Error: expected '# Click the Variables button, above, to create your own variables.\nGET ${exampleVariable1} // _search\n{\n "query": {\n "${exampleVariable2}": {} // match_all\n }\n}' to sort of equal '5' at Assertion.assert (expect.js:100:11) at Assertion.eql (expect.js:244:8) at _console.ts:47:41 at processTicksAndRejections (node:internal/process/task_queues:95:5) at runAttempt (retry_for_success.ts:29:15) at retryForSuccess (retry_for_success.ts:97:21) at RetryService.tryWithRetries (retry.ts:115:12) at Context.<anonymous> (_console.ts:44:7) at Object.apply (wrap_function.js:73:16) ``` (cherry picked from commit 277b3fb)

kibanamachine · 2024-03-19T15:37:30Z

💚 All backports created successfully

Status	Branch	Result
✅	8.13

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…8660) (#178978) # Backport This will backport the following commits from `main` to `8.13`: - [[kbn-ftr-common-functional-services] extend retry service (#178660)](#178660)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Dzmitry Lemechko <[email protected]>

## Summary Resolves: #192201 - Expose `TryWithRetriesOptions` - Tune timeouts to pass ci - Add attempt count debug info to `retry/retry_for_success.ts` - Helps with tuning timeout params - Move exposure of `AlertingApiProvider` from `x-pack/test_serverless/api_integration/services/index.ts` -> `x-pack/test_serverless/shared/services/deployment_agnostic_services.ts` - This exposes the alerting api under Deployment Agnostic Services (DA), and DA is exposed within `x-pack/test_serverless/functional/services/index.ts` (Shared Services [Serverless]) - Collapse helper script functions into just another object literal stanza within `AlertingApiProvider` - Update all references - Refactor alerting api to use `retry` service, instead of p-retry (following [this pr](#178660)) ### Additional debug logging Run in debug mode (add `-v`): ``` node scripts/functional_tests \ --config x-pack/test_serverless/api_integration/test_suites/search/common_configs/config.group1.ts \ --grep "Summary actions" -v ``` #### After ``` │ sill retry.tryWithRetries('Alerting API - waitForDocumentInIndex, retryOptions: {"retryCount":5,"retryDelay":200}', [object AsyncFunction], [object Object]) │ debg --- retry.tryWithRetries error: index_not_found_exception │ Root causes: │ index_not_found_exception: no such index [alert-action-es-query] - Attempt #: 1 │ sill es.search([object Object]) │ debg --- retry.tryWithRetries failed again with the same message... - Attempt #: 2 │ sill es.search([object Object]) │ debg --- retry.tryWithRetries failed again with the same message... - Attempt #: 3 │ sill es.search([object Object]) │ debg --- retry.tryWithRetries failed again with the same message... - Attempt #: 4 │ sill es.search([object Object]) │ debg --- retry.tryWithRetries failed again with the same message... - Attempt #: 5 ... // Msg after all attempts fail: │ Error: retry.tryWithRetries reached the limit of attempts waiting for 'Alerting API - waitForDocumentInIndex, retryOptions: {"retryCount":5,"retryDelay":200}': 5 out of 5 │ ResponseError: index_not_found_exception │ Root causes: │ index_not_found_exception: no such index [alert-action-es-query] │ at SniffingTransport._request (node_modules/@elastic/transport/src/Transport.ts:601:17) │ at processTicksAndRejections (node:internal/process/task_queues:95:5) │ at /Users/trezworkbox/dev/main.worktrees/cleanup-alerting-api/node_modules/@elastic/transport/src/Transport.ts:704:22 │ at SniffingTransport.request (node_modules/@elastic/transport/src/Transport.ts:701:14) │ at Proxy.SearchApi (node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10) │ at alerting_api.ts:123:28 │ at runAttempt (retry_for_success.ts:30:15) │ at retryForSuccess (retry_for_success.ts:99:21) │ at Proxy.tryWithRetries (retry.ts:113:12) │ at Object.waitForDocumentInIndex (alerting_api.ts:120:14) │ at Context.<anonymous> (summary_actions.ts:146:20) │ at Object.apply (wrap_function.js:74:16) │ at Object.apply (wrap_function.js:74:16) │ at onFailure (retry_for_success.ts:18:9) │ at retryForSuccess (retry_for_success.ts:75:7) │ at Proxy.tryWithRetries (retry.ts:113:12) │ at Object.waitForDocumentInIndex (alerting_api.ts:120:14) │ at Context.<anonymous> (summary_actions.ts:146:20) │ at Object.apply (wrap_function.js:74:16) │ at Object.apply (wrap_function.js:74:16) ``` ### Notes Was put back in draft to additional scope detailed in issue linked above. --------- Co-authored-by: Elastic Machine <[email protected]>

## Summary Resolves: elastic#192201 - Expose `TryWithRetriesOptions` - Tune timeouts to pass ci - Add attempt count debug info to `retry/retry_for_success.ts` - Helps with tuning timeout params - Move exposure of `AlertingApiProvider` from `x-pack/test_serverless/api_integration/services/index.ts` -> `x-pack/test_serverless/shared/services/deployment_agnostic_services.ts` - This exposes the alerting api under Deployment Agnostic Services (DA), and DA is exposed within `x-pack/test_serverless/functional/services/index.ts` (Shared Services [Serverless]) - Collapse helper script functions into just another object literal stanza within `AlertingApiProvider` - Update all references - Refactor alerting api to use `retry` service, instead of p-retry (following [this pr](elastic#178660)) ### Additional debug logging Run in debug mode (add `-v`): ``` node scripts/functional_tests \ --config x-pack/test_serverless/api_integration/test_suites/search/common_configs/config.group1.ts \ --grep "Summary actions" -v ``` #### After ``` │ sill retry.tryWithRetries('Alerting API - waitForDocumentInIndex, retryOptions: {"retryCount":5,"retryDelay":200}', [object AsyncFunction], [object Object]) │ debg --- retry.tryWithRetries error: index_not_found_exception │ Root causes: │ index_not_found_exception: no such index [alert-action-es-query] - Attempt #: 1 │ sill es.search([object Object]) │ debg --- retry.tryWithRetries failed again with the same message... - Attempt #: 2 │ sill es.search([object Object]) │ debg --- retry.tryWithRetries failed again with the same message... - Attempt #: 3 │ sill es.search([object Object]) │ debg --- retry.tryWithRetries failed again with the same message... - Attempt #: 4 │ sill es.search([object Object]) │ debg --- retry.tryWithRetries failed again with the same message... - Attempt #: 5 ... // Msg after all attempts fail: │ Error: retry.tryWithRetries reached the limit of attempts waiting for 'Alerting API - waitForDocumentInIndex, retryOptions: {"retryCount":5,"retryDelay":200}': 5 out of 5 │ ResponseError: index_not_found_exception │ Root causes: │ index_not_found_exception: no such index [alert-action-es-query] │ at SniffingTransport._request (node_modules/@elastic/transport/src/Transport.ts:601:17) │ at processTicksAndRejections (node:internal/process/task_queues:95:5) │ at /Users/trezworkbox/dev/main.worktrees/cleanup-alerting-api/node_modules/@elastic/transport/src/Transport.ts:704:22 │ at SniffingTransport.request (node_modules/@elastic/transport/src/Transport.ts:701:14) │ at Proxy.SearchApi (node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10) │ at alerting_api.ts:123:28 │ at runAttempt (retry_for_success.ts:30:15) │ at retryForSuccess (retry_for_success.ts:99:21) │ at Proxy.tryWithRetries (retry.ts:113:12) │ at Object.waitForDocumentInIndex (alerting_api.ts:120:14) │ at Context.<anonymous> (summary_actions.ts:146:20) │ at Object.apply (wrap_function.js:74:16) │ at Object.apply (wrap_function.js:74:16) │ at onFailure (retry_for_success.ts:18:9) │ at retryForSuccess (retry_for_success.ts:75:7) │ at Proxy.tryWithRetries (retry.ts:113:12) │ at Object.waitForDocumentInIndex (alerting_api.ts:120:14) │ at Context.<anonymous> (summary_actions.ts:146:20) │ at Object.apply (wrap_function.js:74:16) │ at Object.apply (wrap_function.js:74:16) ``` ### Notes Was put back in draft to additional scope detailed in issue linked above. --------- Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 9d22e8c)

[kbn-ftr-common-functional-services] extend retry service

006ce91

dmlemeshko mentioned this pull request Mar 13, 2024

Adding a more flexible retry function to the retryService #178535

Closed

dmlemeshko added 3 commits March 18, 2024 10:04

Merge remote-tracking branch 'upstream/main' into ftr-services/extend…

60c888c

…-retry

port utils/retry to retry package

ca96ead

update retry/retry_for_success with retryCount logic

72fa313

dmlemeshko added release_note:skip Skip the PR/issue when compiling release notes FTR v8.14.0 v8.13.1 labels Mar 18, 2024

Merge branch 'main' into ftr-services/extend-retry

c23922f

dmlemeshko marked this pull request as ready for review March 18, 2024 12:33

dmlemeshko requested review from a team as code owners March 18, 2024 12:33

dmlemeshko requested a review from nikitaindik March 18, 2024 12:33

dmlemeshko added 3 commits March 18, 2024 14:01

update export

4f5ea19

make retryCount required option

7b5aa71

Merge branch 'ftr-services/extend-retry' of github.com:dmlemeshko/kib…

b0af5d3

…ana into ftr-services/extend-retry

dmlemeshko requested a review from maryam-saeidi March 18, 2024 15:04

jbudz approved these changes Mar 18, 2024

View reviewed changes

pheyos reviewed Mar 18, 2024

View reviewed changes

packages/kbn-ftr-common-functional-services/services/retry/retry_for_success.ts Outdated Show resolved Hide resolved

dmlemeshko added 2 commits March 18, 2024 17:39

fix typo

2c3cd18

Fix methodName in few calls

86e161f

pheyos reviewed Mar 18, 2024

View reviewed changes

packages/kbn-ftr-common-functional-services/services/retry/retry.ts Outdated Show resolved Hide resolved

pheyos approved these changes Mar 18, 2024

View reviewed changes

Merge branch 'main' into ftr-services/extend-retry

0d1093d

maryam-saeidi approved these changes Mar 19, 2024

View reviewed changes

maximpn reviewed Mar 19, 2024

View reviewed changes

maximpn removed the request for review from nikitaindik March 19, 2024 09:12

jpdjere approved these changes Mar 19, 2024

View reviewed changes

dmlemeshko added 2 commits March 19, 2024 13:10

simplify the logic

749c55d

update docs

2209cdf

dmlemeshko merged commit 277b3fb into elastic:main Mar 19, 2024
17 checks passed

kibanamachine mentioned this pull request Mar 19, 2024

[8.13] [kbn-ftr-common-functional-services] extend retry service (#178660) #178978

Merged

kibanamachine added the v8.13.0 label Mar 19, 2024

maryam-saeidi mentioned this pull request Mar 20, 2024

[Custom threshold] Improve api integration tests by using retry service and alertingApi #179095

Closed

This was referenced Sep 11, 2024

[FTR] Collapse Alerting API Helpers #192201

Closed

[FTR] Collapse Alerting API Helpers Impl #192216

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[kbn-ftr-common-functional-services] extend retry service #178660

[kbn-ftr-common-functional-services] extend retry service #178660

dmlemeshko commented Mar 13, 2024 •

edited by kibanamachine

Loading

dmlemeshko commented Mar 18, 2024

dmlemeshko commented Mar 18, 2024

pheyos left a comment

maryam-saeidi left a comment

maryam-saeidi Mar 19, 2024

maryam-saeidi Mar 19, 2024

dmlemeshko Mar 19, 2024 •

edited

Loading

maximpn left a comment

maximpn Mar 19, 2024

maximpn Mar 19, 2024

maximpn Mar 19, 2024

maximpn Mar 19, 2024

dmlemeshko Mar 19, 2024

maximpn Mar 19, 2024

dmlemeshko Mar 19, 2024 •

edited

Loading

maximpn Mar 21, 2024

maryam-saeidi Mar 21, 2024

maximpn Mar 25, 2024

jpdjere left a comment

kibana-ci commented Mar 19, 2024

API count

kibanamachine commented Mar 19, 2024

[kbn-ftr-common-functional-services] extend retry service #178660

[kbn-ftr-common-functional-services] extend retry service #178660

Conversation

dmlemeshko commented Mar 13, 2024 • edited by kibanamachine Loading

Summary

dmlemeshko commented Mar 18, 2024

dmlemeshko commented Mar 18, 2024

pheyos left a comment

Choose a reason for hiding this comment

maryam-saeidi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmlemeshko Mar 19, 2024 • edited Loading

Choose a reason for hiding this comment

maximpn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmlemeshko Mar 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpdjere left a comment

Choose a reason for hiding this comment

kibana-ci commented Mar 19, 2024

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

API count

History

kibanamachine commented Mar 19, 2024

💚 All backports created successfully

Questions ?

dmlemeshko commented Mar 13, 2024 •

edited by kibanamachine

Loading

dmlemeshko Mar 19, 2024 •

edited

Loading

dmlemeshko Mar 19, 2024 •

edited

Loading