From 070e57d8739659675f046916f92dd7937d0c6c0e Mon Sep 17 00:00:00 2001 From: zhou-yinyuan Date: Wed, 14 Aug 2024 15:07:54 +0800 Subject: [PATCH 1/2] Solution of e2e timed issue [frontend][docs]: solution of e2e timed issue --- .../solution-of-e2e-timed-issue.mdx | 102 ++++++++++++++++++ .../e2e/fixtures/create-new/report-result.ts | 2 +- .../major-path/create-a-new-project.spec.ts | 6 +- .../import-project-from-file.spec.ts | 4 +- 4 files changed, 108 insertions(+), 6 deletions(-) create mode 100644 docs/src/content/docs/en/issue-solutions/solution-of-e2e-timed-issue.mdx diff --git a/docs/src/content/docs/en/issue-solutions/solution-of-e2e-timed-issue.mdx b/docs/src/content/docs/en/issue-solutions/solution-of-e2e-timed-issue.mdx new file mode 100644 index 0000000000..f019f3f63b --- /dev/null +++ b/docs/src/content/docs/en/issue-solutions/solution-of-e2e-timed-issue.mdx @@ -0,0 +1,102 @@ +--- +title: Solution of e2e timed issue +description: Solution of e2e timed issue +--- + +**Issue** + +When running the e2e tests, all e2e tests failed because we can't get build data from BuildKite. + +**Reason** + +![BuildKite Expired](https://cdn.jsdelivr.net/gh/au-heartbeat/data-hosting@main/issue-solution/buildkite-expired.png) + +- for `free user` on BuildKite, BuildKite will save pipeline build data only for 90 days. So after 90 days of the time selected in e2e test, we will not get build data from BuildKite. + +**Solution** + +- Modify e2e test regularly + +**Estimated the time of next modification** + +- the time we select mainly is from `2024-06-03` to `2024-06-07` in the e2e test, so according to my personal estimate that next modified time should be around `2024-09-01`. +- From `2024-09-01` to `2024-09-05`, the steps are obtained but the result will be changed, because the e2e test can get only partial build data. +- After `2024-09-05`, the e2e test can't get any build data from the `2024-06-03` to `2024-06-07` time range. + +> Tips: The estimated time may not be completely accurate + +**Summary steps** + +> Tips: The following steps are based on our experience and are not required to be followed. If there is any more excellent experience, you can modify the following steps. +> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StepContentComments
Step 1. Configure the latest time period in the local browser and get the report data1. Change the time period to a time period that can get BuildKite build dataIt's recommended to change to a time period close to the current date, because this can reduce the number of e2e test modifications. And select multiple time periods, because only selecting one time range can't go to chart page
2. According to the time period, input all tokensSelect all the metrics and input all tokens. If any metrics are not required in the e2e test, we can modify the configuration file
3. Input all information that are required, for example, classification, crews and cycle time. Next go to report pageIt's recommended to select all
4. Export the config file and all the csv fileIn the e2e test, we can directly use the config file to e2e test, and the local csv file can also use these csv files
5. Save the report data to local file, such as imagesWhen you use the config file to e2e test, you will change the report data to assert the report data. And you can directly get these data in the local file without getting the data by executing e2e test multiple times
Step 2. Modify the e2e test1. For the e2e test about create a new project, we will modify the time periods and all the information by config file and modify the report result data from the Step 1None
2. For the e2e test about import from config file, change the config file name to the e2e test config file name, and replace origin input-files/*.template.json fileIf we directly use the origin file name, we will not need to modify the code. And we only upload input-files/*.template.json file to repo rather than input-files/*.json, so we must modify the *.template.json file
3. In the *.template.json file, for data security, we must replace token with placeholder of <E2E_TOKEN_JIRA> , <E2E_TOKEN_BUILD_KITE>，<E2E_TOKEN_GITHUB> like the previous fileWe can copy the previous placeholder to use the new placeholder, And when we run pnpm e2e:local, the generate-config-files.sh file will replace the placeholder with the real token. Tips: there is an e2e test to test the pipeline that is no organization in the unhappy path, and the placeholder is <E2E_TOKEN_PIPELINE_NO_ORG_CONFIG_BUILDKITE>, maybe you need to check if the token is correct
4. Modify local token file. We will modify the origin token to the new token in the .env.local file to ensure that the e2e test can be passed locallyIn the local, the placeholder from the *.template.json file will be replaced with the token in the .env.local file. And the generate-config-files.sh file will create a new json file with the same name by *.template.json file.
5. Run the e2e test, and change the report data and local csv files by the Step 1If the previous steps are correct, the e2e test will go to the report page as in the browser. So you can input the data by Step 1
6. If the download csv files are not equal to local csv files, change the local csv files to download filesWhen these files are not equal, you can print the download filename, and replace the local csv file with download file saved to temp directory. For board csv file, the current jira cards will be added after the blank line in the file, but at different times, these cards are not same, so when we parse the local csv file and download csv file, we set the number of the lines that need to be parsed, so the value that is usually equal to 1 + the number of non-done cards in the BOARD_CSV_COMPARED_LINES of create-new/report-result.ts file may be modified
7. Repeat Step 1 and Step 2, until all e2e tests are passed in the localTips: For special tests, we need to prepare special time periods. For example, in the unhappy path when import file, we will test the function that board configuration don't exist in the metrics page, so the board data should not exist in the time periods that we select, and the time periods usually are holiday, such as from 2024-05-01 to 2024-05-05
8. Close the playwright and rerun the cmd of pnpm e2e:local to re-test the e2e testThe reason is to ensure that all e2e tests are passed in the local env. Becausethe changes of *.template.json will not be hot replacement, however, when we run the pnpm e2e:local command, the json file will be generated by generate-config-files.sh file
Step 3. Modify the remote env to ensure that all e2e tests are passed in the remote env1. Change the Github token to the correct tokenNot sure if permissions are needed
2. Change the AWS token to the correct tokenlogin to AWS by HeartBeatKeyPair.pem in the google drive, and by the cmd of vim /etc/buildkite-agent/hooks/environment to change the env and modify the token to the correct token. Permissions are needed
diff --git a/frontend/e2e/fixtures/create-new/report-result.ts b/frontend/e2e/fixtures/create-new/report-result.ts index 0eb650891b..94740c4dfd 100644 --- a/frontend/e2e/fixtures/create-new/report-result.ts +++ b/frontend/e2e/fixtures/create-new/report-result.ts @@ -740,7 +740,7 @@ export const BOARD_METRICS_REWORK_MULTIPLE_RANGES: IBoardMetricsDetailItem[][] = ], ]; -export const BAORD_CSV_COMPARED_LINES: ICsvComparedLines = { +export const BOARD_CSV_COMPARED_LINES: ICsvComparedLines = { 'board-20240603-20240604': 4, 'board-20240605-20240606': 2, 'board-20240607-20240607': 2, diff --git a/frontend/e2e/specs/major-path/create-a-new-project.spec.ts b/frontend/e2e/specs/major-path/create-a-new-project.spec.ts index 7dcdc0c078..47790171e4 100644 --- a/frontend/e2e/specs/major-path/create-a-new-project.spec.ts +++ b/frontend/e2e/specs/major-path/create-a-new-project.spec.ts @@ -14,7 +14,7 @@ import { } from '../../fixtures/create-new/metrics-step'; import { configWithoutBlockColumn as configWithoutBlockColumnData } from '../../fixtures/create-new/config-step'; import { cycleTimeByStatusFixture } from '../../fixtures/cycle-time-by-status/cycle-time-by-status-fixture'; -import { BAORD_CSV_COMPARED_LINES } from '../../fixtures/create-new/report-result'; +import { BOARD_CSV_COMPARED_LINES } from '../../fixtures/create-new/report-result'; import { config as metricsStepData } from '../../fixtures/create-new/metrics-step'; import { config as configStepData } from '../../fixtures/create-new/config-step'; import { ProjectCreationType } from 'e2e/pages/metrics/report-step'; @@ -91,7 +91,7 @@ test('Create a new project', async ({ homePage, configStep, metricsStep, reportS cycleTimeData: BOARD_METRICS_CYCLE_TIME_MULTIPLE_RANGES, classificationData: BOARD_METRICS_CLASSIFICATION_MULTIPLE_RANGES, reworkData: BOARD_METRICS_REWORK_MULTIPLE_RANGES, - csvCompareLines: BAORD_CSV_COMPARED_LINES, + csvCompareLines: BOARD_CSV_COMPARED_LINES, }); await reportStep.checkDoraMetricsForMultipleRanges(DORA_METRICS_RESULT_MULTIPLE_RANGES); await reportStep.checkDoraMetricsDetailsForMultipleRanges({ @@ -198,7 +198,7 @@ test('Create a new project with design and waiting for deployment in the cycle t cycleTimeData: BOARD_METRICS_WITH_DESIGN_AND_WAITING_FOR_DEPLOYMENT_CYCLE_TIME, classificationData: BOARD_METRICS_CLASSIFICATION_MULTIPLE_RANGES, reworkData: BOARD_METRICS_REWORK_MULTIPLE_RANGES, - csvCompareLines: BAORD_CSV_COMPARED_LINES, + csvCompareLines: BOARD_CSV_COMPARED_LINES, fileNamePrefix, }); await reportStep.checkDoraMetricsForMultipleRanges(DORA_METRICS_RESULT_MULTIPLE_RANGES); diff --git a/frontend/e2e/specs/major-path/import-project-from-file.spec.ts b/frontend/e2e/specs/major-path/import-project-from-file.spec.ts index 4323c98fb4..70ccbb2b1d 100644 --- a/frontend/e2e/specs/major-path/import-project-from-file.spec.ts +++ b/frontend/e2e/specs/major-path/import-project-from-file.spec.ts @@ -7,7 +7,7 @@ import { BOARD_METRICS_CYCLE_TIME_MULTIPLE_RANGES, BOARD_METRICS_CLASSIFICATION_MULTIPLE_RANGES, BOARD_METRICS_REWORK_MULTIPLE_RANGES, - BAORD_CSV_COMPARED_LINES, + BOARD_CSV_COMPARED_LINES, DORA_METRICS_RESULT_MULTIPLE_RANGES, CYCLE_TIME_WITH_ANALYSIS_STATUS_PROJECT_BOARD_METRICS_RESULT, } from '../../fixtures/create-new/report-result'; @@ -76,7 +76,7 @@ test('Import project from file with all ranges API succeed', async ({ cycleTimeData: BOARD_METRICS_CYCLE_TIME_MULTIPLE_RANGES, classificationData: BOARD_METRICS_CLASSIFICATION_MULTIPLE_RANGES, reworkData: BOARD_METRICS_REWORK_MULTIPLE_RANGES, - csvCompareLines: BAORD_CSV_COMPARED_LINES, + csvCompareLines: BOARD_CSV_COMPARED_LINES, }); await reportStep.checkDoraMetricsDetailsForMultipleRanges({ doraMetricsReportData: DORA_METRICS_RESULT_MULTIPLE_RANGES, From 4f78a698c2c3a36b2957c3215de33ca0d47e27ae Mon Sep 17 00:00:00 2001 From: zhou-yinyuan Date: Wed, 14 Aug 2024 17:20:38 +0800 Subject: [PATCH 2/2] Solution of e2e timed issue [docs]: fix issue --- .../en/issue-solutions/solution-of-e2e-timed-issue.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/src/content/docs/en/issue-solutions/solution-of-e2e-timed-issue.mdx b/docs/src/content/docs/en/issue-solutions/solution-of-e2e-timed-issue.mdx index f019f3f63b..f54998fe75 100644 --- a/docs/src/content/docs/en/issue-solutions/solution-of-e2e-timed-issue.mdx +++ b/docs/src/content/docs/en/issue-solutions/solution-of-e2e-timed-issue.mdx @@ -62,16 +62,16 @@ When running the e2e tests, all e2e tests failed because we can't get build data None - 2. For the e2e test about import from config file, change the config file name to the e2e test config file name, and replace origin input-files/*.template.json file - If we directly use the origin file name, we will not need to modify the code. And we only upload input-files/*.template.json file to repo rather than input-files/*.json, so we must modify the *.template.json file + 2. For the e2e test about import from config file, change the config file name to the e2e test config file name, and replace origin input-files/*.template.json file + If we directly use the origin file name, we will not need to modify the code. And we only upload input-files/*.template.json file to repo rather than input-files/*.json, so we must modify the *.template.json file - 3. In the *.template.json file, for data security, we must replace token with placeholder of <E2E_TOKEN_JIRA> , <E2E_TOKEN_BUILD_KITE>，<E2E_TOKEN_GITHUB> like the previous file + 3. In the *.template.json file, for data security, we must replace token with placeholder of <E2E_TOKEN_JIRA> , <E2E_TOKEN_BUILD_KITE>，<E2E_TOKEN_GITHUB> like the previous file We can copy the previous placeholder to use the new placeholder, And when we run pnpm e2e:local, the generate-config-files.sh file will replace the placeholder with the real token. Tips: there is an e2e test to test the pipeline that is no organization in the unhappy path, and the placeholder is <E2E_TOKEN_PIPELINE_NO_ORG_CONFIG_BUILDKITE>, maybe you need to check if the token is correct 4. Modify local token file. We will modify the origin token to the new token in the .env.local file to ensure that the e2e test can be passed locally - In the local, the placeholder from the *.template.json file will be replaced with the token in the .env.local file. And the generate-config-files.sh file will create a new json file with the same name by *.template.json file. + In the local, the placeholder from the *.template.json file will be replaced with the token in the .env.local file. And the generate-config-files.sh file will create a new json file with the same name by *.template.json file. 5. Run the e2e test, and change the report data and local csv files by the Step 1 @@ -87,7 +87,7 @@ When running the e2e tests, all e2e tests failed because we can't get build data 8. Close the playwright and rerun the cmd of pnpm e2e:local to re-test the e2e test - The reason is to ensure that all e2e tests are passed in the local env. Becausethe changes of *.template.json will not be hot replacement, however, when we run the pnpm e2e:local command, the json file will be generated by generate-config-files.sh file + The reason is to ensure that all e2e tests are passed in the local env. Becausethe changes of *.template.json will not be hot replacement, however, when we run the pnpm e2e:local command, the json file will be generated by generate-config-files.sh file Step 3. Modify the remote env to ensure that all e2e tests are passed in the remote env