[APM] e2e are failing when running them locally with some `snapshot` #65422

v1v · 2020-05-06T07:41:05Z

Kibana version:
master branch

commit a108b23c27635ee90382f86de2cc8206bdb18d8d (grafted, HEAD -> master, origin/master, origin/HEAD)
Author: Ryan Keairns <[email protected]>
Date:   Wed May 6 01:39:29 2020 -0500

Elasticsearch version:
master branch

{
name: "122804b8ce33",
cluster_name: "docker-cluster",
cluster_uuid: "v-rouCyHShWZI5Ykl-t4qw",
version: {
number: "8.0.0-SNAPSHOT",
build_flavor: "default",
build_type: "docker",
build_hash: "84116c5132e7ea5a1e63b6a4c4736ea0a7111e4f",
build_date: "2020-05-06T03:08:57.223715Z",
build_snapshot: true,
lucene_version: "8.5.1",
minimum_wire_compatibility_version: "7.8.0",
minimum_index_compatibility_version: "7.0.0"
},
tagline: "You Know, for Search"
}

Server OS version:
macosx 10.15.4

Browser version:
Cypress e2e UI

Browser OS version:
N/A

Original install method (e.g. download page, yum, from source, etc.):
From source
node --version - v10.19.0
nvm --version - 0.34.0

Description of the problem including expected versus actual behavior:

e2e tests are failing with

Steps to reproduce:

Run all the below steps from scratch:

cd /tmp
git clone https://github.com/elastic/kibana --depth 1
docker ps -a
cd kibana
nvm use $(cat .node-version)
cd x-pack/plugins/apm/e2e
yarn install
cd -
yarn kbn clean && yarn kbn bootstrap
nohup node ./scripts/kibana --no-base-path --dev --no-dev-config --config x-pack/plugins/apm/e2e/ci/kibana.e2e.yml > /tmp/kibana.log 2>&1 &
x-pack/plugins/apm/e2e/run-e2e.sh

Since there is an issue when running from scratch, see #61942
I recommend to change the x-pack/plugins/apm/e2e/run-e2e.sh with something like the below diff:

diff --git a/x-pack/plugins/apm/e2e/run-e2e.sh b/x-pack/plugins/apm/e2e/run-e2e.sh
index 818d45ab..aa1aecd8 100755
--- a/x-pack/plugins/apm/e2e/run-e2e.sh
+++ b/x-pack/plugins/apm/e2e/run-e2e.sh
@@ -111,15 +111,26 @@ fi
 ##################################################
 echo "\n${bold}Waiting for Kibana to start...${normal}"
 echo "Note: you need to start Kibana manually. Find the instructions at the top."
-yarn wait-on -i 500 -w 500 http://localhost:$KIBANA_PORT > /dev/null
+yarn wait-on -i 500 -w 500 http://localhost:$KIBANA_PORT/status > /dev/null
 
 echo "\n✅ Setup completed successfully. Running tests...\n"
 
 #
-# run cypress tests
+# run cypress tests a few times
 ##################################################
-yarn cypress run --config pageLoadTimeout=100000,watchForFileChanges=true
-
+n=0
+until [ $n -ge 20 ]
+  do
+    ## for debugging purposes only
+    curl --silent --user admin:changeme \
+          "localhost:${KIBANA_PORT}/api/status" | \
+            jq .status.overall.state | \
+            grep 'green'
+    yarn cypress run --config pageLoadTimeout=100000,watchForFileChanges=true && break || true
+    sleep 10
+    n=$[$n+1]
+  done
+fi

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="0" tests="1" failures="1">
  <testsuite name="Root Suite" timestamp="2020-05-06T07:39:14" tests="0" file="cypress/integration/apm.feature" failures="0" time="0">
  </testsuite>
  <testsuite name="APM" timestamp="2020-05-06T07:39:14" tests="1" failures="1" time="0">
    <testcase name="APM Transaction duration charts" time="0" classname="Transaction duration charts">
      <failure message="Snapshot difference. To update, delete snapshot and rerun test.
&quot;500 ms&quot; =&gt; &quot;350 ms&quot;" type="Error"><![CDATA[Error: Snapshot difference. To update, delete snapshot and rerun test.
"500 ms" => "350 ms"
    at http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:22407:15
    at InternalConstructor.orElse (http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:2433:14)
    at cyRaiser (http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:22396:14)
    at setOrCheckValue (http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:22583:7)
    at snapShotCore (http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:22595:14)
    at setSnapshot (http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:22412:5)
    at Context.snapshot (http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:22431:5)]]></failure>
    </testcase>
  </testsuite>
</testsuites>

Describe the feature:

The text was updated successfully, but these errors were encountered:

elasticmachine · 2020-05-06T07:41:18Z

Pinging @elastic/apm-ui (Team:apm)

sorenlouv · 2020-05-06T10:21:04Z

Try deleting the snapshots.js file. I think something could have happened to that during the NP migration.

v1v · 2020-05-08T14:07:33Z

Do you mean x-pack/plugins/apm/e2e/cypress/integration/snapshots.js ? If so I did delete it and the error is now something else:

What did I do?

cd /tmp
git clone https://github.com/elastic/kibana --depth 1
cd kibana

## Apply patch
cat <<EOT > tweak.patch
diff --git a/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js b/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
deleted file mode 100644
index a462f4a5..00000000
--- a/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
+++ /dev/null
@@ -1,16 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License;
- * you may not use this file except in compliance with the Elastic License.
- */
-
-module.exports = {
-  APM: {
-    'Transaction duration charts': {
-      '1': '500 ms',
-      '2': '250 ms',
-      '3': '0 ms'
-    }
-  },
-  __version: '4.2.0'
-};
diff --git a/x-pack/plugins/apm/e2e/run-e2e.sh b/x-pack/plugins/apm/e2e/run-e2e.sh
index 818d45ab..7aad1412 100755
--- a/x-pack/plugins/apm/e2e/run-e2e.sh
+++ b/x-pack/plugins/apm/e2e/run-e2e.sh
@@ -118,7 +118,12 @@ echo "\n✅ Setup completed successfully. Running tests...\n"
 #
 # run cypress tests
 ##################################################
-yarn cypress run --config pageLoadTimeout=100000,watchForFileChanges=true
+n=0
+until [  -ge 20 ] ; do
+    yarn cypress run --config pageLoadTimeout=100000,watchForFileChanges=true && break || true
+    sleep 10
+    n=\$[\$n+1]
+done
 
 #
 # Run interactively
EOT

git apply  --stat tweak.patch 
git apply tweak.patch 
nvm use $(cat .node-version)
cd x-pack/plugins/apm/e2e
yarn install
cd -
yarn kbn clean && yarn kbn bootstrap
nohup node ./scripts/kibana --no-base-path --dev --no-dev-config --config x-pack/plugins/apm/e2e/ci/kibana.e2e.yml > /tmp/kibana.log 2>&1 &
x-pack/plugins/apm/e2e/run-e2e.sh

sorenlouv · 2020-05-11T11:55:23Z

What happens if you create it as an empty file:

module.exports = {};

v1v · 2020-05-12T07:56:39Z

That's interesting, I actually created the file with the empty content as suggested, and after running the scripts, the file got changed as below:

diff --git a/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js b/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
index dc164f3e..07782866 100644
--- a/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
+++ b/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
@@ -1 +1,10 @@
-module.exports = { }
\ No newline at end of file
+module.exports = {
+  "__version": "4.2.0",
+  "APM": {
+    "Transaction duration charts": {
+      "1": "350 ms",
+      "2": "175 ms",
+      "3": "0 ms"
+    }
+  }
+}

It works with the retry tweak after 6 tries :)

How did I run the e2e?

cd /tmp
git clone https://github.com/elastic/kibana --depth 1
cd kibana

## Apply patch
cat <<EOT > tweak.patch
diff --git a/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js b/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
index a462f4a5..dc164f3e 100644
--- a/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
+++ b/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
@@ -1,16 +1 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License;
- * you may not use this file except in compliance with the Elastic License.
- */
-
-module.exports = {
-  APM: {
-    'Transaction duration charts': {
-      '1': '500 ms',
-      '2': '250 ms',
-      '3': '0 ms'
-    }
-  },
-  __version: '4.2.0'
-};
+module.exports = { }
\ No newline at end of file
diff --git a/x-pack/plugins/apm/e2e/run-e2e.sh b/x-pack/plugins/apm/e2e/run-e2e.sh
index 818d45ab..c6fceaaf 100755
--- a/x-pack/plugins/apm/e2e/run-e2e.sh
+++ b/x-pack/plugins/apm/e2e/run-e2e.sh
@@ -118,7 +118,12 @@ echo "\n✅ Setup completed successfully. Running tests...\n"
 #
 # run cypress tests
 ##################################################
-yarn cypress run --config pageLoadTimeout=100000,watchForFileChanges=true
+n=0
+until [ \$n -ge 20 ] ; do
+    yarn cypress run --config pageLoadTimeout=100000,watchForFileChanges=true && break || true
+    sleep 10
+    n=\$[\$n+1]
+done
 
 #
 # Run interactively
EOT

git apply  --stat tweak.patch 
git apply tweak.patch 
nvm use $(cat .node-version)
cd x-pack/plugins/apm/e2e
yarn install
cd -
yarn kbn clean && yarn kbn bootstrap
nohup node ./scripts/kibana --no-base-path --dev --no-dev-config --config x-pack/plugins/apm/e2e/ci/kibana.e2e.yml > /tmp/kibana.log 2>&1 &
x-pack/plugins/apm/e2e/run-e2e.sh

Do you need any further details?

sorenlouv · 2020-05-12T10:32:49Z

after running the scripts, the file got changed as below:

Great, that's also what I'd expect (it should recreate the snapshot).
It's a shame this only happens when the file is empty - it should also happen when the file doesn't exist. But we can look into that another time.

It works with the retry tweak after 6 tries :)

Ouch. Do you know why it failed the first 5? Is it still because we can't reliably determine when Kibana is up and running?
If so, can you try running Kibana in prod mode instead of dev?

node ./scripts/kibana --no-base-path --config x-pack/plugins/apm/e2e/ci/kibana.e2e.yml

v1v · 2020-05-12T12:14:40Z

Is it still because we can't reliably determine when Kibana is up and running?
If so, can you try running Kibana in prod mode instead of dev?

That's already reported in #61942

A bit of context, I've already tried running Kibana in prod mode instead of dev a few times and did not work:

That's the reason I enabled the retry tweak in #61803 and it does work, the good thing, what people run locally with the e2e it's also validated in the CI, therefore no running discrepancies in the CI and locally.

sorenlouv · 2020-05-12T12:19:51Z

That's already reported in #61942

I'm not following. Are you saying that the retries are still necessary after #61942 is fixed?

sorenlouv · 2020-05-12T12:41:46Z

I think we should find out why the retries are necessary so we can get rid of them.

v1v · 2020-05-25T07:21:24Z

I'll close this done since it's now solved in #66373

v1v added the Team:APM All issues that need APM UI Team support label May 6, 2020

sorenlouv added the [zube]: Backlog label May 6, 2020

v1v closed this as completed May 25, 2020

zube bot added [zube]: Done and removed [zube]: Backlog labels May 25, 2020

sorenlouv removed the [zube]: Done label May 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] e2e are failing when running them locally with some `snapshot` #65422

[APM] e2e are failing when running them locally with some `snapshot` #65422

v1v commented May 6, 2020

elasticmachine commented May 6, 2020

sorenlouv commented May 6, 2020

v1v commented May 8, 2020

sorenlouv commented May 11, 2020 •

edited

Loading

v1v commented May 12, 2020

sorenlouv commented May 12, 2020

v1v commented May 12, 2020

sorenlouv commented May 12, 2020

sorenlouv commented May 12, 2020

v1v commented May 25, 2020

[APM] e2e are failing when running them locally with some snapshot #65422

[APM] e2e are failing when running them locally with some snapshot #65422

Comments

v1v commented May 6, 2020

elasticmachine commented May 6, 2020

sorenlouv commented May 6, 2020

v1v commented May 8, 2020

What did I do?

sorenlouv commented May 11, 2020 • edited Loading

v1v commented May 12, 2020

How did I run the e2e?

sorenlouv commented May 12, 2020

v1v commented May 12, 2020

sorenlouv commented May 12, 2020

sorenlouv commented May 12, 2020

v1v commented May 25, 2020

[APM] e2e are failing when running them locally with some `snapshot` #65422

[APM] e2e are failing when running them locally with some `snapshot` #65422

sorenlouv commented May 11, 2020 •

edited

Loading