test(e2e): Switch from Appium to Maestro #4210

krystofwoldrich · 2024-10-28T12:18:37Z

📢 Type of change

Enhancement
Refactoring

📜 Description

This PR replaces appium and jest with maestro as runner of the e2e tests.

💡 Motivation and Context

part of https://github.com/getsentry/projects/issues/165

💚 How did you test it?

ci, locally

📝 Checklist

I reviewed submitted code
I added tests to verify changes
No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled
All tests passing
No breaking changes

#skip-changelog

github-actions · 2024-10-28T12:38:36Z

Android (legacy) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	356.96 ms	456.55 ms	99.59 ms
Size	17.74 MiB	20.08 MiB	2.34 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`2534337`	394.15 ms	415.12 ms	20.97 ms
`5446992`	403.40 ms	426.70 ms	23.30 ms
`9433f35`	347.64 ms	356.22 ms	8.58 ms
`148f924`	492.65 ms	500.28 ms	7.63 ms
`31fcca2`	391.22 ms	414.78 ms	23.56 ms
`76d1baf`+dirty	335.72 ms	355.52 ms	19.80 ms
`15c80ab`+dirty	336.27 ms	350.58 ms	14.31 ms
`52a8031`+dirty	311.55 ms	321.37 ms	9.82 ms
`0d3e677`	422.82 ms	411.90 ms	-10.92 ms
`d7401ac`+dirty	375.20 ms	383.51 ms	8.31 ms

App size

Revision	Plain	With Sentry	Diff
`2534337`	17.73 MiB	19.84 MiB	2.11 MiB
`5446992`	17.73 MiB	19.85 MiB	2.12 MiB
`9433f35`	17.73 MiB	19.81 MiB	2.08 MiB
`148f924`	17.73 MiB	19.94 MiB	2.21 MiB
`31fcca2`	17.73 MiB	19.90 MiB	2.17 MiB
`76d1baf`+dirty	17.73 MiB	20.04 MiB	2.31 MiB
`15c80ab`+dirty	17.73 MiB	20.04 MiB	2.31 MiB
`52a8031`+dirty	17.73 MiB	20.04 MiB	2.31 MiB
`0d3e677`	17.74 MiB	20.07 MiB	2.34 MiB
`d7401ac`+dirty	17.73 MiB	19.75 MiB	2.02 MiB

github-actions · 2024-10-28T12:39:17Z

iOS (legacy) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1221.41 ms	1220.49 ms	-0.92 ms
Size	2.36 MiB	3.08 MiB	735.65 KiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`c2a4e9b`+dirty	1240.10 ms	1239.22 ms	-0.88 ms
`34aba08`+dirty	1276.78 ms	1308.52 ms	31.74 ms
`b1e8712`+dirty	1256.02 ms	1265.14 ms	9.12 ms
`70caa60`+dirty	1218.27 ms	1230.30 ms	12.03 ms
`2534337`+dirty	1225.08 ms	1230.26 ms	5.17 ms
`e5bc97b`+dirty	1230.63 ms	1234.83 ms	4.20 ms
`4a6664f`+dirty	1209.49 ms	1208.63 ms	-0.86 ms
`15c80ab`+dirty	1223.74 ms	1228.96 ms	5.22 ms
`0db0c72`+dirty	1275.02 ms	1285.84 ms	10.82 ms
`575f9da`+dirty	1266.22 ms	1274.84 ms	8.62 ms

App size

Revision	Plain	With Sentry	Diff
`c2a4e9b`+dirty	2.36 MiB	3.08 MiB	734.00 KiB
`34aba08`+dirty	2.36 MiB	2.85 MiB	495.32 KiB
`b1e8712`+dirty	2.36 MiB	2.84 MiB	488.84 KiB
`70caa60`+dirty	2.36 MiB	2.83 MiB	479.27 KiB
`2534337`+dirty	2.36 MiB	2.88 MiB	525.47 KiB
`e5bc97b`+dirty	2.36 MiB	3.10 MiB	753.14 KiB
`4a6664f`+dirty	2.36 MiB	3.04 MiB	696.39 KiB
`15c80ab`+dirty	2.36 MiB	2.83 MiB	474.49 KiB
`0db0c72`+dirty	2.36 MiB	2.84 MiB	487.01 KiB
`575f9da`+dirty	2.36 MiB	2.87 MiB	520.20 KiB

github-actions · 2024-10-28T12:44:49Z

Android (new) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	413.31 ms	477.76 ms	64.45 ms
Size	7.15 MiB	8.35 MiB	1.20 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`d43a46b`+dirty	417.65 ms	472.98 ms	55.33 ms
`d361d38`+dirty	257.72 ms	318.76 ms	61.04 ms
`0677344`+dirty	288.40 ms	391.44 ms	103.04 ms
`76d1baf`+dirty	339.02 ms	408.65 ms	69.63 ms
`15c80ab`+dirty	276.38 ms	327.54 ms	51.17 ms
`52a8031`+dirty	330.72 ms	358.76 ms	28.03 ms
`575f9da`+dirty	337.15 ms	370.47 ms	33.32 ms
`0ebca77`+dirty	360.94 ms	402.24 ms	41.30 ms
`e2b64fe`+dirty	258.82 ms	304.26 ms	45.44 ms
`484813b`+dirty	411.65 ms	456.02 ms	44.37 ms

App size

Revision	Plain	With Sentry	Diff
`d43a46b`+dirty	7.15 MiB	8.34 MiB	1.19 MiB
`d361d38`+dirty	7.15 MiB	8.08 MiB	959.34 KiB
`0677344`+dirty	7.15 MiB	8.07 MiB	949.80 KiB
`76d1baf`+dirty	7.15 MiB	8.09 MiB	964.41 KiB
`15c80ab`+dirty	7.15 MiB	8.09 MiB	966.13 KiB
`52a8031`+dirty	7.15 MiB	8.09 MiB	965.95 KiB
`575f9da`+dirty	7.15 MiB	8.10 MiB	979.68 KiB
`0ebca77`+dirty	7.15 MiB	8.22 MiB	1.07 MiB
`e2b64fe`+dirty	7.15 MiB	8.07 MiB	947.16 KiB
`484813b`+dirty	7.15 MiB	8.34 MiB	1.19 MiB

github-actions · 2024-10-28T12:51:58Z

iOS (new) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1222.83 ms	1226.51 ms	3.68 ms
Size	2.92 MiB	3.64 MiB	741.21 KiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`c2a4e9b`+dirty	1247.39 ms	1243.04 ms	-4.35 ms
`34aba08`+dirty	1268.58 ms	1276.80 ms	8.22 ms
`b1e8712`+dirty	1284.11 ms	1297.82 ms	13.71 ms
`70caa60`+dirty	1279.08 ms	1281.54 ms	2.46 ms
`2534337`+dirty	1220.87 ms	1221.47 ms	0.60 ms
`e5bc97b`+dirty	1229.17 ms	1227.64 ms	-1.54 ms
`4a6664f`+dirty	1218.77 ms	1221.07 ms	2.30 ms
`15c80ab`+dirty	1248.41 ms	1251.24 ms	2.83 ms
`0db0c72`+dirty	1258.88 ms	1262.52 ms	3.64 ms
`575f9da`+dirty	1272.00 ms	1284.38 ms	12.38 ms

App size

Revision	Plain	With Sentry	Diff
`c2a4e9b`+dirty	2.92 MiB	3.64 MiB	739.91 KiB
`34aba08`+dirty	2.92 MiB	3.41 MiB	499.03 KiB
`b1e8712`+dirty	2.92 MiB	3.40 MiB	494.15 KiB
`70caa60`+dirty	2.92 MiB	3.39 MiB	486.04 KiB
`2534337`+dirty	2.92 MiB	3.43 MiB	529.76 KiB
`e5bc97b`+dirty	2.92 MiB	3.66 MiB	758.40 KiB
`4a6664f`+dirty	2.92 MiB	3.60 MiB	702.09 KiB
`15c80ab`+dirty	2.92 MiB	3.39 MiB	481.56 KiB
`0db0c72`+dirty	2.92 MiB	3.40 MiB	492.71 KiB
`575f9da`+dirty	2.92 MiB	3.43 MiB	524.26 KiB

dev-packages/e2e-tests/cli.mjs

antonis

This is not related with the changes introduced in this PR but I noticed that the Build RN , Upload logs task always fails on Android (example)

Warning: No files were found with the provided path: dev-packages/e2e-tests/react-native-versions/0.76.0/RnDiffApp/ios/*.log. No artifacts will be uploaded.

I think replacing ios with the platform on this line could fix this

path: dev-packages/e2e-tests/react-native-versions/${{ matrix.rn-version }}/RnDiffApp/${{ matrix.platform }}/*.log

krystofwoldrich · 2024-10-30T10:39:27Z

@antonis Good catch, we can just not run it for Android, as there are no logs generated. This should be for iOS only, because the xcodeoutput is saved to file.

…values Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

antonis

LGTM 🚀

lucas-zimerman

LGTM 🚀

krystofwoldrich added 9 commits October 24, 2024 10:55

chore(samples): Update gems

cf16bc8

chore(perf-samples): Remove unused deps and files

d3e7028

fix yarn lock file

4d1b67b

ref(e2e): Fetch received event in the tested app

96f2c86

Merge branch 'main' into kw/tests/fetch-event-in-sim

f5db3a0

Show error as text on screen instead of crashing the app

caba867

TMP Fix build

6bb13b8

tmp fix rn builds

5555f27

test(e2e): Switch from Appium to Maestro

627d23d

krystofwoldrich added 6 commits October 28, 2024 16:19

fix run maestro using e2e cli

b114ce5

fix ios generic build

5925376

Fix path to the app

e48dc72

Fix missing maestro and booted ios simulator

a9cb2a6

longer timeout and add tests ready

5e782ab

Extend wait for initial tests ready

7f5ae03

Base automatically changed from kw/tests/fetch-event-in-sim to main October 29, 2024 14:41

krystofwoldrich marked this pull request as ready for review October 29, 2024 15:41

krystofwoldrich requested review from lucas-zimerman and antonis as code owners October 29, 2024 15:41

antonis reviewed Oct 30, 2024

View reviewed changes

dev-packages/e2e-tests/cli.mjs Outdated Show resolved Hide resolved

krystofwoldrich added 2 commits October 30, 2024 10:47

Merge branch 'main' into kw/tests/switch-to-maestro

d5199ec

remove tmp xcodeproj gem fix

1f95d10

github-advanced-security bot found potential problems Oct 30, 2024

View reviewed changes

dev-packages/e2e-tests/cli.mjs Fixed Show resolved Hide resolved

dev-packages/e2e-tests/cli.mjs Fixed Show resolved Hide resolved

antonis reviewed Oct 30, 2024

View reviewed changes

Fix code scanning alert no. 18: Shell command built from environment …

560283f

…values Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

antonis approved these changes Oct 30, 2024

View reviewed changes

Update cli.mjs

a9f5fb3

lucas-zimerman approved these changes Oct 30, 2024

View reviewed changes

krystofwoldrich merged commit 4c3c7db into main Oct 30, 2024
60 checks passed

krystofwoldrich deleted the kw/tests/switch-to-maestro branch October 30, 2024 16:48

This was referenced Nov 13, 2024

chore(deps): bump github/codeql-action from 3.27.0 to 3.27.2 #4268

Closed

Bump appium version on metrics e2e tests #3923

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(e2e): Switch from Appium to Maestro #4210

test(e2e): Switch from Appium to Maestro #4210

krystofwoldrich commented Oct 28, 2024 •

edited

Loading

github-actions bot commented Oct 28, 2024 •

edited

Loading

Baseline results on branch: main

Startup times

App size

github-actions bot commented Oct 28, 2024 •

edited

Loading

Baseline results on branch: main

Startup times

App size

github-actions bot commented Oct 28, 2024 •

edited

Loading

Baseline results on branch: main

Startup times

App size

github-actions bot commented Oct 28, 2024 •

edited

Loading

Baseline results on branch: main

Startup times

App size

antonis left a comment

krystofwoldrich commented Oct 30, 2024

antonis left a comment

lucas-zimerman left a comment

test(e2e): Switch from Appium to Maestro #4210

test(e2e): Switch from Appium to Maestro #4210

Conversation

krystofwoldrich commented Oct 28, 2024 • edited Loading

📢 Type of change

📜 Description

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

github-actions bot commented Oct 28, 2024 • edited Loading

Android (legacy) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

github-actions bot commented Oct 28, 2024 • edited Loading

iOS (legacy) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

github-actions bot commented Oct 28, 2024 • edited Loading

Android (new) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

github-actions bot commented Oct 28, 2024 • edited Loading

iOS (new) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

antonis left a comment

Choose a reason for hiding this comment

krystofwoldrich commented Oct 30, 2024

antonis left a comment

Choose a reason for hiding this comment

lucas-zimerman left a comment

Choose a reason for hiding this comment

krystofwoldrich commented Oct 28, 2024 •

edited

Loading

github-actions bot commented Oct 28, 2024 •

edited

Loading

github-actions bot commented Oct 28, 2024 •

edited

Loading

github-actions bot commented Oct 28, 2024 •

edited

Loading

github-actions bot commented Oct 28, 2024 •

edited

Loading