Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Functional test cleanup and speedups #5080

Merged
merged 9 commits into from
Nov 30, 2022
Merged

Functional test cleanup and speedups #5080

merged 9 commits into from
Nov 30, 2022

Conversation

oskirby
Copy link
Collaborator

@oskirby oskirby commented Nov 29, 2022

Description

I have become frustrated at the functional tests taking 25 minutes to build. Let's see if we can do better.

Here is a quick breakdown of what I changed, and why it helped:

  • Updating the github runners to ubuntu-22.04 means that we no longer need to bring in a PPA for Qt6 support, and we can instead get those packages from the official repositories, which have much better caching and CDNs behind them.
  • Caching grcov between runs saves us about 5 minuntes of compilation time.
  • Minimizing the profiling data dramatically reduces the storage and bandwidth required to move the build artifacts around (3.5GB down to 250MB!).
  • Now that the PPA is no longer necessary, copying the installed .deb files from build to test runners is actually slower than just installing fresh dependencies from the official repositories.
  • Building the unit tests with Ninja does a much better job of building in parallel.

Some other room for improvement that I did not address:

  • Building the addons could be done during the build stage. This would save CPU time, but not calendar time, and would be a more representative test.
  • Ninja is way faster for Windows too, but it falls apart somewhere in the rust libraries.
  • Do something with ccache between workflow runs? That would be outrageously fast!

Reference

Github #4877 (VPN-3204)

Checklist

  • My code follows the style guidelines for this project
  • I have not added any packages that contain high risk or unknown licenses (GPL, LGPL, MPL, etc. consult with DevOps if in question)
  • I have performed a self review of my own code
  • I have commented my code PARTICULARLY in hard to understand areas
  • I have added thorough tests where needed

@oskirby oskirby changed the title Fix warnings about ::set-output deprecation Functional test cleanup and speedups Nov 29, 2022
@codecov-commenter
Copy link

codecov-commenter commented Nov 29, 2022

Codecov Report

Base: 38.41% // Head: 21.39% // Decreases project coverage by -17.02% ⚠️

Coverage data is based on head (4b586a3) compared to base (24207f3).
Patch has no changes to coverable lines.

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #5080       +/-   ##
===========================================
- Coverage   38.41%   21.39%   -17.03%     
===========================================
  Files         330      328        -2     
  Lines       22145    20912     -1233     
  Branches    11966    11276      -690     
===========================================
- Hits         8508     4474     -4034     
- Misses       5802     9748     +3946     
+ Partials     7835     6690     -1145     
Flag Coverage Δ
functional_test_Addons 0.00% <ø> (?)
functional_test_AuthenticationInApp 0.00% <ø> (?)
functional_test_AuthenticationInBrowser 0.00% <ø> (?)
functional_test_Benchmark 0.00% <ø> (?)
functional_test_CaptivePortal 0.00% <ø> (?)
functional_test_Connection 0.00% <ø> (?)
functional_test_ContactUs 0.00% <ø> (?)
functional_test_Devices 0.00% <ø> (?)
functional_test_Heartbeat 0.00% <ø> (?)
functional_test_MobileOnboarding 0.00% <ø> (?)
functional_test_Multihop 0.00% <ø> (?)
functional_test_Navigation 0.00% <ø> (?)
functional_test_Onboarding 0.00% <ø> (?)
functional_test_PushMessageHandlers 0.00% <ø> (?)
functional_test_Servers 0.00% <ø> (?)
functional_test_Settings 0.00% <ø> (?)
functional_test_Subscription 0.00% <ø> (?)
functional_test_TelemetryView 0.00% <ø> (?)
functional_test_TipsAndTricksIntroModal 0.00% <ø> (?)
functional_test_Tutorials 0.00% <ø> (?)
functional_test_UnsecuredNetworkAlert 0.00% <ø> (?)
functional_tests ?
lottie_tests 23.06% <ø> (ø)
unit_tests 28.97% <ø> (-4.99%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/pingsender.h 0.00% <0.00%> (-100.00%) ⬇️
src/profileflow.h 0.00% <0.00%> (-100.00%) ⬇️
src/externalophandler.h 0.00% <0.00%> (-100.00%) ⬇️
src/websocket/pushmessage.h 0.00% <0.00%> (-100.00%) ⬇️
src/imageproviderfactory.cpp 0.00% <0.00%> (-100.00%) ⬇️
src/addons/manager/addonmanager.h 0.00% <0.00%> (-100.00%) ⬇️
src/captiveportal/captiveportal.h 0.00% <0.00%> (-100.00%) ⬇️
src/captiveportal/captiveportalrequest.h 0.00% <0.00%> (-100.00%) ⬇️
src/tasks/removedevice/taskremovedevice.h 0.00% <0.00%> (-100.00%) ⬇️
src/connectionbenchmark/benchmarktasksentinel.h 0.00% <0.00%> (-100.00%) ⬇️
... and 236 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@oskirby oskirby force-pushed the functional-test-tidy branch from 7be5ca8 to c69e7b4 Compare November 29, 2022 20:00
@oskirby
Copy link
Collaborator Author

oskirby commented Nov 29, 2022

There are still some efficiency gains to be found in moving addon generation into the build stage, but that won't really affect the calendar time much and I'm a little bit intimidated by messing around with the addon build scripts for now.

@oskirby
Copy link
Collaborator Author

oskirby commented Nov 29, 2022

I am also genuinely shocked at how long the unit tests take to build.

@oskirby oskirby force-pushed the functional-test-tidy branch from 4b586a3 to dca3d72 Compare November 30, 2022 05:10
@oskirby oskirby requested a review from mozrokafor November 30, 2022 05:35
@oskirby oskirby marked this pull request as ready for review November 30, 2022 05:36
@bakulf bakulf merged commit 3befd81 into main Nov 30, 2022
@bakulf bakulf deleted the functional-test-tidy branch November 30, 2022 06:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants