Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ccache support #2856

Merged
merged 11 commits into from
Jul 7, 2024
Merged

Add ccache support #2856

merged 11 commits into from
Jul 7, 2024

Conversation

mikee47
Copy link
Contributor

@mikee47 mikee47 commented Jul 7, 2024

This PR adds ccache support to the framework and improves general build performance.

Of particular importance is keeping the CI runs efficient to conserve resources without sacrificing effectiveness. If they complete a bit faster too, that's a bonus :-)

Reduce CI installation overhead

Since we're using more CI cache space to cache builds, it's at a premium. Fortunately there's quite a lot of stuff we can remove so running a script (clean-tools.py) after tool installation (CI only) clears this out so it doesn't end up in the cache. In practice this saves about 1.4GB of (compressed) cache space.

Also, for IDF4.3 we can reduce that to a single run for the standard esp32 SOC.

Installer improvements

Optional packages (e.g. clang-format) only required for Host builds, so don't install them otherwise.

Review Windows install scripts:

  • Install ninja, ccache
  • Check for undefined environment variables
  • Check if tools already installed, skip if so and print message
  • Don't overwrite MinGW if it already exists

Add ccache support

I've never used ccache so wondered what it was doing in the install scripts. Certainly didn't seem to be used anywhere so thinking we could remove it...

Nope. Let's use it. Add ENABLE_CCACHE setting, and enable in CI builds.

Stats (ccache -sv) shows half of the calls aren't cacheable.
Turns out ccache doesn't support -MM which we use for generating header build dependency files. After some investigations turns out we can just use -MMD which has several benefits:

  1. Allows 100% hit rate via ccache
  2. Reduces number of compiler invocations required since .d files are generated at the same time as .o files
  3. Previously, if a source file is moved we got an error such as No rule to make target '...file.cpp', needed by '...file.cpp.d'. Stop.. This no longer happens.
  4. Works on Windows without sed workaround for paths so makefile is simpler/faster.

Note that a typical ccache installation may add soft links which override system compilers.
For example, gcc would actually run /usr/lib64/ccache/gcc instead of /usr/bin/gcc.
If ENABLE_CCACHE isn't set then Sming overrides this behaviour (using CCACHE_DISABLE) to ensure behaviour consistent with other (cross-compiled) architectures.

Fixes

Update ws_parser submodule. During testing I was getting some complaints about the particular commit not being available.
It looks like this repo. got rebased at some point and the current master has changed.
No change to content.

Python script fixes: make ide-vscode fails in Windows, and there's a regex fix to boardtool.py.

Fix fallthrough warning in LiveDebug sample.

@mikee47 mikee47 force-pushed the feature/faster-install branch from 08063ed to 4308127 Compare July 7, 2024 13:03
@slaff slaff added this to the 5.2.0 milestone Jul 7, 2024
@slaff slaff merged commit 045a85a into SmingHub:develop Jul 7, 2024
46 checks passed
@mikee47 mikee47 deleted the feature/faster-install branch July 7, 2024 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants