Skip to content
This repository has been archived by the owner on Nov 6, 2020. It is now read-only.

Switching sccache from local to Redis #10971

Merged
merged 5 commits into from
Oct 2, 2019
Merged

Switching sccache from local to Redis #10971

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2539cf5
Switching cache from local to Radis
General-Beck Aug 14, 2019
984681f
sccache -s won't work. All the images are taken from own registry
TriplEight Aug 14, 2019
408765b
new images addresses
TriplEight Aug 15, 2019
ca20ad9
statisticss after stop sccache server
General-Beck Oct 2, 2019
5d4b192
Merge branch 'master' into sccache-redis
General-Beck Oct 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 3 additions & 27 deletions .gitlab-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@ stages:
- publish
- optional

image: parity/parity-ci-linux:latest
image: registry.parity.io/pub/ci/parity-ci-linux:latest
variables:
GIT_STRATEGY: fetch
GIT_SUBMODULE_STRATEGY: recursive
CI_SERVER_NAME: "GitLab CI"
CARGO_HOME: "/ci-cache/${CI_PROJECT_NAME}/cargo/${CI_JOB_NAME}"
SCCACHE_DIR: "/ci-cache/${CI_PROJECT_NAME}/sccache"
CARGO_TARGET: x86_64-unknown-linux-gnu


.no_git: &no_git # disable git strategy
variables:
GIT_STRATEGY: none
Expand Down Expand Up @@ -41,22 +41,7 @@ variables:
before_script:
- rustup show
- cargo --version
- SCCACHE_ERROR_LOG=/builds/parity/parity-ethereum/sccache_debug.log
RUST_LOG=sccache=debug
sccache --start-server
- sccache -s
after_script:
# sccache debug info
- if test -e sccache_debug.log;
then
echo "_____All crate-types:_____";
grep 'parse_arguments.*--crate-type' sccache_debug.log | sed -re 's/.*"--crate-type", "([^"]+)".*/\1/' | sort | uniq -c;
echo "_____Non-cacheable reasons:_____";
grep CannotCache sccache_debug.log | sed -re 's/.*CannotCache\((.+)\).*/\1/' | sort | uniq -c;
else
echo "_____No logs from sccache_____";
exit 0;
fi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so debugging will be disabled completely?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, sccache debugging was actually helpful only once.
But what's important and bugs me, is that debug log level somehow hinders sccache to communicate with redis.
Also, had to remove sccache -s as it was failing the job because it couldn't get the info from Redis. I tried some ideas, but think that in order to do that, we should install sccache to CI1 and CI2 which is not really necessary, at least for now.

- SCCACHE_REDIS=redis://:${REDIS}@172.17.0.1/0 sccache --start-server
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it wise to use the explicit ip address here?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a default gateway address of a Docker container-internal network (so in practice it's a mapping of host's 127.0.0.1 into a local subnet).
I guess it's safe enough, provided that we're not going to mess with Docker's networking too much in this setup.

As far as I understood from my chat with @General-Beck earlier today, there's no easy way to teach sccache to identify that IP by hostname, and there's no non-hacky lookup flow to be used instead either.

tags:
- linux-docker

Expand All @@ -66,7 +51,6 @@ variables:
<<: *collect_artifacts
script:
- scripts/gitlab/build-linux.sh
- sccache -s
after_script:
- mkdir -p tools
- cp -r scripts/docker/hub/* ./tools
Expand All @@ -84,21 +68,18 @@ cargo-check 0 3:
<<: *docker-cache-status
script:
- time cargo check --target $CARGO_TARGET --locked --no-default-features --verbose --color=always
- sccache -s

cargo-check 1 3:
stage: test
<<: *docker-cache-status
script:
- time cargo check --target $CARGO_TARGET --locked --manifest-path util/io/Cargo.toml --no-default-features --verbose --color=always
- sccache -s

cargo-check 2 3:
stage: test
<<: *docker-cache-status
script:
- time cargo check --target $CARGO_TARGET --locked --manifest-path util/io/Cargo.toml --features "mio" --verbose --color=always
- sccache -s

cargo-audit:
stage: test
Expand All @@ -112,37 +93,32 @@ validate-chainspecs:
<<: *docker-cache-status
script:
- ./scripts/gitlab/validate-chainspecs.sh
- sccache -s

test-cpp:
stage: build
<<: *docker-cache-status
script:
- ./scripts/gitlab/test-cpp.sh
- sccache -s

test-linux:
stage: build
<<: *docker-cache-status
script:
- ./scripts/gitlab/test-linux.sh stable
- sccache -s

test-linux-beta:
stage: build
only: *releaseable_branches
<<: *docker-cache-status
script:
- ./scripts/gitlab/test-linux.sh beta
- sccache -s

test-linux-nightly:
stage: build
only: *releaseable_branches
<<: *docker-cache-status
script:
- ./scripts/gitlab/test-linux.sh nightly
- sccache -s
allow_failure: true

build-android:
Expand Down