Fix caching for cargo cross CI tests #1293

Job	Run time
test_features	21s
Fuzz targets (afl)	2m 19s
test_features (default)	2m 12s
Fuzz targets (cargo-fuzz)	50s
build_benchmarks	52s
test_features (rayon)	35s
test_features (avif)	2m 19s
cargo-deny	30s
clippy	1m 37s
test_features (bmp)	1m 34s
public_private_dependencies	25s
test_features (dds)	49s
rustfmt	10s
test_features (exr)	1m 5s
test_avif_decoding	1m 22s
test_features (ff)	23s
test_other_archs (powerpc-unknown-linux-gnu)	2m 20s
test_features (gif)	31s
test_other_archs (i686-unknown-linux-gnu)	1m 19s
test_toolchains (1.67.1)	34s
test_features (hdr)	14s
test_toolchains (nightly)	1m 22s
test_features (ico)	26s
test_toolchains (beta)	1m 50s
test_features (jpeg)	17s
test_features (png)	57s
test_features (pnm)	16s
test_features (qoi)	31s
test_features (tga)	28s
test_features (tiff)	27s
test_features (webp)	50s
	29m 45s

Provide feedback