Otherwise we get warnings about unused imports on Windows.

This is a bit of a hack.  The integration tests under `src/test/`,
specifically `test_server_compile` have started timing out at some point
after the Windows builds were broken.  It's not obvious to me why it
should be this particular test, and not other tests.  But we're running
on Windows, and presumably running on some kind of VM in CI, so it seems
worth bumping up the timeout here.

Tests on appveyor timeout before the server starts.

BLAKE3 is designed to be a very high performance cryptographic hash. The
BLAKE3 team has shown 8.5x higher single-thread performance than SHA-512
on modern server hardware (AWS `c5.metal`). This change did not result
in a significant improvement to my observed local build times, but
newer hardware may see a meaningful improvement.

Signed-off-by: George Hahn <[email protected]>

This is used by chromium for example.

Even though Actions aren't supported on the main repo, they can still be
supported on personal repos, and people might proactively fix their
Windows bustage if GitHub sends them emails about it.  So let's keep
Actions, but turn off things we haven't enabled on Travis yet.

Co-authored-by: Bert Belder <[email protected]>

Removes a chunk from the readme regarding a false positive
rustc-wrapper entry not being used, which is closed since 1.40.0 .
Cargo issue: rust-lang/cargo#7745

Co-authored-by: Bernhard Schuster <[email protected]>

…pdb.

Signed-off-by: Nicholas Rishel <[email protected]>

This gets us one step into tokio 0.2, with its new scheduler. This makes
things significantly faster on my 32-core 64-threads machine, with
Firefox builds with 100% cache hits at -j64 going down from 3 minutes to
2:24. Also, with the old scheduler, with 100% cache hits, -j16 was
actually faster than -j64, which is not the case anymore.

… windows for test_server_port_in_use"

This reverts commit b43d51f and
commit b437c92.

While tokio_reactor::Handle::current has a deprecation note that says
"semantics were sometimes surprising, use Handle::default()", it turns
out the the semantics of `current` work better for sccache, as sccache
--start-server times out with `default` but not with `current` (although
the server does start).

The error for the test_server_port_in_use switch to being about a
timeout was due to the switch to `current`, so we're now back to a more
normal looking error message.

This was done to support different compilers/languages which
use different compilation flags. For example nvcc supports
`-c`, `-dc`, and `-ptx`. This change allows compilers to determine
what type of compilation they are doing and react correctly.

Started based on the work at https://github.com/yf225/sccache but
more changes have been required to properly handle the different
nvcc compile flags and modes

NVCC doesn't support generating both the dependency information
and the preprocessor output at the same time. So we need to
have the -MT/-MD flags go with a dependency info compiler invocation
and not the pre-process step.

In addition this now allows sccache to work properly with `-MQ` for
gcc like compilers.

Flags such as `expt-relaxed-constexpr` are flags and don't have
any parameters they should capture. Doing this caused compilation
failures when it captured `Xcompiler` and therefore a subsequent flag
for the host compiler was incorrectly given to nvcc instead.

The TODO in server.rs that we didn't fix was somewhat cryptic and I
don't think it would have made the code any clearer even if it could
have been fixed.

Should have caught this previously.

The linked PR in the comment changed the output format, but never
bothered to change sccache to match the new output format.  So when we
stumbled across a case of the new output format, we panicked.

mozilla#674 broke caching for msvc /foo style arguments that weren't
handled in our arg list. So let's expand that list.

Fixes mozilla#725.

Fixes mozilla#747.

It should not prevent caching by itself.

It's not clear to me whether something like this is already happening
inside type derivation or not, but this at least makes things clearer
that `compiler_result` is not being borrowed mutably.

This routine had a couple of problems:

1. The buffer was reinitialized with zeros every time through the loop,
   rather than once outside of the loop.
2. We created a buffered reader, but we are using it to do large block
   reads, which makes the internal buffer essentially useless.  `BufReader`
   is at least smart enough to bypass the internal buffer in such cases,
   but still, we were allocating an additional amount of heap memory that
   we didn't need.

This patch addresses both of those problems.  It is arguably better to use
`std::io::copy`, which currently takes advantage of `Read::initialize`, a
Nightly-only API that attempts to avoid needing to initialize buffers prior to
passing them off to `Read::read`, but the default buffer size of
`std::io::copy` is much smaller than what we use here.  Rust rlibs can get
rather large (e.g. about 80% of rlibs during a Firefox compilation are larger
than 128kb and will therefore go through even the loop we have here at least
twice; ~55% are larger than 700k; about 10% are larger than 12mb (!)) and are
the primary consumers of this method (via `util::hash_all`), so at least
fixing the obvious major issues above seems worthwhile.

The code was already mostly rustfmt'd, so we might as well go the whole
way.

This patch looks like a routine crate version update, but it fixes
something very fundamentally wrong with the tests: `escargot`, to find
the `sccache` binary during tests, would actually run `cargo build` and
parse the output.  There were all manner of hacks to try and build the
binary with the correct set of features, none of which were reliable,
and the hacks were not reliably present in all places, either.

When the hacks were not present, `escargot` would rebuild the entire
sccache crate, which was responsible for the `sccache_cargo` test taking
entirely too long.  When the hacks failed utterly, perfectly fine
patches would mysteriously fail, as seen in mozilla#774.

We can get rid of `escargot` because `assert-cmd` has been updated to a)
not use `escargot` under the hood; and b) do something smarter to locate
the target binary.  The upshot is fewer mysterious test failures and
significantly faster running tests.

This finishes what mozilla#762 started. All `println!`/`eprintln!` calls now
start with `sccache: `, except those involved in printing of requested
data, such as stats.

Stable Rust and the OS X versions on TravisCI do not seem to get along
right now.  I filed a Rust bug (rust-lang/rust#73030) about this, but if
the bug is very OS X-configuration specific, I'm not sure there's much
that can be done about it on the Rust side.

We can at least change the CI configuration so we don't get a bunch of
spurious failures.

The abstraction barrier here is not perfect, because people can make
`get_object` calls on their own.  But at least `CacheRead` is slightly
more structured this way.

The abstraction barrier here is not perfect, because people can make
`put_object` calls on their own.  But at least `CacheWrite` is slightly
more structured this way and the optimization for empty streams is
hidden away from the client.

This seems cleaner than having somebody else manage object extraction.

This patch is the mirror of the previous `CacheRead` change.  Again, not
a huge change, but it's slightly nicer for `CacheWrite` to manage this.

I was looking through old issues, and mozilla#160 mentioned that crc32
checksumming for zip files was showing up in profiles.  zip-rs 0.4 uses
a byte-at-a-time version of crc32; zip-rs 0.5 relies on the `crc32fast`
crate, which features faster implementations.

With tokio-compat being used it is now possible to use dependencies that
use tokio-0.2

Because `anyhow` is what seems to be the most common error library for
applications these days.

- The global `Result` type is now `anyhow::Result`.

- In errors.rs, there's no need for any boilerplate to wrap all the foreign
  errors seen: `hyper::Error`, `io:Error`, etc.

- The internal errors that we care about are now separate types, rather
  than within an enum, because that works better when we need to check for them
  by downcasting an `anyhow::Error`. And it's nice to write
  `Err(ProcessError(output))` rather than
  `Err(ErrorKind::ProcessError(output))`.

- The `Which` error was unused and is removed.

- The most common change is that `.chain_err()` is changed to
  `.context`/`.with_context()`.

- `anyhow!` is used where necessary, mostly to promote a string to an
  `anyhow::Error`.

- Errors within futures: `FutureChainErr`/`.chain_err()` is changed to
  `FutureContext`/`fcontext`/`fwith_context`. The `f` prefix is because I found
  it helpful to distinguish these cases from the simple error cases.

- `BuilderIncoming`, `SchedulerIncoming`, `ServerIncoming` no longer have an
  `Error` associated type, we just use `anyhow::Error` uniformly.

- `e.display_chain()` changes to `format!("{:?}")`, because they both print the
  full cause chain, and the backtrace (if present).

- A few places where the old code was doing something weird or more
  complicated than seemed necessary, I generally tried to replace it with
  something simpler and more typical. Two places used `with_boxed_chain()`,
  which doesn't have an equivalent in `anyhow`, so I did my best to do
  something reasonable.

- In `src/server.rs` we now import `std::task::Context` as `TaskContext` to
  avoid overshadowing the `anyhow::Context` trait :(

Change error handling from `error-chain` to `anyhow`.

We shouldn't contort and rename structs for the purpose of some trait
whose name doesn't matter in practice.

The current trimming of rlibs in the dist-sccache case looks like:

1. Find the `rust.metadata.bin` file in the rlib (archive).
2. Read its data.
3. Create a new archive that will only contain said file.
4. Copy the data into the new archive.

We can do better than this, by doing the following:

1. Find the `rust.metadata.bin` file in the rlib (archive).
2. Create a new archive that will only contain said file.
3. Copy the data from the original rlib directly into the new archive.

We thus save a copy.

All that being said, it looks like recent versions of Rust don't
actually output `rust.metadata.bin` files into rlibs...unless we ought
to be only taking the `lib.rmeta` file that lives inside the archive.
But that would be a separate fix.  (It's also great that we're seemingly
copying the rmeta file twice, once because it lives in the rlib and once
because it lives on disk as a separate file...)

Except redis, which pulls in async-std, which needs some further
consideration.

zstd is faster and gives slightly smaller (~5%) compressed blobs than
deflate does, as measured on a Firefox build.  Rather than inventing our
own compressed archive format, we piggyback on top of zip's "stored"
files to stuff zstd-compressed blobs in the zip archive.

This update should fix the BLAKE3 avx512 compilation issues.

0.1.2 was released earlier this year and includes the fix we need.

I found this using `cargo-udeps`.

This option was getting passed as a preprocessor argument and the
additional arguments that it requires (e.g. `-MT`) were getting passed
as dependency arguments.  So when we went to call the preprocessor, the
compiler complained about the missing arguments.  Instead, we should be
classifying it as specifying a dependency target, which will direct it
to the correct set of arguments.

We also take this opportunity to move it to be a clang-exclusive option;
the same option exists in GCC, but it's Darwin-specific and unlikely to
be actually used there.

mozilla#764 and its followon patch, mozilla#768, failed to take into account that
passthrough arguments actually need to be passed through to the compiler
invocation: they don't prevent caching, but they do need to be
considered when computing the cache key...and when actually performing
the compilation.

It feels like the obvious way would be to ask cmake to use "sccache
gcc" as CMAKE_C_COMPILER, but that option can't take arguments.  Leave
a note for the next person trying to figure it out.

And add enough to support -fprofile-generate.

The patch implements the TODO mentioned.

The server writes the port of on which it is listening to the socket
placed at SCCACHE_STARTUP_NOTIFY.  This port should match the port
expected by the client that requested the startup.

The corresponding environment variable is `SCCACHE_S3_KEY_PREFIX`.

* Add known caveats section to readme

These are pretty serious limitations that I would have liked to read about rather than discover later! I also added an example for `RUST_LOG` so you don't have to go digging in the docs to learn how to use it.

* clang: Allow args for arbitrary plugins

The plugin name may be accepted via Concatenated('-').
However, the following -Xclang argument should be passed through
as the value of the plugin argument. As a proof-of-concept,
identify the following Raw argument and pass it through.

Fixes mozilla#350.