Mark temporary directories as excluded from macOS Spotlight #8686

kornelski · 2020-09-07T02:32:36Z

macOS doesn't support CACHEDIR.TAG, and Cargo doesn't use correct temporary dir locations for macOS. As a workaround, directories can be excluded from macOS Spotlight indexing by adding .noindex suffix.

I've added the suffix only to build and incremental. Ideally .fingerprint and deps should have it too, but deps appears in user-visible paths and .fingerprint seems to be an experimental public API, so I haven't changed these two.

This may be better served by #8063

rust-highfive · 2020-09-07T02:32:39Z

r? @ehuss

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · 2020-09-08T17:16:37Z

We've historically had friction trying to rename build directories due to how baked in the current structure is to other external tools, but is there perhaps a way to exclude items from spotlight with file attributes rather than by filename? (something like what we do for time machine for example)

joshtriplett · 2020-09-09T20:21:34Z

Yeah, having to use the name for this seems painful.

Might be preferable to put it in one of the standard macOS directories that get excluded, or putting it somewhere in the .cargo directory (since those don't get indexed). We've already talked about using .cargo for that for other reasons (shared caches and similar).

joshtriplett · 2020-09-09T20:33:17Z

Another possibility: if there's any concern that something might be looking in these directories, we could use build.noindex and incremental.noindex and similar and then symlink build -> build.noindex and incremental -> incremental.noindex.

kornelski · 2020-09-09T23:15:19Z

@alexcrichton AFAIK there's no filesystem attribute for this. .noindex is the only still-working method that I've found, and it's pretty obscure. It looks like Apple really doesn't want to support temporary files anywhere other than the dedicated system temp/cache directories.

kornelski · 2020-09-09T23:16:18Z

Symlinks sound like an easy way out. How about taking it further and symlinking to ~/Library/Caches/Cargo/<hash of the project directory>?

tesuji · 2020-09-10T01:18:49Z

This doesn't affect old installation on macOS, right ?

alexcrichton · 2020-09-14T17:53:20Z

It's true that a possible alternative is to place everything in the system default location, but Cargo isn't necessarily the best prepared for that. One thing I think we'd need to handle is cargo clean should be sure to have basically the same behavior that it has today. Another worry we've had in the past is that we hard-link a lot of files in Cargo between intermediate locations and their final location. By moving across things like your home directory we run the risk of more frequently spanning volumes, causing hard links to fail. This is mostly only an issue for Linux, though, since I think on macOS multi-volume home directories is much rarer.

Overall this is unfortunately a tricky problem in that renaming build directories is ripe for breakage. When we discussed this the idea of symlinks seemed like they could possibly fix the issue, though, so I think it might be worthwhile to explore what a solution might look like where caches are placed in more system-friendly directories (e.g. ~/Library/Caches/Cargo on macOS, and the equivalents on Windows and Linux) and symlinked into the same locations we've used historically.

alexcrichton · 2020-09-14T17:54:05Z

Or well there's also always the gotcha that symlinks are difficult/different on Windows. I think we can get away with "directory junctions" regardless of user preferences, which may work for our purposes as well, but we'd have to investigate.

kornelski · 2020-09-15T02:35:26Z

Thank you for the input. Here's the plan:

I'll make existing target folders keep everything as-is (real directories) to improve back compat and ease migration.
I'll limit this change to cfg(unix) to avoid the issues of directory junctions. This will make it a platform-dependent behaviour, but Windows is a differently-behaving platform.
New target/ directories will have symlinks to a subdirectory in system's temp if it's on the same device. Otherwise keeps existing behavior.
To keep this change small, the the subdirectory in temp will be unique per workspace to avoid adding new form of sharing of build artifacts (these dirs could be unified later)
I'll verify that cargo clean can clean inside the symlinked folders.

joshtriplett · 2020-09-15T03:01:29Z

On Mon, Sep 14, 2020 at 07:35:39PM -0700, Kornel wrote: Thank you for the input. Here's the plan: * I'll make existing target folders keep everything as-is (real directories) to improve back compat and ease migration.

When you upgrade Rust, you're already going to rebuild everything and you can't do an incremental build, so replacing the build and incremental directories seems perfectly fine.

* New target/ directories will have symlinks to a subdirectory in system's temp *if* it's on the same device. Otherwise keeps existing behavior.

Only for build and incremental, right? Also, if the system's temp directory is not on the same device, how about using adjacent `build.noindex` and `incremental.noindex` directories as the symlink targets?

* To keep this change small, the the subdirectory in temp will be unique per workspace to avoid adding new form of sharing of build artifacts (these dirs could be unified later)

You mentioned using a hash, but please also prefix it with a crate name or similar, for future troubleshooting, so that it's possible to figure out roughly where it came from.

alexcrichton · 2020-09-15T14:13:20Z

That sounds about right I believe yeah. A possible further simplification, though, is that target/debug itself could point to a system-stored location. That way we don't have to worry about cross-device links and such (and we could opportunistically try it on Windows, handling errors and making a directory if it doesn't work). That way everything, not just some folders, would reside in the temporary directory location, but it should look the same to all existing users except that the top-level debug and release folders would actually reside somewhere else.

We'll still want to test this though since env::current_exe() may return a path which no longer resides in the project directory, which may break some projects. We can always tweak if necessary if that comes up though!

kornelski · 2020-09-15T17:56:53Z

target/debug symlink would be a bit easier, but personally I think it's good to separate "build products" (cargo calls them targets) from "build intermediaries", so I think target/debug/$binary belongs where it is, since it's only a handful of files and they shouldn't suddenly disappear when the OS thinks it's time to reclaim disk space. The thousands of other temp files are the problem.

alexcrichton · 2020-09-16T00:44:34Z

I think deps may fill up with a lot of unused stuff as well, but in any case this seems like it's ripe for experimentation to see what works best!

bors · 2020-09-27T09:16:06Z

☔ The latest upstream changes (presumably #8739) made this pull request unmergeable. Please resolve the merge conflicts.

Note that reviewers usually do not review pull requests until merge conflicts are resolved! Once you resolve the conflicts, you should change the labels applied by bors to indicate that your PR is ready for review. Post this as a comment to change the labels:

@rustbot modify labels: +S-waiting-on-review -S-waiting-on-author

soc · 2020-10-12T14:14:36Z

src/cargo/core/compiler/layout.rs

@@ -170,6 +172,7 @@ impl Layout {
            fingerprint: dest.join(".fingerprint"),
            examples: dest.join("examples"),
            doc: root.join("doc"),
+            temp_root: Self::temp_root_path(&root),


The naming is a bit confusing, because cache and temp are usually different things and people people have different conceptions about them.

soc · 2020-10-12T14:16:39Z

src/cargo/util/paths.rs

+        if path.exists() {
+            return Some(path.join("cargo"));
+        }
+        None


Not sure I understand this logic – is there any reason why it ignores the "important" part of XDG and instead does something completely different altogether in the "env not set" case?

More generally, reimplementing the directory-selection logic seems to be a straight-forward route to bikeshed city (as demonstrated in the RFC).

What "important" part do you mean?

I am afraid that touching XDG is going to attract bikeshed fiesta. I'm tempted to remove it for that reason, and go straight for /var/tmp instead.

What "important" part do you mean?

XDG defines default directories if the env var is not set. The env var is probably not set in 99% of the cases, because people are fine with the defaults.

Only "being XDG" when the env var is set ... if I wanted to troll people, that's how I would do it. ;-)

I am afraid that touching XDG is going to attract bikeshed fiesta.

Did you consider simply using a library?

I'm tempted to remove it for that reason, and go straight for /var/tmp instead.

That's a cheap way out for this special case, but I'm not sure you want the next person following this precedent when they try to fix the issue for config/data files.

I've considered using a library, but it this area is full of abandoned projects and half-abandoned forks, so I didn't see any suitable crate.

I think the best option would be to use a cargo-specific variable/setting for customizing this dir.

I've considered using a library, but it this area is full of abandoned projects and half-abandoned forks, so I didn't see any suitable crate.

I think no crate can help you if your intention is to invent your own non-standard scheme anyway.

I think the best option would be to use a cargo-specific variable/setting for customizing this dir.

That's of course technically possible, but is does nothing to resolve rust-lang/rfcs#1615 which is about sane defaults.

I think no crate can help you if your intention is to invent your own non-standard scheme anyway.

Jabs like that are unhelpful. Please don't do that.

Have a nice day.

rust-highfive assigned ehuss Sep 7, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 7, 2020

kornelski force-pushed the tempchurn branch from 107229d to aace08f Compare September 7, 2020 12:02

kornelski marked this pull request as draft September 7, 2020 12:46

kornelski force-pushed the tempchurn branch from aace08f to 5304748 Compare September 9, 2020 23:18

kornelski added 2 commits October 7, 2020 16:25

Add persistent_temp_path

ca648bd

Use system temp directory on unix

b172d53

kornelski force-pushed the tempchurn branch from 5304748 to b172d53 Compare October 10, 2020 19:51

kornelski mentioned this pull request Oct 11, 2020

Let Cargo put data into platform-specific directories rust-lang/rfcs#1615

Closed

soc reviewed Oct 12, 2020

View reviewed changes

kornelski mentioned this pull request Oct 14, 2020

Configurable directory for temp files that isn't the target dir #6227

Open

kornelski closed this Feb 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mark temporary directories as excluded from macOS Spotlight #8686

Mark temporary directories as excluded from macOS Spotlight #8686

kornelski commented Sep 7, 2020 •

edited

Loading

rust-highfive commented Sep 7, 2020

alexcrichton commented Sep 8, 2020

joshtriplett commented Sep 9, 2020 •

edited

Loading

joshtriplett commented Sep 9, 2020

kornelski commented Sep 9, 2020

kornelski commented Sep 9, 2020

tesuji commented Sep 10, 2020

alexcrichton commented Sep 14, 2020

alexcrichton commented Sep 14, 2020

kornelski commented Sep 15, 2020

joshtriplett commented Sep 15, 2020 via email

alexcrichton commented Sep 15, 2020

kornelski commented Sep 15, 2020 •

edited

Loading

alexcrichton commented Sep 16, 2020

bors commented Sep 27, 2020

soc Oct 12, 2020 •

edited

Loading

soc Oct 12, 2020 •

edited

Loading

kornelski Oct 12, 2020

soc Oct 12, 2020 •

edited

Loading

kornelski Oct 13, 2020

soc Oct 13, 2020

kornelski Oct 13, 2020

soc Oct 13, 2020

Mark temporary directories as excluded from macOS Spotlight #8686

Mark temporary directories as excluded from macOS Spotlight #8686

Conversation

kornelski commented Sep 7, 2020 • edited Loading

rust-highfive commented Sep 7, 2020

alexcrichton commented Sep 8, 2020

joshtriplett commented Sep 9, 2020 • edited Loading

joshtriplett commented Sep 9, 2020

kornelski commented Sep 9, 2020

kornelski commented Sep 9, 2020

tesuji commented Sep 10, 2020

alexcrichton commented Sep 14, 2020

alexcrichton commented Sep 14, 2020

kornelski commented Sep 15, 2020

joshtriplett commented Sep 15, 2020 via email

alexcrichton commented Sep 15, 2020

kornelski commented Sep 15, 2020 • edited Loading

alexcrichton commented Sep 16, 2020

bors commented Sep 27, 2020

soc Oct 12, 2020 • edited Loading

Choose a reason for hiding this comment

soc Oct 12, 2020 • edited Loading

Choose a reason for hiding this comment

kornelski Oct 12, 2020

Choose a reason for hiding this comment

soc Oct 12, 2020 • edited Loading

Choose a reason for hiding this comment

kornelski Oct 13, 2020

Choose a reason for hiding this comment

soc Oct 13, 2020

Choose a reason for hiding this comment

kornelski Oct 13, 2020

Choose a reason for hiding this comment

soc Oct 13, 2020

Choose a reason for hiding this comment

kornelski commented Sep 7, 2020 •

edited

Loading

joshtriplett commented Sep 9, 2020 •

edited

Loading

kornelski commented Sep 15, 2020 •

edited

Loading

soc Oct 12, 2020 •

edited

Loading

soc Oct 12, 2020 •

edited

Loading

soc Oct 12, 2020 •

edited

Loading