node_modules being backed up despite being tmignored (?)

[borekb]

I've been using tmignore happily in the past, it always worked well but in the recent backup, I see several node_modules using the BackupLoupe utility:

The yarn-install-demo is actually this public repo: https://github.com/borekb/yarn-install-demo where you can see two .gitignore files:

• root .gitignore:
• packages-v2/.gitignore:

I did a quick tmignore check which looks alright:

$ tmignore list | rg 'yarn-install-demo'
  - /Users/borekb/dev/borekb/yarn-install-demo/packages-v1/app/node_modules/
  - /Users/borekb/dev/borekb/yarn-install-demo/packages-v1/lib/node_modules/
  - /Users/borekb/dev/borekb/yarn-install-demo/packages-v1/node_modules/
  - /Users/borekb/dev/borekb/yarn-install-demo/packages-v2/.yarn/build-state.yml
  - /Users/borekb/dev/borekb/yarn-install-demo/packages-v2/.yarn/cache/
  - /Users/borekb/dev/borekb/yarn-install-demo/packages-v2/.yarn/install-state.gz
  - /Users/borekb/dev/borekb/yarn-install-demo/packages-v2/app/node_modules/
  - /Users/borekb/dev/borekb/yarn-install-demo/packages-v2/lib/node_modules/
  - /Users/borekb/dev/borekb/yarn-install-demo/packages-v2/node_modules/

How comes that .yarn cache files are excluded from the backup but node_modules aren't? I also ran this, which again contained only .yarn/* entries and no node_modules entries:

$ find /Users/borekb/dev/borekb/yarn-install-demo -exec tmutil isexcluded {} + | grep -F "[Excluded]" | sed -E 's/^\[Excluded\][[:space:]]*//'
/System/Volumes/Data/Users/borekb/dev/borekb/yarn-install-demo/packages-v2/.yarn/cache
/System/Volumes/Data/Users/borekb/dev/borekb/yarn-install-demo/packages-v2/.yarn/cache/fb-watchman-npm-2.0.0-98fd97d6b7-192298457e.zip
/System/Volumes/Data/Users/borekb/dev/borekb/yarn-install-demo/packages-v2/.yarn/cache/elegant-spinner-npm-1.0.1-8b799f39a6-69837a8a88.zip
...

(The yarn-install-demo folder on my disk is about 3 days old while the Time Machine backup was run 1 hour ago, so there shouldn't be any problem with the node_modules folder not being known to tmignore yet.)

Any ideas?

[wderezin]

What pattern is in your .gitignore file? I just experienced the same thing with a .terraform directory.

Here are the 2 entries from my .gitignore

**/.terraform/*     # tmignore didn't match
.terraform            # tmignore works.
/**/.terraform/*    # tmignore works

[borekb]

@wderezin The two .gitignore files in my repo are linked above.

[wderezin]

@borekb Could this be your issue below?

For me if I recreate the node_modules (delete/create) the exclusion doesn't stick. You can verify if the file is excluded by using tmutil iscluded. Running tmignore reset put all my exclusions back.

If tmignore added the -p option when doing the tmutil addexclusion This will make the behavior of the exclusion equal the expected behavior of the gitignore.

Currently tmignore is using this mode:
The first kind of exclusion, which is the default behavior for the addexclusion verb, is a location-independent
("sticky") exclusion that follows a file or directory. When the file or directory is moved, the exclusion goes
with the item to the new location. Additionally, when the item is copied, the copy retains the exclusion.

Switch to this mode (-p)
The second kind of exclusion is a fixed-path exclusion. With this, you tell Time Machine that you want a specific
path to be excluded, agnostic of the item at that path. If there is no file or directory at the specified path,
the exclusion has no effect; if the item previously at the path has been moved or renamed, the item is not
excluded, because it does not currently reside at the excluded path. As a consequence of these semantics, moving a
file or directory to the path will cause the item to be excluded--fixed-path exclusions are not automatically
cleaned up when items are moved or deleted and will take effect again once an item exists at an excluded path.

This is the line that needs to be updated:

tmignore/tmignore/TimeMachine.swift

Line 20 in 8f359a4

_ = try exec(program: "/usr/bin/tmutil", arguments: ["addexclusion"] + pathChunk)

[borekb]

@wderezin Interesting.. I didn't have time to experiment with this further but are you saying that tmignore always had the "problem" I reported, i.e., removing and re-adding node_modules (for example, running rm -rf node_modules && yarn) made it not-ignored?

I don't know because I think I've used BackupLoupe to explore my snapshots in the past and they seemed to be without node_modules. And if I understand you correctly, tmignore always worked in the "non -p" manner.

[wderezin]

@borekb Correct that is what I'm experiencing. And is consistent with the tmutil man page for -p. I also am using BackupLoupe to confirm that items where backed up after a delete/add. I am running macOS Big Sure Version 11 so I'm not sure if this functionality changed. I can run another test with detail logs and screen shots if you want.

[borekb]

@wderezin Just to make sure I understand: when you do e.g. cd /example && rm -rf node_modules && npm install, is /example/node_modules tm-ignored or not on your computer?