Buckd does not detect build file changes when symbolic links are involved

[davido]

Gerrit Code Review project uses Buck and can be extended with plugin concept. There are two plugin Buck build modes supported: in Gerrit tree and standalone (Bucklets driven).

I'm seeing weird stale artifact problem in tree plugin build mode: the plugin is only built once. After that, changing the sources doesn't trigger re-build of the artifact anymore. buck clean doesn't help either, only rm -rf buck-out.

I have a very small reproducer, but will try to describe the problem:

Main project directory structure is

gerrit
|--bucklets/gerrit_plugin.bucklet
|--plugins

Plugin:

foo
|--BUCK
|--src/main/[...]

The content of bucklets/gerrit_plugin.bucklet is:

def gerrit_library(
    name,
    srcs = []):
  java_library(
    name = name,
    srcs = srcs,
    visibility = ['PUBLIC'],
  )

The content of foo/BUCK is:

include_defs('//bucklets/gerrit_plugin.bucklet')

gerrit_library(
  name = 'foo',
  srcs = glob(['src/main/java/**/*.java']),
)

Now, when the plugin foo is linked to plugins directory with symbolic link, we can only build once and never again:

$ cd gerrit
$ buck build plugins/foo:foo
Using buckd.
[-] PROCESSING BUCK FILES...FINISHED 0,1s
[-] BUILDING...FINISHED 0,2s (1/1 JOBS, 1 UPDATED, 0,0% CACHE HITS)

Now when we change the source file: plugins/foo/src/main/java/org/ostrovsky/buck/Main.java and say introduce some compile errors, and re-run, nothing happens:

$ buck build plugins/foo:foo
Using buckd.
[-] PROCESSING BUCK FILES...FINISHED 0,1s
[-] BUILDING...FINISHED 0,0s (1/1 JOBS, 0 UPDATED)

buck clean doesn't help either. The only way to re-trigger the build is to drop buck-out directory:

$ rm -rf buck-out/
$ buck build plugins/foo:foo
Using buckd.
[-] PROCESSING BUCK FILES...FINISHED 0,1s
[+] BUILDING...0,0s (0/1 JOBS, 0 UPDATED)
 |=> //plugins/foo:foo...  0,0s (running javac[0,0s])
/home/davido/projects/buck_problem/plugins/foo/src/main/java/org/ostrovsky/buck/Main.java:21: error: illegal start of type
  42
  ^
/home/davido/projects/buck_problem/plugins/foo/src/main/java/org/ostrovsky/buck/Main.java:21: error: ';' expected
  42
    ^
Errors: 2. Warnings: 0.
BUILD FAILED: //plugins/foo:foo failed with exit code 1:

This problem only occurs, when foo is a symbolic link to some directory parallel to gerrit:

$ ls -all plugins/
lrwxrwxrwx 1 davido users    9 Jun 12 22:26 foo -> ../../foo

All is fine when foo is a real directory and not a symbolic link.

[davido]

cc @wez

So I had outdated watchman version installed on this machine: 2.9.8. Current is 3.2.0. Unfortunately, watchman upgrade havn't fixed this bug. However after disabling watchman all was working as expected:

$ buck build plugins/foo:foo
Not using buckd because watchman isn't installed.
[-] PROCESSING BUCK FILES...FINISHED 0,0s
[-] BUILDING...FINISHED 0,2s (1/1 JOBS)

Introduce compile error and re-run:

$ buck build plugins/foo:foo
Not using buckd because watchman isn't installed.
[-] PROCESSING BUCK FILES...FINISHED 0,0s
[+] BUILDING...0,3s (0/1 JOBS)
 |=> //plugins/foo:foo...  0,1s (running javac[0,1s])
/home/davido/projects/buck_problem/plugins/foo/src/main/java/org/ostrovsky/buck/Main.java:21: error: illegal start of type
  42
  ^

[wez]

Watchman does not follow symlinks. facebook/watchman#105 has some context. We have no plans to magically and generally make symlinks "work" because it seems way too complex to make it work as you might expect. We may be able to work something out that will make it easier for tools to deal with this situation though.

In the absence of any helpers, it sounds like buck should resolve symlink(s) and watch-project on the target path(s) and feed those subscriptions into its watcher implementation.

We'll need to brainstorm on this.

[Coneko]

We have logic in the parser to not use the in memory cache when symlinks are involved, but it only applies to input files, not to build files themselves.

https://github.com/facebook/buck/blob/master/src/com/facebook/buck/parser/Parser.java#L1138-L1150

This is something we should fix in Buck.

[zivkov]

I also hit this issue a couple of times when building Gerrit with plugins. Sometimes I want to just symlink a plugin from Gerrit's plugin folder instead of having to clone it inside the plugins folder. Unfortunately, this bug prevents me from doing that.

[somehibs]

I suppose the suggested workaround at present is to uninstall watchman when working on symblinked dependencies.

[Coneko]

You can set NO_BUCKD=1 as an environment variable to stop Buck from using watchman, then you won't need to uninstall it.

[davido]

Isn't the variable name is NO_BUCKD?

[Coneko]

Yes, NO_BUCKD is correct, I edited the comment.

[davido]

Was this issue fixed? After upgrading to 6a19cf9 we are getting this warning:

Disabling caching for target //plugins/wip:wip__plugin, because one or
more input files are under a symbolic link
({plugins/wip=/home/davido/projects/wip}). This will severely impact
performance! To resolve this, use separate rules and declare
dependencies instead of using symbolic links.

[Coneko]

It should be fixed in the sense that it will never be cached now, so you should be able to use buckd, but there's no way of disabling that warning if you want to keep that behaviour.

There is a setting to whitelist certain paths to be symlinks and be cached, but that's for symlinking into read only file systems.

[davido]

Thanks for fixing it.

[...] but there's no way of disabling that warning if you want to keep that behaviour.

Given that this is a normal workflow, at least in our use cases of Gerrit Code Review plugins, I would expect existence of some i_know_what_i_am_doing setting that silence those warnings.

[Coneko]

Oh sorry, my bad, there actually is.

It's project.allow_symlinks: https://buckbuild.com/concept/buckconfig.html#project.allow_symlinks .

[davido]

Unfortunately it doesn't silent the warning. If I'm reading the documentation correctly, the default for project.allow_symlinks is allow, but:

$ buck --version
buck version 6a19cf921c107ea0f1751e4f84c639e5050e2da7
$ watchman --version
4.3.0

And wip plugin is a symbolic link:

$ ls -all plugins/wip
lrwxrwxrwx 1 davido users 9 Aug 22 08:42 plugins/wip -> ../../wip

and i do not have the mentioned option in .buckconfig:

$ grep allow_symlinks .buckconfig

Still the warning is not suppressed during the build:

$ buck build plugins/wip
Waiting for Watchman query [[watch-project, /home/davido/projects/gerrit/.]]...
Disabling caching for target //plugins/wip:wip__plugin, because one or more input files are under a symbolic link ({plugins/wip=/home/davido/projects/wip}). This will severely impact performance! To resolve this, use separate rules and declare dependencies instead of using symbolic links.
[-] PROCESSING BUCK FILES...FINISHED 0.5s [100%] 🐳  Symlink caused cache invalidation
[-] DOWNLOADING... (0.00 B/S AVG, TOTAL: 0.00 B, 0 Artifacts)
[-] BUILDING...FINISHED 18.9s [100%] (259/259 JOBS, 258 UPDATED, 67 [25.9%] CACHE MISS)

To reproduce, clone Gerrit Code Review from here: [1] and Work In Progress plugin from here: [2], and link wip directory in gerrit's plugins directory.

[1] https://gerrit.googlesource.com/gerrit
[2] https://gerrit.googlesource.com/plugins/wip

[Coneko]

The default isn't allow, it's warn. Let me fix the docs.

[davido]

Thanks.

[jaimeagudo]

Docs arent' fixed yet guys #1189 Anyway it would be nice to solve this