Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bazel-0.22.0 regression: remote execution lost support for symlink? #7212

Closed
or-shachar opened this issue Jan 22, 2019 · 2 comments
Closed

bazel-0.22.0 regression: remote execution lost support for symlink? #7212

or-shachar opened this issue Jan 22, 2019 · 2 comments
Assignees
@philwo @buchgr
Labels
P0 This is an emergency and more important than other current work. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team untriaged

Comments

@or-shachar
Copy link
Contributor

or-shachar commented Jan 22, 2019 •
edited
Loading

Description of the problem:

We tested bazel 0.22.0 rc3 (#6494 ) with our repo and noticed a regression:
one of our resources jar (java_library without srcs) has a symlink in its resources to another place in the repo.
Until now it all worked fine. But on bazel 0.22.0.rc3 we started to fail with:

15:22:57 ERROR: /home/builduser/workspace/Test-bazel-version/run-bazel-custom-version/src/test/resources/BUILD.bazel:1:1: Building Java resource jar failed (Exit 1)
15:22:57 singlejar: external/bazel_tools/src/tools/singlejar/mapped_file_posix.inc:41: open src/test/resources/readme_link.txt:: No such file or directory
15:22:57 singlejar: external/bazel_tools/src/tools/singlejar/output_jar.cc:907: stat src/test/resources/readme_link.txt:: No such file or directory
15:22:57 singlejar: external/bazel_tools/src/tools/singlejar/output_jar.cc:941: src/test/resources/readme_link.txt: No such file or directory

The issue does not reproduce when building locally (without remote execution)

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I created a repro here but need to run it with remote execution configuration:
https://github.com/or-shachar/bazel-sample-repo (branch: symlink-check)

Try to test the target: //src/test/java/com/example:javatest

What operating system are you running Bazel on?

linux

What's the output of bazel info release?

bazel 0.22.0 rc3
@irengrig irengrig added P0 This is an emergency and more important than other current work. (Assignee required) untriaged team-Remote-Exec Issues and PRs for the Execution (Remote) team labels Jan 22, 2019
@or-shachar
Copy link
Contributor Author

@buchgr can we confirm it's a bug and not an intended change?
cc: @EladWeinson

@buchgr
Copy link
Contributor

buchgr commented Jan 22, 2019

@or-shachar thanks for reporting. It's a bug, I ll role that change back. You can use --incompatible_remote_symlinks=false to workaround this issue.

buchgr added a commit to buchgr/bazel that referenced this issue Jan 22, 2019
@buchgr
Revert "Enabling Bazel to generate input symlinks as defined by RE AP…
67b4065 
…I V2."

This reverts commit baa1786.

The symlink resolution in this change is broken, as it does not take into
account parent symlinks. Consider the following structure on the filesystem:

a/d/file
a/b/c/symlink -> ../../d/file

And action inputs as follows:

a/d/file
(f -> a/b/c)/symlink -> ../../d/file

with (f -> a/b/c) denoting that f is a symlink to directory c.

This change then builds the following merkle tree:

a
  d
    file
f
  symlink -> ../../d/file

My guesstimate is that there are a number of additional problems with
this change, as symlink resolution is unfortunately more complicated
than calling stat() on a symlink. See for example Bazel's symlink
resolution in FileFunction [1].

A real world example of this error is: bazelbuild#7212
Fixes bazelbuild#7212

[1] https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/skyframe/FileFunction.java;l=114?q=FileFunction
@buchgr buchgr mentioned this issue Jan 22, 2019
Closed
buchgr added a commit to buchgr/bazel that referenced this issue Jan 22, 2019
@buchgr
Revert "Enabling Bazel to generate input symlinks as defined by RE AP…
bbd456f 
…I V2."

This reverts commit baa1786.

The symlink resolution in this change is broken, as it does not take into
account parent symlinks. Consider the following structure on the filesystem:

a/d/file
a/b/c/symlink -> ../../d/file

And action inputs as follows:

a/d/file
(f -> a/b/c)/symlink -> ../../d/file

with (f -> a/b/c) denoting that f is a symlink to directory c.

This change then builds the following merkle tree:

a
  d
    file
f
  symlink -> ../../d/file

My guesstimate is that there are a number of additional problems with
this change and we should think hard about them when attempting to
roll foward this change in the future. See for example Bazel's symlink
resolution in FileFunction [1].

A real world example of this error is: bazelbuild#7212
Fixes bazelbuild#7212

[1] https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/skyframe/FileFunction.java;l=114?q=FileFunction
@buchgr buchgr mentioned this issue Jan 22, 2019
Closed
aehlig pushed a commit that referenced this issue Jan 23, 2019
@buchgr @aehlig
Revert "Enabling Bazel to generate input symlinks as defined by RE AP?
0449941 
?I V2."

This reverts commit baa1786.

The symlink resolution in this change is broken, as it does not take into
account parent symlinks. Consider the following structure on the filesystem:

```
a/d/file
a/b/c/symlink -> ../../d/file
```
And action inputs as follows:
```
a/d/file
(f -> a/b/c)/symlink -> ../../d/file
```
with (f -> a/b/c) denoting that f is a symlink to directory c.

This change then builds the following merkle tree:
```
a
  d
    file
f
  symlink -> ../../d/file
```
My guesstimate is that there are a number of additional problems with
this change and we should think hard about them when attempting to
roll foward this change in the future. See for example Bazel's symlink
resolution in FileFunction [1].

A real world example of this error is: #7212
Fixes #7212

[1] https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/skyframe/FileFunction.java;l=114?q=FileFunction

Closes #7216.

PiperOrigin-RevId: 230527829
aehlig pushed a commit that referenced this issue Jan 23, 2019
@buchgr @aehlig
Revert "Enabling Bazel to generate input symlinks as defined by RE AP?
7bafcec 
?I V2."

This reverts commit baa1786.

The symlink resolution in this change is broken, as it does not take into
account parent symlinks. Consider the following structure on the filesystem:

```
a/d/file
a/b/c/symlink -> ../../d/file
```
And action inputs as follows:
```
a/d/file
(f -> a/b/c)/symlink -> ../../d/file
```
with (f -> a/b/c) denoting that f is a symlink to directory c.

This change then builds the following merkle tree:
```
a
  d
    file
f
  symlink -> ../../d/file
```
My guesstimate is that there are a number of additional problems with
this change and we should think hard about them when attempting to
roll foward this change in the future. See for example Bazel's symlink
resolution in FileFunction [1].

A real world example of this error is: #7212
Fixes #7212

[1] https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/skyframe/FileFunction.java;l=114?q=FileFunction

Closes #7216.

PiperOrigin-RevId: 230527829
weixiao-huang pushed a commit to weixiao-huang/bazel that referenced this issue Jan 31, 2019
@buchgr @weixiao-huang
Revert "Enabling Bazel to generate input symlinks as defined by RE AP?
063eb0d 
?I V2."

This reverts commit baa1786.

The symlink resolution in this change is broken, as it does not take into
account parent symlinks. Consider the following structure on the filesystem:

```
a/d/file
a/b/c/symlink -> ../../d/file
```
And action inputs as follows:
```
a/d/file
(f -> a/b/c)/symlink -> ../../d/file
```
with (f -> a/b/c) denoting that f is a symlink to directory c.

This change then builds the following merkle tree:
```
a
  d
    file
f
  symlink -> ../../d/file
```
My guesstimate is that there are a number of additional problems with
this change and we should think hard about them when attempting to
roll foward this change in the future. See for example Bazel's symlink
resolution in FileFunction [1].

A real world example of this error is: bazelbuild#7212
Fixes bazelbuild#7212

[1] https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/skyframe/FileFunction.java;l=114?q=FileFunction

Closes bazelbuild#7216.

PiperOrigin-RevId: 230527829
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@philwo philwo

@buchgr buchgr

Labels
P0 This is an emergency and more important than other current work. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@philwo @buchgr @or-shachar @irengrig