Respect main repository mapping in `platform_mappings` #17133

fmeum · 2023-01-04T06:30:28Z

fmeum · 2023-01-04T10:51:43Z

I'm rather clueless about the remaining test failures as these tests don't use non-trivial repository mappings:

Unrecoverable error while evaluating node 'PACKAGE:external' (requested by nodes 'Key{repoName=@, rootModuleShouldSeeWorkspaceRepos=true}')
java.lang.RuntimeException: 
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:657)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:382)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: The Package for external had a different set of targets (<targetsInPkg> - <targetsInNewlyLoadedPkg> = [], <targetsInNewlyLoadedPkg> - <targetsInPkg> = [//external:android_gmaven_r8, //external:com_google_protobuf, //external:local_config_xcode, //external:rules_java]) when loaded normally during execution of the current test than it did when loaded via BazelPackageLoader (done automatically by the BazelPackageLoadingListenerForTesting hook). This either means: (i) Skyframe package loading semantics have diverged from BazelPackageLoader semantics (ii) The test in question is doing something that confuses BazelPackageLoadingListenerForTesting.
	at com.google.devtools.build.lib.testutil.BazelPackageLoadingListenerForTesting.sanityCheckBazelPackageLoader(BazelPackageLoadingListenerForTesting.java:104)
	at com.google.devtools.build.lib.testutil.BazelPackageLoadingListenerForTesting.onLoadingCompleteAndSuccessful(BazelPackageLoadingListenerForTesting.java:65)
	at com.google.devtools.build.lib.packages.PackageFactory.afterDoneLoadingPackage(PackageFactory.java:583)
	at com.google.devtools.build.lib.skyframe.PackageFunction.getExternalPackage(PackageFunction.java:356)
	at com.google.devtools.build.lib.skyframe.PackageFunction.compute(PackageFunction.java:395)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:575)
	... 4 more

Wyverald · 2023-01-04T13:23:06Z

last time I tried to fix this, I ran into some circular dependency problems in Bazel's test setup code. Don't remember exactly what was in the cycle, but I gave up fairly quickly and thought nobody would be using platform mappings anyway. Guess I was wrong :P

fmeum · 2023-01-04T13:25:57Z

I do rely on them for macOS cross-compilation. I will take another stab at fixing these failures.

fmeum · 2023-01-09T17:17:30Z

@katre Do you happen to have an idea what to do about the test failures?

katre · 2023-01-09T17:20:18Z

Not offhand, but I'll try and take a look and see if I can figure anything out.

brentleyjones · 2023-02-21T22:28:28Z

Friendly bump on this.

katre · 2023-02-27T19:26:42Z

Okay, I rebased this to the current master and looked at the failures.

The core problem is that the expected error (in the case of StarlarkRuleContextTest.testLoadBlockRepositoryRedefinition, the error for redefining a repository) is being swallowed by a different error, and either the message is dropped or the type is changed.

I don't know why this PR caused a change in error handling: possibly things are being evaluated earlier? The fix is probably to handle the errors properly, but I don't know how.

@Wyverald, any ideas?

brentleyjones · 2023-08-04T19:19:53Z

Ping 🙏

Wyverald · 2023-08-04T19:40:16Z

I'll try to carve some time next week to look at this again. AFAIK what was hard about it last time (see above) is still hard today (see test failures...).

fmeum · 2023-08-07T10:54:16Z

I got this down to a single type of failure:

1) testMappingFileIsRead_fromAlternatePackagePath(com.google.devtools.build.lib.skyframe.PlatformMappingFunctionTest)
java.lang.AssertionError: ERROR /DEFAULT.WORKSPACE:11:17: fetching local_repository rule //external:rules_java_builtin: java.io.IOException: The repository's path is "/workspace/rules_java" (absolute: "/workspace/rules_java") but it does not exist or is not a directory.
	at org.junit.Assert.fail(Assert.java:89)
	at com.google.devtools.build.lib.testutil.FoundationTestCase$1.handle(FoundationTestCase.java:61)
	at com.google.devtools.build.lib.events.Reporter.handle(Reporter.java:127)
	at com.google.devtools.build.lib.events.Event.reportTo(Event.java:71)
	at com.google.devtools.build.skyframe.ParallelEvaluatorContext$NestedSetEventReceiver.accept(ParallelEvaluatorContext.java:210)
	at com.google.devtools.build.skyframe.ParallelEvaluatorContext$NestedSetEventReceiver.accept(ParallelEvaluatorContext.java:200)
	at com.google.devtools.build.lib.collect.nestedset.NestedSetVisitor.visitRaw(NestedSetVisitor.java:77)
	at com.google.devtools.build.lib.collect.nestedset.NestedSetVisitor.visit(NestedSetVisitor.java:56)
	at com.google.devtools.build.skyframe.SkyFunctionEnvironment.reportEventsAndGetEventsToStore(SkyFunctionEnvironment.java:317)
	at com.google.devtools.build.skyframe.SkyFunctionEnvironment.commitAndGetParents(SkyFunctionEnvironment.java:882)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:582)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:399)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

@meteorcloudy This looks like something you may know how to solve.

meteorcloudy · 2023-08-07T12:45:48Z

This is because for the new workspace, it's missing setup from https://cs.opensource.google/bazel/bazel/+/refs/heads/master:src/test/java/com/google/devtools/build/lib/analysis/util/BuildViewTestCase.java;l=405-408

I tired to reuse the WORKSPACE file, it seems working:

diff --git a/src/test/java/com/google/devtools/build/lib/skyframe/PlatformMappingFunctionTest.java b/src/test/java/com/google/devtools/build/lib/skyframe/PlatformMappingFunctionTest.java
index 7de265990e..d9ff4cbfb5 100644
--- a/src/test/java/com/google/devtools/build/lib/skyframe/PlatformMappingFunctionTest.java
+++ b/src/test/java/com/google/devtools/build/lib/skyframe/PlatformMappingFunctionTest.java
@@ -127,7 +127,7 @@ public final class PlatformMappingFunctionTest extends BuildViewTestCase {
   @Test
   public void testMappingFileIsRead_fromAlternatePackagePath() throws Exception {
     scratch.setWorkingDir("/other/package/path");
-    scratch.file("WORKSPACE");
+    scratch.copyFile(rootDirectory.getRelative("WORKSPACE").getPathString(), "WORKSPACE");
     setPackageOptions("--package_path=/other/package/path");
     scratch.file(
         "my_mapping_file",

fmeum · 2023-08-07T12:56:57Z

@meteorcloudy Thanks, that fixed the test.

I could of course add that line to

bazel/src/test/java/com/google/devtools/build/lib/skyframe/PlatformMappingFunctionTest.java

Line 150 in 36d0275

public void handlesNoWorkspaceFile() throws Exception {

but the whole point of that test seems to be that there is no WORKSPACE file. I am not sure how this can work, but I also don't know anything about --package_path. Can this test be dropped?

meteorcloudy · 2023-08-07T13:02:41Z

I believe --package_path is never meant for Bazel, but is used internally. /cc @justinhorvitz @lberki

Could adding a --override_repository=rules_java_builtin=/workspace/rules_java_workspace work?

fmeum · 2023-08-21T11:41:06Z

Uh, sorry. I don't know anything off the bat. My best idea would be useLoadingOptions("--override_repository=rules_java_builtin=<some scratch file system path>"), but I assume you tried that already?

I did, that fails with Unrecognized option: ....

How come this becomes a problem with this pull request, but is not one at HEAD? I have only given the change the most cursory of glances, but it looks like it doesn't add anything new about Java, but then how come that test works now, but this pull request makes it require a @rules_java_builtin repository?

This happens because PlatformMappingFunction now depends on RepositoryMappingFunction to get the main repository mapping, which in turn triggers the evaluation of the WORKSPACE file.

lberki · 2023-08-21T12:09:30Z

Oh, I see. That's unfortunate. I don't have any better idea than disabling the test :(

If I really wanted to, I'd poke at SkyframeExecutor to see if it's possible to inject a mock WORKSPACE value, but then it'll get further away from production code... so I vote for disabling the test and adding a comment that tells why that is.

fmeum · 2023-08-21T13:55:49Z

@Wyverald With @lberki's okay, this should be ready to review and merge.

Wyverald · 2023-08-24T21:30:40Z

src/main/java/com/google/devtools/build/lib/analysis/config/CoreOptionConverters.java

@@ -232,12 +232,19 @@ private static Label convertOptionsLabel(String input, @Nullable Object conversi
        return Label.parseCanonical(input);
      }
      Preconditions.checkArgument(
-          conversionContext instanceof RepositoryMapping,
+          conversionContext instanceof RepositoryMapping


quick question -- this change shouldn't be necessary, right? everything in the platform mapping file should be anchored to the main repo? (it looks like you're passing around a mainRepoContext that always has RepositoryName.MAIN, instead of a mainRepoMapping; I think the latter would be less cognitive burden.)

I found it confusing that RepoContext is not a valid conversionContext, but RepositoryMapping is - it doesn't matter that much though and RepoContext does have redundant information, so I reverted this part.

fmeum · 2023-09-07T10:50:22Z

@Wyverald Friendly ping

Wyverald

thanks for fixing this! and sorry for the long delay.

I'm still not entirely sure how you worked around the nasty dependency cycles in the tests, but for now I'll just be happy that you did and move along :)

fmeum · 2023-09-07T19:49:39Z

I'm still not entirely sure how you worked around the nasty dependency cycles in the tests, but for now I'll just be happy that you did and move along :)

Glad that you aren't sure since I have no idea :-D I suspect that something changed on master between my two attempts. This may blow up when we try to cherry-pick it.

brentleyjones · 2023-09-08T18:45:36Z

@bazel-io flag

iancha1992 · 2023-09-08T20:47:47Z

@bazel-io fork 6.4.0

iancha1992 · 2023-09-11T22:45:54Z

@fmeum @brentleyjones @Wyverald @lberki @katre @meteorcloudy We tried to cherry-pick this change to release-6.4.0. However, there were some conflicts. Can you please take a look?

src/main/java/com/google/devtools/build/lib/skyframe/PlatformMappingFunction.java
We currently have in release-6.4.0 branch,

static Mappings parse(List<String> lines) throws PlatformMappingException {

However, we expected the below before the cherry-picking

static Mappings parse(Environment env, List<String> lines) throws PlatformMappingException {

src/main/java/com/google/devtools/build/lib/skyframe/PlatformMappingFunction.java

if (platformsToFlags == null) {
  return null;
}

is missing, but should be present already in the release-6.4.0 branch before cherry-picking

src/main/java/com/google/devtools/build/lib/skyframe/PlatformMappingFunction.java

private static class SkyframeTargetLoader implements StarlarkOptionsParser.BuildSettingLoader

is missing, but should be present already in the release-6.4.0 branch before cherry-picking

src/main/java/com/google/devtools/build/lib/skyframe/PlatformMappingValue.java
In the current release-6.4.0 branch, we have below

private OptionsParsingResult parse(Iterable<String> args) throws OptionsParsingException {

But we expected below before the cherry-picking

private OptionsParsingResult parse(NativeAndStarlarkFlags args) throws OptionsParsingException {

src/test/java/com/google/devtools/build/lib/analysis/RunfilesRepoMappingManifestTest.java

public void actionRerunsOnRepoMappingChange_workspaceName() throws Exception {

should already be in the release-6.4.0 branch.

src/test/java/com/google/devtools/build/lib/skyframe/PlatformMappingFunctionParserTest.java
We currently have below in the release-6.4.0 branch

return PlatformMappingFunction.parse(ImmutableList.copyOf(lines));

But, expected below before cherry-picking

return PlatformMappingFunction.parse(/* env= */ null, ImmutableList.copyOf(lines));

src/test/java/com/google/devtools/build/lib/skyframe/PlatformMappingFunctionTest.java

public void starlarkFlagMapping() throws Exception {

is currently missing in the release-6.4.0 branch

src/test/java/com/google/devtools/build/lib/skyframe/PlatformMappingValueTest.java
We currently have below in the release-6.4.0 branch

private static final Label PLATFORM1 = Label.parseAbsoluteUnchecked("//platforms:one");
private static final Label PLATFORM2 = Label.parseAbsoluteUnchecked("//platforms:two");

But we expected below before cherry-picking

private static final Label PLATFORM1 = Label.parseCanonicalUnchecked("//platforms:one");
  private static final Label PLATFORM2 = Label.parseCanonicalUnchecked("//platforms:two");

cc: @bazelbuild/triage

Fixes bazelbuild#17127 Closes bazelbuild#17133. PiperOrigin-RevId: 563803704 Change-Id: I0f49fbfce624207b81a6ca4f7e564d9f3b525d54

fmeum · 2023-09-12T09:15:56Z

@iancha1992 I cherry-picked this manually in #19495.

fmeum mentioned this pull request Jan 4, 2023

Repo names in platform_mappings aren't handled with bzlmod #17127

Closed

sgowroji added team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. area-Bzlmod Bzlmod-specific PRs, issues, and feature requests awaiting-review PR is awaiting review from an assigned reviewer labels Jan 4, 2023

fmeum force-pushed the 17127-repo-mapped-platform-mapping branch 3 times, most recently from 9239013 to ad0b3d4 Compare January 4, 2023 10:43

fmeum mentioned this pull request Jan 9, 2023

platform_mappings file repo mapping is literal in some cases #17171

Closed

meteorcloudy requested a review from Wyverald January 17, 2023 13:25

fmeum force-pushed the 17127-repo-mapped-platform-mapping branch from ad0b3d4 to 2fb16a0 Compare July 20, 2023 13:10

fmeum requested a review from a team as a code owner July 20, 2023 13:10

fmeum requested review from aranguyen and removed request for a team July 20, 2023 13:10

fmeum force-pushed the 17127-repo-mapped-platform-mapping branch from 2fb16a0 to 76382a2 Compare August 1, 2023 16:13

fmeum force-pushed the 17127-repo-mapped-platform-mapping branch 2 times, most recently from bbb840e to c78c5a1 Compare August 7, 2023 10:30

fmeum force-pushed the 17127-repo-mapped-platform-mapping branch from c78c5a1 to 36d0275 Compare August 7, 2023 12:54

fmeum removed the request for review from aranguyen August 21, 2023 13:55

Wyverald reviewed Aug 24, 2023

View reviewed changes

fmeum force-pushed the 17127-repo-mapped-platform-mapping branch from 1aea2de to c9d7cb0 Compare August 25, 2023 07:44

Respect main repository mapping in platform_mappings

d4a88cc

fmeum requested a review from Wyverald August 25, 2023 07:57

fmeum force-pushed the 17127-repo-mapped-platform-mapping branch from c9d7cb0 to d4a88cc Compare August 25, 2023 07:57

lberki mentioned this pull request Sep 7, 2023

Flip --incompatible_enable_cc_toolchain_resolution #19441

Closed

Wyverald approved these changes Sep 7, 2023

View reviewed changes

Wyverald added awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally and removed awaiting-review PR is awaiting review from an assigned reviewer labels Sep 7, 2023

meteorcloudy mentioned this pull request Sep 8, 2023

Register apple toolchains bazelbuild/rules_swift#1106

Closed

copybara-service bot closed this in c0fc433 Sep 8, 2023

github-actions bot removed the awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally label Sep 8, 2023

bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Sep 8, 2023

fmeum deleted the 17127-repo-mapped-platform-mapping branch September 8, 2023 20:26

bazel-io removed the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Sep 8, 2023

bazel-io mentioned this pull request Sep 8, 2023

[6.4.0] Respect main repository mapping in platform_mappings #19469

Closed

fmeum added a commit to fmeum/bazel that referenced this pull request Sep 12, 2023

Respect main repository mapping in platform_mappings

7e655a7

Fixes bazelbuild#17127 Closes bazelbuild#17133. PiperOrigin-RevId: 563803704 Change-Id: I0f49fbfce624207b81a6ca4f7e564d9f3b525d54

fmeum mentioned this pull request Sep 12, 2023

[6.4.0] Respect main repository mapping in platform_mappings #19495

Closed

Wyverald added this to the 7.0.0 branch cut milestone Sep 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Respect main repository mapping in `platform_mappings` #17133

Respect main repository mapping in `platform_mappings` #17133

fmeum commented Jan 4, 2023

fmeum commented Jan 4, 2023 •

edited

Loading

Wyverald commented Jan 4, 2023

fmeum commented Jan 4, 2023

fmeum commented Jan 9, 2023

katre commented Jan 9, 2023

brentleyjones commented Feb 21, 2023

katre commented Feb 27, 2023

brentleyjones commented Aug 4, 2023

Wyverald commented Aug 4, 2023

fmeum commented Aug 7, 2023

meteorcloudy commented Aug 7, 2023

fmeum commented Aug 7, 2023

meteorcloudy commented Aug 7, 2023 •

edited

Loading

fmeum commented Aug 21, 2023

lberki commented Aug 21, 2023

fmeum commented Aug 21, 2023

Wyverald Aug 24, 2023

fmeum Aug 25, 2023

fmeum commented Sep 7, 2023

Wyverald left a comment

fmeum commented Sep 7, 2023

brentleyjones commented Sep 8, 2023

iancha1992 commented Sep 8, 2023

iancha1992 commented Sep 11, 2023

fmeum commented Sep 12, 2023

Respect main repository mapping in platform_mappings #17133

Respect main repository mapping in platform_mappings #17133

Conversation

fmeum commented Jan 4, 2023

fmeum commented Jan 4, 2023 • edited Loading

Wyverald commented Jan 4, 2023

fmeum commented Jan 4, 2023

fmeum commented Jan 9, 2023

katre commented Jan 9, 2023

brentleyjones commented Feb 21, 2023

katre commented Feb 27, 2023

brentleyjones commented Aug 4, 2023

Wyverald commented Aug 4, 2023

fmeum commented Aug 7, 2023

meteorcloudy commented Aug 7, 2023

fmeum commented Aug 7, 2023

meteorcloudy commented Aug 7, 2023 • edited Loading

fmeum commented Aug 21, 2023

lberki commented Aug 21, 2023

fmeum commented Aug 21, 2023

Wyverald Aug 24, 2023

Choose a reason for hiding this comment

fmeum Aug 25, 2023

Choose a reason for hiding this comment

fmeum commented Sep 7, 2023

Wyverald left a comment

Choose a reason for hiding this comment

fmeum commented Sep 7, 2023

brentleyjones commented Sep 8, 2023

iancha1992 commented Sep 8, 2023

iancha1992 commented Sep 11, 2023

fmeum commented Sep 12, 2023

Respect main repository mapping in `platform_mappings` #17133

Respect main repository mapping in `platform_mappings` #17133

fmeum commented Jan 4, 2023 •

edited

Loading

meteorcloudy commented Aug 7, 2023 •

edited

Loading