Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tests] Compile BlocksRuntimeStub with stdlib compiler settings #34859

Merged

Conversation

drodriguez
Copy link
Contributor

@drodriguez drodriguez commented Nov 21, 2020

BlocksRuntimeStub was being build in the middle of test/CMakeLists.txt and using a stdlib function, however, the environment in test/CMakeLists.txt is not the same as the environment inside stdlib/ where that function is normally used. To avoid problems, BlocksRuntimeStub is moved into its own folder under stdlib/private/, where the compiler environment will be correct.

Most of the time this was working because the host compiler is modern enough to compile the simple stub in a compatible way, however for the case of Android using newer unified NDKs, the host compiler in Ubuntu 18.04 would not work.

As a step before the main change, some pieces of the test/CMakeLists.txt are extracted into helper functions in test/cmake/modules/SwiftTestUtils.cmake to allow them to be used in the new stdlib/private/BlocksRuntimeStub/CMakeLists.txt.

Also a small cleanup is done after the main change in which the indentation of test/CMakeLists.txt is corrected in a foreach block.

This is mostly moving code around to different files. It might be easier to check the changes per commit instead of as a whole. It might also be easier to check ignoring the white space changes (adding ?w=0 to the URL, IIRC).

@compnerd: you were the original author of the BlocksRuntimeStub bits. I think that besides using the compiler that the stdlib is using, these changes should not change anything.

@devincoughlin: this modified around the build flavor for macCatalyst. I think the changes are non-functional around those values, and nothing at all should change, but I would appreciate your feedback if something looks wrong.

@buttaface: this should allow the tests to compile with newer NDKs. I tried your changes on top of this and it the CI build-script invocation seems to perform the testing correctly.

@drodriguez
Copy link
Contributor Author

@swift-ci please test

@compnerd
Copy link
Member

Does it perhaps make sense to move the test library into the stdlib directory? That would reduce a significant amount of duplication, we just need to ensure that we clearly indicate that the BlocksRuntime here is for testing, so we could create it as test-BlocksRuntimeStubs, which follows the pattern for the other test tools as well. WDYT?

@drodriguez
Copy link
Contributor Author

Does it perhaps make sense to move the test library into the stdlib directory? That would reduce a significant amount of duplication, we just need to ensure that we clearly indicate that the BlocksRuntime here is for testing, so we could create it as test-BlocksRuntimeStubs, which follows the pattern for the other test tools as well. WDYT?

I can look into that. I looked a little bit there, but it wasn't very clear where it would have gone.

However, there will be a strange bit of the stdlib/ that will important helpers from the test/ directory, which seems "backwards" (while importing a little bit of the stdlib/ in the test/ directory doesn't sound like breaking the encapsulation that much).

In any case, I will look into it, and we can decide over the code if it looks right or not.

@compnerd
Copy link
Member

I think that stdlib/private would work, or perhaps we can just be even more explicitly - stdlib/test can also work well. I think that the latter has the advantage of being much easier to build only if tests are enabled.

Some pieces of the main CMakeLists.txt file are useful to future
CMakeLists.txt. In order to avoid duplication, move those pieces into a
new SwiftTestsUtils.cmake to be able to include the functions from other
files.

This should not modify anything in the behaviour of the build.
BlocksRuntimeStubs is built for the target SDK(s), but it was using one
of the stdlib CMake helpers without setting up the same environment as
the stdlib/ folder is setting.

To avoid problems, like BlocksRuntimeStubs being built with the host
compiler, instead of the just built compiler, and other difficult to
understand problems, create a folder for it in stdlib/private/. This
will allow the usage of the stdlib CMake function in the same environment
that is normally used.
Just adding indentation at the foreach block for the build_flavors,
which was not applying extra indentation.

This should not modify the behaviour of the build.
@drodriguez drodriguez force-pushed the test-blocks-runtime-stub-as-stdlib branch from abaa66d to 0de902e Compare November 24, 2020 23:33
@drodriguez
Copy link
Contributor Author

@swift-ci please test

@drodriguez drodriguez merged commit 517f4b7 into swiftlang:main Nov 30, 2020
@drodriguez drodriguez deleted the test-blocks-runtime-stub-as-stdlib branch November 30, 2020 17:24
@davezarzycki
Copy link
Contributor

This broke building without the blocks runtime (Fedora 33 x86-64):

FAIL: Swift(linux-x86_64) :: Reflection/capture_descriptors.sil (3318 of 13779)
******************** TEST 'Swift(linux-x86_64) :: Reflection/capture_descriptors.sil' FAILED ********************
Script:
--
: 'RUN: at line 7';   rm -rf "/home/dave/b/u/t/tools/swift/test-linux-x86_64/Reflection/Output/capture_descriptors.sil.tmp" && mkdir -p "/home/dave/b/u/t/tools/swift/test-linux-x86_64/Reflection/Output/capture_descriptors.sil.tmp"
: 'RUN: at line 8';   /home/dave/b/u/t/bin/swiftc -target x86_64-unknown-linux-gnu -toolchain-stdlib-rpath  -module-cache-path /home/dave/b/u/t/swift-test-results/x86_64-unknown-linux-gnu/clang-module-cache -swift-version 4  -Xfrontend -ignore-module-source-info  /home/dave/s/u/swift/test/Reflection/capture_descriptors.sil -emit-module -emit-library -module-name capture_descriptors -o /home/dave/b/u/t/tools/swift/test-linux-x86_64/Reflection/Output/capture_descriptors.sil.tmp/capture_descriptors.so -L/home/dave/b/u/t/tools/swift/test-linux-x86_64/Reflection/Output/capture_descriptors.sil.tmp/../../.. -lBlocksRuntime
: 'RUN: at line 9';   /home/dave/b/u/t/bin/swift-reflection-dump -arch x86_64 -binary-filename /home/dave/b/u/t/tools/swift/test-linux-x86_64/Reflection/Output/capture_descriptors.sil.tmp/capture_descriptors.so | /usr/bin/python3.9 /home/dave/s/u/swift/utils/PathSanitizingFileCheck --sanitize BUILD_DIR=/home/dave/b/u/t/tools/swift --sanitize SOURCE_DIR=/home/dave/s/u/swift --use-filecheck /home/dave/b/u/t/bin/FileCheck  /home/dave/s/u/swift/test/Reflection/capture_descriptors.sil
--
Exit Code: 1

Command Output (stderr):
--
/usr/bin/ld.gold: error: cannot find -lBlocksRuntime
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)

--

********************

@davezarzycki
Copy link
Contributor

There really ought to either be some CMake additions to verify that the blocks runtime is installed or the test harness needs a new "REQUIRES: block_runtime" mode.

@CodaFi
Copy link
Contributor

CodaFi commented Nov 30, 2020

@davezarzycki Could you add the appropriate gates?

@davezarzycki
Copy link
Contributor

I could but I'm not an expert at CMake or lit, so it'd take me some time. Also, I've already worked around the problem by installing the blocks runtime, so this won't be a high priority. If somebody else feels comfortable doing this and can, then please :-)

@drodriguez
Copy link
Contributor Author

@davezarzycki: so I imagine that you don't have a libBlocksRuntime.so in /home/dave/b/u/t/tools/swift/test-linux-x86_64/? (can you also check if it is somewhere in the build tree?)

From what I understand, as long as you are building the dynamic stdlib and the tests, the stubs should be build, and the -lBlocksRuntime should have found the library. This seems to try to avoid having BlocksRuntime installed at all.

This mostly changes the CMake files, but should not have changed anything the output in the build directory, so one should have expected the same results before and after.

PD: I might be misremembering, but you didn't use the build script, right? Can you share your build setup?

@davezarzycki
Copy link
Contributor

I don't build the blocks runtime nor do I normally have it installed, so I don't need to check. It's not there.

I just use cmake with a unified build setup (of the minimum set of projects). Nothing fancy.

@drodriguez
Copy link
Contributor Author

This is not about building or having Blocks Runtime installed. This is a library with an empty function stubbing the necessary pieces of Blocks Runtime that it is used during testing (precisely to avoid having to install or build blocks runtime). I also don't have Blocks Runtime installed in the system (but I am using Ubuntu).

These stubs should end up in test-linux-x86_64 as long as SWIFT_BUILD_STDLIB, SWIFT_BUILD_DYNAMIC_STDLIB, and SWIFT_INCLUDE_TESTS are true (I think those are enough, but there might be more), and there are configured SDKs. Good conditions will lead to the compilation happening in stdlib/private/BlocksRuntimeStubs/CMakeLists.txt. The results of that will be left in <build dir>/test-<os>-<architecture> as a dynamic library. Since you are telling me that's not there, I imagine some of the steps leading to there had failed. Can you imagine if some of those variables might be false?

About the unified build setup, since it is not very standard, and there might be many variants, do you have some document explaining it? I would try to setup a virtual machine and help, but without a setup to reproduce it feels that I am going to be losing time in unnecessary detours.

@davezarzycki
Copy link
Contributor

davezarzycki commented Dec 1, 2020

I see. I'm sorry, I hadn't realized that a stub library was/is being built. Now that I've checked, it seems like it lives in a different location than you asked about. So NOT here: /home/dave/b/u/t/tools/swift/test-linux-x86_64/ . It actually lives here: /home/dave/b/u/t/test-linux-x86_64

It also seems like the latter directory would be empty / not exist if it weren't for that library, so ya, it seems like the build system is confused and tools/swift isn't getting inserted while building up a file/directory path.

As to unified builds, it isn't that complicated in theory but in practice one can run into lots of gotchas. The short summary is that one creates a checkout of llvm-project and then one checks out cmark and swift inside of that. From there, you build LLVM like normal but with -DLLVM_EXTERNAL_PROJECTS="swift;cmark"

Oh, and Swift tends to refer to unified builds as being "not standalone" if that helps you better understand the build setup.

@davezarzycki
Copy link
Contributor

Oh, it just occurred to me that it's really easy when hacking on Swift's cmake setup (or really any LLVM derived build system) to assume/forget that the CMake output directories are not always the same as the project output directories. For example, SWIFT_BINARY_DIR and CMAKE_BINARY_DIR are the same when built standalone but different when built unified / not standalone.

@drodriguez
Copy link
Contributor Author

Oh, it just occurred to me that it's really easy when hacking on Swift's cmake setup (or really any LLVM derived build system) to assume/forget that the CMake output directories are not always the same as the project output directories. For example, SWIFT_BINARY_DIR and CMAKE_BINARY_DIR are the same when built standalone but different when built unified / not standalone.

That's probably the problem. Sorry for not realizing the values in unified builds change slightly. The original path was relative to CMAKE_CURRENT_BINARY_DIR, but when I was asked to move things into stdlib/private I didn't want to add a lot of ../.. in the path and switched to CMAKE_BINARY_DIR. I didn't know about SWIFT_BINARY_DIR. Sorry about that.

I am going to be away from the keyboard for a while, maybe without connectivity. If you can make that change in stdlib/private/BlocksRuntimeStubs/CMakeLists.txt it would be great. And again, sorry for the inconvenience.

davezarzycki added a commit to davezarzycki/swift that referenced this pull request Dec 1, 2020
ainu-bot added a commit to google/swift that referenced this pull request Dec 1, 2020
* 'main' of github.com:apple/swift:
  [test] Mark two SILOptimizer tests requiring asserts
  [test] Disable IRGen/unmanaged_objc_throw_func.swift with non-optimized stdlib
  [NFC] Move Migrated SDK Target List into StdlibDeploymentTarget
  [stdlib] Unbreak unified builds after swiftlang#34859
  Update a comment in EscapeAnalysis.
  [concurrency] IRGen: update task/executor/context on every suspend point
  [auto-diff] Fix a bunch of places in the *Cloners where we were not closing borrow scopes.
  [test] Adjusting stdlib ocurrences where class syntax is used for protocol inheritance
  [test] Adding specific tests for the warning for protocol inheritance class keyword syntax deprecation
  [test] Adjusting test files where class syntax is used for protocol inheritance
  [Sema] Adding deprecation warning for protocol inheritance class keyword syntax
  [stdlib] Simplify buffer pointer initialization.
  [cxx-interop] Bail on functions that use unimportable types.
  Sema: Remove a dead param on TypeChecker::conformsToProtocol
  [CSBinding] Attempt to join any existing and viable bindings with new binding
  [cmake] Semi-parametrize manpage location.
beccadax added a commit to beccadax/swift that referenced this pull request Dec 3, 2020
A CMake refactoring in swiftlang#34859 caused failures in Mac Catalyst configurations which are not tested in open source. This commit corrects these issues.

Fixes rdar://71897958.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants