Support NDK r19

[jin]

Expected release: Q4 2018

Changes: https://android.googlesource.com/platform/ndk/+/master/docs/Roadmap.md#ndk-r19

• Make all toolchains be standalone toolchains

Now that the NDK is down to a single compiler and STL, if we just taught the Clang driver to emit -D__ANDROID_API__=foo and to link libc.so.18 instead of libc.so, standalone toolchains would be obsolete because the compiler would already be a standalone toolchain. The NDK toolchain would Just Work regardless of build system, and the logic contained in each build system could be greatly reduced.

Related to this (but maybe occurring in a later release), we'll want to switch from libgcc to libcompiler-rt and our own unwinder.

See the corresponding bug make all toolchains standalone toolchains for detailed discussion of the implementation and sub-tasks.

[jin]

r19 beta 2 is available, r19 is expected to ship soon.

https://developer.android.com/ndk/downloads/

[jin]

NDK 19 has dropped: https://developer.android.com/ndk/downloads/#stable-downloads

[jin]

Link to changelog: https://github.com/android-ndk/ndk/wiki/Changelog-r19

[jin]

After speaking with @hlopko, the right way forward here is to implement the NDK CROSSTOOL in Starlark. This isn't a necessary blocker for NDK 19, since we can adapt the NDK 18 configuration for 19, but it would be nice to turn down proto CROSSTOOL generation to move to the modern approach.

@hlopko's suggestion:

write a rule ndk_cc_toolchain_config that has attributes for things that are different between versions, and only instantiate that in BUILD files.

More information:

• incompatible_disable_crosstool_file: Disable loading data from CROSSTOOL files #7320
• https://docs.bazel.build/versions/master/cc-toolchain-config-reference.html

[steeve]

@jin do you plan on supporting lld too ?

[jin]

@steeve definitely. We want the Android rules to be 100% compatible with the recommended set of tools in the NDK.

[jin]

The NDK CROSSTOOL has been ported to Starlark with massive help from @hlopko - thanks!

Going through the changelog:

Announcements

Developers should begin testing their apps with LLD. AOSP has switched to using LLD by default and the NDK will use it by default in the next release. BFD and Gold will be removed once LLD has been through a release cycle with no major unresolved issues (estimated r21). Test LLD in your app by passing -fuse-ld=lld when linking.

Note: lld does not currently support compressed symbols on Windows. See Issue 888. Clang also cannot generate compressed symbols on Windows, but this can be a problem when using artifacts built from Darwin or Linux.

We will need to add support for this.

The Play Store will require 64-bit support when uploading an APK beginning in August 2019. Start porting now to avoid surprises when the time comes. For more information, see this blog post.

We may want to add a warning message about this when building for non-64-bit.

Issue 780: Standalone toolchains are now unnecessary. Clang, binutils, the sysroot, and other toolchain pieces are now all installed to $NDK/toolchains/llvm/prebuilt/ and Clang will automatically find them. Instead of creating a standalone toolchain for API 26 ARM, instead invoke the compiler directly from the NDK:

$ $NDK/toolchains/llvm/prebuilt//bin/armv7a-linux-androideabi26-clang++ src.cpp
For r19 the toolchain is also installed to the old path to give build systems a chance to adapt to the new layout. The old paths will be removed in r20.

We must definitely support this, but the effects of it should be transparent to end users. It's not clear what the impact of having a standalone toolchain will be on the Starlark crosstool configuration.

The make_standalone_toolchain.py script will not be removed. It is now unnecessary and will emit a warning with the above information, but the script will remain to preserve existing workflows.

Not applicable, we don't use this.

If you're using ndk-build, CMake, or a standalone toolchain, there should be no change to your workflow. This change is meaningful for maintainers of third-party build systems, who should now be able to delete some Android-specific code. For more information, see the Build System Maintainers guide.

More reading: https://android.googlesource.com/platform/ndk/+/ndk-release-r19/docs/BuildSystemMaintainers.md

ndk-depends has been removed. We believe that ReLinker is a better solution to native library loading issues on old Android versions.

Not applicable, we don't use this.

Issue 862: The GCC wrapper scripts which redirected to Clang have been removed, as they are not functional enough to be drop in replacements.

This should be addressed by the standalone toolchain support.

---

r19c

• Issue 912: Prevent the CMake toolchain file from clobbering a user
  specified CMAKE_FIND_ROOT_PATH.
• Issue 920: Fix clang wrapper scripts on Windows.

r19b

• Issue 855: ndk-build automatically disables multithreaded linking for LLD
  on Windows, where it may hang. It is not possible for the NDK to detect this
  situation for CMake, so CMake users and custom build systems must pass
  -Wl,--no-threads when linking with LLD on Windows.

We'll need to take note of this for the lld support.

• Issue 849: Fixed unused command line argument warning when using standalone
  toolchains to compile C code.
• Issue 890: Fixed CMAKE_FIND_ROOT_PATH. CMake projects will no longer
  search the host's sysroot for headers and libraries.
• Issue 906: Explicitly set -march=armv7-a for 32-bit ARM to workaround
  Clang not setting that flag automatically when using -fno-integrated-as.
  This fix only affects ndk-build and CMake. Standalone toolchains and custom
  build systems will need to apply this fix themselves.
• Issue 907: Fixed find_path for NDK headers in CMake.

Changes

• Updated Clang to r339409.
  • C++ compilation now defaults to C++14.

Need to update default std to -std=c++14.

• [Issue 780]: A complete NDK toolchain is now installed to the Clang
  directory. See the announcements section for more information.

• ndk-build no longer removes artifacts from NDK_LIBS_OUT for ABIs not
  present in APP_ABI. This enables workflows like the following:

  for abi in armeabi-v7a arm64-v8a x86 x86_64; do
      ndk-build APP_ABI=$abi
  done

  Prior to this change, the above workflow would remove the previously built
  ABI's artifacts on each successive build, resulting in only x86_64 being
  present at the end of the loop.

• ndk-stack has been rewritten in Python.

• Issue 776: To better support LLD, ndk-build and CMake no longer pass
  -Wl,--fix-cortex-a8 by default.

  • CPUs that require this fix are uncommon in the NDK's supported API range
    (16+).
  • If you need to continue supporting these devices, add
    -Wl,--fix-cortex-a8 to your APP_LDFLAGS or CMAKE_C_FLAGS, but note
    that LLD will not be adding support for this workaround.
  • Alternatively, use the Play Console to blacklist Cortex-A8 CPUs to
    disallow your app from being installed on those devices.

Our ARM crosstool add this --fix-cortex-a8 linker flag, so we should drop them here.

src/main/java/com/google/devtools/build/lib/bazel/rules/android/ndkcrosstools/r18/ArmCrosstools.java
149:        .addLinkerFlag("-Wl,--fix-cortex-a8")

• Issue 798: The ndk-build and CMake options to disable RelRO and noexecstack
  are now ignored. All code is built with RelRO and non-executable stacks.

Not sure how/if we do this, will need to investigate.

• Issue 294: All code is now built with a subset of compiler-rt to provide
  a complete set of compiler built-ins for Clang.

Known Issues

• This is not intended to be a comprehensive list of all outstanding bugs.
• Issue 888: lld does not support compressed symbols on Windows. Clang also
  cannot generate compressed symbols on Windows, but this can be a problem when
  using artifacts built from Darwin or Linux.
• Issue 360: thread_local variables with non-trivial destructors will cause
  segfaults if the containing library is dlcloseed on devices running M or
  newer, or devices before M when using a static STL. The simple workaround is
  to not call dlclose.
• Issue 70838247: Gold emits broken debug information for AArch64. AArch64
  still uses BFD by default.
• Issue 855: LLD may hang on Windows when using multithreaded linking.
  ndk-build will automatically disable multithreaded linking in this situation,
  but CMake users and custom build systems should pass -Wl,--no-threads when
  using LLD on Windows. The other linkers and operating systems are unaffected.

Addressed above.

• Issue 884: Third-party build systems must pass -fno-addrsig to Clang for
  compatibility with binutils. ndk-build, CMake, and standalone toolchains
  handle this automatically.

Looks like standalone toolchains support this, but will be good to verify.

• Issue 906: Clang does not pass -march=armv7-a to the assembler when using
  -fno-integrated-as. This results in the assembler generating ARMv5
  instructions. Note that by default Clang uses the integrated assembler which
  does not have this problem. To workaround this issue, explicitly use
  -march=armv7-a when building for 32-bit ARM with the non-integrated
  assembler, or use the integrated assembler. ndk-build and CMake already
  contain these workarounds.

We'll need to add this workaround.

• This version of the NDK is incompatible with the Android Gradle plugin
  version 3.0 or older. If you see an error like
  No toolchains found in the NDK toolchains folder for ABI with prefix: mips64el-linux-android,
  update your project file to use plugin version 3.1 or newer. You will also
  need to upgrade to Android Studio 3.1 or newer.

[jin]

Addressed in #8524