[FR] remove superfluous LLVM libraries

[ur4t]

Description

I am curious about how LLVM toolchain distributed with NDK are built and find some potential improvements in current versions (r26.2/r487747e and r28/r522817):

1. Plugin support in Clang was disabled to reduce binary size, but...
2. libclang.so, libclang-cpp.so and libLLVM-17.so are built, beacuse LLVM_BUILD_LLVM_DYLIB is ON since the build scripts are refactored, and...
3. Those shared libraries are not linked to any binaries because LLVM_LINK_LLVM_DYLIB is not set (OFF by default since introduced by llvm/llvm-project@9211396).
4. Currently the build scripts perform BOLT themselves, while upstream CMake has added BOLT support (llvm/llvm-project@3dab7fe and llvm/llvm-project@076240f).

By the way, I find LLVM_ENABLE_THREADS is explicitly defined as ON, which is default on most platforms supported by LLVM.

Those are not bugs because functionalities are not broken, so this issue is labeled with "enhancement". As for "plugin support in Clang", it would be nice to compile and load out-of-tree passes independently rather than compile the whole modified LLVM toolchain.

[pirama-arumuga-nainar]

Description

1. Plugin support in Clang was disabled to reduce binary size, but...
2. libclang.so, libclang-cpp.so and libLLVM-17.so are built, beacuse LLVM_BUILD_LLVM_DYLIB is ON since the build scripts are refactored, and...

Thanks for flagging this. The libclang and libLLVM libraries are used by clang-based tools used in Android platform build. @DanAlbert They can be removed from the ndk.

4. Currently the build scripts perform BOLT themselves, while upstream CMake has added BOLT support (llvm/llvm-project@3dab7fe and llvm/llvm-project@076240f).

This is by design. AOSP clang collects PGO and BOLT profiles when building Android so all backends get exercised and profiled. The CMake support only builds for the host architecture.

By the way, I find LLVM_ENABLE_THREADS is explicitly defined as ON, which is default on most platforms supported by LLVM.

https://r.android.com/3037873

As for "plugin support in Clang", it would be nice to compile and load out-of-tree passes independently rather than compile the whole modified LLVM toolchain.

For now, we plan to leave plugin support off considering the significant size reduction and build speedup, esp. for users on old hardware.

[DanAlbert]

https://android-review.googlesource.com/c/platform/ndk/+/3038133. Will cherry-pick to r27 since that also shaves about half a GB off the install size. We had some code that stripped those (the toolchain you give me has those unstripped for whatever reason), but they either moved or whoever wrote that code in the first place didn't test it, because it was stripping the wrong location :(

[pirama-arumuga-nainar]

I am not sure why the libraries are not stripped. Filed internal bug b/333762307 to track this down.

[ur4t]

@pirama-arumuga-nainar @DanAlbert Thanks for your kindly response introducing more NDK design perspectives!

FYI, when LLVM_LINK_LLVM_DYLIB is enabled, we have a dependency list:

components	dependencies
clang*	libclang-cpp.so and libLLVM.so
llvm* lld	libLLVM.so
lldb	liblldb.so
libclang.so liblldb.so	libclang-cpp.so and libLLVM.so
libclang-cpp.so	libLLVM.so

That means currently those bloated binaries and libraries, whose size are around 100MB and more, contain multiple copies of (at least part of) libclang-cpp.so and libLLVM.so.

With LLVM_LINK_LLVM_DYLIB enabled, we have:

• Most binaries, libLTO.so and libclang.so below 1MB;
• liblldb.so, clangd and clang-tidy around 20MB;
• libclang-cpp.so around 70MB.

So I believe enabling LLVM_LINK_LLVM_DYLIB is able to introduce more size reduction and build speedup (no more redundant static linking), which makes it negligible to enable plugin support in clang.

The problem is whether optimizers, especially BOLT, work with dynamic libraries. I have noticed that both upstream CMake and our build scripts only perform BOLT over the clang binary itself. Is it possible to perform BOLT over underlying libraries as well?

[DanAlbert]

A change in that policy would be up to @pirama-arumuga-nainar, as I wouldn't be the one paying the maintenance costs for it.

Closing as we've cleaned up the misleading mess, and there's currently no plan to add plugin support. If the toolchain team decides they want to take on the costs of plugin support, we'll get an FR filed to track it (or it'll just show up in the changelog).