Windows lld hangs when multi-threading is enabled

[darkspace]

In both NDK r18b and r19 beta1 for Windows 64-bit, ld.lld.exe will hang most of the time when linking our project (1078 source files). Possibly it's the same bug reported here:

https://bugs.llvm.org/show_bug.cgi?id=34806

Disabling threads prevents the hang but we would like multi-threading as link times are slow with our project without it. A snippet of CMakeLists.txt to workaround the hang and another unrelated issue with fix-cortex-a8:

set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fuse-ld=lld -Wl,--no-threads")

string(REPLACE "-Wl,--fix-cortex-a8" "" CMAKE_SHARED_LINKER_FLAGS ${CMAKE_SHARED_LINKER_FLAGS})
string(REPLACE "-Wl,--fix-cortex-a8" "" CMAKE_MODULE_LINKER_FLAGS ${CMAKE_MODULE_LINKER_FLAGS})
string(REPLACE "-Wl,--fix-cortex-a8" "" CMAKE_EXE_LINKER_FLAGS ${CMAKE_EXE_LINKER_FLAGS})

I downloaded and compiled the latest lld and it does not hang however is also does not appear to be multithreaded even after removing the --no-threads flag, as CPU usage remains at 25% on my quad-core during the link stage. Explicitly setting the --threads flag to lld didn't help. However it's possible I didn't compile it correctly with multi-threading enabled:

<installed LLVM tools, llvm-config.exe in my path>
<downloaded source from https://android.googlesource.com/toolchain/lld>
cd lld
mkdir build
cd build
cmake -G "Visual Studio 15 2017" -A x64 -T host=x64 ..
cmake --build . --target ALL_BUILD --config Release
copy Release\bin\ld.lld.exe \AndroidTools\android-ndk-r18b\toolchains\llvm\prebuilt\windows-x86_64\bin /Y

Environment:
Windows 10 64-bit
Intel i3 8300, 16GB ram, 250GB NVMe drive
ABI= tried armeabi-v7a, armeabi-v8a
Android build tools= 28.0.3
Android platform API= tried 21, 26, 28
Gradle= 4.6

[DanAlbert]

@pirama-arumuga-nainar @stephenhines the upstream the OP mentioned seems unlikely considering that it was marked fixed in October 2017... any thoughts on this?

[mstorsjo]

I believe this is related to this bug in winpthreads: https://sourceforge.net/p/mingw-w64/bugs/774/

It's pretty easy to work around within LLVM with a change like this:
mstorsjo/llvm@winpthreads-workaround
(Untested right now but recreated from memory since I debugged that issue.)

There's another similar bug with LLD threading on Windows (https://reviews.llvm.org/D53968), that only happens if the core LLVM libraries are linked in a different DLL than LLD, but that doesn't seem to be the case within the NDK so that's not the issue here.

The reason for winpthreads being involved at all is that libstdc++ requires it for implementing C++11 threads, which LLVM/LLD rely on.

A completely different approach, with larger effort of course, would be to build the NDK with a toolchain with libc++ instead of libstdc++. My own mingw toolchains at https://github.com/mstorsjo/llvm-mingw are set up that way (and use clang/lld and other llvm tools throughout the whole toolchain).

[pirama-arumuga-nainar]

A completely different approach, with larger effort of course, would be to build the NDK with a toolchain with libc++ instead of libstdc++. My own mingw toolchains at https://github.com/mstorsjo/llvm-mingw are set up that way (and use clang/lld and other llvm tools throughout the whole toolchain).

@mstorsjo Thanks for pointing this out. Didn't realize the connection to winpthreads. The windows Clang builds are still being built with gcc-4.8/libstdc++. My plan is to move to using Clang/libc++/win32-threads within this quarter. That should circumvent this issue.

[mstorsjo]

It would of course be good to verify that this really is the culprit before jumping to conclusions. But the symtoms do match (hanging sometimes but not always).

[DanAlbert]

We won't be getting a new toolchain in the next week, so moving to r21.

[rprichard]

The lld.exe in the clang-r353983c prebuilt uses libc++ instead of libstdc++/winpthreads, and I've verified that it works on a test program with --threads where it used to hang. I also verified with Process Explorer that it is actually creating worker threads.

Once the NDK upgrades to this toolchain, I think we can close this issue.

[mstorsjo]

If I read things correctly, NDK r20 does include the updated toolchain, and this bug should now be gone? (The NDK r20 changelog still included this as a known issue.)

[DanAlbert]

r20 has Clang r346389c, which is not new enough to have the fix for this. The reason we don't have an r21 beta out yet is actually because we are waiting for a new toolchain that has this fix, among other. r353983c mentioned above had too many regressions for us to ship, so we're waiting for the new new toolchain.

[mstorsjo]

Ah, sorry for the misread - I just read the last few chars of the Clang rev number, and mixed up "...983c" with "...389c". Ok, then things are clear.

[rprichard]

Here's a repro program with a run_test.bat script in it. It hangs on NDK r20.

lld_hang.zip

[DanAlbert]

Fix is in r21.