[llvmlinux] "make test" for x86_64 target just hung there, why?

Daniel Sanders Daniel.Sanders at imgtec.com
Sat Sep 12 11:15:11 UTC 2015


> nice work on MIPS and the LLVM/Clang's integrated assembler works on MIPS :-).

Thanks. We still need to fix a few more things before LLVMLinux successfully builds using the integrated assembler but we're down to the last few issues now. Hopefully it will boot once it builds but I'm expecting to find more bugs at that point.

> I fell over your patch [1] as I "backported" it for Linux v4.1.
> Do you remember how you made that "inline" macros "empty"?
>
> You point exactly where I want to play with...
> There is a include/linux/compiler-gcc.h file where inlining is defined
> (see below) plus LLVMLinux has an own include/linux/compiler-clang.h
> file.

You've found the right place. It seems they aren't just inlining (I'd forgotten about notrace and __maybe_unused) so we don't want to make them completely empty. We just want to remove inline (and it's alternate underscored spellings) and the always_inline attribute from the expansion like this:
#define inline          inline          __attribute__((always_inline)) notrace __maybe_unused
becomes:
#define inline          notrace __maybe_unused
and likewise for all the spellings of the inline and always_inline macros.  Leave the noinline one alone though since that's preventing inlining for the trivial cases compilers tend to inline automatically.

I'm fairly certain it was compiler-gcc.h but you can double check by adding a '#error "foo"' to one of them and see if you get this forced error when you rebuild. If you do, then the header is included in your build. You can also use this technique to find the exact macro that gets defined by placing the #error next to it.

> Talking on #llvm (OFTC) with two guys (echristo and majnemer)...
> Both confirmed the defined behaviour in compiler-gcc.h seems to be
> correct, but they needed more context as they have no clue about the
> Linux-kernel :-).

I agree that compiler-gcc.h is correct. However, its current definition hinders debugging this kind of issue. We only want to change it temporarily to see if GCC can be made to see the same bug.

> From this discussion... LLVM/Clang's default behaviour is to do an
> "always_inline".

As far as I know, always_inline is always obeyed and has precedence over everything. The inline keyword is a bit softer. It will usually inline but the standard allows compilers to ignore it if they think it's a bad idea.

> I asked for a compiler-flag to change or disable that behaviour, but
> that seems not to exist.

There's -fno-inline but I found that the keywords and attributes in the source had precedence in LLVM.

> Before writing a LLVM/Clang bug-report I wanted to see if it is doable
> on the kernel-side.
>
> As said building a llvmlinux-patched Linux v4.2 with gcc v4.9 boots on
> my machine!
>
> As a conclusion, it looks like a LLVM/Clang (inline optimization) bug to me.

If it's like the bug I had, improving inlining would probably optimize the bug away like GCC did but the real bug would still be present in the kernel source. If disabling inlining as above causes the same bug in GCC then it's probably a kernel bug.
________________________________________
From: Sedat Dilek [sedat.dilek at gmail.com]
Sent: 12 September 2015 10:30
To: Daniel Sanders
Cc: pageexec at gmail.com; llvmlinux at lists.linuxfoundation.org; David Woodhouse
Subject: Re: [llvmlinux] "make test" for x86_64 target just hung there, why?

On Sat, Sep 12, 2015 at 5:36 AM, Sedat Dilek <sedat.dilek at gmail.com> wrote:
> On Fri, Sep 11, 2015 at 8:37 PM, Daniel Sanders
> <Daniel.Sanders at imgtec.com> wrote:
>> It could be that GCC does more inlining and optimises the bug away. This happened to me with the slab allocator initialization on the Mips arch.
>>
>> At the time, I was able to reproduce it in GCC by making the inline and always_inline macros empty. I don't recall the exact spelling of those macros, they may have additional underscores.
>>
>
> Hi Daniel,
>
> nice work on MIPS and the LLVM/Clang's integrated assembler works on MIPS :-).
>
> YUPP, the inline (always_inline) area is where I also think is the problem.
>
> I fell over your patch [1] as I "backported" it for Linux v4.1.
> Do you remember how you made that "inline" macros "empty"?
>
> You point exactly where I want to play with...
> There is a include/linux/compiler-gcc.h file where inlining is defined
> (see below) plus LLVMLinux has an own include/linux/compiler-clang.h
> file.
>
> [ include/linux/compiler-gcc.h ]
> ...
> /*
>  * Force always-inline if the user requests it so via the .config,
>  * or if gcc is too old:
>  */
> #if !defined(CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING) ||                \
>     !defined(CONFIG_OPTIMIZE_INLINING) || (__GNUC__ < 4)
> #define inline          inline          __attribute__((always_inline))
> notrace __maybe_unused
> #define __inline__      __inline__      __attribute__((always_inline))
> notrace __maybe_unused
> #define __inline        __inline        __attribute__((always_inline)) notrace
> #else
> /* A lot of inline functions can cause havoc with function tracing */
> #define inline          inline          notrace __maybe_unused
> #define __inline__      __inline__      notrace __maybe_unused
> #define __inline        __inline        notrace __maybe_unused
> #endif
>
> #define __always_inline inline __attribute__((always_inline))
> #define  noinline       __attribute__((noinline))
> ...
>
> I saw that for x86_64 arch setting CONFIG_OPTIMIZE_INLINING had no
> effect with clang v3.7 (lib/bitmap always BROKEN).
> x86_64 linux-arch has always CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y.
>
> So, how can I force one of the three inline "types" or make them "empty"?
> Where is the right place - in compiler-gcc.h or in compiler-clang.h file?
> Just for testing!
>
> Talking on #llvm (OFTC) with two guys (echristo and majnemer)...
> Both confirmed the defined behaviour in compiler-gcc.h seems to be
> correct, but they needed more context as they have no clue about the
> Linux-kernel :-).
> From this discussion... LLVM/Clang's default behaviour is to do an
> "always_inline".
> I asked for a compiler-flag to change or disable that behaviour, but
> that seems not to exist.
>
> Before writing a LLVM/Clang bug-report I wanted to see if it is doable
> on the kernel-side.
>
> As said building a llvmlinux-patched Linux v4.2 with gcc v4.9 boots on
> my machine!
>
> As a conclusion, it looks like a LLVM/Clang (inline optimization) bug to me.
>
> Please, give me some clear instructions on setting inline macros or
> how to make them "empty".
>
> Thanks.
>

When doing a change like this...

[ include/linux/bitmap.h ]
...
-extern int __bitmap_weight(const unsigned long *bitmap, unsigned int nbits);
+extern noinline int __bitmap_weight(const unsigned long *bitmap,
unsigned int nbits);
...

...I get the same objdump like with gcc v4.9...

$ objdump -drw lib/bitmap.o | grep bitmap_weight
0000000000000670 <__bitmap_weight>:
     681:       74 26                   je     6a9 <__bitmap_weight+0x39>
     693:       e8 00 00 00 00          callq  698
<__bitmap_weight+0x28>       694: R_X86_64_PC32
__sw_hweight64-0x4
     6a4:       75 ea                   jne    690 <__bitmap_weight+0x20>
     6ad:       74 1c                   je     6cb <__bitmap_weight+0x5b>
     6c2:       e8 00 00 00 00          callq  6c7
<__bitmap_weight+0x57>       6c3: R_X86_64_PC32
__sw_hweight64-0x4
     f11:       e8 00 00 00 00          callq  f16 <bitmap_remap+0x56>
 f12: R_X86_64_PC32      __bitmap_weight-0x4
     f5b:       e8 00 00 00 00          callq  f60 <bitmap_remap+0xa0>
 f5c: R_X86_64_PC32      __bitmap_weight-0x4
    104f:       e8 00 00 00 00          callq  1054
<bitmap_bitremap+0x24>      1050: R_X86_64_PC32
__bitmap_weight-0x4
    106e:       e8 00 00 00 00          callq  1073
<bitmap_bitremap+0x43>      106f: R_X86_64_PC32
__bitmap_weight-0x4

- Sedat -

> Regards,
> - Sedat -
>
> [1] http://git.linuxfoundation.org/?p=llvmlinux.git;a=blob_plain;f=arch/all/patches/ARCHIVE/correct-size_index-table-before-replacing-the-bootst.patch;hb=HEAD
> [2] http://git.linuxfoundation.org/?p=llvmlinux.git;a=blob;f=arch/all/patches/compiler-gcc.patch
>
>> Hope that helps.
>> ________________________________________
>> From: llvmlinux-bounces at lists.linuxfoundation.org [llvmlinux-bounces at lists.linuxfoundation.org] on behalf of Sedat Dilek [sedat.dilek at gmail.com]
>> Sent: 11 September 2015 17:10
>> To: pageexec at gmail.com
>> Cc: llvmlinux at lists.linuxfoundation.org; David Woodhouse
>> Subject: Re: [llvmlinux] "make test" for x86_64 target just hung there, why?
>>
>> On Wed, Sep 9, 2015 at 10:24 PM, PaX Team <pageexec at gmail.com> wrote:
>>> On 9 Sep 2015 at 12:12, Sedat Dilek wrote:
>>>
>>>> I can boot into a CLANG v3.7 compiled kernel when lib/bitmap is
>>>> compiled with GCC v4.9.
>>>
>>> are you sure it's not lib/hweight.o instead? under gcc it's compiled
>>> with special flags (CONFIG_ARCH_HWEIGHT_CFLAGS) which clang doesn't
>>> support and we used to patch that out but i have no idea about the
>>> current state of affairs.
>>>
>>
>> Hi pipacs :-),
>>
>> if I switch the compiler with the same llvmlinux-patchset and
>> kernel-config GCC v4.9 builds fine.
>> So, I am not sure if the CLANG v3.7 compiler is BROKEN when doing
>> (always-)inlining functions.
>>
>> Can you give some aid, how to verify if it is a COMPILER problem or a
>> kernel-side one?
>>
>> Regards,
>> - Sedat -
>> _______________________________________________
>> LLVMLinux mailing list
>> LLVMLinux at lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/llvmlinux


More information about the LLVMLinux mailing list