[llvmlinux] "make test" for x86_64 target just hung there, why?

Sedat Dilek sedat.dilek at gmail.com
Sat Sep 12 11:29:16 UTC 2015


On Sat, Sep 12, 2015 at 1:15 PM, Daniel Sanders
<Daniel.Sanders at imgtec.com> wrote:
>> nice work on MIPS and the LLVM/Clang's integrated assembler works on MIPS :-).
>
> Thanks. We still need to fix a few more things before LLVMLinux successfully builds using the integrated assembler but we're down to the last few issues now. Hopefully it will boot once it builds but I'm expecting to find more bugs at that point.
>
>> I fell over your patch [1] as I "backported" it for Linux v4.1.
>> Do you remember how you made that "inline" macros "empty"?
>>
>> You point exactly where I want to play with...
>> There is a include/linux/compiler-gcc.h file where inlining is defined
>> (see below) plus LLVMLinux has an own include/linux/compiler-clang.h
>> file.
>
> You've found the right place. It seems they aren't just inlining (I'd forgotten about notrace and __maybe_unused) so we don't want to make them completely empty. We just want to remove inline (and it's alternate underscored spellings) and the always_inline attribute from the expansion like this:
> #define inline          inline          __attribute__((always_inline)) notrace __maybe_unused
> becomes:
> #define inline          notrace __maybe_unused
> and likewise for all the spellings of the inline and always_inline macros.  Leave the noinline one alone though since that's preventing inlining for the trivial cases compilers tend to inline automatically.
>
> I'm fairly certain it was compiler-gcc.h but you can double check by adding a '#error "foo"' to one of them and see if you get this forced error when you rebuild. If you do, then the header is included in your build. You can also use this technique to find the exact macro that gets defined by placing the #error next to it.
>

Thanks for your answer, I will try that.

Just found this doc (see [1]):

"C99 inline functions

By default, Clang builds C code in GNU C11 mode, so it uses standard
C99 semantics for the inline keyword. These semantics are different
from those in GNU C89 mode, which is the default mode in versions of
GCC prior to 5.0. For example, consider the following code:

inline int add(int i, int j) { return i + j; }

int main() {
  int i = add(4, 5);
  return i;
}

In C99, inline means that a function's definition is provided only for
inlining, and that there is another definition (without inline)
somewhere else in the program. That means that this program is
incomplete, because if add isn't inlined (for example, when compiling
without optimization), then main will have an unresolved reference to
that other definition. Therefore we'll get a (correct) link-time error
like this:

Undefined symbols:
  "_add", referenced from:
      _main in cc-y1jXIr.o

By contrast, GNU C89 mode (used by default in older versions of GCC)
is the C89 standard plus a lot of extensions. C89 doesn't have an
inline keyword, but GCC recognizes it as an extension and just treats
it as a hint to the optimizer.

There are several ways to fix this problem:

(1) Change add to a static inline function. This is usually the right
solution if only one translation unit needs to use the function.
static inline functions are always resolved within the translation
unit, so you won't have to add a non-inline definition of the function
elsewhere in your program.

(2) Remove the inline keyword from this definition of add. The inline
keyword is not required for a function to be inlined, nor does it
guarantee that it will be. Some compilers ignore it completely. Clang
treats it as a mild suggestion from the programmer.

(3) Provide an external (non-inline) definition of add somewhere else
in your program. The two definitions must be equivalent!

(4) Compile in the GNU C89 dialect by adding -std=gnu89 to the set of
Clang options. This option is only recommended if the program source
cannot be changed or if the program also relies on additional
C89-specific behavior that cannot be changed.

All of this only applies to C code; the meaning of inline in C++ is
very different from its meaning in either GNU89 or C99."

I played with all (1)..(4).

So, from kernel-side -std=gnu89 [2] is set in main Makefile which is
equivalent to add -fgnu89-inline.
I tried also to re-compile - generate bitmap.o - with passing
-std=gnu99 which is default behaviour for LLVM/Clang's inlining.

Not sure if I have to non-inline both __bitmap_weight() and
bitmap_weight() in include/linux/bitmap.h.
Unfortunately, all changes to bitmap.h requires a rebuild of my
Linux-kernel :-(.

Still playing...

Thanks again.

- Sedat -


[1] http://clang.llvm.org/compatibility.html#inline
[2] https://gcc.gnu.org/onlinedocs/gcc/Inline.html


>> Talking on #llvm (OFTC) with two guys (echristo and majnemer)...
>> Both confirmed the defined behaviour in compiler-gcc.h seems to be
>> correct, but they needed more context as they have no clue about the
>> Linux-kernel :-).
>
> I agree that compiler-gcc.h is correct. However, its current definition hinders debugging this kind of issue. We only want to change it temporarily to see if GCC can be made to see the same bug.
>
>> From this discussion... LLVM/Clang's default behaviour is to do an
>> "always_inline".
>
> As far as I know, always_inline is always obeyed and has precedence over everything. The inline keyword is a bit softer. It will usually inline but the standard allows compilers to ignore it if they think it's a bad idea.
>
>> I asked for a compiler-flag to change or disable that behaviour, but
>> that seems not to exist.
>
> There's -fno-inline but I found that the keywords and attributes in the source had precedence in LLVM.
>
>> Before writing a LLVM/Clang bug-report I wanted to see if it is doable
>> on the kernel-side.
>>
>> As said building a llvmlinux-patched Linux v4.2 with gcc v4.9 boots on
>> my machine!
>>
>> As a conclusion, it looks like a LLVM/Clang (inline optimization) bug to me.
>
> If it's like the bug I had, improving inlining would probably optimize the bug away like GCC did but the real bug would still be present in the kernel source. If disabling inlining as above causes the same bug in GCC then it's probably a kernel bug.
> ________________________________________
> From: Sedat Dilek [sedat.dilek at gmail.com]
> Sent: 12 September 2015 10:30
> To: Daniel Sanders
> Cc: pageexec at gmail.com; llvmlinux at lists.linuxfoundation.org; David Woodhouse
> Subject: Re: [llvmlinux] "make test" for x86_64 target just hung there, why?
>
> On Sat, Sep 12, 2015 at 5:36 AM, Sedat Dilek <sedat.dilek at gmail.com> wrote:
>> On Fri, Sep 11, 2015 at 8:37 PM, Daniel Sanders
>> <Daniel.Sanders at imgtec.com> wrote:
>>> It could be that GCC does more inlining and optimises the bug away. This happened to me with the slab allocator initialization on the Mips arch.
>>>
>>> At the time, I was able to reproduce it in GCC by making the inline and always_inline macros empty. I don't recall the exact spelling of those macros, they may have additional underscores.
>>>
>>
>> Hi Daniel,
>>
>> nice work on MIPS and the LLVM/Clang's integrated assembler works on MIPS :-).
>>
>> YUPP, the inline (always_inline) area is where I also think is the problem.
>>
>> I fell over your patch [1] as I "backported" it for Linux v4.1.
>> Do you remember how you made that "inline" macros "empty"?
>>
>> You point exactly where I want to play with...
>> There is a include/linux/compiler-gcc.h file where inlining is defined
>> (see below) plus LLVMLinux has an own include/linux/compiler-clang.h
>> file.
>>
>> [ include/linux/compiler-gcc.h ]
>> ...
>> /*
>>  * Force always-inline if the user requests it so via the .config,
>>  * or if gcc is too old:
>>  */
>> #if !defined(CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING) ||                \
>>     !defined(CONFIG_OPTIMIZE_INLINING) || (__GNUC__ < 4)
>> #define inline          inline          __attribute__((always_inline))
>> notrace __maybe_unused
>> #define __inline__      __inline__      __attribute__((always_inline))
>> notrace __maybe_unused
>> #define __inline        __inline        __attribute__((always_inline)) notrace
>> #else
>> /* A lot of inline functions can cause havoc with function tracing */
>> #define inline          inline          notrace __maybe_unused
>> #define __inline__      __inline__      notrace __maybe_unused
>> #define __inline        __inline        notrace __maybe_unused
>> #endif
>>
>> #define __always_inline inline __attribute__((always_inline))
>> #define  noinline       __attribute__((noinline))
>> ...
>>
>> I saw that for x86_64 arch setting CONFIG_OPTIMIZE_INLINING had no
>> effect with clang v3.7 (lib/bitmap always BROKEN).
>> x86_64 linux-arch has always CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y.
>>
>> So, how can I force one of the three inline "types" or make them "empty"?
>> Where is the right place - in compiler-gcc.h or in compiler-clang.h file?
>> Just for testing!
>>
>> Talking on #llvm (OFTC) with two guys (echristo and majnemer)...
>> Both confirmed the defined behaviour in compiler-gcc.h seems to be
>> correct, but they needed more context as they have no clue about the
>> Linux-kernel :-).
>> From this discussion... LLVM/Clang's default behaviour is to do an
>> "always_inline".
>> I asked for a compiler-flag to change or disable that behaviour, but
>> that seems not to exist.
>>
>> Before writing a LLVM/Clang bug-report I wanted to see if it is doable
>> on the kernel-side.
>>
>> As said building a llvmlinux-patched Linux v4.2 with gcc v4.9 boots on
>> my machine!
>>
>> As a conclusion, it looks like a LLVM/Clang (inline optimization) bug to me.
>>
>> Please, give me some clear instructions on setting inline macros or
>> how to make them "empty".
>>
>> Thanks.
>>
>
> When doing a change like this...
>
> [ include/linux/bitmap.h ]
> ...
> -extern int __bitmap_weight(const unsigned long *bitmap, unsigned int nbits);
> +extern noinline int __bitmap_weight(const unsigned long *bitmap,
> unsigned int nbits);
> ...
>
> ...I get the same objdump like with gcc v4.9...
>
> $ objdump -drw lib/bitmap.o | grep bitmap_weight
> 0000000000000670 <__bitmap_weight>:
>      681:       74 26                   je     6a9 <__bitmap_weight+0x39>
>      693:       e8 00 00 00 00          callq  698
> <__bitmap_weight+0x28>       694: R_X86_64_PC32
> __sw_hweight64-0x4
>      6a4:       75 ea                   jne    690 <__bitmap_weight+0x20>
>      6ad:       74 1c                   je     6cb <__bitmap_weight+0x5b>
>      6c2:       e8 00 00 00 00          callq  6c7
> <__bitmap_weight+0x57>       6c3: R_X86_64_PC32
> __sw_hweight64-0x4
>      f11:       e8 00 00 00 00          callq  f16 <bitmap_remap+0x56>
>  f12: R_X86_64_PC32      __bitmap_weight-0x4
>      f5b:       e8 00 00 00 00          callq  f60 <bitmap_remap+0xa0>
>  f5c: R_X86_64_PC32      __bitmap_weight-0x4
>     104f:       e8 00 00 00 00          callq  1054
> <bitmap_bitremap+0x24>      1050: R_X86_64_PC32
> __bitmap_weight-0x4
>     106e:       e8 00 00 00 00          callq  1073
> <bitmap_bitremap+0x43>      106f: R_X86_64_PC32
> __bitmap_weight-0x4
>
> - Sedat -
>
>> Regards,
>> - Sedat -
>>
>> [1] http://git.linuxfoundation.org/?p=llvmlinux.git;a=blob_plain;f=arch/all/patches/ARCHIVE/correct-size_index-table-before-replacing-the-bootst.patch;hb=HEAD
>> [2] http://git.linuxfoundation.org/?p=llvmlinux.git;a=blob;f=arch/all/patches/compiler-gcc.patch
>>
>>> Hope that helps.
>>> ________________________________________
>>> From: llvmlinux-bounces at lists.linuxfoundation.org [llvmlinux-bounces at lists.linuxfoundation.org] on behalf of Sedat Dilek [sedat.dilek at gmail.com]
>>> Sent: 11 September 2015 17:10
>>> To: pageexec at gmail.com
>>> Cc: llvmlinux at lists.linuxfoundation.org; David Woodhouse
>>> Subject: Re: [llvmlinux] "make test" for x86_64 target just hung there, why?
>>>
>>> On Wed, Sep 9, 2015 at 10:24 PM, PaX Team <pageexec at gmail.com> wrote:
>>>> On 9 Sep 2015 at 12:12, Sedat Dilek wrote:
>>>>
>>>>> I can boot into a CLANG v3.7 compiled kernel when lib/bitmap is
>>>>> compiled with GCC v4.9.
>>>>
>>>> are you sure it's not lib/hweight.o instead? under gcc it's compiled
>>>> with special flags (CONFIG_ARCH_HWEIGHT_CFLAGS) which clang doesn't
>>>> support and we used to patch that out but i have no idea about the
>>>> current state of affairs.
>>>>
>>>
>>> Hi pipacs :-),
>>>
>>> if I switch the compiler with the same llvmlinux-patchset and
>>> kernel-config GCC v4.9 builds fine.
>>> So, I am not sure if the CLANG v3.7 compiler is BROKEN when doing
>>> (always-)inlining functions.
>>>
>>> Can you give some aid, how to verify if it is a COMPILER problem or a
>>> kernel-side one?
>>>
>>> Regards,
>>> - Sedat -
>>> _______________________________________________
>>> LLVMLinux mailing list
>>> LLVMLinux at lists.linuxfoundation.org
>>> https://lists.linuxfoundation.org/mailman/listinfo/llvmlinux


More information about the LLVMLinux mailing list