[Ksummit-2013-discuss] [ATTEND] Use vprintk_emit() for userspace event communication

[Hidehiro Kawai]

(2013/07/19 0:37), Hannes Reinecke wrote:> On 07/18/2013 02:14 PM, Hidehiro Kawai wrote:
>> Hannes Reinecke wrote:
>>> Syslog, OTOH, is well used to be flooded with tons and tons of
>>> messages so it does not suffer from this problem.
>>> And with the latest printk updates vprintk_emit() has now two
>>> different buffers, one for the logging message and another one for
>>> structured data. So it would be possible to issue a vprintk_emit()
>>> with no message, just the structured data.
>>> Such a message would not show up in syslog, but would be available
>>> for tools accessing the structured buffer directly.
>>> This approach would have the benefit that we would not have to
>>> invent yet another mechanism but could use existing, defined interfaces.
>>>
>>> On the kernel summit I would like to discuss this approach to figure
>>> out if this use-case of using vprintk_emit() with just a dictionary
>>> meets with general approval or if there are alternative routes for
>>> signalling events to userspace.
>>
>> It's interesting.  We are trying to handle errors in user space by
>> adding a hash value to structured printk output.  Since the hash value is
>> generated from the message format, user space can identify the message
>> easily and consistently (as long as the message doesn't change).
>> This feature aims at general errors detected by kernel, and also
>> SCSI device error is our target.
>>
>> By the way, my RFC patch can be found here:
>> http://thread.gmane.org/gmane.linux.kernel/1519633
>>
> Hmm. Yes, I dimly remember that this was the original idea from Kay
> Sievers (who presented it at KS two years back).

Thanks for the information.  I didn't know that such discussion had been
held, although I knew IBM guys' works in 2008.

> But for this to work you'd have to audit all kernel messages to ensure
> that each message is in fact unique, otherwise you'd be having two
> messages with the same hash, but generated at different places.

In the current implementation of my patch, __FILE__ and __LINE__ are
also added to /dev/kmsg output in addition to the hash.  So if you
want to identify each message uniquely, just use these information.

But practically we don't always need to make hashes unique, I think.
If the same message appears in multiple places, the user space can
handle them as the same event in the most cases.  Some small messages
with one word have different meanings even if the same message, but
users would have few interests in such messages.

If a hash collision happens between different messages, user space
just do message matching in the traditional way (e.g. regexp).
A message-and-hash catalog is generated at build time, and it gives
the user space a finite set of hash values.  So the user space can know
in advance which ones collide and require message matching.  There is
no collision in my 5000+ message case, so I think the cases needing
such message matching is not so much.

> As for your patch I really do think it would be more sensible to add the
> hash as an entry to the dictionary for vprintk_emit() instead of adding
> yet another field to printk itself.
> That way you wouldn't have to modify the existing caller.

Sorry, my explanation might be misleading.  There is no need to
modify the existing caller although we have to change printk variants
like dev_printk.  Message hashes are added transparently.

> As a sidenote, wouldn't it be sufficient to generate a hash over the
> source code line,
> ie the output of the __func__ macro?
> You still could extract the message string if you so chooses, but the
> overall generation would be far easier as you wouldn't care about the
> actual string.

Hmm, sometimes a printk message is printed by multiple printks without
newline.  In this case, my approach doesn't work well,
but the approach you suggested can handle it because those multiple
printks resides within a same function in the most cases.

However, there is no guarantee that the hashes of __func__ are unique,
while __FILE__ + __LINE__ pairs are always unique.  And we can
produce __func__ information from __FILE__ + __LINE__ in user space.
So I think the current implementation (i.e. outputting hash of message
format, __FILE__, and __LINE__) is more feasible.

> So you would avoid the problem of having duplicate strings in the
> printk() output.

Again, thanks for your reply!

--
Hidehiro Kawai
Hitachi, Yokohama Research Laboratory
Linux Technology Center