[lsb-discuss] Making app checker more useful in some circumstances
Wichmann, Mats D
mats.d.wichmann at intel.com
Wed Jan 20 10:53:58 PST 2010
We had another round of discussion about this during today's call,
prompted by a new bug (2877), which has this descripton:
"This is a rollup bug for bugs that affect App Checker's
usefulness when checking native builds."
I want to try to flush this out a bit further, even though
it's been discussed many times before. My opinion has evolved
a little from where it was during the call, even.
The basic issue seems to be that someone with an existing
app binary would like to get some information about the
portability of that app. We have reporting modes for that in
the app checker, but it seems what people can't get past is that
lsbappchk will report a lot of problems on any application
which is not compiled in LSB mode, and as a collection tool
the app checker may let a lot of that show (and if you use
appchk or pkgchk directly you get them for sure). If the
developer wants a reading on whether it's even worth proceeding,
these messages may end up overwhelming the capacity of the
developer in question to pursue the topic.
My view of this is that lsbappchk is very stupid in
this regard - it's "test case based" which means that
each individual event is reported as a separate and
distinct failure. That's the essence of assertion-based
testing that our main toolkit, TET, uses, and we've
followed that mode of behavior in appchk, which is not
There are two approaches to attacking the TMI (Too
Much Information) problem: teach appchk to be smarter,
or filter appchk's conclusions through a higher-level tool -
putting the smarts there.
The higher level tool already exists, it's the aforementioned
app checker and does exactly this filtering. I'm comfortable
with the type of output the app checker produces, but it appears
that's not a universally shared opinion. There's an LSB bug
that may illustrate this dilemma:
It we follow the link in this bug, there's a page showing
a report of checking a package. There are many binary
objects in this package: at least 10 execuable programs
and a number of shared libraries, at least 20. Since none
of this is built in LSB mode, checking the entire package can
be expected to produce a fair bit of reportage from the lower
level tool. To me the level of information is just about right
(but I'm not the target audience), with one exception.
- at the top we see a test summary
- lower down we see a detailed report, which shows a breakout
for each category, which can be expanded if desired
What's my "one exception"? There's a column for Comments and
it's empty everywhere - it has a generic greyed out "No information"
But if we as experienced users look at this, we can explain almost
all of these, which means we do have information.
Now this example isn't a true representation of the magnitude
of the problem, as this app is very close to conforming. But
I think we could profit by looking at how the organization of
the information display could be improved for various use cases,
and this is one of them: we have a lot of repeated messages
due to the fact that there are a lot of objects being checked.
So why not shuffle the groupings a bit? The app checker must
know it's in "package" mode here, so why not group by error?
In "Non-LSB ELF section" there are 10 different sections reported
on. Why not collapse this list to 10 entries, one per section,
with a count of occurrences, and have a button to expand each
one if wanted. One could even go further and collapse it another
level so that the nine debug sections were collapsed down to a
single top-level entry.
More importantly, and this is what I was trying to argue in the
call, we have a collective knowledge base about LSB, but the
electronic database used by the app checker is too much smaller.
We *know* the debug sections are not in the LSB, that there's no
good description of them, and that it really doesn't matter
(that's why they are warnings, not errors) because they don't
contribute to program loading and the linker can/does just ignore them.
We should be able to identify many of the issues we know are
"this is because you compiled without lsbcc" and actually report
on them, and the information should be sufficient that in app
checker one could click a button to switch between "conformance
mode" where such items are displayed, and "investigation mode"
where they are hidden. I expect this may need some extension
or tweaking of the database schema, and resulting adjustment to
scripts which process the database, in addition to actually
capturing the data. We actually do have a fair bit of data that
fits the decsription "we know about this but it's not in the LSB"
so maybe there isn't much that has to be done, but this is the
topic I'd really like to tease out: how can we do better at
identifying and properly reporting on non-LSB things that we
actually do know a fair bit about.
More information about the lsb-discuss