[Ksummit-discuss] [CORE TOPIC] Testing

[Mark Brown]

On Tue, Jul 07, 2015 at 08:25:21AM -0700, Guenter Roeck wrote:
> On 07/07/2015 02:24 AM, Mark Brown wrote:

> >The main things I'm aware of that are happening at the minute are
> >kselftest development, the 0day tester, plus kernelci.org and the other
> >build and boot/test bots that are running against various trees.

> Maybe list all known ones as a start ?

Off the top of my head the automated ones I'm aware of are Olof's build
& boot test, Dan running smatch and I think some other static analysis
stuff, someone (not sure who?) running some coccinelle stuff, Coverity
and I've got a builder too.

> >In terms of discussion topics some of the issues I'm seeing are:

> >  - Can we pool resources to share the workload of running things and
> >    interpreting results, ideally also providing some central way for
> >    people to discover what results are out there for them to look at
> >    for a given kernel in the different systems?

> That might be quite useful. However, I have seen that it doesn't really
> help to just provide the test results. kissb test results have been
> available for ages, and people just don't look at it. Even the regular
> "Build regression" e-mails sent out by Geert seem to be widely ignored.

> What I really found to help is to bisect new problems and send an e-mail
> to the responsible maintainer and to the submitter of the patch which
> introduced it. I'd like to automate that with my test system, but
> unfortunately I just don't have the time to do it.

Yes, that's the "and interpreting" bit in the above - this only really
works with people actively pushing.  You do start to get people checking
themselves once things are perceived as something people care about but
it does take active work to establish and maintain that.  

It also really helps if things are delivered promptly, and against trees
people are actively developing for.  But even with clear reports and
sometimes patches not everyone shows an interest.  As we get more and
more actual testing running that's going to start to become more
serious, breaking the build or boot will also mean that automated tests
don't get to run.

This is one of the things 0day gets really right, when it kicks in it'll
e-mail people directly and promptly.

> >  - Should we start carrying config fragments upstream designed to
> >    support testing, things like the distro config fragments that keep
> >    getting discussed are one example here but there's other things like
> >    collections of debug options we could be looking at.  Should we be
> >    more generally slimming defconfigs and moving things into fragments?

> >and there's always the the perennial ones about what people would like
> >to see testing for.

> Sharing as many test bot configuration scripts and relevant configurations
> as possible would be quite helpful. For example, I am building various
> configurations for all architectures, but I don't really know if they
> are relevant. Also, I would like to run more qemu configurations,
> but it is really hard to find working ones.

Grant (just CCed) was working intermittently on the qemu bit.  I think
the last plan was to enhance the scripts Kevin has for driving his build
farm.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: Digital signature
URL: <http://lists.linuxfoundation.org/pipermail/ksummit-discuss/attachments/20150707/32468c32/attachment.sig>