[Ksummit-discuss] [CORE TOPIC] Testing

[Julia Lawall]

On Tue, 7 Jul 2015, Mark Brown wrote:

> On Tue, Jul 07, 2015 at 08:25:21AM -0700, Guenter Roeck wrote:
> > On 07/07/2015 02:24 AM, Mark Brown wrote:
>
> > >The main things I'm aware of that are happening at the minute are
> > >kselftest development, the 0day tester, plus kernelci.org and the other
> > >build and boot/test bots that are running against various trees.
>
> > Maybe list all known ones as a start ?
>
> Off the top of my head the automated ones I'm aware of are Olof's build
> & boot test, Dan running smatch and I think some other static analysis
> stuff, someone (not sure who?) running some coccinelle stuff, Coverity
> and I've got a builder too.

The 0day service runs Coccinelle.

Coccinelle does not need the build to succeed.

julia

> > >In terms of discussion topics some of the issues I'm seeing are:
>
> > >  - Can we pool resources to share the workload of running things and
> > >    interpreting results, ideally also providing some central way for
> > >    people to discover what results are out there for them to look at
> > >    for a given kernel in the different systems?
>
> > That might be quite useful. However, I have seen that it doesn't really
> > help to just provide the test results. kissb test results have been
> > available for ages, and people just don't look at it. Even the regular
> > "Build regression" e-mails sent out by Geert seem to be widely ignored.
>
> > What I really found to help is to bisect new problems and send an e-mail
> > to the responsible maintainer and to the submitter of the patch which
> > introduced it. I'd like to automate that with my test system, but
> > unfortunately I just don't have the time to do it.
>
> Yes, that's the "and interpreting" bit in the above - this only really
> works with people actively pushing.  You do start to get people checking
> themselves once things are perceived as something people care about but
> it does take active work to establish and maintain that.
>
> It also really helps if things are delivered promptly, and against trees
> people are actively developing for.  But even with clear reports and
> sometimes patches not everyone shows an interest.  As we get more and
> more actual testing running that's going to start to become more
> serious, breaking the build or boot will also mean that automated tests
> don't get to run.
>
> This is one of the things 0day gets really right, when it kicks in it'll
> e-mail people directly and promptly.
>
> > >  - Should we start carrying config fragments upstream designed to
> > >    support testing, things like the distro config fragments that keep
> > >    getting discussed are one example here but there's other things like
> > >    collections of debug options we could be looking at.  Should we be
> > >    more generally slimming defconfigs and moving things into fragments?
>
> > >and there's always the the perennial ones about what people would like
> > >to see testing for.
>
> > Sharing as many test bot configuration scripts and relevant configurations
> > as possible would be quite helpful. For example, I am building various
> > configurations for all architectures, but I don't really know if they
> > are relevant. Also, I would like to run more qemu configurations,
> > but it is really hard to find working ones.
>
> Grant (just CCed) was working intermittently on the qemu bit.  I think
> the last plan was to enhance the scripts Kevin has for driving his build
> farm.
>