[Ksummit-discuss] Fwd: Rough notes from testing unconference

Grant Likely grant.likely at secretlab.ca
Fri Aug 22 14:19:27 UTC 2014


Rough notes from the Testing discussion during the workshop day of
Kernel Summit. Big thanks to Paul McKenney for writing all of this
down.

g.

Testing session (unconference)

o       In-kernel or out of kernel?  Bisection?  Can do either way,
        one approach is to have two clones of the git tree, and another
        is to "git checkout" a specific version of the tools directory.

o       Levels of tests:  1) get a login prompt, 2) kselftest.

o       Should we have specified userspace?  No, quickly gets into
        the distribution-building business.

o       Will Deacon: Can script using busybox, inittab, and so on.
        Can run trinity on such a setup.

o       Ted: Would be nice if other test frameworks built on top of
        kselftest.  Therefore kselftest should not make any asssumptions.

o       Shua: Goal of kselftest isn't to create something new, but
        rather to gather up what we already have.

o       Ted: Much of the code in tools/testing/selftest is unit tests
        for a tiny part of the kernel.  These don't really belong
        in LTP.

o       Olof: Let's not reinvent the test-infrastructure wheel.

o       Aneesh: Autotest.  Others: Lots of such tests, but lots of
        different ones that are relatively difficult to set up.

o       Paul: Don't need rootfs.  rcutorture runs out of initrd.

o       Olof: Often easier just to write a script.

o       Ted: If we are going to lightweight, need to focus on kernel
        unit tests.

o       What kind of tool can we depend on?  Busybox?  Just busybox,
        has everything it needs.  (This means scripting must be in
        dash.)

o       Arnd: klibc has its own set of binaries, cannot support
        busybox.  Need libc.

o       Josh: Only C and standard C library supported.

o       Paul:  Host or guest?

o       Discussion: apparently need both.  Assume full-fledged distro
        on host, minimal on guest/target.  UART, retrieve a file.
        Some targets don't have persistent mass storage.

o       Grant:  Needs to work on separate hardware, on qemu, on
        fast models.

o       Need stdio output, input optional (some tests take input
        from kernel boot parameters.)

o       How to parse success or failure?  How to know what to ignore?
        Ted: What are high-level requirements rather than low-level
        implementation?  Need host to easily parse the information
        from the target on the host.

o       Mauro: Need something for performance tests as well.

o       Ted: Group the tests.  "Quick" group.  Groups associated with
        given function (e.g., suspend-resume).  Have profiles based
        on types of tests needed and time allotted.

o       Josh: Add test types to the MAINTAINERS file.

o       Tim: Requirements around cross-building?  People make their
        own?  We supply some in-tree?  We supply pointers to known
        good cross-build toolchains?  Grant: "Yes".

o       Tim:  The need is to enable people who have never cross-built
        in their life to successfully cross-build and test with
        minimal effort.

Next steps:  Grant to send call for volunteers for various efforts
        to ksummit-discuss, interested people to reply.

o       Josh: Can we pick an existing test output format that already
        has tools, infrastructure, and so on.

        Tim: Hudson!

        Andy: PASS, FAIL, XFAIL, ...

        Shua: Must fit into the primary goal of being a good kernel
        developer unit test.

        Ted: Proposed requirement: Simplest use case must have simple
        tool that parses output.  Must not include a JRE for output
        parsing.

        Tim: Must be human reasable.

        Andy: Must be short boilerplate, so that it can be generated
        and read quickly.

        Paul: Willing to make reasonable adaptations to rcutorture output.
        But it must not require an in-kernel JRE.

        Arnd: Would like self-test entry for drivers.

        Ben: Have seen similar things in other operating systems, would
        be good in Linux.  Mauro: Would like to separate hardware and
        software tests for drivers.   Arnd: Could be separate profile.


More information about the Ksummit-discuss mailing list