[Ksummit-discuss] Fwd: Rough notes from testing unconference
Grant Likely
grant.likely at secretlab.ca
Fri Aug 22 14:19:27 UTC 2014
Rough notes from the Testing discussion during the workshop day of
Kernel Summit. Big thanks to Paul McKenney for writing all of this
down.
g.
Testing session (unconference)
o In-kernel or out of kernel? Bisection? Can do either way,
one approach is to have two clones of the git tree, and another
is to "git checkout" a specific version of the tools directory.
o Levels of tests: 1) get a login prompt, 2) kselftest.
o Should we have specified userspace? No, quickly gets into
the distribution-building business.
o Will Deacon: Can script using busybox, inittab, and so on.
Can run trinity on such a setup.
o Ted: Would be nice if other test frameworks built on top of
kselftest. Therefore kselftest should not make any asssumptions.
o Shua: Goal of kselftest isn't to create something new, but
rather to gather up what we already have.
o Ted: Much of the code in tools/testing/selftest is unit tests
for a tiny part of the kernel. These don't really belong
in LTP.
o Olof: Let's not reinvent the test-infrastructure wheel.
o Aneesh: Autotest. Others: Lots of such tests, but lots of
different ones that are relatively difficult to set up.
o Paul: Don't need rootfs. rcutorture runs out of initrd.
o Olof: Often easier just to write a script.
o Ted: If we are going to lightweight, need to focus on kernel
unit tests.
o What kind of tool can we depend on? Busybox? Just busybox,
has everything it needs. (This means scripting must be in
dash.)
o Arnd: klibc has its own set of binaries, cannot support
busybox. Need libc.
o Josh: Only C and standard C library supported.
o Paul: Host or guest?
o Discussion: apparently need both. Assume full-fledged distro
on host, minimal on guest/target. UART, retrieve a file.
Some targets don't have persistent mass storage.
o Grant: Needs to work on separate hardware, on qemu, on
fast models.
o Need stdio output, input optional (some tests take input
from kernel boot parameters.)
o How to parse success or failure? How to know what to ignore?
Ted: What are high-level requirements rather than low-level
implementation? Need host to easily parse the information
from the target on the host.
o Mauro: Need something for performance tests as well.
o Ted: Group the tests. "Quick" group. Groups associated with
given function (e.g., suspend-resume). Have profiles based
on types of tests needed and time allotted.
o Josh: Add test types to the MAINTAINERS file.
o Tim: Requirements around cross-building? People make their
own? We supply some in-tree? We supply pointers to known
good cross-build toolchains? Grant: "Yes".
o Tim: The need is to enable people who have never cross-built
in their life to successfully cross-build and test with
minimal effort.
Next steps: Grant to send call for volunteers for various efforts
to ksummit-discuss, interested people to reply.
o Josh: Can we pick an existing test output format that already
has tools, infrastructure, and so on.
Tim: Hudson!
Andy: PASS, FAIL, XFAIL, ...
Shua: Must fit into the primary goal of being a good kernel
developer unit test.
Ted: Proposed requirement: Simplest use case must have simple
tool that parses output. Must not include a JRE for output
parsing.
Tim: Must be human reasable.
Andy: Must be short boilerplate, so that it can be generated
and read quickly.
Paul: Willing to make reasonable adaptations to rcutorture output.
But it must not require an in-kernel JRE.
Arnd: Would like self-test entry for drivers.
Ben: Have seen similar things in other operating systems, would
be good in Linux. Mauro: Would like to separate hardware and
software tests for drivers. Arnd: Could be separate profile.
More information about the Ksummit-discuss
mailing list