[Ksummit-discuss] [CORE TOPIC] kernel testing standard

[Masami Hiramatsu]

(2014/05/24 9:30), Theodore Ts'o wrote:
> On Fri, May 23, 2014 at 09:05:02AM -0500, Justin M. Forbes wrote:
>> All good suggestions. As nice as it would be if tests were in tree, this
>> might be unmanageable. But even out of tree tests could be automatically
>> brought in provided they are listed somewhere in tree.  Ideally you
>> would be able to "make tests" and get all in tree tests run, or "make
>> alltests" and have it grab/build/run out of tree tests with git urls as
>> well. 
> 
> Um.... how long do you expect "make alltests" to run?
> 
> And how do you deal with tests that require specific hardware?
> 
> For ext4, just doing a light smoke test takes about 30 minutes.  For
> me to run the full set of tests using multiple file system
> configurations, it takes about 12 to 16 hours.  And that's just for
> one file system.  (I do the tests using KVM, with a 90 megabyte
> compressed root file system, and 55 gigabytes worth of scratch
> partitions.)

Agreed. That is why I didn't say uniforming test-harness, but
making a standard way to test. I think we'd better organize maintainers
to test their tree; clarify what they will run and what will not do,
when each branch is tested (e.g. before/after merge, or nightly) and how.

If we force to unify the test frameworks, it will neither be maintained
nor used. Instead, if maintainers state what test they will run and how
to maintain it, we are sure that each test will be done on the each
subsystem branch, and before release, we can avoid to run all tests which
requires many hardware and long time but just run a small number of tests
(e.g. LTP, trinity, etc.)

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt at hitachi.com