[cgl_discussion] Proposal for WG validation team

Lynch, Rusty rusty.lynch at intel.com
Mon Nov 4 15:03:00 PST 2002

In last weeks validation subgroup phone meeting, we had a lengthy
discussion about what the role of the validation team should be in the
context of the new PoC goals.  We kicked around a few high level ideas
ranging from a team that provided free validation consulting to various
open source projects, to a team that implements a LTP like validation
framework for test that are out-of-scope for LTP.

Since testing tends to be the one part of software development that open
source regularly ignores (in the same way that documentation usually gets
little attention), there seems to be a definite need that the validation
team could help out in making Linux ready for a carrier grade environment. 
The hard part is how to assist the community in a meaningful way without
taking on the burden of validating the whole world.

This morning I was chatting with Julie over some coffee, trying to explain
how the validation team fits into the rest of the working group and the
various autonomous patch set projects, when an idea started to take shape
that seems like a good compromise.  At a very high level, the validation
team would have three primary jobs:

1. Provide quality assurance surveys and gap analysis on the 
   state of testing efforts on open source projects that the PoC has an
   interest in.
2. Provide tools and documentation to assist anyone wanting to test a 
   Linux project.
3. When the certification team knows what it wants to do, the validation
   team will create certification test on certifiable requirements. 

Specifics on Analysis:

For this job I see the validation team as a board of engineers who would
take as input a prioritized list of open source projects from the PoC, and
would return (given enough time and resources) evaluations for each of the
projects that would summarize the current state of testing on that project,
along with a proposal detailing what would need to be added to the project
to bring it up to acceptable levels (in the opinion of the team) and a
resource and time estimate on what it would take to fulfill the

This report would be feedback to the PoC for consideration, and if the PoC
considered the proposal worth resourcing, then the PoC (with the aid of the
steering committee) would find volunteers from the members of the working
group to do the work.  The validation team could provide further
clarification to who ever volunteers to do the work, but the validation
team would not be on the hook to do the work.  If the working group is
unable to get anyone to volunteer to do the work, then by definition the
project was not really a priority project to the working group.

Specifics on tools and documentation:

As the validation team surveys existing projects, there is going to be a
build-up of FAQ, best known methods, and HOWTO type of information that
would be useful to anyone trying to beef up testing on an open source
project.  The validation team would make all of this information available
(maybe on the new developer.osdl.org site?).

In addition to information, the validation team could provide tools to aid
in testing... basically any utilities, scripts, frameworks that could be
by anyone. 

Specifics on creating certification test:

It is really too early to get into specifics on certification, but it is 
fairly safe to assume that some kind of certification test will need to be
written for requirements that provide sufficient information.  When the time
comes, the validation team will need to provide this code.  Additional 
resources will be required at that time if the working group wants the test
to be completed in any reasonable amount of time.

Rusty Lynch              
Staff Software Engineer,  Intel Corporation
rusty.lynch at intel.com

*  This email message solely contains my own personal views, and not 
   necessarily those of my employer

More information about the cgl_discussion mailing list