[cgl_discussion] Proposal for WG validation team

Craig Thomas craiger at osdl.org
Mon Nov 4 16:07:35 PST 2002


One point came up during last weeks validation subgroup phone meeting 
that stuck with me... that is to have a single place to go when you want
to find the test cases for all of the projects that comprise the CGL.
I feel that the validation team could use their resources effectively in
this issue.  Maintaining a site that at least points to all of the test
repositories for the CGL projects is a minimal goal.  I do know that
there is an effort underway to create an environment to help manage the
patch sets and to run tests against those patch sets.  This should help
out the CGL proof of concept well.  The validation group could be the
evangelists for this project.

On Mon, 2002-11-04 at 15:03, Lynch, Rusty wrote:
> 
> In last weeks validation subgroup phone meeting, we had a lengthy
> discussion about what the role of the validation team should be in the
> context of the new PoC goals.  We kicked around a few high level ideas
> ranging from a team that provided free validation consulting to various
> open source projects, to a team that implements a LTP like validation
> framework for test that are out-of-scope for LTP.
> 
> Since testing tends to be the one part of software development that open
> source regularly ignores (in the same way that documentation usually gets
> little attention), there seems to be a definite need that the validation
> team could help out in making Linux ready for a carrier grade environment. 
> The hard part is how to assist the community in a meaningful way without
> taking on the burden of validating the whole world.
> 
> This morning I was chatting with Julie over some coffee, trying to explain
> how the validation team fits into the rest of the working group and the
> various autonomous patch set projects, when an idea started to take shape
> that seems like a good compromise.  At a very high level, the validation
> team would have three primary jobs:
> 
> 1. Provide quality assurance surveys and gap analysis on the 
>    state of testing efforts on open source projects that the PoC has an
>    interest in.
> 2. Provide tools and documentation to assist anyone wanting to test a 
>    Linux project.
> 3. When the certification team knows what it wants to do, the validation
>    team will create certification test on certifiable requirements. 

I see more...

  4.  provide test cases to the projects (actually write test cases to
      support the projects -- but I may be getting confused between a
      validation subgroup member and a member of an open source project)
  5.  Help to define the certifiable requirements by assisting in the
      specification documentation so the requirements are certifiable.
> 
> Specifics on Analysis:
> 
> For this job I see the validation team as a board of engineers who would
> take as input a prioritized list of open source projects from the PoC, and
> would return (given enough time and resources) evaluations for each of the
> projects that would summarize the current state of testing on that project,
> along with a proposal detailing what would need to be added to the project
> to bring it up to acceptable levels (in the opinion of the team) and a
> resource and time estimate on what it would take to fulfill the
> recommendation. 
> 
> This report would be feedback to the PoC for consideration, and if the PoC
> considered the proposal worth resourcing, then the PoC (with the aid of the
> steering committee) would find volunteers from the members of the working
> group to do the work.  The validation team could provide further
> clarification to who ever volunteers to do the work, but the validation
> team would not be on the hook to do the work.  If the working group is
> unable to get anyone to volunteer to do the work, then by definition the
> project was not really a priority project to the working group.
> 
> Specifics on tools and documentation:
> 
> As the validation team surveys existing projects, there is going to be a
> build-up of FAQ, best known methods, and HOWTO type of information that
> would be useful to anyone trying to beef up testing on an open source
> project.  The validation team would make all of this information available
> (maybe on the new developer.osdl.org site?).
> 
> In addition to information, the validation team could provide tools to aid
> in testing... basically any utilities, scripts, frameworks that could be
> used
> by anyone. 
> 
> Specifics on creating certification test:
> 
> It is really too early to get into specifics on certification, but it is 
> fairly safe to assume that some kind of certification test will need to be
> written for requirements that provide sufficient information.  When the time
> comes, the validation team will need to provide this code.  Additional 
> resources will be required at that time if the working group wants the test
> to be completed in any reasonable amount of time.

Will the validation team need to provide the code to the certification
group or will the certification group be able to collect the various
test cases from the projects themselves? (I'm making the assumption that
CGL will be comprised of projects; each project has its own set of
tests).  

I see the validation team and the certification team as two separate
bodies.  If the validation team is to provide code to the certification
group, then why doesn't the validation team just begin the work of
defining the certification process and begin to create the certification
suite?

> 
> ----------------------------------------------------------------------------
> Rusty Lynch              
> Staff Software Engineer,  Intel Corporation
> rusty.lynch at intel.com
> 
> *  This email message solely contains my own personal views, and not 
>    necessarily those of my employer
>  
> 
> _______________________________________________
> cgl_discussion mailing list
> cgl_discussion at lists.osdl.org
> http://lists.osdl.org/mailman/listinfo/cgl_discussion
-- 
Craig Thomas                         phone: 503-626-2455  ext. 33
Open Source Development Labs         email: craiger at osdl.org
15275 SW Koll Pkwy, Suite H
Beaverton, OR  97006




More information about the cgl_discussion mailing list