[Cgl_discussion] Draft outline of the validation framework specification

Craig Thomas craiger at osdl.org
Fri Mar 29 11:50:56 PST 2002

Below is a draft outline of content that would be included in the
CGLinux Validation Framework Specification.  Although it is a bit
early to finalize the content, it may be sufficient to distribute 
to the groups and solicit feedback.

Draft Outline of document "Implementing the CGL Validation
Framework and Validation Suites"

Possible Subjects to cover:

 1.  Introduction
       a) Carrier Grade Linux project description, problem
          to be solved, etc.
       b) Validation Framework group and charter
       c) Scope of this document
             + defines the framework and tools
             + defines the test categories
             + defines the roadmap and milestones for test development
       d) note that this doc is evolving; changes are made as
          requirements change, or as milestones are completed

 2.  Requirements of Carrier Grade Linux
       a) describe test approach and methodology
             + provide robustness to carrier grade systems
               running linux
             + provides performance benchmarks
             + provides certification?
       b) define test categories - provide a brief prose description of
          the kinds of tests that are need to support the requirements
          categories defined in the requirements document
             + performance
             + standards
             + availability
             + interoperability
             + install/reboot
             + etc.

 3.  Framework Architecture
       a) environment needed to run tests
       b) tools used to execute tests
       c) the test framework (TET, PAN, STAF, etc)
       d) File system structure of test environment (results,
          wrappers, logs, test cases, etc)

 4.  Test Suite descriptions
       a) detailed each test category and list the actual tests
          contained within that category.
            + describe each test in some detail
            + cross reference each test with the requirement that
              it satisfies
       b) list tests that still need to be developed (if not created)
            + describe what the test needs to accomplish to satisfy
              a requirement
            + provide a milestone date when it would need to be
              complete and delivered to the test framework

 5.  Execution of the Test Framework
       a)  downloading the Framework
       b)  execution of the tests
       c)  analyzing the results
       d)  running the framework at OSDL under STP (as a test option)
       e)  posting results to the Carrier Grade Linux community

 6.  Contributing Tests to the Framework
       a) how to contribute tests
       b) how to validate tests
            + code reviews?
            + successful execution on UP, 2 proc at minimum?
       c) how to notify the Carrier Grade Linux community that new
          tests exist
       d) procedure for code check-ins
       e) procedure for updating this framework document (test
          case description and the requirements it satisfies,
          roadmap table)

 7.  Controling release versions of the framework
       a) policies for releasing
            + semi-anual, quarterly, etc.
       b) policies of what is bundled and released
       c) notification of new framework version release

 8.  Roadmap of tests under development
       a) table of all tests and their current status
            + complete
            + in progress
            + not started
       b) general dates for estimated completion of deliverables

 9.  References and links to other projects
       a) references to standards to be complied with
       b) references of test projects
            + LSB
            + LTP
            + OSDL STP

10.  Revision History (could go at the top)

11.  Appendicies

Craig Thomas
Open Source Development Labs
15275 SW Koll Pkwy, Suite H
Beaverton, OR  97006
phone: 503-626-2455  ext. 33
email: craiger at osdl.org

More information about the cgl_discussion mailing list