[cgl_discussion] Minutes taken from the CGL Valildation team meeting 9-4-02

Craig Thomas craiger at osdl.org
Wed Sep 4 16:14:13 PDT 2002

Craig Thomas                         phone: 503-626-2455  ext. 33
Open Source Development Labs         email: craiger at osdl.org
15275 SW Koll Pkwy, Suite H
Beaverton, OR  97006
-------------- next part --------------

    The discussion started with Julie's enhancements to the validation
    framework directives to support a --abat switch in the wrapper.sh
    script for each validation suite.  Rusty then descirbed the ABAT
    execution process at a high level: ABAT is a test harness loaded onto
    a system after a new kernel is loaded.  Once rebooted, abat is started
    and it looks to a specific directory to run tests.  Each test is run,
    results are reported to STDOUT and mailed to a "receiving" system.

    Each validation suite would include code in its wrapper.sh script so
    that the test maintainer could identify a set of "smoke" tests for that
    suite that ABAT would be able to run as a quick sanity check for the 
    latest build.

    We came up with idea about reworking ABAT so that it has the same
    consistent look and feel as the rest of the suite in terms of the
    results output.  From there other ideas came to mind regarding how to
    add enhancemetns to ABAT.  The plan is to create a 'ToDo' list for 
    feature improvements for the Validation suite and ABAT.  Initial
    considerations are:

     + modify abat to produce results the same way as the validation suite
     + enhance abat to utilize a separate monitor/driver that can tell when
       a system reboots after the load of a new kernel

    We discussed whether or not ABAT should be moved under the cglvalidation
    CVS tree or kept on sourceforge as an open project.  The advantages of
    bringing ABAT under the CGL validation tree is that the two could be
    more tightly coupled and ABAT could take advantage of the defined
    directory layout of the validation suite.  The disadvantage would be
    extra code to maintain.  It was later thought that it would be best to
    keep ABAT separate from the CGL validation suite to allow a more general
    use of ABAT.

    In discussing the status of integrating the CGL validation suite into 
    STP,  Craig stated that no work has gone into adding the suite to STP, but
    initial investigation has occurred.  Resources at OSDL are on other
    projects at this time, but it is expected that by next week, work can begin
    to create a selectable test called cgl_validation (or something similar)
    that would trigger a validation run with a seleted number of CGL tests.
    The plan is to create the entire scaffolding to make it easy to add the
    CGL test suites later.  Craig then went into a descirption of the STP
    framework and process: STP is web based, and is generally used in two

    scenario 1:  developer builds new kernel based on new patches.  Uploads
    kernel onto osdl.org and then goes to the STP web page and logs on (need
    to be an OSDL associate -- which is a free signup).  Once logged in, the
    tester selects 'new test' link and then chooses the kernel to run, the
    type of system, and the test to run (which in this case would be a cgl
    test) then  submits the run.  STP automatically queues the test and runs
    it.  STP wipes the disks clean on the system under test, loads the new 
    kernel, loads the selected test, then executes.  Results are captured
    and the user is notified via email when the test is complete.  The tester
    can then check the results though the OSDL web site.

    scenario 2:  a tester wants to select any kernel build and run all of
    the cgl tests.  STP can be set up to automatically run the entire suite
    of tests when a new kernel is build.  We would want to configure STP to
    do this (or at least run the set of ABAT tests for a smoke test).

    Abat and STP can not handle multiple nodes at this time.  Work needs
    to be done here.  Would STP push the XML output?  Would STP or CGL
    need to change to support the validation suite?  These questions need
    to be addressed in order to allow the tools we use to work within the
    specifiations of the validation suite.

    Need to create a 'todo' list for ABAT features and validattion suite
    features rather than logging defects.

    Discussion of the developer.org validation page:  The projects that are
    currently defined need to have the other projects and requirements added
    to the web site.  There was a thought about sending an email about cross
    referencing the current projects with those identified in the CGL
    project management milestones and with the requirements document.  Julie
    remembered an email already sent out that did preliminary work for
    identifying missing projects on the validation web page.  Rusty noted that
    work needs to be done to update the validation web site.  Therefore, a
    cross check will be postponed until the validation web site could be
       - integrate the new requriements approved in 8/23 minutes.

    Craig will write a strategy plan for using STP for the validation suites.

    We never had a chance to cover the status of the current maintainers
    for the various validation suites.  Our conference call ended.

More information about the cgl_discussion mailing list