[lsb-discuss] Ways for us to work more efficiently

Theodore Tso tytso at MIT.EDU
Thu Jan 24 15:23:33 PST 2008

On Thu, Jan 24, 2008 at 02:08:43PM -0800, Wichmann, Mats D wrote:
> It would take us closer to the way the project operated some
> years ago; many of the more formal procedures were pared down
> over time as staff dwindled and we ended up with more overhead
> than various people thought was needed.  As a very basic example,
> we had separate project teams for distinct functional areas - 
> this now lives on as a memory in bugzilla categories, and in the 
> distinct package signing keys that were used by the various teams.

Yep, I remember.  I mean "project" in a slightly different way,
though, which would be more granular in nature.  For example, one such
project might be "Uplift glibc to newer version".  And who might work
on it might be one person, or it might get handed off from person to
person as it moves from "evaluate which version of glibc", to the spec
work, to "updating the tests" --- or some portions of it might be done
in parallel.  The idea would be that all of the information related to
project of uplifting glibc would be tracked on the wiki, since it's
probably too complicated of a task where bugzilla would be a good
match; there would be all sorts of information that we might want to
keep up to date as we go.

> In general, the wiki is apallingly stale, and I assert that
> at least part of the reason is that it's now too hard to use,
> so you don't get the same level of "drive-by updates" that we 
> used to - the whole purpose of a wiki.  The authentication 
> mechanism is extremely slow (e.g. it takes several minutes for
> me to complete a login), and for some reason nobody has ever
> explained to me it times out your login very frequently. 

Hmm.  That seems like an IT issue that we should have David look at.
I've not noticed it take a long time to login, but we should
definitely have that looked at.

Also, as far as I know the wiki isn't timing out a login; I've been
left logged in for hours at a time.  There does seem to be an issue
where if you take too long editing a particular wiki page, it times
out the edit --- but you don't need to login again; all you have to do
click on the "try again" link

> Always buildable, and ready for release, are clearly two different
> points - both very important.  

Yes, very true, and a good point.

> Ready for release, of course, was a completely different story.  Yes,
> testing feedback from lots of different distros, released &&
> in-development,
> is crucial.  I've been hoping we could have some of those resources
> be external to LF, it would be pretty tricky to set up and maintain
> the right farm and I think would cost quite a bit of admin time where
> we're doing both nightly updates of the distro and of the tests (with
> the former not infrequently requiring a reboot for a enw kernel). 

To the extent that we can get help from a distro to run their our test
suites every day, that would be highly desirable, of course.  And if
we can make it really easy for them to download the latest distro test
suites, and run it on (a) their currently certified systems, and (b)
their latest bleeding edge distro, that would be a very good thing.
We should definitely encourage that whenever possible.

On the other hand, I'm not sure we can count on all distro's doing
this, and so if they can't we may have to run the tests ourselves.
Yes, that will mean that we may need a much bigger set of test farms.
And that will cost money, and server space, and admin.  I would not be
surprised if we have to hire another I/T staff member in addition
David to help run these machines.

> Maybe it's not practical, but I'd love to see partnerships where at
> least in-development distros would host an instance of the
> autotester and push results back.

So would I!  I'm just not sure how likely that is to be.

> Even if you get all this data, the hard part is analysis - you could
> argue that we now have access to too much data.  We don't yet have 
> very good comparative tools.  If you have two test journals from 
> two different runs, you can easily do a comparison with tjreport.
> But as the number of tests have swelled such that underneath the
> nice dtk-manager interface it's anything but monolithic - a current
> full certification-ready run will produce 27 distinct journal files -
> it's not that easy to indentify a failure that is unique to one or
> a subset of distros, and then drill down on what's going on.  

Yes, we very clearly need better tools here.  Ideally what I would
like is a web page that can summarize the number of failures on a
per-distro basis, and where we can look at the data multiple ways,
i.e., with the number of failures over time, so we can see when
failures started happening, from a high-level summary point of view,
either over time or by distro/architecture type, as well as letting us
drill down so we can see the failures on a particular day.

I don't know if this is something that the ISPRAS folks might be able
to help us with, or whether we would have to hire a contractor to set
up such a system, but I think it would be a major help.

> I'll
> give just one example - we've been flummoxed by changes in behavior
> in Gtk, where suddenly tests start returning different results in
> later releases.  The test journal for this component doesn't include
> versioning information for the components, so once you've identified
> which distributions are failing you have to start digging to see if
> you can identify a common thread such as gtk-2.14 "fails", anything
> earlier "works".  THEN you can start anlysis of whether there was
> an incompatible behavior change upstream, or whether the test made
> bad assumptions, or....

Yeah, this sounds like something that perhaps an enhanced Database
Navigator could help with, but it would need to get a much larger test
feed if we wanted it to work against a constantly changing community
release, such as Debian Unstable in the extreme case.  The plus side
is that we would get advance warning of problems much, much sooner!

The other way that we might be able to get some relief on this front
is to work with the upstream to see if they would be willing to
include our tests as part of their regression test suite (i.e., that
would be run via "make check" or "make test" in the upstream source
tree).  But that is also going to take a lot of work....

> Conclusion: to go from autobuilt-nightly to ready-to-release-nightly 
> I do think we need to come up with much more sophisticated data 
> analysis mechanisms that can help us understand what that night's
> auto-test runs mean. 

I think you're absolutely right.  Can you think of any other analysis
tools that would be helpful for us to develop?

      	   	    	    	      - Ted

More information about the lsb-discuss mailing list