[Ksummit-discuss] [MAINTAINER SUMMIT] Stable trees and release time

Jan Kara jack at suse.cz
Wed Sep 5 08:31:04 UTC 2018


On Tue 04-09-18 19:54:34, Guenter Roeck wrote:
> On 09/04/2018 06:45 PM, Laura Abbott wrote:
> > On 09/04/2018 04:35 PM, Guenter Roeck wrote:
> > > On 09/04/2018 03:06 PM, Laura Abbott wrote:
> > > > On 09/04/2018 02:49 PM, Guenter Roeck wrote:
> > > > > On 09/04/2018 01:58 PM, Laura Abbott wrote:
> > > > > > I'd like to start a discussion about the stable release cycle.
> > > > > > 
> > > > > > Fedora is a heavy user of the most recent stable trees and we
> > > > > > generally do a pretty good job of keeping up to date. As we
> > > > > > try and increase testing though, the stable release process
> > > > > > gets to be a bit difficult. We often run into the problem where
> > > > > > release .Z is officially released and then .Z+1 comes
> > > > > > out as an -rc immediately after. Given Fedora release processes,
> > > > > > we haven't always finished testing .Z by the time .Z+1 comes
> > > > > > out. What to do in this situation really depends on what's in
> > > > > > .Z and .Z+1 and how stable we think things are. This usually
> > > > > > works out fine but a) sometimes we guess wrong and should have
> > > > > > tested .Z more b) we're only looking to increase testing.
> > > > > > 
> > > > > > What I'd like to see is stable updates that come on a regular
> > > > > > schedule with a longer -rc interval, say Sunday with
> > > > > > a one week -rc period. I understand that much of the current
> > > > > > stable schedule is based on Greg's schedule. As a distro
> > > > > > maintainer though, a regular release schedule with a longer
> > > > > > testing window makes it much easier to plan and deliver something
> > > > > > useful to our users. It's also a much easier sell for encouraging
> > > > > > everyone to pick up every stable update if there's a known
> > > > > > schedule. I also realize Greg is probably reading this with a very
> > > > > > skeptical look on his face so I'd be interested to hear from
> > > > > > other distro maintainers as well.
> > > > > > 
> > > > > 
> > > > > For my part, a longer -rc interval would not help or improve the
> > > > > situation. Given the large number of security fixes, it would
> > > > > actually make the situation worse: In many cases I could no longer
> > > > > wait for a fix to be available in a release. Instead, I would have
> > > > > to pick and pre-apply individual patches from a pending release.
> > > > > 
> > > > 
> > > > Fedora does this already. We frequently carry patches which have
> > > > not yet made it into a stable release. Sometimes they only stay
> > > > around for one release but we've had ones that stayed around for
> > > > multiple releases.
> > > > 
> > > Sure, but having to pull them from release candidates adds additional
> > > work and increases risk.
> > > 
> > > > > I like the idea of having (no more than) one release per week with
> > > > > the exception of security fixes, but longer -rc intervals would be
> > > > > problematic.
> > > > > 
> > > > 
> > > > Security fixes are an interesting question. The problem is that
> > > > not every security issue is actually equal and even patches
> > > > that fix CVEs can cause regressions.
> > > > 
> > > 
> > > We do have a pretty well defined process for handling CVEs depending
> > > on their severity. The preferred handling for all CVEs is to get the
> > > fixes through stable releases.
> > > 
> > 
> > Yes, I agree CVEs should eventually go through a stable release
> > for the same reason all fixes are security fixes. There's also a
> > difference between a CVE that should be picked up urgently and one that
> > can be applied as part of a regular update cycle.
> > 
> > > As for regressions, only a system with no patches applied is safe from
> > > regressions. Otherwise regressions are unavoidable. The key is to improve
> > > testing to a point where the pain from regressions is acceptable.
> > 
> > This may just be kernel tree philosophy but I'm not sure any regression
> > in the stable tree should be acceptable. In Greg's blog post
> > http://www.kroah.com/log/blog/2018/08/24/what-stable-kernel-should-i-use/
> > he suggested "Server: Latest stable release or latest LTS release"
> > I don't think anyone wants their server regressing. I've talked
> > with the CoreOS team about their experience using stable kernels
> > and it gets tricky to convince users to update when there are
> > regressions.
> > 
> 
> I understand that philosophy very well. Each and every regression is an argument
> to not use stable releases in the first place. We have been there. My solution
> is to do everything I can to improve testing to the point where regressions
> are all but non-existent.
> 
> However, realistically, there will _always_ be regressions, and some of them
> _will_ be found post-release. If zero regressions is your absolute must-have
> condition for a release, the only guaranteed means to accomplish that is to
> make zero changes in that release.

Agreed. Even enterprise distro kernels with all the QA and review and
experienced people doing the work regress occasionally. In the end what
matters to users is: "If I update, how big is the probability that my
current workload stops working?" And you can make that 1%, 0.1% or 0.01%
(with adequately increasing costs to achieve that) but never 0.

								Honza
-- 
Jan Kara <jack at suse.com>
SUSE Labs, CR


More information about the Ksummit-discuss mailing list