[Ksummit-discuss] [MAINTAINER SUMMIT] Stable trees and release time

[Benjamin Gilbert]

On Tue, Sep 04, 2018 at 06:17:46PM -0700, Laura Abbott wrote:
> On 09/04/2018 04:43 PM, Guenter Roeck wrote:
> > On 09/04/2018 04:14 PM, Sasha Levin wrote:
> >> Maybe some concrete numbers will help here. Do you maybe know how many
> >> commits in the past year snuck past the -rc cycle into a stable release
> >> and found as buggy by Fedora's testing pipeline?
> > 
> > ... and how many bugs were found during the existing test cycle ?
> > 
> > The next question would be how many regressions were reported by users
> > after a release was published.
> > 
> > The statistics I carried until early this year suggested a regression rate
> > of around 0.15% for stable releases, where regression means that a bug was
> > found post-release and had to be fixed later. It would indeed be interesting
> > to know how many of those were found by (automated ?) testing and how many
> > were found by users.
> 
> I'd have to do some digging through bugzilla to get numbers. Some
> of this is also motivated by discussions with the CoreOS team who
> have also tried to use the stable kernels and ran into problems.
> I'll see if I can get some numbers.

We've shipped 11 different 4.14.x kernels on the CoreOS Container Linux
stable channel, all in this calendar year.  Five of them had user-impacting
regressions that had to be fixed via OS updates:

4.14.30 - vxlan panic
          https://github.com/coreos/bugs/issues/2382
4.14.42 - Failure to set MTU in xen-netfront
          https://github.com/coreos/bugs/issues/2443
4.14.44 - Failure to bring up hv_netvsc interface after it was brought down
          https://github.com/coreos/bugs/issues/2454
4.14.48 - Integer overflow causing tiny TCP receive windows
          https://github.com/coreos/bugs/issues/2457
4.14.55 - Broken CIFS client
          https://github.com/coreos/bugs/issues/2480
4.14.55 - Failure to mount ext4 filesystems 3 TB or larger
          https://github.com/coreos/bugs/issues/2485

The TCP window bug was particularly exciting, since affected machines would
have downloaded the fixed OS image at ~300 bytes/sec.  To avoid that, we had
to implement several workarounds in our update infrastructure, only the
second time we've had to do that in the history of Container Linux.

--Benjamin Gilbert