F16 bug count is out of control at this point. We’re still managing to close them faster than they’re coming in, but the sheer number of bugs still open is overwhelming. There still exist a lot of older bugs that may have been fixed but the user hasn’t updated the bug. As we’re coming across these bugs that have been in needinfo state for a while we’re closing them out if we have a reasonable expectation that the bug is fixed.

There’s also still a huge number of duplicate bugs, where it’s not immediately obvious that they are duplicates. The end frame in the call chain may be the same, but how we got there differs for example. Or we see multiple instances of a particular trace, from various hardware. (A good example of this case being the numerous irqpoll bugs open for which we still have no real explanation. Likewise softlockup reports).

A curious observation: We’ve had a slight uptick lately in the number of bugs that seem to be caused by bad memory or other hardware problems. Sometimes these stick out like a sore thumb (a single bit flips changing an address from ffffffff81c8bca0 to ffefffff81c8bca0 for example). Other times, they aren’t as obvious.
I have been wondering though if we see more of these in Fedora just due to the extra debug options we leave enabled even in the production builds (like linked list debug, which is pretty much free).

Something that has been interesting though when we’ve found a user with a bad ram report, is searching for other bugs that user filed. Suddenly “weird” oopses make a lot more sense.

This week saw a lot of activity in Fedora 16. We pushed out a rebase to 3.3. Yesterday I did a mass-update to bugzilla requesting people retest. (Which screwed up when I got a bugzilla proxy error, causing everything to be posted 3 times. Apologies to those who got all those mails).

The good news is we seem to be closing a lot of bugs quickly from this rebase.
Around 50 or so bugs got closed just since yesterday.
As with any rebase, some new bugs are showing up, but some of the more common ones (like the bluetooth oopses) I think we’ve got a handle on, and will get fixed in an update out next week.

Next week, 3.3.1 should also land, and that will likely coincide with Fedora 15 also getting a rebase to this kernel. It’s been a while since we’ve had three releases all on the same version. (I think last time was F12/F13/F14).

Despite all this activity, the overall bug count still remains very high. We still have no real answer for all the memory corruption problems caused by hibernate (which accounts for dozens of open bugs now, plus probably a bunch that we haven’t yet attributed to this problem).

It’s pretty clear to see that f16 has been seeing most of the attention this week. (As it has been for most of the month).

Overall bug count creeping up, but some of the long standing bugs like the various corruption bugs mention in last weeks posts are slowly edging their way to closure. Andrea Arcangeli fixed bug in the transparent huge page code which explains a bunch of the page table corruption issues we saw reported. The i915/hibernate/memory corruption bug has at least some theories on a potential solution now, and we’ve been able to attribute another bunch of the ‘weird’ bugs to post-hibernate corruption now that we know what patterns to look for.

With 3.3 likely being released next week, we’ll be moving F16 to it (and F15 a week later). Hopefully we get to close out a bunch more of the longer standing bugs that haven’t been backported to -stable.