This is how the year begins for the Fedora kernel.

To be continued..

NOTE: This position has now been filled. The rest of this post is left for posterity only. If you’re interested in working at Red Hat in some other role, check out jobs.redhat.com and/or send me an email, and I’ll forward it to the right people.

We’re hiring for a position on the Fedora kernel team again.
If you are interested, email me a CV (davej at redhat) and we’ll see where things go. (we don’t have a posting on jobs.redhat.com yet)

Some of the tasks of the job include (but are not limited to):
* diagnosing incoming bugs from users
* backporting fixes from Linus’ latest tree to Fedora
* for bugs unfixed in Linus’ tree, work on fixing them, with the upstream maintainers.
* interacting with the -stable team to make sure those same fixes go there.
* helping keep Fedora on the cutting edge of kernel development by pushing new releases

Hiring distribution kernel maintainers is never easy. It is a job that requires knowledge of the various parts of the kernel. A generalist as opposed to a specialist. The bugs you’re staring at one day could be completely different the next.

With Fedora things are even harder due to the rapid rate of change upstream. By contrast, a RHEL kernel maintainer could spend time working on a bug for a month without the code underneath him changing.

It’s a demanding job, with an unrelenting torrent of new things that need fixing / working on. If this doesn’t dissuade you, you could be just what we’re looking for!

(oh, and the job doesn’t involve you having to move to Boston. Though if you wanted to I’m sure we could help make that happen).

Over the long weekend, I downloaded a copy of Red Star Linux, the official operating system of North Korea.
Because license violations seem to be part of the Juche idea, there’s no known source code online, so proper analysis of what goes into it is difficult.

The rpm headers alone reveal quite a lot of interesting information though.
It seems that Red Star is forked from a version of Fedora somewhere around the Fedora 10 or 11 timeframe.

Here’s the changelog embedded in the kernel rpm..

* Fri Mar 27 14:00:00 2009 Jong Song Jin
- patched linux-2.6.25-drivers-video.patch(for video capture saa7134 onboard driver)

* Wed Apr 30 14:00:00 2008 Dave Airlie 2.6.25-14
- fix radeon fast-user-switch oops + i915 breadcrumb oops


Some interesting things here.
- All changelog timestamps are on the hour. Suggesting they’ve been sanitised, or generated from another source. All 374 changelog entries had been munged in this way, including the ones from the original Fedora release.
- No email addresses for the changelog entries (no surprise)
- The actual changelogs are quite cryptic. “change machanism for pci device information.” why?
- ‘fixed the 8250 serial driver for modem.’ wtf ?
- They decided tuxonice is the way forward for hibernation. Perhaps it works better on dear leaders laptop.
- ‘apply jipsam algorithm’. This is a crypto module that isn’t in mainline (and apparently doesn’t exist outside North Korea). I bet it’s good though. No backdoor master keys or anything similar.
- ‘implement the usb filtering through user authentification’.
What does that even mean ?

Browsing through the rest of the distribution, a lot of packages are renamed. OpenOffice became UriOffice. Gimp became ImageProcessor. Wine became CrossWin2.0.

It also comes with AntiVirus 2.0. Which comes with a rtscan.ko kernel module, which judging by the symbols it uses, does some magic with jprobes to hook various functions for on-open scanning.

Another curious thing, is that throughout the distro whenever you do see an email address or hostname, it has a .kp TLD, that never seem to resolve. I’m assuming that the DNS servers in North Korea show different results if you’re in North Korea or not.

This past weekend I’ve been thinking of reviving an idea that has come up countless times. Producing RPM builds of the rawhide kernel for our already released Fedoras. The reason for not doing them so far has come down to bandwidth. (in terms of build system throughput, disk space, mirroring, and people bandwidth).

What I’m toying with doing is some devel kernels for Fedora 11 that are built outside of the Fedora build system. The Fedora kernel team now has enough build bandwidth for x86-[64] that we can actually get builds for those architectures done faster than koji.

Disk space – I’m thinking of just keeping the last 2-3 builds available.

Mirroring – Instead of having these be part of Fedora proper, I think an external repo on something like fedorapeople.org will suffice.

Which just leaves people bandwidth. For the most part the work is going to be just regularly syncing the devel/ branch with a CVS branch of F-11/ For some of this work, some scripting could be done to alleviate some of the pain. Also the frequency at which we push out these builds will determine the pain point. Perhaps every -git isn’t particularly valuable anyway. One build every handful of -git’s should be sufficient for bisecting.

There does remain one additional barrier. Occasionally we introduce something in rawhide builds which just won’t work on F11. For example, the kernel modesetting patches are tied closely to Xorg packages. Sometimes upstream changes require changes in mkinitrd or udev or some other ‘plumbing’. Some of these are regressions, and hopefully by identifying them sooner we can get them reverted/fixed upstream. Sometimes however, things get deprecated, and we need to change these packages. I’m not sure how to cope with this yet in a devel-for-F11 scenario.

One other thing that might be fun to throw into this would be the generation of -vanilla packages. The only reasons we don’t do these as part of the regular kernel builds is the various bandwidth concerns above. The specfile copes with spitting out RPMs with very little work needed. Josh Boyer has been occasionally doing these builds, though there hasn’t been a huge uptake. It’s unclear if this is due to lack of interest, or just a lack of publicity.

Another question to be answered is whether we go the route of enabling debugging in all builds as we do in rawhide, or do separate -debug builds. I’m leaning towards the latter.

I’m not committing anything to this for sure just yet, but it’s something I’ve been giving quite a bit of thought. There are still a bunch of unanswered questions.

One of the longest living patchsets we’ve carried in the Fedora kernel is that of execshield. Over time, bits of it have gone upstream (in particular, some of the randomisation bits). In F11, it’s still a 1000 line 30K diff, touching all manner of core kernel functionality. To try and get more of it pushed upstream, I’ve been working on splitting it up into its component parts.

The current state of the diffs is at http://www.codemonkey.org.uk/projects/execshield/.

The emulate-NX-with-segment-limits chunk is unlikely to ever go upstream. A bit of a shame given it’s the largest part of execshield remaining. Linus wasn’t thrilled by it, and it is a pretty nasty hack.
Also, with modern CPUs having hardware-NX, it becomes less useful over time. (Though we still need to carry it judging by the number of old-school 686 users we still have).

So if we do have to keep execshield, we should at least try to make it cleaner and smaller. Every time I poke at it, I manage to shave off another hundred lines or so.

Linus just released the 2.6.28 kernel. It’s already compiling for tomorrows rawhide. Fedora 9 & 10 will probably move to it in a few weeks. Typically, we wait until the dust settles and the first -stable release comes out. I was asked recently what bits we’re excited about in .28 for Fedora. To be honest, I didn’t give a great answer. It’s just not a “OMG, THIS RELEASE IS AWESOME” kind of release. There’s nothing in there that I was disappointed not to get into .27 for F10′s release. In fact, lots of the bits in there we were already carrying in the Fedora kernel (the DRM bits for example). Asides from that, it’s the usual churn of bug fixes, new drivers, and probably some interesting new bugs.

What about F11 ? Looking at the current schedule, we’ll get at least .29 in. I’m not sure we’ll have enough time to pull in .30 at this stage. All depends on how quickly .29 stabilises. Version numbers are so hand-wavy anyway. I wish when people asked me ‘what version is fX going to be’, they’d really ask ‘is feature xyz going to be merged by fX’. But people sure are hung up on numbers.

People tend not to notice kernel features these days for the most part. Which in a way is a good thing. (means it’s working). Unless it’s something that gets a lot of press like “unified x86 architecture” “tickless kernel” “modesetting”. There are dozens of features every release, but people don’t really get excited about a lot of them, and for good reason. They’re mostly dull from a userspace programmer/end-user perspective.