To be continued..

New year, new bugs. is a post from: codemonkey.org.uk

No related posts.

Some of the tasks of the job include (but are not limited to):
* diagnosing incoming bugs from users
* backporting fixes from Linus’ latest tree to Fedora
* for bugs unfixed in Linus’ tree, work on fixing them, with the upstream maintainers.
* interacting with the -stable team to make sure those same fixes go there.
* helping keep Fedora on the cutting edge of kernel development by pushing new releases

Hiring distribution kernel maintainers is never easy. It is a job that requires knowledge of the various parts of the kernel. A generalist as opposed to a specialist. The bugs you’re staring at one day could be completely different the next.

With Fedora things are even harder due to the rapid rate of change upstream. By contrast, a RHEL kernel maintainer could spend time working on a bug for a month without the code underneath him changing.

It’s a demanding job, with an unrelenting torrent of new things that need fixing / working on. If this doesn’t dissuade you, you could be just what we’re looking for!

(oh, and the job doesn’t involve you having to move to Boston. Though if you wanted to I’m sure we could help make that happen).

Hiring for a position on the Fedora kernel team. is a post from: codemonkey.org.uk

Related posts:

1. Red Star kernel. Over the long weekend, I downloaded a copy of Red...

The rpm headers alone reveal quite a lot of interesting information though.
It seems that Red Star is forked from a version of Fedora somewhere around the Fedora 10 or 11 timeframe.

Here’s the changelog embedded in the kernel rpm..

* Fri Mar 27 14:00:00 2009 Jong Song Jin
- patched linux-2.6.25-drivers-video.patch(for video capture saa7134 onboard driver)

* Wed Apr 30 14:00:00 2008 Dave Airlie 2.6.25-14
- fix radeon fast-user-switch oops + i915 breadcrumb oops


Some interesting things here.
- All changelog timestamps are on the hour. Suggesting they’ve been sanitised, or generated from another source. All 374 changelog entries had been munged in this way, including the ones from the original Fedora release.
- No email addresses for the changelog entries (no surprise)
- The actual changelogs are quite cryptic. “change machanism for pci device information.” why?
- ‘fixed the 8250 serial driver for modem.’ wtf ?
- They decided tuxonice is the way forward for hibernation. Perhaps it works better on dear leaders laptop.
- ‘apply jipsam algorithm’. This is a crypto module that isn’t in mainline (and apparently doesn’t exist outside North Korea). I bet it’s good though. No backdoor master keys or anything similar.
- ‘implement the usb filtering through user authentification’.
What does that even mean ?

Browsing through the rest of the distribution, a lot of packages are renamed. OpenOffice became UriOffice. Gimp became ImageProcessor. Wine became CrossWin2.0.

It also comes with AntiVirus 2.0. Which comes with a rtscan.ko kernel module, which judging by the symbols it uses, does some magic with jprobes to hook various functions for on-open scanning.

Another curious thing, is that throughout the distro whenever you do see an email address or hostname, it has a .kp TLD, that never seem to resolve. I’m assuming that the DNS servers in North Korea show different results if you’re in North Korea or not.

Red Star kernel. is a post from: codemonkey.org.uk

No related posts.

The project began after seeing the program ‘cpuid’ by Phil Karn. I had sent a few patches to Phil but never got any reply, and no new release of cpuid appeared until months later (not including my patches). As I ended up forking more and more of the code, I decided to push it out as ‘x86info’ for the first time. Phil did a few more releases, up until 2002, when the program was abandoned.

Over the years, x86info has seen 739 commits from at least 19 contributors (some patches written by others were committed in the CVS days without correct attribution). The bulk of the commits were unsurprisingly mine.

The commit rate year by year is interesting.

2001 197
2002 119
2003 88
2004 17
2005 37
2006 46
2007 34
2008 74
2009 65
2010 27
2011 54

2003 was when I got hired at Red Hat, and the Fedora kernel pretty much became my life. 2004 I was doing RHEL4 too. The drop-off around that time is significant, and doesn’t recover until several years later.
This year seems to be off to a good start

x86info has been ported to Solaris and FreeBSD, and at one time, even to Microsoft Windows. That support was ultimately removed due to the number of invasive ifdefs present. The various ports sometimes get broken due to lack of testing during development time, but afaik they are available in ‘ports’ with whatever fixes need to be present.

Asides from the Linux kernel, this is the oldest project I’m involved in that is still seeing development.

10 years of x86info. is a post from: codemonkey.org.uk

No related posts.

Even though I was on vacation, and there solely to take in the latest and greatest in musical instruments, I was curious to find out just how much stuff is running Linux. Most of the manufacturers I spoke with wouldn’t tell me anything (and appeared quite puzzled that I’d even care). The only obvious ones were from manufacturers I already knew about (like the Muse receptor, Korg’s etc..)

There was one revealing moment however that I found amusing. In 1979, the Fairlight CMI was *the* sampling synth. Back then, its lightpen input device must have seemed like something from the future. (Here’s Quincy Jones demoing it). Anyway.. Fairlight instruments have returned, and created a retro style reproduction of the original CMI. One of my friends managed to capture a photograph of it when the synth UI crashed. Yup, looks like Ubuntu to me.

There’s really no escape from Linux. It’s everywhere you look.

I returned last night refreshed from time off from daily work, but also inspired in many ways by new things, and new people I met. Escaping the frozen wasteland of Boston for the sunnier climate did wonders for my general mood too. As draining as the whole experience was, I’m sure I’ll be going back next year.

(For those who care about any of the other things I saw there, I took a bunch of photographs too).

Winter NAMM 2011. is a post from: codemonkey.org.uk

No related posts.

CVE-2010-4256: Pipe fcntl local denial of service.
The inode struct in the kernel contains this union..

union {
struct pipe_inode_info *i_pipe;
struct block_device *i_bdev;
struct cdev *i_cdev;
};


A missing “is this a pipe” check in pipe_fcntl allowed a user to perform pipe ioctl’s on inodes that were not pipes. (ie, block/char devs). Things blew up pretty quickly.

CVE-2010-4347: Incorrect sysfs permissions
One of the things the fuzzer does is to pass random file descriptors to syscalls that expect them. At first, it generated a few itself on startup by creating a bunch of files. I changed this to open any files that were readable/writable from sysfs, procfs and /dev. It prints out what it managed to open on startup. I immediately noticed something that stood out like a sore thumb.
/sys/kernel/debug/acpi/custom_method was world writable. As this file allows a user to upload new ACPI tables to the kernel, this is a fairly obvious local root. Thankfully debugfs isn’t mounted by default on most systems.
This discovery prompted a further investigation into S_IWUGO users in the kernel, which led to a slew of fixes (mostly in the staging tree). A patch is also pending to checkpatch.pl to warn about any new additions, as exporting a file from the kernel with this mode is nearly always a bad idea.

There’s still a lot more work to do, mostly annotating each and every system call.
An annotated syscall looks like this. It’s time consuming to read through every syscall and add every possible argument, so right now there are only a dozen or so (out of ~300) fully annotated syscalls. I suspect more kernel bugs to be found as coverage increases.

After lwn picked up on my last post, I got a number of interesting mails, some offering more suggestions for improvements. One interesting mail I got was from Tavis Ormandy at google, who has been working on a similar tool. We’ve taken some different design decisions which has meant that both tools may pick up on different bugs, so I don’t think either is really a replacement for the other. I’ve definitely found it inspirational to read through a ‘competing’ tool though. Worth checking out if you found my fuzzer interesting at all.

Finally, I changed the name of the project. There were a number of other syscall fuzzers out there all called ‘scrashme’. To differentiate, mine is now named ‘Trinity’. (Cloning information at original post has been updated). I chose the name after rat-holing on wikipedia for several hours, reading about nuclear tests. I always liked the Oppenheimer quote “Now I am become Death, the destroyer of worlds”. For a project that intentionally attempts to destroy, it seemed fitting.

Now, back to annotating..

System call fuzzing continued. is a post from: codemonkey.org.uk

No related posts.

One common thing throughout all of this though, has been the problem of regressions. Someone who had suspend/resume on Fedora work out of the box on one Fedora release, expects it to work just as well if not better on the next release.

A common refrain from users who see regressions like this is “suspend still works in Windows after I apply updates”.
The big difference of course is that these other operating systems aren’t on six month release cycles, with a kernel that’s still seeing huge rate of change every 12 weeks.

But the nature of suspend/resume itself is an incredibly fragile process on any operating system.
I’ve seen both Windows and OS X fail to resume after installing updates and/or third party drivers. I.e., as soon as something is introduced that wasn’t the ‘tested gold master’ configuration. It turns out that it’s a really, really tough problem to support every possible hardware platform out there, even if you’re a huge company like Microsoft, or if you’re a company that controls the hardware you run on, like Apple.

The big problem with diagnosing problems with suspend/resume though is that they typically need a developer with hands-on access to the machine in question. Short of cornering Matthew Garrett at a conference though, the only options are back and forth via email/bugzilla, which isn’t necessarily the most optimal way to debug things. Additionally, the methods for diagnosing suspend/resume failures are still kinda in the dark ages.

Five years ago, things were pretty hopeless. Today, they aren’t great but they’re a lot better. Maybe in another five years things will be robust enough that we can take it for granted as a feature that always works. I personally feel that they have to for features like aggressive runtime power management to become more prevalent.

Musings on suspend & resume regressions. is a post from: codemonkey.org.uk

No related posts.

With the syscall fuzzer running out of things to break in 2.6.36, it became more pressing to move the laptop onto something more recent, which meant I had to track it down. Initially I suspected kernel bug. It froze during booting just after initializing usb, printing two messages about disabling interrupts. I bisected it using a bunch of old rpm’s from koji, and found the exact kernel version that it broke on. During that day, a bunch of acpi patches got merged, so I figured one of those was to blame. Hours and hours of bisecting/compiling/rebooting later, bisect told me lies.
Linus pointed out that maybe one of my assumptions was incorrect. Perhaps 2.6.36rc6 which I marked as ‘good’ was broken too. So I did a local compile of the kernel that was working fine as an rpm, and sure enough it was also broken.

I checked the compiler version. Hadn’t changed. Then it dawned on me that maybe there was something in the initramfs that was breaking. I diffed a working vs broken initramfs, and got a ton of ‘Binary files foo differs’ messages, but nothing in the dracut scripts themselves. My thought at this point: something that got updated during f14 broke initramfs’s, and the version that I created my rc6 kernel from was made using an earlier toolset. To test this prognosis, I regenerated the working initramfs with the current tools, rebooted, and sure enough.. boom.

At this point I had dozens of possible suspect packages. I ended up doing a complete f14 reinstall. Booted a test kernel just fine. yum updated to current. boom. Now I knew for sure I was on the right path. I reinstalled again. Then went through a painful cycle of..

update one package
remove and then install kernel package (thus regenerating the initramfs)
reboot

hours later… I was fully updated, and everything works fine.

I have no idea why the bug isn’t manifesting now. I have this nagging feeling it’ll be back at some point.
(This blog entry is mostly to remind myself what happened in case it does rear its head again).

Some days I hate computers.

chasing a heisenbug. is a post from: codemonkey.org.uk

No related posts.

For a while I’ve been working on a system call fuzzing tool. A really long time ago, Kurt Garloff wrote one, which I added some improvements to, which just blasted random crap in every register, and called random calls. After it triggered some really stupid bugs which got fixed pretty quickly, it rarely found anything again. Sometimes when a new syscall was added, someone would miss something, and it would trigger some new bug, but for the most part every call just -EINVAL’d immediately.

So I started exploring the idea of writing a tool that instead of passing random junk, actually passed semi sensible data. If the first thing a syscall does is check if a value is between 0 and 3, then passing rand() % 3 is going to get us further into the function than it would if we had just passed rand() unmasked. There are a bunch of other things that can be done too. If a syscall expects a file descriptor, pass one. If it expects an address of a structure, pass it realistic looking addresses (kernel addresses, userspace addresses, ‘weird’ looking addresses).

The new tool just found its first real bug. If perf is running, and mprotect is being fuzzed, then we get an oops. Oops indeed.

I suspect it’ll find some more bugs over time as I add more twists on ‘random’ data. I already have a bunch of ideas that I want to implement (some are listed in the TODO). Seeing a bunch of people at the kernel summit / plumbers conf last week gave me a whole bunch more ideas too.

The git tree for the tool is at git://git.codemonkey.org.uk/trinity.git
(gitweb)
Snapshot tarballs are at http://codemonkey.org.uk/projects/trinity/

system call abuse. is a post from: codemonkey.org.uk

No related posts.

Asides from one problem. Every time I did a yum update that pulled in an selinux policy update, it would consistently exhaust all the ram in the machine. I filed a bug on this, and as usual, Dan Walsh dropped some selinux knowledge that I had no idea about.

You can customize the bzip block size and “small” flag via
/etc/selinux/semanage.conf. After applying you can add entries like these to
your /etc/selinux/semanage.conf to trade off memory vs disk space (block size)
and to trade off memory vs runtime (small):

bzip-blocksize=4
bzip-small=true

You can also disable bzip compression altogether for your module store
via:
bzip-blocksize=0

Since I put that first tweak in place, it’s survived several policy updates without a hiccup.

SELinux on low memory systems. is a post from: codemonkey.org.uk

No related posts.