Even though I was on vacation, and there solely to take in the latest and greatest in musical instruments, I was curious to find out just how much stuff is running Linux. Most of the manufacturers I spoke with wouldn’t tell me anything (and appeared quite puzzled that I’d even care). The only obvious ones were from manufacturers I already knew about (like the Muse receptor, Korg’s etc..)

There was one revealing moment however that I found amusing. In 1979, the Fairlight CMI was *the* sampling synth. Back then, its lightpen input device must have seemed like something from the future. (Here’s Quincy Jones demoing it). Anyway.. Fairlight instruments have returned, and created a retro style reproduction of the original CMI. One of my friends managed to capture a photograph of it when the synth UI crashed. Yup, looks like Ubuntu to me.

There’s really no escape from Linux. It’s everywhere you look.

I returned last night refreshed from time off from daily work, but also inspired in many ways by new things, and new people I met. Escaping the frozen wasteland of Boston for the sunnier climate did wonders for my general mood too. As draining as the whole experience was, I’m sure I’ll be going back next year.

(For those who care about any of the other things I saw there, I took a bunch of photographs too).

Winter NAMM 2011. is a post from: codemonkey.org.uk

No related posts.

CVE-2010-4256: Pipe fcntl local denial of service.
The inode struct in the kernel contains this union..

union {
struct pipe_inode_info *i_pipe;
struct block_device *i_bdev;
struct cdev *i_cdev;
};


A missing “is this a pipe” check in pipe_fcntl allowed a user to perform pipe ioctl’s on inodes that were not pipes. (ie, block/char devs). Things blew up pretty quickly.

CVE-2010-4347: Incorrect sysfs permissions
One of the things the fuzzer does is to pass random file descriptors to syscalls that expect them. At first, it generated a few itself on startup by creating a bunch of files. I changed this to open any files that were readable/writable from sysfs, procfs and /dev. It prints out what it managed to open on startup. I immediately noticed something that stood out like a sore thumb.
/sys/kernel/debug/acpi/custom_method was world writable. As this file allows a user to upload new ACPI tables to the kernel, this is a fairly obvious local root. Thankfully debugfs isn’t mounted by default on most systems.
This discovery prompted a further investigation into S_IWUGO users in the kernel, which led to a slew of fixes (mostly in the staging tree). A patch is also pending to checkpatch.pl to warn about any new additions, as exporting a file from the kernel with this mode is nearly always a bad idea.

There’s still a lot more work to do, mostly annotating each and every system call.
An annotated syscall looks like this. It’s time consuming to read through every syscall and add every possible argument, so right now there are only a dozen or so (out of ~300) fully annotated syscalls. I suspect more kernel bugs to be found as coverage increases.

After lwn picked up on my last post, I got a number of interesting mails, some offering more suggestions for improvements. One interesting mail I got was from Tavis Ormandy at google, who has been working on a similar tool. We’ve taken some different design decisions which has meant that both tools may pick up on different bugs, so I don’t think either is really a replacement for the other. I’ve definitely found it inspirational to read through a ‘competing’ tool though. Worth checking out if you found my fuzzer interesting at all.

Finally, I changed the name of the project. There were a number of other syscall fuzzers out there all called ‘scrashme’. To differentiate, mine is now named ‘Trinity’. (Cloning information at original post has been updated). I chose the name after rat-holing on wikipedia for several hours, reading about nuclear tests. I always liked the Oppenheimer quote “Now I am become Death, the destroyer of worlds”. For a project that intentionally attempts to destroy, it seemed fitting.

Now, back to annotating..

System call fuzzing continued. is a post from: codemonkey.org.uk

No related posts.

One common thing throughout all of this though, has been the problem of regressions. Someone who had suspend/resume on Fedora work out of the box on one Fedora release, expects it to work just as well if not better on the next release.

A common refrain from users who see regressions like this is “suspend still works in Windows after I apply updates”.
The big difference of course is that these other operating systems aren’t on six month release cycles, with a kernel that’s still seeing huge rate of change every 12 weeks.

But the nature of suspend/resume itself is an incredibly fragile process on any operating system.
I’ve seen both Windows and OS X fail to resume after installing updates and/or third party drivers. I.e., as soon as something is introduced that wasn’t the ‘tested gold master’ configuration. It turns out that it’s a really, really tough problem to support every possible hardware platform out there, even if you’re a huge company like Microsoft, or if you’re a company that controls the hardware you run on, like Apple.

The big problem with diagnosing problems with suspend/resume though is that they typically need a developer with hands-on access to the machine in question. Short of cornering Matthew Garrett at a conference though, the only options are back and forth via email/bugzilla, which isn’t necessarily the most optimal way to debug things. Additionally, the methods for diagnosing suspend/resume failures are still kinda in the dark ages.

Five years ago, things were pretty hopeless. Today, they aren’t great but they’re a lot better. Maybe in another five years things will be robust enough that we can take it for granted as a feature that always works. I personally feel that they have to for features like aggressive runtime power management to become more prevalent.

Musings on suspend & resume regressions. is a post from: codemonkey.org.uk

No related posts.

With the syscall fuzzer running out of things to break in 2.6.36, it became more pressing to move the laptop onto something more recent, which meant I had to track it down. Initially I suspected kernel bug. It froze during booting just after initializing usb, printing two messages about disabling interrupts. I bisected it using a bunch of old rpm’s from koji, and found the exact kernel version that it broke on. During that day, a bunch of acpi patches got merged, so I figured one of those was to blame. Hours and hours of bisecting/compiling/rebooting later, bisect told me lies.
Linus pointed out that maybe one of my assumptions was incorrect. Perhaps 2.6.36rc6 which I marked as ‘good’ was broken too. So I did a local compile of the kernel that was working fine as an rpm, and sure enough it was also broken.

I checked the compiler version. Hadn’t changed. Then it dawned on me that maybe there was something in the initramfs that was breaking. I diffed a working vs broken initramfs, and got a ton of ‘Binary files foo differs’ messages, but nothing in the dracut scripts themselves. My thought at this point: something that got updated during f14 broke initramfs’s, and the version that I created my rc6 kernel from was made using an earlier toolset. To test this prognosis, I regenerated the working initramfs with the current tools, rebooted, and sure enough.. boom.

At this point I had dozens of possible suspect packages. I ended up doing a complete f14 reinstall. Booted a test kernel just fine. yum updated to current. boom. Now I knew for sure I was on the right path. I reinstalled again. Then went through a painful cycle of..

update one package
remove and then install kernel package (thus regenerating the initramfs)
reboot

hours later… I was fully updated, and everything works fine.

I have no idea why the bug isn’t manifesting now. I have this nagging feeling it’ll be back at some point.
(This blog entry is mostly to remind myself what happened in case it does rear its head again).

Some days I hate computers.

chasing a heisenbug. is a post from: codemonkey.org.uk

No related posts.

For a while I’ve been working on a system call fuzzing tool. A really long time ago, Kurt Garloff wrote one, which I added some improvements to, which just blasted random crap in every register, and called random calls. After it triggered some really stupid bugs which got fixed pretty quickly, it rarely found anything again. Sometimes when a new syscall was added, someone would miss something, and it would trigger some new bug, but for the most part every call just -EINVAL’d immediately.

So I started exploring the idea of writing a tool that instead of passing random junk, actually passed semi sensible data. If the first thing a syscall does is check if a value is between 0 and 3, then passing rand() % 3 is going to get us further into the function than it would if we had just passed rand() unmasked. There are a bunch of other things that can be done too. If a syscall expects a file descriptor, pass one. If it expects an address of a structure, pass it realistic looking addresses (kernel addresses, userspace addresses, ‘weird’ looking addresses).

The new tool just found its first real bug. If perf is running, and mprotect is being fuzzed, then we get an oops. Oops indeed.

I suspect it’ll find some more bugs over time as I add more twists on ‘random’ data. I already have a bunch of ideas that I want to implement (some are listed in the TODO). Seeing a bunch of people at the kernel summit / plumbers conf last week gave me a whole bunch more ideas too.

The git tree for the tool is at git://git.codemonkey.org.uk/trinity.git
(gitweb)
Snapshot tarballs are at http://codemonkey.org.uk/projects/trinity/

system call abuse. is a post from: codemonkey.org.uk

No related posts.

Asides from one problem. Every time I did a yum update that pulled in an selinux policy update, it would consistently exhaust all the ram in the machine. I filed a bug on this, and as usual, Dan Walsh dropped some selinux knowledge that I had no idea about.

You can customize the bzip block size and “small” flag via
/etc/selinux/semanage.conf. After applying you can add entries like these to
your /etc/selinux/semanage.conf to trade off memory vs disk space (block size)
and to trade off memory vs runtime (small):

bzip-blocksize=4
bzip-small=true

You can also disable bzip compression altogether for your module store
via:
bzip-blocksize=0

Since I put that first tweak in place, it’s survived several policy updates without a hiccup.

SELinux on low memory systems. is a post from: codemonkey.org.uk

No related posts.

What the hell was going on ? I started wondering if I could blame it on software. Maybe there was something in the driver that I could tweak. Maybe Pulseaudio was doing something wrong. I spent an afternoon looking for things to configure, going as far as disabling power management features in the hope that was the cause. In the end, I gave up. I just decided that the “High Definition Audio Controller” built into the ICH7 chipset, or some other components in the audio signal path on the motherboard was crap.

A few months ago, Chris Lee visited, and brought with him a NuForce Icon uDAC. (He also brought a pair of $1500 headphones for which he took much ridicule for being an audiophile). I got the chance to try out his setup at the time, and I admit it did sound great (even with my cheapo $99 headphones).

Remembering all this, I decided to pick up a udac, and give it a shot. As suspected, it worked perfectly. Complete plug and play experience, with no complications, and the crystal clear audio that I wanted. I can hear bass frequencies again. High frequencies are reproduced in a manner that doesn’t sound like tinnitus.

It’s weird. I used to think that the days of add-in sound cards were over with the advent of onboard motherboard sound. For as long as there exist motherboard implementations that sound this bad, I’m thankful that you can still pick up inexpensive quality solutions.

nuforce udac is a post from: codemonkey.org.uk

No related posts.

Russian spies. Ad-hoc networks. is a post from: codemonkey.org.uk

No related posts.

The ‘fast startup’ slides are interesting too. The ‘logoff & hibernate’ is the only real ‘new’ feature there afaics. It’s interesting to see terms used that have become passe in Linux like ‘cache prefetching’ and ‘parallel startup’.

interesting windows 8 leaked info. is a post from: codemonkey.org.uk

No related posts.

As you can see in this screenshot, sometimes the ‘incomplete’ message gets printed in green, even though the echo is prefixed with the escape codes for red.

lazyweb , wtf is going on here?

UPDATE: Duhh. typo on line 37. The ‘incomplete’ should say ‘complete’
(When trying to debug this, I replaced ‘complete’ and ‘incomplete’ with ‘red’ and ‘green’. *headdesk*)

Spot the shell bug. is a post from: codemonkey.org.uk

No related posts.