Trinity

A Linux System call fuzz tester.

What is it?.

The basic idea is fairly simple. As 'fuzz testing' suggests, we call syscalls at random, with random arguments.
Not an original idea, and one that has been done many times before on Linux, and on other operating systems.
Where Trinity differs is that the arguments it passes are not purely random.

We found some bugs in the past by just passing random values, but once the *really* dumb bugs were found, these dumb fuzzers would just run and run. The problem was if a syscall took for example a file descriptor as an argument, one of the first things it would try to do was validate that fd. Being garbage, the kernel would just reject it as -EINVAL of course. So on startup, Trinity creates a list of file descriptors, by opening pipes, scanning sysfs, procfs, /dev, and creates a bunch of sockets using random network protocols. Then when a syscall needs an fd, it gets passed one of these at random.

File descriptors aren't the only thing Trinity knows about. Every syscall has its arguments annotated, and where possible it tries to provide something at least semi-sensible. "Length" arguments for example get passed one of a whole bunch of potentially interesting values. (Powers of 2 +/-1 are a good choice for triggering off-by-one bugs it seems).

Trinity also shares those file descriptors between multiple threads, which causes havoc sometimes.

If a child process successfully creates an mmap, the pointer is stored, and fed to subsequent syscalls, sometimes with hilarious results.

Trinity supports Alpha, Aarch64, ARM, i386, IA-64, MIPS, PowerPC-32, PowerPC-64, S390, S390x, SPARC-64, x86-64.
Adding support for additional architectures is a small amount of work mostly involving just defining the order of the syscall table.
See Documentation/HACKING for details.

How do I use it?.

If you run Trinity without any arguments as a non-root user, it will scan for fd's as mentioned above, then create a number of child processes (depending on how many processors you have). These child processes are where the 'call the syscall' happens. It's done in child processes so that if a syscall causes a segfault, trinity will respawn a new thread to take its place. Likewise, if a syscall hangs for more than a few seconds, it will get killed.

I recommend not running it on machines containing data you care about. It's entirely plausible that trinity could start calling unlink() on files it randomly finds. You might also want to make sure that there are no nfs or similar filesystems mounted for the same reason.

With that warning out of the way: Trinity has a neat feature called 'victim files'. You can point at a directory with -V and it will use anything it finds in there as potential sources when it needs a file descriptor. If this is somewhere on NFS for example, this will cause a bunch of filesystem related syscalls over NFS to occur, which might shake out NFS related bugs.

There's a more extensive README in the source distribution which details all the extra parameters. Also recommended reading, are the examples in scripts/ which should give some ideas.

Known problems ?
  • There are almost always new kernel bugs being triggered by trinity. I used to maintain a list of the bugs found/fixed, but it got to be too much work to keep track of.
  • Sometimes, trinity causes the oom-killer to trigger. As long as the oom-killer picks the right process (ie, one of the trinity children, and not dbus, or NetworkManager etc) then this is working as designed.
  • There's a fair amount of "noise" printk's in the kernel that Trinity triggers. I keep a set of patches to remove the most common ones in git. It should always apply to Linus' latest tree. For older kernels, you may have to fix up some parts.
  • If you do find and report a new kernel bug, please Cc me, and mention that you found it with Trinity.

Downloads:
  • Latest release is v1.5, released on March 2nd 2015.
  • The git tree for trinity lives on github. (Please don't send me github pull requests, send email instead).
  • snapshot tarballs are created every hour.

Major changes in git since the last (1.5) tarball release.
  • Assorted improvements to the tuned random number generation.
  • Various networking related improvements/fixes:
    • tcp: add TCP_TIMESTAMP, TCP_NOTSENT_LOWAT & TCP_CC_INFO socket options.
    • ipv6: Improved generation of random addresses. (No longer just localhost)
    • ipv6: Added 14 missing socket options.
    • ipv6: Now passes correct lengths for socket options. (Note: This change may break older glibcs: See this patch.)
    • Beginnings of some better proto-alg sockaddr generation.
    • Recognise PF_IB and PF_MPLS network protocols
    • Socket generation improvements. (Picks right socket type to go with protocol)
    • Now supports an ARG_SOCKETINFO for syscalls that operate primarily on sockets. (Still occasionally passes random fd's)
    • accept,accept4,bind,connect,getpeername,getsockname,recv,setsockopt,send converted to use ARG_SOCKETINFO.
    • setsockopt now also matches the protocol of the socket passed to the right setsockopt args.
    • netlink socket generation fix (pid is a portid, not a process id)
    • The -P parameter no longer accepts the incomprehensible numeric form of arguments, just names.
    • The PF_ prefix to the -P parameter is now optional, so you can just say 'UNIX' instead of 'PF_UNIX'.
  • Updates to keep up with new upstream kernel changes.
    • Updated perf_event_open syscall to include 4.1 changes
    • Updated syscall lists
      alpha: execveat, getrandom, memfd_create
      s390[x]: execveat
      parisc: execveat
    • mips: add new prctls for PR_SET_FP_MODE / PR_GET_FP_MODE
    • Support for new fallocate flags (FALLOC_FL_INSERT_RANGE)
  • Watchdog:
    • Remove some false-positive triggering checks from the watchdog.
    • Watchdog process is now nice'd to -19
    • Monitor how many processes are currently stalled.
    • If all child processes are stalled, send SIGKILLs to 50%
  • Misc:
    • New fd generators for drm dumb buffers & inotify watches.
    • blacklist /dev/sd* from the fd list, so we can be a bit safer when running as root with --dropprivs
    • Fixed the 'bind process to CPU' code to only pick online CPUs.
    • Self-corruption checks added to child processes, like the watchdog code already did.
    • Remove guard pages around shm.
    • In debug mode, write protect the shm before making syscalls.
    • Refactoring of logging code.
    • Various code cleanups as usual.
    • No longer tries to mmap 1GB pages if running with less than 8GB free.

Contact:
There is a mailing list for trinity hosted at vger.kernel.org.
To subscribe, send a mail with "subscribe trinity" in the body to [email protected]
If you get oopses, send them to [email protected] with the relevant maintainer Cc'd.
(I'd appreciate a Cc too <davej atsign codemonkey.org.uk>, and please mention that you used Trinity to trigger it).

Like this?
Since 2015, Trinity is an unfunded entirely spare-time developed project.
Your donations go directly towards buying and running new hardware to run continuous testing.

bitcoin: 15UnuJ42WcJHVg8biotD81y2rKQYMZsuFQ

Web www.codemonkey.org.uk




back to Dave Jones home page..