syzkaller reproducers

syzbot aims at providing stand-alone C reproducers for all reported bugs. However, sometimes it can’t extract a reproducer at all, or can only extract a syzkaller reproducer. syzkaller reproducers are programs in a special syzkaller notation and they can be executed on the target system with a little bit more effort. See this for instructions.

A syskaller program can also give you an idea as to what syscalls with what arguments were executed (note that some calls can actually be executed in parallel).

A syzkaller program can be converted to an almost equivalent C source using syz-prog2c utility. syz-prog2c has lots of flags in common with syz-execprog, e.g. -threaded which controls if the syscalls are executed sequentially or in parallel. An example invocation:

syz-prog2c -prog repro.syz.txt -enable=all -threaded -repeat -procs=8 -sandbox=namespace -segv -tmpdir -waitrepeat

However, note that if syzbot did not provide a C reproducer, it wasn’t able to trigger the bug using the C program (though, it can be just because the bug is triggered by a subtle race condition).

Downloadable assets

Syzbot shares links to the bootable disk image, kernel object file and other relevant files that can be used to reproduce and debug the issue locally.

See this tutorial on how to use them.

Crash does not reproduce?

If the provided reproducer does not work for you, most likely it is related to the fact that you have slightly different setup than syzbot. syzbot has obtained the provided crash report on the provided reproducer on a freshly-booted machine, so the reproducer worked for it somehow.

Note: if the report contains userspace arch: i386, then the program needs to be built with -m32 flag.

syzbot uses GCE VMs for testing, but usually it is not important.

If the reproducer exits quickly, try to run it several times, or in a loop. There can be some races involved.

Latest compiler used by syzbot is contained in gcr.io/syzkaller/syzbot:gcc-10.2.1 docker image. For in-tree kernel build in current directory it can be used as follows:

docker pull gcr.io/syzkaller/syzbot:gcc-10.2.1
docker run -it --user $(id -u ${USER}):$(id -g ${USER}) \
	--volume "$PWD:/syzkaller/pwd" --workdir /syzkaller/pwd \
	gcr.io/syzkaller/syzbot:gcc-10.2.1
make

For an out-of-tree build one needs to proxy more host directories using --volume flag.

Older compilers used by syzbot can be found here:

• gcc 7.1.1 20170620 (245MB)
• gcc 8.0.1 20180301 (286MB)
• gcc 8.0.1 20180412 (33MB)
• gcc 9.0.0 20181231 (30MB)
• gcc 10.1.0-syz (20200507) (220MB)
• gcc 10.2.0 (165MB)
• clang 7.0.0 (trunk 329060) (44MB)
• clang 7.0.0 (trunk 334104) (44MB)
• clang 8.0.0 (trunk 343298) (45MB)
• clang 10.0.0 (c2443155)
• clang 11.0.0 (git ca2dcbd030e) (682MB)
• clang 11.0.1

A QEMU-suitable Debian Stretch image can be found here (241 MB). A reference qemu command line to run it is as follows:

qemu-system-x86_64 -smp 2 -m 4G -enable-kvm -cpu host \
    -net nic -net user,hostfwd=tcp::10022-:22 \
    -kernel arch/x86/boot/bzImage -nographic \
    -device virtio-scsi-pci,id=scsi \
    -device scsi-hd,bus=scsi.0,drive=d0 \
    -drive file=stretch-amd64.img,format=raw,if=none,id=d0 \
    -append "root=/dev/sda1 console=ttyS0 earlyprintk=serial"

And then you can ssh into it using:

ssh -p 10022 root@localhost

Note: before March 25th 2020 Debian Wheezy image was used for testing, so some of the bugs reported before that date might only be reproducible on Wheezy. That image is here and the key for it is here.

No reproducer at all?

Reproducers are best-effort. syzbot always tries to create reproducers, and once it has one it adds it to the bug. If there is no reproducer referenced in a bug, a reproducer does not exist. There are multiple reasons why syzbot can fail to create a reproducer: some crashes are caused by subtle races and are very hard to reproduce in general; some crashes are caused by global accumulated state in kernel (e.g. lockdep reports); some crashes are caused by non-reproducible coincidences (e.g. an integer 0x12345 happened to reference an existing IPC object) and there is long tail of other reasons.

Moderation queue

Bugs with reproducers are automatically reported to kernel mailing lists. Bugs without reproducers are first staged in moderation queue to filter out invalid, unactionable or duplicate reports. Staged bugs are shown on dashboard in moderation section and mailed to syzkaller-upstream-moderation mailing list. Staged bugs accept all commands supported for reported bugs (fix, dup, invalid) with a restriction that bugs reported upstream can’t be dup-ed onto bugs in moderation queue. Additionally, staged bugs accept upstream command:

#syz upstream

which sends the bug to kernel mailing lists.

KMSAN bugs

KMSAN is a dynamic, compiler-based tool (similar to KASAN) that detects uses of uninitialized values. As compared to (now deleted) KMEMCHECK which simply detected loads of non-stored-to memory, KMSAN tracks precise propagation of uninitialized values through memory and registers and only flags actual eventual uses of uninitialized values. For example, KMSAN will detect a branch on or a copy_to_user() of values that transitively come from uninitialized memory created by heap/stack allocations. This ensures /theoretical/ absence of both false positives and false negatives (with some implementation limitations of course). Note that KMSAN requires clang compiler.

KMSAN is not upstream yet, though, we want to upstream it later. For now, it lives in github.com/google/kmsan and is based on a reasonably fresh upstream tree. As the result, any patch testing requests for KMSAN bugs need to go to KMSAN tree (https://github.com/google/kmsan.git repo, master branch). A standard way for triggering the test with KMSAN tree is to send an email to syzbot+HASH address containing the following line:

#syz test: https://github.com/google/kmsan.git master

and attach/inline your test patch in the same email.

Report explanation. The first call trace points to the use of the uninit value (which is usually a branching or copying it to userspace). Then there are 0 or more “Uninit was stored to memory at:” stacks which denote how the unint value traveled through memory. Finally there is a “Uninit was created at:” section which points either to a heap allocation or a stack variable which is the original source of uninitialized-ness.

USB bugs

syzkaller has an ability to perform fuzzing of the Linux kernel USB stack, see the details here. As of now all kernel changes required for USB fuzzing have been merged into the mainline (the last one during the 5.8-rc1 merge window), so the USB fuzzing instance has been switched to target the usb-testing tree.

Testing kernel patches on the USB instance follows the same principle as on the mainline instances, with a few caveats:

1. You may specify any kernel tree for syz test as long as it includes all mainline patches up to 5.8-rc1.

2. Some of the bugs have reproducers generated on kernel versions with custom kernel (when fuzzing was performed with non-yet-mainlined kernel patches), thus those reproducers might no longer work. The recommended workflow is to: first, execute a syz test command on a target tree to make sure that the bug reproduces, and then execute a syz test command with a fix/debug patch.

If the bug was triggered on the KMSAN tree, follow the instructions above, with the exception that you must also use commit-hash instead of the master branch when testing patches.

Memory leaks

syzbot uses KMEMLEAK to find memory leaks in the Linux kernel. KMEMLEAK kernel config is stored here. See KMEMLEAK docs for general info, algorithm overview and usage instructions.

Memory leaks may be tricky to debug because we have only the allocation stack, but we don’t see where kernel code forgot to free the object or drop a reference. KMEMLEAK can have some false positives on tricky kernel code that hides pointers to live objects and due to memory scanning non-atomicity. But don’t write off everything to false positives right away, the rate of false positives is observed to be very low. In particular, KMEMLEAK is not confused by pointers stored in a middle of another object; and it’s not confused if several pointer low bits are used as flags because a pointer into a middle of an object also marks the target as reachable.

A useful litmus test is to remove KMEMLEAK code from the reproducer and run it for longer and/or multiple times. If memory consumption and number of live objects in /proc/slabinfo steadily grow, most likely the leak is real.

KCSAN bugs

The Kernel Concurrency Sanitizer (KCSAN) is a dynamic data-race detector. Reproduction of data-races is currently unsupported, and syzbot is unable to test patches.

No custom patches

While syzbot can test patches that fix bugs, it does not support applying custom patches during fuzzing. It always tests vanilla unmodified git trees. There are several reasons for this:

• custom patches may not apply tomorrow
• custom patches may not apply to all of the tested git trees
• it’s hard to communicate exact state of the code with bug reports (not just hash anymore)
• line numbers won’t match in reports (which always brings suspicion as to the quality of reports)
• custom patches can also introduce bugs, and even if they don’t a developer may (rightfully) question validity of and may not want to spend time on reports obtained with a number of out-of-tree patches
• order of patch application generally matters, and at some point patches need to be removed, there is nobody to manage this

We’ve experimented with application of custom patches in the past and it lead to unrecoverable mess. If you want syzbot to pick up patches sooner, ask tree maintainers for priority handling.

However, syzbot kernel config always includes CONFIG_DEBUG_AID_FOR_SYZBOT=y setting, which is not normally present in kernel. What was used for particularly elusive bugs in the past is temporary merging some additional debugging code into linux-next under this config setting (e.g. more debug checks and/or debug output) and waiting for new crash reports from syzbot.

One can also always run syzkaller locally on any kernel for better stress testing of a particular subsystem and/or patch.

Kernel configs

Kernel configs, sysctls and command line arguments that syzbot uses are available in /dashboard/config.

Is syzbot code available?

Yes, it is here.