syz-testbed

syz-testbed is the tool that simplifies the process of evaluating the performance of different syzkaller versions (or configurations) against each other. The tool automates checking out syzkaller repos, building them, running syz-managers and collecting/summarizing their results.

Configuring syz-testbed

syz-testbed requires a JSON config file. An example of such a file:

{
  "workdir": "/tmp/syz-testbed-workdir/",
  "corpus": "/tmp/corpus.db",
  "target": "syz-manager",
  "max_instances": 5,
  "run_time": "24h",
  "http": "0.0.0.0:50000",
  "checkouts": [
    {
      "name": "first",
      "repo": "https://github.com/google/syzkaller.git",
    },
    {
      "name": "second",
      "repo": "https://github.com/google/syzkaller.git",
      "branch": "some-dev-branch",
    }
  ],
  "manager_config": {
	  "target": "linux/amd64",
	  "kernel_obj": "/tmp/linux-stable",
	  "image": "/tmp/kernel-image/bullseye.img",
	  "sshkey": "/tmp/kernel-image/bullseye.id_rsa",
	  "procs": 8,
	  "type": "qemu",
	  "vm": {
          "count": 2,
          "kernel": "/tmp/linux-stable/arch/x86/boot/bzImage",
          "cpu": 2,
          "mem": 2048
	  }
  }
}

Given such a configuration file, syz-testbed will do the following:

1. Check out the master branch of https://github.com/google/syzkaller.git into /tmp/syz-testbed-workdir/checkouts/first/ and build it.
2. Check out the some-dev-branch of https://github.com/google/syzkaller.git into /tmp/syz-testbed-workdir/checkouts/second/ and build it.
3. Set up and run 3 instances of first and 2 instances of second (max_instances = 5).

The directory structure looks as follows:

/tmp/syz-testbed-workdir/
└── checkouts
    ├── first
    │   ├── run-first-0
    │   │   ├── log.txt
    │   │   ├── manager.cfg
    │   │   └── workdir
    │   ├── run-first-1
    │   │   ├── log.txt
    │   │   ├── manager.cfg
    │   │   └── workdir
    │   └── run-first-4
    │   │   ├── log.txt
    │   │   ├── manager.cfg
    │   │   └── workdir
    └── second
        ├── run-second-2
        │   ├── log.txt
        │   ├── manager.cfg
        │   └── workdir
        └── run-second-3
            ├── log.txt
            ├── manager.cfg
            └── workdir

4. After 24 hours (as run_hours is 24), stop those 5 instances.
5. Create and run 2 instances of first and 3 instances of second.

The tool stops after receiving a SIGINT (e.g. after Ctrl+C) or a SIGTERM signal. Also, if one of the instances has exited due to some error, this also stops the whole experiment.

Web interface

The tool has a simple web interface that displays the current information about the experiment (the number of active and finished instances, the time until instances are stopped, etc.) and the latest statistics collected from the syz-managers.

If the benchmp parameter points to the syz-benchcmp executable, then the web interface can also generate graphs of various parameters over time or the number of executions.

In order to enable the interface, set the http parameter to the IP address and port to which syz-testbed should bind. E.g. "http": "0.0.0.0:50000".

Statistics

syz-testbed provides two “views” of the statistics:

1. complete - only includes data from the finished instances (i.e. those that have been running for run_hours).
2. all - also includes the data from the currently active instances. The statistics from the finished instances is winded back to match the current uptime of the active instances.

Therefore, the statistics is laid out the following way.

$ tree -L 2 /tmp/syz-testbed-workdir/
/tmp/syz-testbed-workdir/
├── stats_all
│   ├── benches
│   │   ├── avg_first.txt
│   │   ├── avg_second.txt
│   ├── bugs.csv
│   ├── checkout_stats.csv
│   └── instance_stats.csv
├── stats_completed
│   ├── benches
│   │   ├── avg_first.txt
│   │   ├── avg_second.txt
│   ├── bugs.csv
│   ├── checkout_stats.csv
│   └── instance_stats.csv
└── testbed.csv

1. bugs.csv contains all the bugs found by the running instances. If a single checkout has several instances (i.e. count > 1), syz-testbed takes a union of bugs found by them. The purpose is ultimately to collect all bugs that could be found by that version of syzkaller.
2. Statistics that is generated by individual syz-managers is saved into instance_stats.csv. The same data is also averaged among instances that belong to the same checkouts and saved into the checkout_stats.csv file.
3. Bench files (see tools/syz-benchcmp) of all syz-managers belonging to a single checkout are averaged and saved into the corresponding files in the benches folder.

The statics is updated once every 90 seconds.

Running syz-testbed

First, checkout the most recent version of syzkaller itself:

$ git clone https://github.com/google/syzkaller.git

Then, build syz-testbed:

$ cd syzkaller/tools/syz-testbed/
$ go build

Write and save the configuration file (e.g. into the config.json file). Then, syz-testbed can be run using the following command:

$ ./syz-testbed -config config.json

Stopping the syz-testbed process results in stopping all the syzkaller instances.

Testing syz-repro

syz-testbed can also be used to test syzkaller’s ability to reproduce bugs. To do so, set the target property in the syz-testbed’s config file to syz-repro.

One can also specify the source of the crash log files. This is either just a folder, whose files will be treated accordingly or it can be a syzkaller’s workdir. input_logs must point to the folde with crash logs - syz-testbed will traverse it and treat each file as an input. Otherwise, input_workdir must be used.

For example:

  "repro_config": {
    "input_workdir": "/tmp/some-syzkaller-workdir",
    "crashes_per_bug": 2,
    "skip_bugs": ["SYZFAIL", "no output", "corrupted", "lost connection"]
  },

In this case, syz-testbed will traverse all bugs found by the syzkaller, skip those that match “SYZFAIL”, “no output”, “corrupted” or “lost connection”, then pick 2 random crash logs for each such bug for later processing.

syz-testbed will check out and compile the specified syzkaller instances and will go on executing their syz-repros on each picked up crash log file, as long as the tool is not stopped.