编辑

How to verify bugs and bisect regressions

How to verify bugs and bisect regressions

This document describes how to check if some Linux kernel problem occurs in code currently supported by developers — to then explain how to locate the change causing the issue, if it is a regression (e.g. did not happen with earlier versions).

The text aims at people running kernels from mainstream Linux distributions on commodity hardware who want to report a kernel bug to the upstream Linux developers. Despite this intent, the instructions work just as well for users who are already familiar with building their own kernels: they help avoid mistakes occasionally made even by experienced developers.

The essence of the process (aka ‘TL;DR’)

[If you are new to building or bisecting Linux, ignore this section and head over to the ‘step-by-step guide <introguide_bissbs>{.interpreted-text role=“ref”}’ below. It utilizes the same commands as this section while describing them in brief fashion. The steps are nevertheless easy to follow and together with accompanying entries in a reference section mention many alternatives, pitfalls, and additional aspects, all of which might be essential in your present case.]

In case you want to check if a bug is present in code currently supported by developers, execute just the preparations and segment 1; while doing so, consider the newest Linux kernel you regularly use to be the ‘working’ kernel. In the following example that’s assumed to be 6.0, which is why its sources will be used to prepare the .config file.

In case you face a regression, follow the steps at least till the end of segment 2. Then you can submit a preliminary report — or continue with segment 3, which describes how to perform a bisection needed for a full-fledged regression report. In the following example 6.0.13 is assumed to be the ‘working’ kernel and 6.1.5 to be the first ‘broken’, which is why 6.0 will be considered the ‘good’ release and used to prepare the .config file.

Step-by-step guide on how to verify bugs and bisect regressions {#introguide_bissbs}

This guide describes how to set up your own Linux kernels for investigating bugs or regressions you intend to report. How far you want to follow the instructions depends on your issue:

Execute all steps till the end of segment 1 to verify if your kernel problem is present in code supported by Linux kernel developers. If it is, you are all set to report the bug — unless it did not happen with earlier kernel versions, as then your want to at least continue with segment 2 to check if the issue qualifies as regression which receive priority treatment. Depending on the outcome you then are ready to report a bug or submit a preliminary regression report; instead of the latter your could also head straight on and follow segment 3 to perform a bisection for a full-fledged regression report developers are obliged to act upon.

Preparations: set up everything to build your own kernels <introprep_bissbs>{.interpreted-text role=“ref”}.

Segment 1: try to reproduce the problem with the latest codebase <introlatestcheck_bissbs>{.interpreted-text role=“ref”}.

Segment 2: check if the kernels you build work fine <introworkingcheck_bissbs>{.interpreted-text role=“ref”}.

Segment 3: perform a bisection and validate the result <introbisect_bissbs>{.interpreted-text role=“ref”}.

Complementary tasks: cleanup during and after following this guide <introclosure_bissbs>{.interpreted-text role=“ref”}.

Optional tasks: test reverts, patches, or later versions <introoptional_bissbs>{.interpreted-text role=“ref”}.

The steps in each segment illustrate the important aspects of the process, while a comprehensive reference section holds additional details for almost all of the steps. The reference section sometimes also outlines alternative approaches, pitfalls, as well as problems that might occur at the particular step — and how to get things rolling again.

For further details on how to report Linux kernel issues or regressions check out Documentation/admin-guide/reporting-issues.rst, which works in conjunction with this document. It among others explains why you need to verify bugs with the latest ‘mainline’ kernel (e.g. versions like 6.0, 6.1-rc1, or 6.1-rc6), even if you face a problem with a kernel from a ‘stable/longterm’ series (say 6.0.13).

For users facing a regression that document also explains why sending a preliminary report after segment 2 might be wise, as the regression and its culprit might be known already. For further details on what actually qualifies as a regression check out Documentation/admin-guide/reporting-regressions.rst.

If you run into any problems while following this guide or have ideas how to improve it, please let the kernel developers know <submit_improvements>{.interpreted-text role=“ref”}.

Preparations: set up everything to build your own kernels {#introprep_bissbs}

The following steps lay the groundwork for all further tasks.

Note: the instructions assume you are building and testing on the same machine; if you want to compile the kernel on another system, check Build kernels on a different machine <buildhost_bis>{.interpreted-text role=“ref”} below.

::: {#backup_bissbs}

::: {#vanilla_bissbs}

::: {#secureboot_bissbs}

::: {#rangecheck_bissbs}

::: {#bootworking_bissbs}

::: {#diskspace_bissbs}

::: {#buildrequires_bissbs}

::: {#sources_bissbs}

::: {#stablesources_bissbs}

:::

::: {#oldconfig_bissbs}

::: {#localmodconfig_bissbs}

::: {#tagging_bissbs}

::: {#debugsymbols_bissbs}

::: {#configmods_bissbs}

::: {#saveconfig_bissbs}

Segment 1: try to reproduce the problem with the latest codebase {#introlatestcheck_bissbs}

The following steps verify if the problem occurs with the code currently supported by developers. In case you face a regression, it also checks that the problem is not caused by some .config change, as reporting the issue then would be a waste of time. [details <introlatestcheck_bisref>{.interpreted-text role=“ref”}]

::: {#checkoutmaster_bissbs}

::: {#build_bissbs}

::: {#install_bissbs}

::: {#storagespace_bissbs}

::: {#kernelrelease_bissbs}

:::

::: {#tainted_bissbs}

::: {#recheckbroken_bissbs}

::: {#recheckstablebroken_bissbs}

Do you follow this guide to verify if a problem is present in the code currently supported by Linux kernel developers? Then you are done at this point. If you later want to remove the kernel you just built, check out Complementary tasks: cleanup during and after following this guide <introclosure_bissbs>{.interpreted-text role=“ref”}.

In case you face a regression, move on and execute at least the next segment as well.

Segment 2: check if the kernels you build work fine {#introworkingcheck_bissbs}

In case of a regression, you now want to ensure the trimmed configuration file you created earlier works as expected; a bisection with the .config file otherwise would be a waste of time. [details <introworkingcheck_bisref>{.interpreted-text role=“ref”}]

::: {#recheckworking_bissbs}

Segment 3: perform the bisection and validate the result {#introbisect_bissbs}

With all the preparations and precaution builds taken care of, you are now ready to begin the bisection. This will make you build quite a few kernels — usually about 15 in case you encountered a regression when updating to a newer series (say from 6.0.13 to 6.1.5). But do not worry, due to the trimmed build configuration created earlier this works a lot faster than many people assume: overall on average it will often just take about 10 to 15 minutes to compile each kernel on commodity x86 machines.

::: {#bisectstart_bissbs}

::: {#bisectbuild_bissbs}

::: {#bisecttest_bissbs}

::: {#bisectlog_bissbs}

::: {#revert_bissbs}

Complementary tasks: cleanup during and after the bisection {#introclosure_bissbs}

During and after following this guide you might want or need to remove some of the kernels you installed: the boot menu otherwise will become confusing or space might run out.

::: {#makeroom_bissbs}

::: {#finishingtouch_bissbs}

Optional: test reverts, patches, or later versions {#introoptional_bissbs}

While or after reporting a bug, you might want or potentially will be asked to test reverts, debug patches, proposed fixes, or other versions. In that case follow these instructions.

[details <introoptional_bisref>{.interpreted-text role=“ref”}]

Conclusion {#submit_improvements}

You have reached the end of the step-by-step guide.

Did you run into trouble following any of the above steps not cleared up by the reference section below? Did you spot errors? Or do you have ideas how to improve the guide?

If any of that applies, please take a moment and let the maintainer of this document know by email (Thorsten Leemhuis <linux@leemhuis.info>), ideally while CCing the Linux docs mailing list (linux-doc@vger.kernel.org). Such feedback is vital to improve this text further, which is in everybody’s interest, as it will enable more people to master the task described here — and hopefully also improve similar guides inspired by this one.

Reference section for the step-by-step guide

This section holds additional information for almost all the items in the above step-by-step guide.

Preparations for building your own kernels

The steps in this section lay the groundwork for all further tests. [... <introprep_bissbs>{.interpreted-text role=“ref”}]

The steps in all later sections of this guide depend on those described here.

[back to step-by-step guide <introprep_bissbs>{.interpreted-text role=“ref”}].

Prepare for emergencies {#backup_bisref}

Create a fresh backup and put system repair and restore tools at hand. [... <backup_bissbs>{.interpreted-text role=“ref”}]

Remember, you are dealing with computers, which sometimes do unexpected things — especially if you fiddle with crucial parts like the kernel of an operating system. That’s what you are about to do in this process. Hence, better prepare for something going sideways, even if that should not happen.

[back to step-by-step guide <backup_bissbs>{.interpreted-text role=“ref”}]

Remove all software that depends on externally developed kernel drivers or builds them automatically. [...<vanilla_bissbs>{.interpreted-text role=“ref”}]

Externally developed kernel modules can easily cause trouble during a bisection.

But there is a more important reason why this guide contains this step: most kernel developers will not care about reports about regressions occurring with kernels that utilize such modules. That’s because such kernels are not considered ‘vanilla’ anymore, as Documentation/admin-guide/reporting-issues.rst explains in more detail.

[back to step-by-step guide <vanilla_bissbs>{.interpreted-text role=“ref”}]

Deal with techniques like Secure Boot {#secureboot_bisref}

On platforms with ‘Secure Boot’ or similar techniques, prepare everything to ensure the system will permit your self-compiled kernel to boot later. [... <secureboot_bissbs>{.interpreted-text role=“ref”}]

Many modern systems allow only certain operating systems to start; that’s why they reject booting self-compiled kernels by default.

You ideally deal with this by making your platform trust your self-built kernels with the help of a certificate. How to do that is not described here, as it requires various steps that would take the text too far away from its purpose; ‘Documentation/admin-guide/module-signing.rst’ and various web sides already explain everything needed in more detail.

Temporarily disabling solutions like Secure Boot is another way to make your own Linux boot. On commodity x86 systems it is possible to do this in the BIOS Setup utility; the required steps vary a lot between machines and therefore cannot be described here.

On mainstream x86 Linux distributions there is a third and universal option: disable all Secure Boot restrictions for your Linux environment. You can initiate this process by running mokutil --disable-validation; this will tell you to create a one-time password, which is safe to write down. Now restart; right after your BIOS performed all self-tests the bootloader Shim will show a blue box with a message ‘Press any key to perform MOK management’. Hit some key before the countdown exposes, which will open a menu. Choose ‘Change Secure Boot state’. Shim’s ‘MokManager’ will now ask you to enter three randomly chosen characters from the one-time password specified earlier. Once you provided them, confirm you really want to disable the validation. Afterwards, permit MokManager to reboot the machine.

[back to step-by-step guide <secureboot_bissbs>{.interpreted-text role=“ref”}]

Boot the last kernel that was working {#bootworking_bisref}

Boot into the last working kernel and briefly recheck if the feature that regressed really works. [...<bootworking_bissbs>{.interpreted-text role=“ref”}]

This will make later steps that cover creating and trimming the configuration do the right thing.

[back to step-by-step guide <bootworking_bissbs>{.interpreted-text role=“ref”}]

Space requirements {#diskspace_bisref}

Ensure to have enough free space for building Linux. [... <diskspace_bissbs>{.interpreted-text role=“ref”}]

The numbers mentioned are rough estimates with a big extra charge to be on the safe side, so often you will need less.

If you have space constraints, be sure to hay attention to the step about debug symbols' <debugsymbols_bissbs>{.interpreted-text role=“ref”} and its accompanying reference section' <debugsymbols_bisref>{.interpreted-text role=“ref”}, as disabling then will reduce the consumed disk space by quite a few gigabytes.

[back to step-by-step guide <diskspace_bissbs>{.interpreted-text role=“ref”}]

Bisection range {#rangecheck_bisref}

Determine the kernel versions considered ‘good’ and ‘bad’ throughout this guide. [...<rangecheck_bissbs>{.interpreted-text role=“ref”}]

Establishing the range of commits to be checked is mostly straightforward, except when a regression occurred when switching from a release of one stable series to a release of a later series (e.g. from 6.0.13 to 6.1.5). In that case Git will need some hand holding, as there is no straight line of descent.

That’s because with the release of 6.0 mainline carried on to 6.1 while the stable series 6.0.y branched to the side. It’s therefore theoretically possible that the issue you face with 6.1.5 only worked in 6.0.13, as it was fixed by a commit that went into one of the 6.0.y releases, but never hit mainline or the 6.1.y series. Thankfully that normally should not happen due to the way the stable/longterm maintainers maintain the code. It’s thus pretty safe to assume 6.0 as a ‘good’ kernel. That assumption will be tested anyway, as that kernel will be built and tested in the segment ‘2’ of this guide; Git would force you to do this as well, if you tried bisecting between 6.0.13 and 6.1.15.

[back to step-by-step guide <rangecheck_bissbs>{.interpreted-text role=“ref”}]

Install build requirements {#buildrequires_bisref}

Install all software required to build a Linux kernel. [...<buildrequires_bissbs>{.interpreted-text role=“ref”}]

The kernel is pretty stand-alone, but besides tools like the compiler you will sometimes need a few libraries to build one. How to install everything needed depends on your Linux distribution and the configuration of the kernel you are about to build.

Here are a few examples what you typically need on some mainstream distributions:

These commands install a few packages that are often, but not always needed. You for example might want to skip installing the development headers for ncurses, which you will only need in case you later might want to adjust the kernel build configuration using make the targets ‘menuconfig’ or ‘nconfig’; likewise omit the headers of Qt6 if you do not plan to adjust the .config using ‘xconfig’.

You furthermore might need additional libraries and their development headers for tasks not covered in this guide — for example when building utilities from the kernel’s tools/ directory.

[back to step-by-step guide <buildrequires_bissbs>{.interpreted-text role=“ref”}]

Download the sources using Git {#sources_bisref}

Retrieve the Linux mainline sources. [...<sources_bissbs>{.interpreted-text role=“ref”}]

The step-by-step guide outlines how to download the Linux sources using a full Git clone of Linus’ mainline repository. There is nothing more to say about that — but there are two alternatives ways to retrieve the sources that might work better for you:

Downloading Linux mainline sources using a bundle {#sources_bundle_bisref}

Use the following commands to retrieve the Linux mainline sources using a bundle:

wget -c \
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/clone.bundle
git clone --no-checkout clone.bundle ~/linux/
cd ~/linux/
git remote remove origin
git remote add mainline \
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git fetch mainline
git remote add -t master stable \
  https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

In case the ‘wget’ command fails, just re-execute it, it will pick up where it left off.

[back to step-by-step guide <sources_bissbs>{.interpreted-text role=“ref”}] [back to section intro <sources_bisref>{.interpreted-text role=“ref”}]

Downloading Linux mainline sources using a shallow clone {#sources_shallow_bisref}

First, execute the following command to retrieve the latest mainline codebase:

git clone -o mainline --no-checkout --depth 1 -b master \
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/
cd ~/linux/
git remote add -t master stable \
  https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

Now deepen your clone’s history to the second predecessor of the mainline release of your ‘good’ version. In case the latter are 6.0 or 6.0.13, 5.19 would be the first predecessor and 5.18 the second — hence deepen the history up to that version:

git fetch --shallow-exclude=v5.18 mainline

Afterwards add the stable Git repository as remote and all required stable branches as explained in the step-by-step guide.

Note, shallow clones have a few peculiar characteristics:

[back to step-by-step guide <sources_bissbs>{.interpreted-text role=“ref”}] [back to section intro <sources_bisref>{.interpreted-text role=“ref”}]

Start defining the build configuration for your kernel {#oldconfig_bisref}

Start preparing a kernel build configuration (the ‘.config’ file). [... <oldconfig_bissbs>{.interpreted-text role=“ref”}]

Note, this is the first of multiple steps in this guide that create or modify build artifacts. The commands used in this guide store them right in the source tree to keep things simple. In case you prefer storing the build artifacts separately, create a directory like ’~/linux-builddir/’ and add the parameter “O=~/linux-builddir/“ to all make calls used throughout this guide. You will have to point other commands there as well — among them the “./scripts/config […]“ commands, which will require “—file ~/linux-builddir/.config“ to locate the right build configuration.

Two things can easily go wrong when creating a .config file as advised:

Note, the olddefconfig target will set any undefined build options to their default value. If you prefer to set such configuration options manually, use make oldconfig instead. Then for each undefined configuration option you will be asked how to proceed; in case you are unsure what to answer, simply hit ‘enter’ to apply the default value. Note though that for bisections you normally want to go with the defaults, as you otherwise might enable a new feature that causes a problem looking like regressions (for example due to security restrictions).

Occasionally odd things happen when trying to use a config file prepared for one kernel (say 6.1) on an older mainline release — especially if it is much older (say 5.15). That’s one of the reasons why the previous step in the guide told you to boot the kernel where everything works. If you manually add a .config file you thus want to ensure it’s from the working kernel and not from a one that shows the regression.

In case you want to build kernels for another machine, locate its kernel build configuration; usually ls /boot/config-$(uname -r) will print its name. Copy that file to the build machine and store it as ~/linux/.config; afterwards run make olddefconfig to adjust it.

[back to step-by-step guide <oldconfig_bissbs>{.interpreted-text role=“ref”}]

Trim the build configuration for your kernel {#localmodconfig_bisref}

Disable any kernel modules apparently superfluous for your setup. [... <localmodconfig_bissbs>{.interpreted-text role=“ref”}]

As explained briefly in the step-by-step guide already: with localmodconfig it can easily happen that your self-built kernels will lack modules for tasks you did not perform at least once before utilizing this make target. That happens when a task requires kernel modules which are only autoloaded when you execute it for the first time. So when you never performed that task since starting your kernel the modules will not have been loaded — and from localmodonfig’s point of view look superfluous, which thus disables them to reduce the amount of code to be compiled.

You can try to avoid this by performing typical tasks that often will autoload additional kernel modules: start a VM, establish VPN connections, loop-mount a CD/DVD ISO, mount network shares (CIFS, NFS, …), and connect all external devices (2FA keys, headsets, webcams, …) as well as storage devices with file systems you otherwise do not utilize (btrfs, ext4, FAT, NTFS, XFS, …). But it is hard to think of everything that might be needed — even kernel developers often forget one thing or another at this point.

Do not let that risk bother you, especially when compiling a kernel only for testing purposes: everything typically crucial will be there. And if you forget something important you can turn on a missing feature manually later and quickly run the commands again to compile and install a kernel that has everything you need.

But if you plan to build and use self-built kernels regularly, you might want to reduce the risk by recording which modules your system loads over the course of a few weeks. You can automate this with modprobed-db. Afterwards use LSMOD=<path> to point localmodconfig to the list of modules modprobed-db noticed being used:

yes '' | make LSMOD='${HOME}'/.config/modprobed.db localmodconfig

That parameter also allows you to build trimmed kernels for another machine in case you copied a suitable .config over to use as base (see previous step). Just run lsmod > lsmod_foo-machine on that system and copy the generated file to your build’s host home directory. Then run these commands instead of the one the step-by-step guide mentions:

yes '' | make LSMOD=~/lsmod_foo-machine localmodconfig

[back to step-by-step guide <localmodconfig_bissbs>{.interpreted-text role=“ref”}]

Tag the kernels about to be build {#tagging_bisref}

Ensure all the kernels you will build are clearly identifiable using a special tag and a unique version identifier. [... <tagging_bissbs>{.interpreted-text role=“ref”}]

This allows you to differentiate your distribution’s kernels from those created during this process, as the file or directories for the latter will contain ‘-local’ in the name; it also helps picking the right entry in the boot menu and not lose track of you kernels, as their version numbers will look slightly confusing during the bisection.

[back to step-by-step guide <tagging_bissbs>{.interpreted-text role=“ref”}]

Decide to enable or disable debug symbols {#debugsymbols_bisref}

Decide how to handle debug symbols. [... <debugsymbols_bissbs>{.interpreted-text role=“ref”}]

Having debug symbols available can be important when your kernel throws a ‘panic’, ‘Oops’, ‘warning’, or ‘BUG’ later when running, as then you will be able to find the exact place where the problem occurred in the code. But collecting and embedding the needed debug information takes time and consumes quite a bit of space: in late 2022 the build artifacts for a typical x86 kernel trimmed with localmodconfig consumed around 5 Gigabyte of space with debug symbols, but less than 1 when they were disabled. The resulting kernel image and modules are bigger as well, which increases storage requirements for /boot/ and load times.

In case you want a small kernel and are unlikely to decode a stack trace later, you thus might want to disable debug symbols to avoid those downsides. If it later turns out that you need them, just enable them as shown and rebuild the kernel.

You on the other hand definitely want to enable them for this process, if there is a decent chance that you need to decode a stack trace later. The section ‘Decode failure messages’ in Documentation/admin-guide/reporting-issues.rst explains this process in more detail.

[back to step-by-step guide <debugsymbols_bissbs>{.interpreted-text role=“ref”}]

Adjust build configuration {#configmods_bisref}

Check if you may want or need to adjust some other kernel configuration options:

Depending on your needs you at this point might want or have to adjust some kernel configuration options.

Distro specific adjustments {#configmods_distros_bisref}

Are you running [... <configmods_bissbs>{.interpreted-text role=“ref”}]

The following sections help you to avoid build problems that are known to occur when following this guide on a few commodity distributions.

Debian:

[back to step-by-step guide <configmods_bissbs>{.interpreted-text role=“ref”}]

Individual adjustments {#configmods_individual_bisref}

If you want to influence the other aspects of the configuration, do so now. [... <configmods_bissbs>{.interpreted-text role=“ref”}]

At this point you can use a command like make menuconfig or make nconfig to enable or disable certain features using a text-based user interface; to use a graphical configuration utility, run make xconfig instead. Both of them require development libraries from toolkits they are rely on (ncurses respectively Qt5 or Qt6); an error message will tell you if something required is missing.

[back to step-by-step guide <configmods_bissbs>{.interpreted-text role=“ref”}]

Put the .config file aside {#saveconfig_bisref}

Reprocess the .config after the latest changes and store it in a safe place. [... <saveconfig_bissbs>{.interpreted-text role=“ref”}]

Put the .config you prepared aside, as you want to copy it back to the build directory every time during this guide before you start building another kernel. That’s because going back and forth between different versions can alter .config files in odd ways; those occasionally cause side effects that could confuse testing or in some cases render the result of your bisection meaningless.

[back to step-by-step guide <saveconfig_bissbs>{.interpreted-text role=“ref”}]

Try to reproduce the problem with the latest codebase {#introlatestcheck_bisref}

Verify the regression is not caused by some .config change and check if it still occurs with the latest codebase. [... <introlatestcheck_bissbs>{.interpreted-text role=“ref”}]

For some readers it might seem unnecessary to check the latest codebase at this point, especially if you did that already with a kernel prepared by your distributor or face a regression within a stable/longterm series. But it’s highly recommended for these reasons:

[back to step-by-step guide <introlatestcheck_bissbs>{.interpreted-text role=“ref”}]

Check out the latest Linux codebase {#checkoutmaster_bisref}

Check out the latest Linux codebase. [... <checkoutmaster_bissbs>{.interpreted-text role=“ref”}]

In case you later want to recheck if an ever newer codebase might fix the problem, remember to run that git fetch --shallow-exclude [...] command again mentioned earlier to update your local Git repository.

[back to step-by-step guide <checkoutmaster_bissbs>{.interpreted-text role=“ref”}]

Build your kernel {#build_bisref}

Build the image and the modules of your first kernel using the config file you prepared. [... <build_bissbs>{.interpreted-text role=“ref”}]

A lot can go wrong at this stage, but the instructions below will help you help yourself. Another subsection explains how to directly package your kernel up as deb, rpm or tar file.

Dealing with build errors

When a build error occurs, it might be caused by some aspect of your machine’s setup that often can be fixed quickly; other times though the problem lies in the code and can only be fixed by a developer. A close examination of the failure messages coupled with some research on the internet will often tell you which of the two it is. To perform such investigation, restart the build process like this:

make V=1

The V=1 activates verbose output, which might be needed to see the actual error. To make it easier to spot, this command also omits the -j $(nproc --all) used earlier to utilize every CPU core in the system for the job — but this parallelism also results in some clutter when failures occur.

After a few seconds the build process should run into the error again. Now try to find the most crucial line describing the problem. Then search the internet for the most important and non-generic section of that line (say 4 to 8 words); avoid or remove anything that looks remotely system-specific, like your username or local path names like /home/username/linux/. First try your regular internet search engine with that string, afterwards search Linux kernel mailing lists via lore.kernel.org/all/.

This most of the time will find something that will explain what is wrong; quite often one of the hits will provide a solution for your problem, too. If you do not find anything that matches your problem, try again from a different angle by modifying your search terms or using another line from the error messages.

In the end, most issues you run into have likely been encountered and reported by others already. That includes issues where the cause is not your system, but lies in the code. If you run into one of those, you might thus find a solution (e.g. a patch) or workaround for your issue, too.

Package your kernel up

The step-by-step guide uses the default make targets (e.g. ‘bzImage’ and ‘modules’ on x86) to build the image and the modules of your kernel, which later steps of the guide then install. You instead can also directly build everything and directly package it up by using one of the following targets:

This is just a selection of available make targets for this purpose, see make help for others. You can also use these targets after running make -j $(nproc --all), as they will pick up everything already built.

If you employ the targets to generate deb or rpm packages, ignore the step-by-step guide’s instructions on installing and removing your kernel; instead install and remove the packages using the package utility for the format (e.g. dpkg and rpm) or a package management utility build on top of them (apt, aptitude, dnf/yum, zypper, …). Be aware that the packages generated using these two make targets are designed to work on various distributions utilizing those formats, they thus will sometimes behave differently than your distribution’s kernel packages.

[back to step-by-step guide <build_bissbs>{.interpreted-text role=“ref”}]

Put the kernel in place {#install_bisref}

Install the kernel you just built. [... <install_bissbs>{.interpreted-text role=“ref”}]

What you need to do after executing the command in the step-by-step guide depends on the existence and the implementation of /sbin/installkernel executable on your distribution.

If installkernel is found, the kernel’s build system will delegate the actual installation of your kernel image to this executable, which then performs some or all of these tasks:

You have to take care of some or all of the tasks yourself, if your distribution lacks a installkernel script or does only handle part of them. Consult the distribution’s documentation for details. If in doubt, install the kernel manually:

sudo install -m 0600 $(make -s image_name) /boot/vmlinuz-$(make -s kernelrelease)
sudo install -m 0600 System.map /boot/System.map-$(make -s kernelrelease)

Now generate your initramfs using the tools your distribution provides for this process. Afterwards add your kernel to your bootloader configuration and reboot.

[back to step-by-step guide <install_bissbs>{.interpreted-text role=“ref”}]

Storage requirements per kernel {#storagespace_bisref}

Check how much storage space the kernel, its modules, and other related files like the initramfs consume. [... <storagespace_bissbs>{.interpreted-text role=“ref”}]

The kernels built during a bisection consume quite a bit of space in /boot/ and /lib/modules/, especially if you enabled debug symbols. That makes it easy to fill up volumes during a bisection — and due to that even kernels which used to work earlier might fail to boot. To prevent that you will need to know how much space each installed kernel typically requires.

Note, most of the time the pattern ‘/boot/*$(make -s kernelrelease)*’ used in the guide will match all files needed to boot your kernel — but neither the path nor the naming scheme are mandatory. On some distributions you thus will need to look in different places.

[back to step-by-step guide <storagespace_bissbs>{.interpreted-text role=“ref”}]

Check if your newly built kernel considers itself ‘tainted’ {#tainted_bisref}

Check if the kernel marked itself as ‘tainted’. [... <tainted_bissbs>{.interpreted-text role=“ref”}]

Linux marks itself as tainted when something happens that potentially leads to follow-up errors that look totally unrelated. That is why developers might ignore or react scantly to reports from tainted kernels — unless of course the kernel set the flag right when the reported bug occurred.

That’s why you want check why a kernel is tainted as explained in Documentation/admin-guide/tainted-kernels.rst; doing so is also in your own interest, as your testing might be flawed otherwise.

[back to step-by-step guide <tainted_bissbs>{.interpreted-text role=“ref”}]

Check the kernel built from a recent mainline codebase {#recheckbroken_bisref}

Verify if your bug occurs with the newly built kernel. [... <recheckbroken_bissbs>{.interpreted-text role=“ref”}]

There are a couple of reasons why your bug or regression might not show up with the kernel you built from the latest codebase. These are the most frequent:

[back to step-by-step guide <recheckbroken_bissbs>{.interpreted-text role=“ref”}]

Check the kernel built from the latest stable/longterm codebase {#recheckstablebroken_bisref}

Are you facing a regression within a stable/longterm release, but failed to reproduce it with the kernel you just built using the latest mainline sources? Then check if the latest codebase for the particular series might already fix the problem. [... <recheckstablebroken_bissbs>{.interpreted-text role=“ref”}]

If this kernel does not show the regression either, there most likely is no need for a bisection.

[back to step-by-step guide <recheckstablebroken_bissbs>{.interpreted-text role=“ref”}]

Ensure the ‘good’ version is really working well {#introworkingcheck_bisref}

Check if the kernels you build work fine. [... <introworkingcheck_bissbs>{.interpreted-text role=“ref”}]

This section will reestablish a known working base. Skipping it might be appealing, but is usually a bad idea, as it does something important:

It will ensure the .config file you prepared earlier actually works as expected. That is in your own interest, as trimming the configuration is not foolproof —and you might be building and testing ten or more kernels for nothing before starting to suspect something might be wrong with the build configuration.

That alone is reason enough to spend the time on this, but not the only reason.

Many readers of this guide normally run kernels that are patched, use add-on modules, or both. Those kernels thus are not considered ‘vanilla’ — therefore it’s possible that the thing that regressed might never have worked in vanilla builds of the ‘good’ version in the first place.

There is a third reason for those that noticed a regression between stable/longterm kernels of different series (e.g. 6.0.13..6.1.5): it will ensure the kernel version you assumed to be ‘good’ earlier in the process (e.g. 6.0) actually is working.

[back to step-by-step guide <introworkingcheck_bissbs>{.interpreted-text role=“ref”}]

Build your own version of the ‘good’ kernel {#recheckworking_bisref}

Build your own variant of the working kernel and check if the feature that regressed works as expected with it. [... <recheckworking_bissbs>{.interpreted-text role=“ref”}]

In case the feature that broke with newer kernels does not work with your first self-built kernel, find and resolve the cause before moving on. There are a multitude of reasons why this might happen. Some ideas where to look:

Note, if you found and fixed problems with the .config file, you want to use it to build another kernel from the latest codebase, as your earlier tests with mainline and the latest version from an affected stable/longterm series were most likely flawed.

[back to step-by-step guide <recheckworking_bissbs>{.interpreted-text role=“ref”}]

Perform a bisection and validate the result

With all the preparations and precaution builds taken care of, you are now ready to begin the bisection. [... <introbisect_bissbs>{.interpreted-text role=“ref”}]

The steps in this segment perform and validate the bisection.

[back to step-by-step guide <introbisect_bissbs>{.interpreted-text role=“ref”}].

Start the bisection {#bisectstart_bisref}

Start the bisection and tell Git about the versions earlier established as ‘good’ and ‘bad’. [... <bisectstart_bissbs>{.interpreted-text role=“ref”}]

This will start the bisection process; the last of the commands will make Git check out a commit round about half-way between the ‘good’ and the ‘bad’ changes for you to test.

[back to step-by-step guide <bisectstart_bissbs>{.interpreted-text role=“ref”}]

Build a kernel from the bisection point {#bisectbuild_bisref}

Build, install, and boot a kernel from the code Git checked out using the same commands you used earlier. [... <bisectbuild_bissbs>{.interpreted-text role=“ref”}]

There are two things worth of note here:

[back to step-by-step guide <bisectbuild_bissbs>{.interpreted-text role=“ref”}]

Bisection checkpoint {#bisecttest_bisref}

Check if the feature that regressed works in the kernel you just built. [... <bisecttest_bissbs>{.interpreted-text role=“ref”}]

Ensure what you tell Git is accurate: getting it wrong just one time will bring the rest of the bisection totally off course, hence all testing after that point will be for nothing.

[back to step-by-step guide <bisecttest_bissbs>{.interpreted-text role=“ref”}]

Put the bisection log away {#bisectlog_bisref}

Store Git’s bisection log and the current .config file in a safe place. [... <bisectlog_bissbs>{.interpreted-text role=“ref”}]

As indicated above: declaring just one kernel wrongly as ‘good’ or ‘bad’ will render the end result of a bisection useless. In that case you’d normally have to restart the bisection from scratch. The log can prevent that, as it might allow someone to point out where a bisection likely went sideways — and then instead of testing ten or more kernels you might only have to build a few to resolve things.

The .config file is put aside, as there is a decent chance that developers might ask for it after you report the regression.

[back to step-by-step guide <bisectlog_bissbs>{.interpreted-text role=“ref”}]

Try reverting the culprit {#revert_bisref}

Try reverting the culprit on top of the latest codebase to see if this fixes your regression. [... <revert_bissbs>{.interpreted-text role=“ref”}]

This is an optional step, but whenever possible one you should try: there is a decent chance that developers will ask you to perform this step when you bring the bisection result up. So give it a try, you are in the flow already, building one more kernel shouldn’t be a big deal at this point.

The step-by-step guide covers everything relevant already except one slightly rare thing: did you bisected a regression that also happened with mainline using a stable/longterm series, but Git failed to revert the commit in mainline? Then try to revert the culprit in the affected stable/longterm series — and if that succeeds, test that kernel version instead.

[back to step-by-step guide <revert_bissbs>{.interpreted-text role=“ref”}]

Cleanup steps during and after following this guide

During and after following this guide you might want or need to remove some of the kernels you installed. [... <introclosure_bissbs>{.interpreted-text role=“ref”}]

The steps in this section describe clean-up procedures.

[back to step-by-step guide <introclosure_bissbs>{.interpreted-text role=“ref”}].

Cleaning up during the bisection {#makeroom_bisref}

To remove one of the kernels you installed, look up its ‘kernelrelease’ identifier. [... <makeroom_bissbs>{.interpreted-text role=“ref”}]

The kernels you install during this process are easy to remove later, as its parts are only stored in two places and clearly identifiable. You thus do not need to worry to mess up your machine when you install a kernel manually (and thus bypass your distribution’s packaging system): all parts of your kernels are relatively easy to remove later.

One of the two places is a directory in /lib/modules/, which holds the modules for each installed kernel. This directory is named after the kernel’s release identifier; hence, to remove all modules for one of the kernels you built, simply remove its modules directory in /lib/modules/.

The other place is /boot/, where typically two up to five files will be placed during installation of a kernel. All of them usually contain the release name in their file name, but how many files and their exact names depend somewhat on your distribution’s installkernel executable and its initramfs generator. On some distributions the kernel-install remove... command mentioned in the step-by-step guide will delete all of these files for you while also removing the menu entry for the kernel from your bootloader configuration. On others you have to take care of these two tasks yourself. The following command should interactively remove the three main files of a kernel with the release name ‘6.0-rc1-local-gcafec0cacaca0’:

rm -i /boot/{System.map,vmlinuz,initr}-6.0-rc1-local-gcafec0cacaca0

Afterwards check for other files in /boot/ that have ‘6.0-rc1-local-gcafec0cacaca0’ in their name and consider deleting them as well. Now remove the boot entry for the kernel from your bootloader’s configuration; the steps to do that vary quite a bit between Linux distributions.

Note, be careful with wildcards like ’*’ when deleting files or directories for kernels manually: you might accidentally remove files of a 6.0.13 kernel when all you want is to remove 6.0 or 6.0.1.

[back to step-by-step guide <makeroom_bissbs>{.interpreted-text role=“ref”}]

Cleaning up after the bisection

::: {#finishingtouch_bisref}

Once you have finished the bisection, do not immediately remove anything you set up, as you might need a few things again. [... <finishingtouch_bissbs>{.interpreted-text role=“ref”}] :::

When you are really short of storage space removing the kernels as described in the step-by-step guide might not free as much space as you would like. In that case consider running rm -rf ~/linux/* as well now. This will remove the build artifacts and the Linux sources, but will leave the Git repository (~/linux/.git/) behind — a simple git reset --hard thus will bring the sources back.

Removing the repository as well would likely be unwise at this point: there is a decent chance developers will ask you to build another kernel to perform additional tests — like testing a debug patch or a proposed fix. Details on how to perform those can be found in the section Optional tasks: test reverts, patches, or later versions <introoptional_bissbs>{.interpreted-text role=“ref”}.

Additional tests are also the reason why you want to keep the ~/kernel-config-working file around for a few weeks.

[back to step-by-step guide <finishingtouch_bissbs>{.interpreted-text role=“ref”}]

Test reverts, patches, or later versions {#introoptional_bisref}

While or after reporting a bug, you might want or potentially will be asked to test reverts, patches, proposed fixes, or other versions. [... <introoptional_bissbs>{.interpreted-text role=“ref”}]

All the commands used in this section should be pretty straight forward, so there is not much to add except one thing: when setting a kernel tag as instructed, ensure it is not much longer than the one used in the example, as problems will arise if the kernelrelease identifier exceeds 63 characters.

[back to step-by-step guide <introoptional_bissbs>{.interpreted-text role=“ref”}].

Additional information

Build kernels on a different machine {#buildhost_bis}

To compile kernels on another system, slightly alter the step-by-step guide’s instructions:

This approach even works when building for another architecture: just install cross-compilers and add the appropriate parameters to every invocation of make (e.g. make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- [...]).

Additional reading material

原文链接： https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst

选题： tttturtle-russ

本文将由 HCTT 翻译团队 原创翻译，华中科技大学开放原子开源俱乐部荣誉推出。