Coder Social home page Coder Social logo

gghh / kernel-source Goto Github PK

View Code? Open in Web Editor NEW

This project forked from suse/kernel-source

0.0 0.0 0.0 697.13 MB

openSUSE sched-ext kernel

License: GNU General Public License v2.0

Shell 31.25% Ruby 0.21% Python 49.43% Perl 17.72% C 0.02% Awk 0.35% Makefile 0.47% Dockerfile 0.55%

kernel-source's Introduction

SUSE Kernel Repository

Overview

The kernel-source repository contains sources, configuration files, package definitions and supporting scripts for the SUSE kernels.

The SUSE kernels are generated from the upstream Linux kernel sources found at https://kernel.org/, on top of which a number of patches are applied. The expanded kernel source tree is configured and built, resulting in a binary kernel.

Getting started

Make sure you have the git and quilt tools installed.

Introduce yourself if you haven't done so already:

$ git config --global user.name "Your Name"
$ git config --global user.email your@email

If you omit the --global option, the setting will only apply to this repository clone.

Set up some Git hooks and helpers:

$ ./scripts/install-git-hooks

To hack on the kernel sources:

$ ./scripts/sequence-patch.sh
$ cd tmp/linux-$version-$branch
$ quilt new patches.suse/fix-foo-and-bar.patch
$ quilt edit some/file.c
$ ./refresh_patch.sh
$ quilt header -e # see next chapter

Refer to the Quilt documentation for details. When you are done, add the new patch to an appropriate place in the series.conf file and run ./scripts/log to commit it. Patches should be named such that they consist of alphanumeric characters, '-' and '.'. Typically, patches are named by filtering the Subject of the patch to a lower-case, dash-separated form like the one in the example above.

To build RPM packages:

$ ./scripts/tar-up.sh

This creates a source package in the kernel-source directory. Use

$ ./scripts/osc_wrapper [kernel-source/kernel-$flavor.spec]

to build a kernel package locally, or

$ ./scripts/osc_wrapper upload [--ibs]

to have all flavors and architectures built by the Open Build Service. The --ibs option uses the SUSE internal instance.

Patch headers

Each patch must have an RFC822-style header that at a minimum describes what the patch does, who wrote it and who inside SUSE can be contacted about problems with the patch. The rules for patch headers are:

  • Each patch must have a From tag that identifies the author of the patch.

  • Each patch must have a Subject tag that briefly describes what the patch does. A brief summary that could appear in a change log makes the most sense in most cases.

  • Unless the author specified in the From tag has a @suse.com, @suse.de or @suse.cz address, the patch must include a Signed-off-by, Acked-by or Reviewed-by header which identifies the person in one of these domains who feels responsible for the patch inside the company.

  • The patch must include a Patch-mainline tag that identifies where the patch came from (for backports from mainline) or when it is expected to be added to mainline. The format is one of:

    For backports from mainline:

    Patch-mainline: <upstream version, for instance, "v6.5-rc7">
    Git-commit: <git hash>
    

    If the commit is from a maintainer repository or some other repository that isn't Linus's:

    Patch-mainline: Queued in subsystem maintainer repository
    Git-repo: <url>
    Git-commit: <git hash>
    

    If the patch is not upstream, depending on the situation:

    Patch-mainline: Submitted, <timestamp - destination>
    
    Patch-mainline: Not yet, <reason>
    
    Patch-mainline: Never, <reason>
    
  • The patch should include a References tag that identifies the Bugzilla bug number, JIRA issue ID, etc. where the patch is discussed. Please prefix bugzilla.suse.com bug numbers with bsc# and JIRA issue IDs with jsc#. Make sure you specify a JIRA Implementation task when referencing JIRA features, not its Epic ID. Have a look at https://en.opensuse.org/openSUSE:Packaging_Patches_guidelines#Current_set_of_abbreviations for a full list of abbreviations.

  • The patch header should include a more extensive description of what the patch does, why and how. The idea is to allow others to quickly identify what each patch is about and to give enough information for reviewing.

More details about valid patch headers can be found in scripts/patch-tag-template. The helper script scripts/patch-tag can be used for managing these tags. Documentation for patch-tag can be found at the top of the script itself.

Example usage of scripts/patch-tag-template:

$ cp scripts/patch-tag-template ~/.patchtag
[ Edit ~/.patchtag with any default values you want. ]
$ patch-tag -e file.diff

Example patch header:

From: Pablo Neira Ayuso <[email protected]>
Date: Tue, 15 Aug 2023 15:39:01 +0200
Subject: netfilter: nf_tables: GC transaction race with netns dismantle
Patch-mainline: v6.5-rc7
Git-commit: 02c6c24402bf1c1e986899c14ba22a10b510916b
References: CVE-2023-4563 bsc#1214727

Use maybe_get_net() since GC workqueue might race with netns exit path.

Fixes: 5f68718b34a5 ("netfilter: nf_tables: GC transaction API to avoid race with control plane")
Signed-off-by: Pablo Neira Ayuso <[email protected]>
Signed-off-by: Florian Westphal <[email protected]>
Acked-by: Michal Kubecek <[email protected]>

Patch sorting

Patches added to the "sorted patches" section of series.conf must be sorted according to the upstream order of the commits that they backport.

After you've added a patch file to the main patches.suse/ or a different patch directory, and supplemented the required tags described in the section Patch headers, run

$ ./scripts/git_sort/series_insert.py <patch>

to insert an entry for a new patch file to the sorted section of series.conf.

For more information, please read scripts/git_sort/README.md.

Before you commit -- things to check

Make sure that all patches still apply after your changes. One way of doing this is using scripts/sequence-patch.sh:

$ export SCRATCH_AREA=/var/tmp/scratch
$ ./scripts/sequence-patch.sh
Creating tree in /var/tmp/scratch/linux-5.14-SLE15-SP5
Cleaning up from previous run
Linking from /var/tmp/scratch/linux-5.14.orig
...
[ Tree: /var/tmp/scratch/linux-5.14-SLE15-SP5 ]
[ Generating Module.supported ]
[ Copying config/x86_64/default ]

Note the "Tree:" line output by the sequence-patch.sh script which specifies the location of the expanded kernel tree that is configured for local build. Please test-compile the kernel or even test-build kernel packages, depending on the impact of your changes. Use scripts/tar-up.sh for creating an OBS package directory.

The kernel source tree that scripts/sequence-patch.sh creates can be test-compiled. Before that, make sure all prerequisites are installed. These include libopenssl-devel, libelf-devel and dwarves. Have a look into rpm/kernel-binary.spec.in for a complete list. Then, the compilation can be done as follows:

$ cd /var/tmp/scratch/linux-5.14-SLE15-SP5
$ make oldconfig
$ make

When committing a patch series, try to make the series easily bisectable. In other words, when applying only the first x patches (1 <= x <= n, n being the number of patches in the series), the kernel should be still buildable and functional.

This means especially that just adding upstream patches unmodified to a series and doing a cleanup patch at the end of the series to ensure the kernel is buildable and functional is to be avoided. Each patch from upstream should be modified as required to fit into the kernel it is backported to, both for build time and runtime.

Applying all patches in the tree with scripts/sequence-patch.sh can take a significant amount of time. The --rapid option is present to speed up the process and tells the script to use Rapidquilt instead of the regular Quilt. The Rapidquilt implementation applies patches in parallel and typically produces an expanded tree in a fraction of the original time. A Rapidquilt package can be obtained from https://download.opensuse.org/repositories/Kernel:/tools/.

Config option changes

SUSE kernel packages for various architectures and configurations are built from the same sources. Each such kernel has its own configuration file in config/$ARCH/$FLAVOR. Checks are in place that abort the kernel build when those configuration files are missing necessary config options.

When adding patches that introduce new kernel config options, please also update all config files as follows:

$ ./scripts/sequence-patch.sh
$ cd /var/tmp/scratch/linux-5.14-SLE15-SP5
$ ./patches/scripts/run_oldconfig.sh

Committing and log messages

Every commit to the kernel source repository should be properly documented. Tool scripts/tar-up.sh obtains change descriptions from a Git commit log and automatically produces .changes files for use by the target RPM packages. All commits which affect the kernel package have their description collected, only changes modifying internals of the repository such as helper scripts are skipped.

When recording your changes to the repository, you should use scripts/log rather than running git commit directly in order to produce a commit description in the expected format.

What is the kernel ABI?

All symbols that the kernel exports for use by modules and all symbols that modules export for use by other modules are associated with a so-called modversion. It is a checksum of the type of the symbol, including all sub-types involved. Symbols that a module imports are associated with the identical checksum.

When a module is loaded, the kernel makes sure that the checksums of the symbols imported by the module match the checksums of the target symbols. In case of a mismatch, the kernel rejects to load the module.

Kernel packages additionally set an RPM dependency in the form ksym($FLAVOR:$SYMBOL) = $CHECKSUM for every exported/imported symbol. A Provides dependency is present for each symbol exported by kernel binaries in the package and a Requires dependency is recorded for each imported symbol. This mechanism allows checking module dependencies early at the package installation time.

Kernel ABI changes

SUSE kernels maintain stable kABI during a lifetime of each service pack. An exported symbol can be changed only if a strong reason arises.

When building an RPM kernel package, the build logic checks the modversions of the resulting kernel and compares them against the reference stored in kabi/$ARCH/symvers-$FLAVOR and kabi/$ARCH/symtypes-$FLAVOR. If any updated or removed symbol is found, the build reports an error and aborts. Symbols explicitly ignored in kabi/severities are excluded from this check. See rpm/kabi.pl for details.

To update the reference files, use scripts/update-symvers:

$ ./scripts/update-symvers kernel-default-5.14.21-150500.55.31.1.x86_64.rpm \
      kernel-default-devel-5.14.21-150500.55.31.1.x86_64.rpm ...

Updating the kabi files is typically done by the branch maintainers. Please always ask them for permission before touching these files.

Ignoring kernel ABI changes

It might sometimes be needed to tolerate particular kernel ABI changes and not abort the build. At the same time, you may not want to update the reference symvers and symtypes files in order to monitor the relative changes.

A specific kernel can be marked so that kernel ABI changes are ignored. This is done by creating a kabi/$ARCH/ignore-$FLAVOR file, for example, kabi/x86_64/ignore-default. The kernel ABI checks are still performed, but the build does not abort if a problem is found. The content of the ignore file does not matter.

All kernel ABI changes in all kernel packages can be ignored by creating a file called IGNORE-KABI-BADNESS in the kernel-source/ sub-directory of the repository that scripts/tar-up.sh creates. Doing this may occasionally be necessary for PTF kernels.

Embargoed patches

At certain times during development, the kernel may include "embargoed" patches, which are patches that must not be made available to parties outside of SUSE before an agreed-upon time. Such patches usually have a date of publication that has been coordinated among Linux distributors and other stakeholders. These patches must not be committed to the usual branches, because these are pushed to a public mirror, but instead to a branch named with an _EMBARGO suffix, for example, SLE15-SP5_EMBARGO. The KOTD scripts will testbuild such branches but won't publish them. Once the fix becomes public, the branch needs to be merged back info the "mainline" branch.

Related information

Internal:

Public:

kernel-source's People

Contributors

afaerber avatar agraf avatar ajaeger avatar andreas-schwab avatar dirkmueller avatar ggardet avatar gobenji avatar gregkh avatar hramrach avatar hreinecke avatar jankara avatar jbeulich avatar jdelvare avatar jeffmahoney avatar jiribohac avatar kdave avatar mantel avatar matwey avatar mbgg avatar michal42 avatar mkubecek avatar neilbrown avatar okirch avatar olafhering avatar petrpavlu avatar tiwai avatar tonyj-suse avatar watologo1 avatar werkov avatar ykaukab avatar

kernel-source's Issues

No irq handler for vector.

@Xornee reported the following error while running the build from 12821a0:

smytych@localhost:~> 
Message from syslogd@localhost at Jul 22 11:27:35 ...
 kernel:[27597.992299][    C0] call_irq_handler: 0.55 No irq handler for vector

Message from syslogd@localhost at Jul 22 14:52:28 ...
 kernel:[38559.299527][    C0] call_irq_handler: 0.55 No irq handler for vector
^C

It is unlikely that this is due to the sched-ext patches, as the message indicates an interrupt with vector number 55 was received on cpu 0 (that is the meaning of 0.55), and for such vector number there is no registered interrupt handler. Here the body of the call_irq_handler from arch/x86/kernel/irq.c which is raising the error:

static __always_inline int call_irq_handler(int vector, struct pt_regs *regs)
{
	struct irq_desc *desc;
	int ret = 0;

	desc = __this_cpu_read(vector_irq[vector]);
	if (likely(!IS_ERR_OR_NULL(desc))) {
		handle_irq(desc, regs);
	} else {
		ret = -EINVAL;
		if (desc == VECTOR_UNUSED) {
			pr_emerg_ratelimited("%s: %d.%u No irq handler for vector\n",
					     __func__, smp_processor_id(),
					     vector);
		} else {
			__this_cpu_write(vector_irq[vector], VECTOR_UNUSED);
		}
	}

	return ret;
}

I'm logging this in the issue tracker in case it happens again and the correlation with sched-ext becomes more clear, but for the moment I don't think sched-ext is the reason. Searching google for "irq vector 55" yields several similar reports, see https://www.google.com/search?q=irq+vector+55

Create test OBS project for staging builds.

There is need for an OBS project to try builds out, eg. to share with testers/users and confirm that some issue is fixed, or simply for conveniently make sure that the source builds without error (building locally requires re-creating the complete OBS environment, which may be inconvenient). With a staging area for test builds, the compilation and validation could be done without pushing out a package that isn't proven to work.

Make changelog in wikipage more prominent.

Instructions on how to install the sched-ext enabled kernel on openSUSE are at the wiki page https://en.opensuse.org/Pluggable_CPU_schedulers , which is a MediaWiki instance.

The wiki offers tracking of past changes, but it would be beneficial if the list of most important changes to the set of instructions could appear on the page body, instead only via the "show history" button. First, the MediaWiki history may contain many minor changes such as grammar fixes that aren't interesting, and second, looking at the history isn't a natural reflex for those looking for installation instructions.

The "changelog section" would only show the list of major changes and their date, such as changes in the download links, recommended compilation steps, package names etcetera. The purpose is to make it less confusing for when someone, who followed say version N of the instructions, visits the wiki again (which now has undergone heavy changes and is at some version N+3), and is disoriented as a result. They would look at the "changelog section", and understand that the recommended installation steps have changed significantly.

`scx_lavd` doesn't load, `bpf_iter_scx_dsq_new` missing

@Xornee reports that installing the package built from 12821a0 on openSUSE Leap, the "lavd" scheduler doesn't load.

smytych@localhost:~/bin> sudo ./scx_lavd
Error: Failed to load BPF program

Caused by:
    Invalid argument (os error 22)
smytych@localhost:~/bin>

scx_lavd -v shows the following error:

; bpf_for_each(scx_dsq, p_wait, LAVD_GLOBAL_DSQ, 0) { @ main.bpf.c:1883
258: (bf) r1 = r6                     ; frame1: R1_w=fp-64 R6_w=fp-64
259: (b7) r2 = 0                      ; frame1: R2_w=0
260: (b7) r3 = 0                      ; frame1: R3_w=0
261: <invalid kfunc call> 
kfunc 'bpf_iter_scx_dsq_new' is referenced but wasn't resolved

and Andrea Righi suggests the kernel is missing bpf_iter_scx_dsq_new. The full output from @Xornee's scx_lavd -v is attached.
output.txt

Basic automated testing for new builds.

There probably is already some sort of test suite in the user-space component of sched-ext, eg. to make sure that the kernel API is responding correctly, that all schedulers load, etc. Incorporate that into some continuous integration for the openSUSE builds, which at the moment make the kernel only, and in the future will include user space as well.

Test multiple architectures (at lest Arm, in addition to x86_64).

OBS creates builds for x86_64, Arm, ppc64le and s390. Explore the feasibility of some basic automated functional testing on those architectures, at least for Arm which has the highest chance of being used for personal computing. Is the openQA available for community projects? Does it have hosts for the desired architectures? See http://open.qa .

Service integration with systemd

The CachyOS Linux distribution has made considerable progress in integrating sched-ext with systemd services. See how much of their work is applicable to openSUSE.

Enhance kernel builds with progressive version numbering.

For standard kernel build, the kernel version (such as 6.10-rc5) is sufficient to provide ordering and let the package manager understand which version is the most recent.

In the eventuality of multiple consecutive build based on the same kernel version, an additional number should be provided, as to maintain order among the set of available builds, for example 6.10-rc5-1, 6.10-rc5-2 etc.

Make separate builds for Leap and Tumbleweed.

When installing the Tumbleweed build 12821a0 on Leap 15.6, one gets the following error:

# zypper install kernel-default-6.10~rc2-1.1.g12821a0.x86_64

Problem: 1: the to be installed kernel-default-6.10~rc2-1.1.g12821a0.x86_64 conflicts with
                     'filesystem < 16' provided by the installed filesystem-15.0-11.8.1.x86_64
 Solution 1: do not install kernel-default-6.10~rc2-1.1.g12821a0.x86_64

Choose the above solution using '1' or cancel using 'c' [1/c/d/?] (c): 

This is because the build assume a filesystem layout with the so called "/usr merge", see systemd.io, which is found on Tumbleweed but not on Leap 15.6. With "/usr merge", several directories normally found at the root "/" of the filesystem are moved under "/usr".

Until Leap 15.6 undergoes the "/usr merge" as well, separate builds are needed.

Split the sched-ext diff into individual patches, as posted upstream.

As of commit 12821a0, we're using a 33 MB "jumbo diff" to lift the tree from 6.10-rc5 to the for-6.11 branch at Tejun Heo's repository at kernel.org. The diff is large as it includes the entire bpf/for-next branch.

The diff would be more manageable if it was split into individual patches, and the massive bpf/for-next for reduced at the bare minimum required by sched-ext. It would be desirable to periodically merge this tree into SUSE's master branch (and not rebase, as to preserve commit hashes), and a fine-grain diff decomposed into individual patches would make that process more practical.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.