Training > Linux Kernel Development > Linux Kernel Debugging (LFD445)
INSTRUCTOR-LED COURSE

Linux Kernel Debugging (LFD445)

Master Linux kernel debugging to become an indispensable asset in the tech industry. This course equips you with the skills to solve complex problems, unlocking new career opportunities and enhancing your value to both your team and company.

Who Is It For
This course is designed for current or aspiring kernel developers, device driver developers, and anyone interested in shortening the development cycle by taking advantage of existing tools and facilities.
read less read more
What You’ll Learn
You will learn techniques for local debugging by configuring a system (physical or virtual) for remote debugging from a second system and working with the basic components of the Linux kernel that underlie the built-in debugging frameworks.
read less read more
What It Prepares You For
To deliver improved productivity in Linux kernel development environments by diagnosing and fixing kernel issues more efficiently.
read less read more
Course Outline
Expand All
Collapse All
Introduction
- Objectives
- Who You Are
- The Linux Foundation
- Copyright and No Confidential Information
- The Linux Foundation Training
- Certification Programs and Digital Badging
- The Linux Foundation THRIVE-ONE Subscriptions
- Linux Distributions
- Platforms
- Preparing Your System
- Using and Downloading a Virtual Machine
- Things Change in Linux and Open Source Projects
- Documentation and Links
Preliminaries
- Procedures
- Kernel Versions
- Kernel Sources and Use of git
- Hardware
- Staging Tree
- Labs
How to Work in OSS Projects **
- Overview on How to Contribute Properly
- Know Where the Code is Coming From: DCO and CLA
- Stay Close to Mainline for Security and Quality
- Study and Understand the Project DNA
- Figure Out What Itch You Want to Scratch
- Identify Maintainers and Their Work Flows and Methods
- Get Early Input and Work in the Open
- Contribute Incremental Bits, Not Large Code Dumps
- Leave Your Ego at the Door: Don't Be Thin-Skinned
- Be Patient, Develop Long Term Relationships, Be Helpful
Debugging Overview
- What is Debugging?
- OOPS and Panic Messages
- Output in dmesg
- Hidden Errors
- Course Flow
OOPS Messages
- What is an Oops Message?
- Finding the Code That Triggered the Oops
- Putting it All Together
- Background - Modules **
- Background - Module Compilation **
- Background - Macros and Inlined Functions **
- Lab Exercises
Oops Messages With Source Code
- Without Source Code
- With Source Code - Compiling
- With Source Code - Disassembly
- Summary
Kernel Features
- Components of the Kernel
- User-Space vs. Kernel-Space
- What are System Calls?
- Available System Calls
- Scheduling Algorithms and Task Structures
- Process Context
- Labs
Kernel Programming Preview
- Task Structure
- Memory Allocation
- Transferring Data between User and Kernel Spaces
- Object-Oriented Inheritance - Sort Of
- Linked Lists
- Jiffies
- Labs
Kernel Deprecated Interfaces
- Why Deprecated
- __deprecated
- BUG() and BUG_ON()
- Computed Sizes for kmalloc()
- simple_strtol() Family of Routines
- strcpy(), strncpy(), strlcpy()
- printk() %p Format Specifier
- Variable Length Arrays
- Switch Case Fall-Through
- Zero-Length and One-Element Arrays in Structs
Race Conditions and Synchronization Methods
- Concurrency and Synchronization Methods
- Atomic Operations
- Bit Operations
- Spinlocks
- Seqlocks
- Disabling Preemption
- Mutexes
- Semaphores
- Completion Functions
- Read-Copy-Update (RCU)
- Reference Counts
- Labs
Process Address Space
- Allocating User Memory and Address Spaces
- Locking Pages
- Memory Descriptors and Regions
- Access Rights
- Allocating and Freeing Memory Regions
- Page Faults
- Labs
Printk
- Debugging with printk
- Format Specifiers in printk
- no_hash_pointers Command Line Option
- Using early_printk
- Labs
Kernel Initialization
- Overview of System Initialization
- System Boot
- Das U-Boot for Embedded Systems**
- Kernel Startup
Monitoring and Debugging
- Debuginfo Packages
- Tracing and Profiling
- sysctl
- SysRq Key
- oops Messages
- Kernel Debuggers
- debugfs
- Labs
Ftrace
- What is ftrace?
- ftrace, trace-cmd and kernelshark
- Available Tracers
- Using ftrace
- Files in the Tracing Directory
- Tracing Options
- Printing with trace_printk()
- Trace Markers
- Dumping the Buffer
- trace-cmd
- Labs
Kernel and git Bisection
- What is kernel bisection?
- Bisection basics
- Scripting kernel bisection
- Labs
Kernel Development Tools
- KASAN
- KCSAN
- KFENCE
- Kmemleak
- KMSAN
- UBSAN
- Summary
Perf
- What is perf?
- perf list
- perf stat
- perf record
- perf report
- perf annotate
- perf top
- Labs
kprobes
- kprobes
- kretprobes
- SystemTap **
- Labs
eBPF
- BPF
- eBPF
- Installation
- bcc Tools
- bpftrace
- Labs
QEMU
- What is QEMU?
- Emulated Architectures
- Image Formats
- Third Party Hypervisor Integration
- Labs
Linux Kernel Debugging Tools
- Linux Kernel (built-in) tools and helpers
- kdb
- qemu+gdb
- kgdb: hardware+serial+gdb
- Labs
Crash
- Crash
- Main Commands
- Labs
Kernel gdb Scripts
- Kernel gdb Scripts
- Setup
- Commands
- Functions
- Labs
kexec
- kexec
- Kernel Configuration
- kexec-tools
- Using kexec
- Labs
Kernel Core Dumps
- Producing and Analyzing Kernel Core Dumps
- Labs
Closing and Evaluation Survey
- Evaluation Survey
Kernel Architecture I
- UNIX and Linux **
- Monolithic and Micro Kernels
- Object-Oriented Methods
- Main Kernel Components
- User-Space and Kernel-Space
System Calls
- What are System Calls?
- Available System Calls
- How System Calls are Implemented
- Adding a New System Call
- Labs
Kernel Style and General Considerations
- Coding Style
- Using Generic Kernel Routines and Methods
- Making a Kernel Patch
- sparse
- Using likely() and unlikely()
- Writing Portable Code, CPU, 32/64-bit, Endianness
- Writing for SMP
- Writing for High Memory Systems
- Power Management
- Keeping Security in Mind
- Labs
The proc Filesystem **
- What is the proc Filesystem?
- Creating and Removing Entries
- Reading and Writing Entries
- The seq_file Interface **
- Labs
Unified Device Model and sysfs
- Unified Device Model
- Basic Structures
- Real Devices
- sysfs
- kset and kobject examples
- Labs
**
These sections may be considered in part or in whole as optional. They contain either background reference material, specialized topics, or advanced subjects. The instructor may choose to cover or not cover them depending on classroom experience and time constraints.
Prerequisites
To make the most of this course, you should:
  • Be proficient in the C programming language.
  • Be familiar with basic Linux (UNIX) utilities such as ls, grep and tar.
  • Be comfortable using any of the available text editors (e.g. emacs, vi, etc.).
  • Experience with any major Linux distribution is helpful but not strictly required.
  • Have experience equivalent to having taken Linux Kernel Internals and Development (LFD420).
Lab Info
Systems should have 2+ CPUs and 4GB+ of RAM, as verified by the ready-for.sh script; the more resources available, the faster and smoother the labs will be.
Reviews
Jun 2025
I like the up-to-date information on recent kernels, and that everything is compactly compiled in the booklet.
Mar 2025
I became familiar with most of the kernel concepts and debug tools, and learnt about each concept in depth.
Feb 2025
Fantastic overview of tools and methods used for testing in the kernel.
$3495
Dates don't work? Check out our partner schedule
100% Money Back Guarantee
Includes
Live Online (Virtual) or Live (Classroom)
3 days of Instructor-led class time
Hands-on Labs & Assignments
Resources & Course Manual
Certificate of Completion
Digital Badge
Experience Level: Advanced
Get a Quote
Training more than 10 people?Get a corporate quote
How To Register Someone Other Than YourselfInstructions