Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 29

Deferred Work: Softirqs, Tasklets, and Workqueues

Recommended resources

Introduction

In our last lesson, we established the fundamental "why" of deferred work in the Linux kernel—the top-half/bottom-half model. We saw that to keep interrupt latency low, the interrupt service routine (ISR) must be as fast as possible, deferring any longer-running tasks to a later time.

Today, we address the "how." This lesson focuses on the learning outcome: Compare softirqs, tasklets, and workqueues, and select the appropriate deferred work mechanism for a given scenario. We will explore the three primary mechanisms the kernel provides for deferring work, analyzing their characteristics, performance trade-offs, and constraints. For an aspiring kernel developer, mastering this choice is crucial; it's a core design decision that directly impacts driver performance, correctness, and system stability.

1. The Fundamental Dividing Line: Execution Context

The most important factor distinguishing these mechanisms is their execution context. In the kernel, code can run in one of two major contexts:

  1. Interrupt Context: This is a special, restricted context. Code running here is not associated with a specific process. It executes with interrupts enabled but is not allowed to do anything that might sleep or block. This includes acquiring a mutex, allocating memory with kmalloc (which can sleep), or performing blocking I/O. Softirqs and tasklets run in this context.

  2. Process Context: This is the normal execution context where the kernel is running on behalf of a process (e.g., during a system call or as a kernel thread). Code in this context is allowed to sleep and block. Workqueues run in this context.

This distinction is the root of most of the trade-offs we'll discuss.

Linux Kernel Execution Contexts and Deferred Work Mechanisms
This diagram visually separates Interrupt Context from Process Context. Notice how Hard IRQs (top-halves) have the highest priority, followed by Softirqs and Tasklets within the interrupt context. Workqueues reside in the process context, managed by kernel threads.

The kernel itself needs to track which context the CPU is currently in. It does this using a per-CPU variable called preempt_count. Different bits within this variable are incremented when the kernel enters a hard IRQ, soft IRQ, or NMI context.

Linux Kernel `thread_info->preempt_count` Structure and Context Tracking
This image shows the structure of `preempt_count`. Functions like `local_bh_disable()` (which disables softirq processing) and `irq_enter()` manipulate this counter. Helper functions like `in_softirq()` or `in_atomic()` check this counter to enforce rules like "don't sleep in atomic context."

2. The Deferred Work Mechanisms

Let's now dive into the details of each mechanism. We will start with the lowest-level and highest-performance option and move towards the most flexible.

Softirq, Tasklets and Workqueues

To start, let's get a high-level overview of the three mechanisms. This document from the "Linux Inside" series provides detailed explanations of each.

Please begin by reading the introductory section, which sets the stage for our discussion.

A. Softirqs

Softirqs are the most fundamental and highest-priority deferred work mechanism. They are designed for the most performance-critical and high-frequency tasks in the kernel.

Softirq, Tasklets and Workqueues

Now, let's focus on softirqs using the "Linux Inside" guide. This section goes into the implementation details that are valuable for an in-depth understanding.

Read the "Softirqs" section. Pay close attention to these key characteristics: They are statically allocated at compile time (NR_SOFTIRQS). They are associated with per-CPU ksoftirqd kernel threads, which run pending softirqs if they are being triggered too frequently. A softirq is "raised" using raise_softirq(). You can see their activity in /proc/softirqs.

Key Properties of Softirqs:

  • Context: Interrupt context (cannot sleep).
  • Concurrency: A softirq handler must be reentrant. The same softirq (e.g., NET_RX_SOFTIRQ for network packet reception) can run simultaneously on multiple CPUs. This means any data it accesses must be protected with per-CPU variables or spinlocks.
  • Allocation: Static. There's a fixed number of softirq types defined in the kernel (enum in include/linux/interrupt.h). You cannot dynamically register a new type of softirq from a loadable module.
  • Use Cases: Reserved for core, high-performance subsystems like networking, timers, and RCU. You should almost never use a softirq directly in a device driver.

B. Tasklets

Tasklets are a more flexible mechanism built on top of softirqs. They are designed to simplify deferred work for most driver developers.

Softirq, Tasklets and Workqueues

Next, let's explore tasklets. They are a layer of abstraction over softirqs that makes them much easier to use.

Read the "Tasklets" section. Focus on the key differences from softirqs: They are implemented using two dedicated softirqs: TASKLET_SOFTIRQ and HI_SOFTIRQ. They can be dynamically allocated, making them suitable for modules. The same tasklet is guaranteed not to run on multiple CPUs simultaneously. If a tasklet is scheduled to run, it will run on the CPU it was scheduled on, and another attempt to schedule it will be ignored until it has finished.

Key Properties of Tasklets:

  • Context: Interrupt context (cannot sleep).
  • Concurrency: A specific tasklet handler is serialized. It is guaranteed to only run on one CPU at a time. This greatly simplifies locking, as you don't need to worry about the handler re-entering itself. However, different tasklets can still run in parallel on different CPUs.
  • Allocation: Dynamic. You can initialize a tasklet_struct at runtime with tasklet_init().
  • Use Cases: The "classic" bottom-half mechanism. Good for short, atomic tasks that are deferred from an ISR but don't need the complexity or re-entrancy of a softirq, and most importantly, do not need to sleep.

C. Workqueues

Workqueues are the most different and, for most modern drivers, the most useful of the three. They move the deferred work out of interrupt context entirely.

Softirq, Tasklets and Workqueues

Finally, we examine workqueues, the most flexible and powerful of the deferred work mechanisms.

Read the "Workqueues" section. The most critical takeaway is: Workqueue functions run in process context via kernel worker threads (kworker). This means they are allowed to sleep and block.

Key Properties of Workqueues:

  • Context: Process context (can sleep).
  • Concurrency: Handled by the scheduler. By default, a work item is executed by a shared kworker thread. Since the handler can sleep, you can use standard blocking primitives like mutexes for synchronization.
  • Allocation: Dynamic. You can initialize a work_struct and schedule it with schedule_work().
  • Use Cases: The default and recommended choice for almost all deferred work in device drivers. If your deferred task needs to perform I/O, allocate memory, or acquire a mutex, you must use a workqueue.

3. Summary and Practical Demonstration

Let's consolidate what we've learned. The following reading provides a very clear "cheat sheet" that summarizes the key differences.

Interrupts — The Linux Kernel documentation

This documentation page from the Linux Kernel Labs provides a succinct summary that is perfect for reinforcement.

Please read the sections titled "Deferrable actions", "Soft IRQs", "Tasklets", "Workqueues", and especially the "Deferrable actions summary". This summary table is a perfect key takeaway for comparing the mechanisms.

A Real-World Example: Tracing Network Softirqs

Theory is useful, but seeing these mechanisms in action provides a much deeper understanding. The networking subsystem is a heavy user of softirqs for processing incoming packets. The following video uses kernel tracing to show this in action.

Linux Networking: How The Kernel Handles A TCP Connection

This video from Deep Linux traces a simple TCP connection. It brilliantly illustrates the hand-off from a hardware interrupt to the network softirq (NET_RX_SOFTIRQ).

Please watch the following segments. As you watch, correlate the events in the trace with the concepts we've discussed: 27:43 - 29:28: The presenter first observes a softirq/net-rx being triggered and links it back to a hardware interrupt. This is the top-half to bottom-half delegation in action. 55:42 - 1:05:05: A more detailed walkthrough. Notice the sequence: Hardware IRQ -> irq_handler_entry -> softirq is raised -> ksoftirqd (or another context) runs the softirq handler (net_rx_action) -> the packet is processed by the network stack -> finally, the application is woken up. 1:15:22 - 1:18:21: A final, concise recap of the entire flow from hardware interrupt to softirq processing to unblocking the user application. \nThis demonstrates why softirqs are used for networking: packet processing is a high-frequency, performance-critical task that fits the softirq model perfectly.

4. Interview Scenario: Choosing the Right Mechanism

Now, let's apply this knowledge to a typical interview question designed to test your judgment as a kernel developer.

Scenario: You are writing a Linux driver for a device on a shared I2C bus. The device is a pressure sensor that signals via an interrupt when a new reading is available. Your top-half ISR has confirmed the interrupt and now needs to schedule a bottom half to read the 16-bit pressure value from the device's register over the I2C bus and make it available to user space.

Which deferred work mechanism (tasklet or workqueue) would you choose and why? What would be the consequence of choosing the wrong one?

Click for Analysis

  • The Correct Choice: Workqueue

    The key to this problem lies in the nature of the I2C bus. I2C bus transactions are inherently blocking. When you call a function like i2c_transfer(), the kernel may need to:

    1. Acquire a lock to gain exclusive access to the I2C bus adapter.
    2. Wait for the previous transaction on the bus to complete.
    3. Sleep while waiting for the hardware to complete the byte transfers.

    Because the I2C read operation can sleep, it must be performed in process context. Therefore, a workqueue is the only correct choice.

  • Consequence of Choosing a Tasklet (The Wrong Choice):

    A tasklet runs in interrupt context, where sleeping is strictly forbidden. If you were to call i2c_transfer() from a tasklet handler, the system would detect that you are trying to sleep in an atomic context. This would trigger a kernel warning (a "BUG" message) in the system logs, often looking something like "BUG: sleeping function called from invalid context" and print a full stack trace. In a non-debug kernel, this could lead to unpredictable behavior, deadlocks, or system crashes.

  • Why a Softirq is Even More Wrong:

    While also running in interrupt context, using a softirq would be extreme overkill. They are not designed for simple driver tasks, require static allocation, and demand complex reentrant-safe logic that is completely unnecessary here. An interviewer would see this as a sign of not understanding the purpose and complexity of the different mechanisms.

This scenario demonstrates that understanding the "can it sleep?" rule is the single most important factor when choosing between tasklets and workqueues.

Conclusion

We have now explored the "how" of deferred work in the Linux kernel. By understanding the trade-offs between softirqs, tasklets, and workqueues, you can make informed design decisions that lead to robust and efficient drivers.

Key Takeaways:

  • The primary distinction is execution context: softirqs and tasklets run in interrupt context (cannot sleep), while workqueues run in process context (can sleep).
  • Softirqs are statically allocated, can run concurrently on multiple CPUs, and are reserved for core, high-performance subsystems.
  • Tasklets are dynamically allocated, serialized (the same tasklet won't run on two CPUs at once), and are a good choice for quick, non-sleeping deferred tasks.
  • Workqueues are the most flexible and safest option. They are the go-to choice for any deferred work that might block, such as performing I/O, allocating memory, or acquiring mutexes.
  • The Golden Rule: When in doubt, use a workqueue. The performance overhead is usually negligible compared to the risk of deadlocking the system by sleeping in an atomic context.

In our next lesson, we will put this theory into practice. Our goal will be to implement a kernel module that registers an interrupt handler and defers work using a workqueue, tying together the concepts from this lesson and the previous one into a complete, working example.