Choosing Synchronization Primitives for Thread and Interrupt Contexts

Start with the problem shape, not the API name

A synchronization primitive does more than prevent a race. It expresses a protocol. Choosing a binary semaphore merely because it “acts like a lock,” for example, loses important semantics: ownership, priority inheritance, and the ability to diagnose misuse.

The following distinction is the foundation.

The mutex image below captures the mutual-exclusion relationship: Task A and Task B must first acquire the mutex before entering the guarded resource.

A critical distinction:

• A mutex protects a critical section or resource.
• A semaphore generally represents permits, available units, or notifications.
• A completion/event represents a state transition: “the operation has finished” or “data is ready.”
• An atomic operation safely changes one synchronization variable; it is not a general substitute for protecting an object graph, descriptor ring, or multi-field state.

The DigiKey mutex explanation is a quick conceptual refresher. Focus on the fact that acquiring the mutex must itself be atomic, and that other tasks wait rather than entering the protected region.

The first discriminator: may the caller block?

A thread waiting on a mutex can be removed from the run queue until the mutex becomes available. This is desirable when the protected operation might take longer than a few instructions, such as configuring a device, changing driver state, accessing a shared control structure, or performing an operation that may allocate memory.

A hard interrupt handler has no such option. It must complete promptly and cannot wait for another execution context to release a mutex. The same rule applies while holding a conventional spinlock: code in that critical section must not call anything that can sleep.

That means these combinations are wrong:

• Taking a mutex in a hard ISR.
• Waiting on a completion from a hard ISR.
• Calling a potentially sleeping allocator while holding a spinlock.
• Holding a spinlock while performing a long register-polling loop, copying a large buffer, or invoking an opaque function that might sleep.

The Linux kernel documentation states the mutex-versus-spinlock distinction and the restriction on sleeping functions clearly. Read the selected portions as a context-selection reference, rather than trying to memorize its older softirq/tasklet terminology.

The five primitives as protocols

1. Mutex: exclusive ownership among sleepable threads

Use a mutex when multiple normal threads need exclusive access to a shared object and lock contention may occur.

A driver-oriented example is a control path that performs a PHY reset and changes configuration state:

mutex_lock(&priv->config_lock);

disable_mac();
program_phy_registers();
rebuild_rx_configuration();
enable_mac();

mutex_unlock(&priv->config_lock);

The protected sequence is longer than a simple atomic update and may need to call helper functions that sleep. A mutex makes the intended ownership explicit:

• One task owns it at a time.
• The owner releases it.
• A waiting task sleeps.
• Many RTOS mutexes and Linux mutexes support priority inheritance, reducing unbounded priority inversion caused by a low-priority owner.

Do not use a mutex simply because two contexts share data. If one of those contexts is a hard ISR, redesign the handoff or use a short spin-protected structure.

A useful interview statement is:

“This is thread-to-thread resource ownership, and the critical section may block, so I would use a mutex. I would not use a semaphore because I need owner semantics and, where available, priority inheritance.”

2. Spinlock: short mutual exclusion where sleeping is illegal

A spinlock is appropriate when a shared item can be accessed from interrupt context, softirq-like deferred contexts, or other non-sleepable code. The waiting CPU loops until the owner releases the lock; therefore, the critical section must be extremely short.

Typical examples include:

• Enqueuing a received descriptor from an ISR.
• Protecting a ring’s producer/consumer indices and a short linked list.
• Updating a few shared fields accessed by both a polling path and an interrupt path.
• Protecting state shared across CPUs where an atomic operation alone cannot preserve a multi-step invariant.

Consider a receive path:

/* Hard IRQ handler: record work, then defer processing */
spin_lock(&priv->rx_lock);
enqueue_completed_rx(priv, desc);
spin_unlock(&priv->rx_lock);

schedule_deferred_rx_work(priv);

The ISR does only the minimum: acknowledge hardware, record ownership/state, and trigger later work. It does not process a packet under the lock.

On a conventional Linux kernel, if a process-context path and an IRQ handler share a lock, the process-side code normally needs an IRQ-disabling variant:

unsigned long flags;

spin_lock_irqsave(&priv->rx_lock, flags);
/* update shared RX list or indices */
spin_unlock_irqrestore(&priv->rx_lock, flags);

The hard-IRQ handler can ordinarily use the plain lock because it already executes with the relevant interrupt context constrained:

spin_lock(&priv->rx_lock);
/* short shared-state update */
spin_unlock(&priv->rx_lock);

Why is the process-side irqsave form needed? On a single CPU, a process could acquire a plain spinlock and then be interrupted by the ISR. If that ISR tries to acquire the same lock, it spins forever while the interrupted process cannot resume to release it. Disabling the relevant local interrupts while holding the lock prevents that self-deadlock.

Two qualifications matter in production work:

• Use the narrowest correct IRQ or bottom-half protection variant for the contexts that actually share the data; do not reflexively disable all interrupts everywhere.
• Do not casually replace ordinary driver spinlock_t with raw_spinlock_t to “make it work.” On PREEMPT_RT, ordinary spinlock behavior differs, while raw spinlocks are restricted to low-level cases that genuinely require strict atomic behavior.

The most valuable constraint to say aloud in an interview is:

“A spinlock protects only the small shared-state transition. I release it before any operation that can sleep, copy extensively, poll hardware, or invoke a complex callback.”

3. Semaphore: permits or notification, not ownership

A semaphore holds a count. give increments or makes a unit available; take decrements it when a unit is available. Unlike a mutex, the context that gives a semaphore need not be the one that takes it.

This makes semaphores useful for two related patterns:

1. Counting available resources: for example, there are $N$ independent DMA buffers, tokens, or hardware channels.
2. Counting pending work: each received item increments the available-work count; consumers take one count before removing an item.

For a bounded producer-consumer queue, the usual design may need both counting and mutual exclusion:

• items_available: counting semaphore, initially $0$
• slots_free: counting semaphore, initially queue capacity
• queue_lock: mutex for task-only access, or spinlock if an ISR can manipulate the queue

The semaphore tells a consumer that an item exists; it does not necessarily protect the queue structure’s head, tail, and storage from simultaneous modification. That is why “I used a semaphore” is not automatically a complete race fix.

A binary semaphore has a count of zero or one. It is often suitable for ISR-to-thread notification:

/* ISR */
capture_rx_status(priv);
give_rx_ready_from_isr(priv);

/* Worker thread */
wait_for_rx_ready(priv);
drain_receive_queue(priv);

This is a notification protocol, not a resource-ownership protocol. If events can arrive faster than the worker handles them, a binary semaphore can coalesce multiple notifications into one pending signal. That is often fine when the worker drains a queue or checks a device status register. It is wrong if every event must be counted individually; use a counting semaphore, queue, counter, or ring-based accounting instead.

The following video segment makes the mutex/semaphore distinction directly, including the ISR-notification use case.

4. Completion or event: wait for a condition, then proceed

A completion or event is usually clearer than a semaphore when the question is not “how many units are available?” but rather:

“Has this operation reached the required state?”

Common driver cases include:

• A thread submits a command to hardware and waits for command completion.
• A probe or initialization path waits for firmware loading or a calibration sequence.
• An ISR signals that a DMA transfer has completed.
• A worker waits until another state machine reaches “link configured.”

In Linux terminology, a completion is commonly used for a wait-for-finish relationship:

reinit_completion(&priv->tx_done);

start_dma(priv, buffer, length);

if (!wait_for_completion_timeout(&priv->tx_done, timeout))
    return -ETIMEDOUT;

/* DMA completion has been reported */

The ISR performs the non-blocking signal:

complete(&priv->tx_done);

This does not make the completion a lock. There is no exclusive owner and no protected resource. The wait and signal simply coordinate a state transition.

An RTOS event group or event flag serves a related purpose, often allowing a thread to wait for one or more named conditions, such as RX_READY, LINK_UP, or ERROR. The exact wake-up and clearing semantics vary by OS, so check whether the event is level-triggered, edge-like, auto-clearing, or broadcast-capable before using it for a data path.

A reliable selection rule:

• Mutex: “Only the owner may access or update this shared resource.”
• Completion/event: “Wake me when this operation or condition becomes true.”
• Semaphore: “A unit of capacity or a unit of work is available.”

5. Atomic operation: one variable, one invariant

Use an atomic operation when the shared state is genuinely simple:

• Incrementing a statistics counter.
• Reference counting an object.
• Setting or clearing a bit.
• Changing an enum-like state via compare-and-exchange.
• Claiming a one-time initialization responsibility.

For example, a reference count is naturally atomic:

if (atomic_dec_and_test(&obj->refcount))
    free_object(obj);

No mutex is needed merely to make the decrement indivisible. But reference counting alone does not protect all fields in obj from concurrent modification, nor does it solve object-lifetime protocols by itself in every design.

Likewise, a lock-free state transition can use compare-and-exchange:

enum state expected = IDLE;

if (atomic_compare_exchange_strong(&state, &expected, STARTING)) {
    start_hardware();
}

Only one caller can change IDLE to STARTING. The losing callers observe a different state and can return, retry, or wait according to the protocol.

Connect this to the preceding lesson: if an atomic state variable also transfers ownership of associated ordinary data, it needs appropriate ordering. For example, a producer writes a descriptor and then release-publishes its ownership bit; the consumer acquire-loads that bit before reading the descriptor. An atomic read-modify-write solves indivisibility, but its memory order determines whether it safely publishes surrounding data.

The kernel locking guide’s reference-count example captures the appropriate scope for atomics: a simple counter becomes easier without a lock, while more complex state often remains clearer under a lock.

Select by context: a practical decision procedure

When given a scenario, classify every participant first.

Now apply this sequence:

1. Does the caller include a hard ISR or another non-sleepable context?

   • Yes: eliminate mutex acquisition and blocking waits.
   • No: a mutex is normally the default for complex thread-to-thread mutual exclusion.

2. Is the goal exclusive ownership or notification?

   • Ownership: mutex or spinlock.
   • Notification: completion/event or semaphore.

3. Must events be counted?

   • One coalesced “work exists” notification: binary semaphore or event.
   • Every unit must be accounted for: counting semaphore, queue, or ring with atomic counters.

4. Can the invariant fit into a single atomic state transition?

   • Yes: use an atomic operation, with suitable memory ordering if it publishes data.
   • No: use a lock for the multi-step invariant.

5. How long can the critical section last?

   • Potentially long or sleeping: mutex.
   • A handful of instructions with no blocking: spinlock if interrupt-safe exclusion is required.

The Linux lock-types documentation adds two important refinements: sleeping locks belong in preemptible task context, and semaphores lack an owner, which prevents priority inheritance.

Four driver scenarios, reasoned through

Scenario A: Two configuration threads change MAC settings

A management thread changes MTU and offload settings while a recovery thread can reset and reinitialize the MAC. Each path may call helpers that sleep, allocate, or wait for hardware.

Choose: mutex.

The resource is the device’s configuration and lifecycle state. Both participants are threads, and the operation is too large for a spinlock. A state machine may prevent invalid transitions, but the mutex protects compound changes to related state.

A strong answer includes scope:

“I would hold the mutex around the shared configuration-state transition, but not across avoidable long waits. If hardware completion is asynchronous, I would submit the operation under the mutex, release it as appropriate for the design, and wait through a completion or event rather than spin.”

Scenario B: RX DMA interrupt records a completed descriptor

A hard ISR observes a completion bit, records descriptor ownership, and schedules packet processing. A worker later builds packets from the descriptors.

Choose: a short spinlock for shared queue/index metadata, plus an event/semaphore/work trigger for notification.

The ISR cannot take a mutex or wait for the worker. Its work should be limited to:

1. Acknowledge or capture device status.
2. Update the shared descriptor/ring state under a brief spinlock if needed.
3. Signal deferred processing.
4. Return.

The worker can perform cache handling where applicable, packet construction, protocol processing, and resource allocation. Do not say “spinlock only” if the worker must be awakened; the lock and the notification solve different problems.

Scenario C: Three independently usable hardware engines

A driver has three identical hardware channels. A thread needs one channel, programs it, waits for completion, then returns the channel to the pool.

Choose: counting semaphore for channel availability; mutex/spinlock for the channel pool’s metadata if the data structure itself needs protection; completion/event for each asynchronous command.

The number three is meaningful. That makes a counting semaphore appropriate for permits. It does not replace the need to protect a free-list or channel-selection structure if its updates are not intrinsically atomic.

Scenario D: Prevent duplicate start of link training

Several threads can request link training, but only the first caller should initiate it. Other callers should see “already starting” and either return or wait for the eventual result.

Choose: atomic compare-and-exchange for the state claim, plus a completion/event if callers must wait.

The one-time claim is a single state transition:

That is ideal for compare-and-exchange. Once training finishes, the owner publishes UP or FAILED and signals completion. A mutex would also work, but an atomic state machine can avoid serializing callers that merely need to observe that another request is already in progress.

Common wrong answers and how to repair them

“Use a mutex in the ISR, but with a zero timeout.”

Still wrong. A mutex has ownership semantics and its implementation is not generally safe in interrupt context. A non-blocking attempt does not erase those constraints.

Repair: In the ISR, use a non-blocking signal and defer work; use a spinlock only for the tiny shared-state update if required.

“Use a binary semaphore as a mutex.”

It may prevent simultaneous entry in a narrow happy path, but it lacks ownership semantics. One task can give it while another took it, and it may not provide the mutex’s priority-inheritance behavior.

Repair: Use a mutex for exclusive resource ownership among tasks. Use a binary semaphore for an ISR-to-task notification or a simple readiness handoff.

“Use an atomic counter to protect the descriptor ring.”

An atomic counter may protect the counter itself but not the relationship among descriptor contents, producer index, consumer index, ownership bits, and buffer lifetime.

Repair: Use atomics only where the entire invariant is one atomic state transition. Use a spinlock or a deliberately designed lock-free ring protocol for multi-field shared state.

“Hold a spinlock while waiting for hardware.”

This wastes CPU and may deadlock the system if completion depends on another context that needs the same lock.

Repair: Update state under the spinlock, release it, then wait in a sleepable context on a completion/event. In a hard ISR, return and defer the wait-capable work.

“The event means data is protected.”

An event only says a condition was signaled. It does not automatically serialize access to a queue, guarantee that a descriptor remains owned by the consumer, or count every arriving unit.

Repair: State the ownership protocol separately: who owns the buffer before and after notification, and what lock, index protocol, or atomic release/acquire operation protects that transfer.

Interview-ready summary

Use this compact explanation when you need to reason aloud:

“First I classify the contexts. Because a hard ISR participates, it cannot sleep or take a mutex. If the ISR and worker share multi-field queue state, I protect the brief update with a spinlock, using the appropriate IRQ-safe variant in thread context. The ISR then signals a semaphore, event, or completion and returns. The worker performs the longer processing. If the shared state were only a counter or a one-way state transition, I would prefer an atomic operation with the required acquire-release ordering.”

Key takeaways:

• Mutex: task-to-task exclusive ownership; sleeping is allowed; typically preferred for substantial critical sections.
• Spinlock: tiny non-sleeping critical section, especially where interrupt context participates.
• Semaphore: counts permits or work; a binary semaphore is useful for notification but is not a mutex.
• Completion/event: represents a condition or completed operation, not ownership of a resource.
• Atomic operation: ideal for one variable and one invariant; insufficient for compound shared structures.
• Lock + notification are often both needed: one protects shared state, the other wakes or coordinates another context.
• The prior lesson’s acquire-release reasoning remains necessary whenever a flag, index, or ownership bit publishes associated descriptor or buffer contents.

Next, you will apply these primitive choices to a real-time scheduling problem: priority inversion, priority inheritance, and bounded blocking.