Shared Memory & IPIs for Inter-Processor Communication

The Two Pillars of Inter-Processor Communication (IPC)

At its heart, the challenge of IPC is for one processor core to communicate information to another. This task can be broken down into two distinct problems:

1. The Data Channel: Where do you put the data that you want to share? The answer is shared memory—a region of RAM that is accessible to both cores.
2. The Signaling Channel: How do you notify the other core that new data is available for it to read? While you could have the second core constantly check a flag (a method called polling), this is highly inefficient. A far better method is to use an Inter-Processor Interrupt (IPI).

You can think of it like a physical mailbox at your house. The mailbox itself is the shared memory. You could walk outside and check it every five minutes for new mail (polling), or you can wait for the postal worker to ring your doorbell when a package arrives (an interrupt). The interrupt-driven approach lets you get on with other things, saving time and energy.

1. Shared Memory: The Data Channel

Let's first explore the two main models of IPC, shared memory and message passing, to understand why shared memory is so common in embedded systems.

As the video explains, shared memory offers the highest performance because once the memory region is set up, cores can access it directly without the overhead of kernel system calls for every transaction. In an embedded multi-core system, setting up a shared memory region involves:

• Memory Allocation: A specific section of SRAM is designated as the shared region. This is typically configured in the linker control file for each core's application image, ensuring that both cores map the same physical RAM block to a known address in their virtual address spaces.
• Data Protocol: The communicating cores must agree on the structure of the data within the shared region. This is a purely software convention. It could be as simple as a single flag and a data buffer, or a more complex structure like a circular buffer (also known as a ring buffer).
• Synchronization: As we discussed in a previous lesson on spinlocks, whenever you have multiple writers or a writer and multiple readers accessing a shared resource, you need a synchronization mechanism. Access to the shared memory data structures must be protected by locks (like spinlocks using LDREX/STREX) to prevent race conditions.
• Coherency and Ordering: Thanks to the hardware cache coherency we studied, you can be confident that a write by one core will eventually be visible to others. However, you still need to use memory barriers (e.g., DMB) to enforce the correct order of operations, ensuring, for example, that the data is written to the buffer before a "data ready" flag is set.

2. Inter-Processor Interrupts (IPIs): The Signaling Channel

Once Core A has placed data in shared memory, it needs an efficient way to signal Core B. This is the job of an IPI. On ARM systems, IPIs are typically implemented using Software Generated Interrupts (SGIs) within the Generic Interrupt Controller (GIC).

An SGI is exactly what it sounds like: an interrupt triggered not by an external peripheral, but by software running on a core. The process is straightforward:

1. Trigger: Core A writes to a specific, memory-mapped register in the GIC (e.g., the GICD_SGIR register in older GICs, or through redistributor registers in GICv3/v4). This write specifies the interrupt ID (from 0-15 for SGIs) and which core(s) to target.
2. Delivery: The GIC hardware routes this request and asserts the corresponding interrupt line for the target core(s).
3. Handling: Core B receives the interrupt just like any other peripheral interrupt. It suspends its current execution, saves its context, and jumps to the pre-configured Interrupt Service Routine (ISR) for that SGI.

The following resource, though for ARMv8, clearly defines the different interrupt types and the role of SGIs.

The key takeaway is that the GIC provides a standardized hardware mechanism for one core to programmatically cause an exception on another, making it the perfect signaling channel for IPC.

3. Putting It All Together: A Bare-Metal IPC Implementation

Let's outline how you would implement a simple, one-way communication channel from Core 0 to Core 1 using these building blocks.

Shared Components (in a shared header file):

• A struct defining the communication channel, placed at a fixed memory address by the linker.
• A spinlock for protecting access to the struct.
• The SGI interrupt number used for signaling.

// Placed at a fixed shared memory address via linker script
typedef struct {
    char data_buffer[256];
    volatile bool data_ready;
} shared_ipc_buffer_t;

Core 0 (Sender) Logic:

void send_ipc_message(const char* message) {
    // 1. Acquire the lock to get exclusive access to the buffer
    acquire_spinlock(&ipc_lock);

    // 2. Wait for the receiver to have processed the previous message
    while (shared_buffer->data_ready == true) {
        // Spin or yield, depending on system design
    }

    // 3. Copy the new message into the shared buffer
    strncpy(shared_buffer->data_buffer, message, sizeof(shared_buffer->data_buffer));

    // 4. Use a Data Memory Barrier to ensure the buffer write completes
    //    before the 'data_ready' flag is visible to the other core.
    __asm__ volatile ("dmb sy" : : : "memory");

    // 5. Set the flag to indicate a new message is available
    shared_buffer->data_ready = true;

    // 6. Release the lock
    release_spinlock(&ipc_lock);

    // 7. Trigger an SGI to notify Core 1
    trigger_sgi_for_core(SGI_NUMBER, /* target_core */ 1);
}

Core 1 (Receiver) Logic:

// ISR for the SGI, configured in the vector table for Core 1
void sgi_isr_handler(void) {
    // It's good practice to keep ISRs short.
    // Here, we just notify a background task that data is ready.
    if (shared_buffer->data_ready == true) {
        // A real implementation would use a semaphore or other RTOS primitive
        // to wake up a handler task.
        g_ipc_message_received = true;
    }
    
    // Clear the SGI interrupt at the source in the GIC
    clear_sgi_pending_bit(SGI_NUMBER);
}

// Main loop or task on Core 1
void process_ipc_messages(void) {
    if (g_ipc_message_received) {
        acquire_spinlock(&ipc_lock);

        // Process the message...
        printf("Core 1 received: %s\n", shared_buffer->data_buffer);
        
        // Mark the buffer as consumed
        shared_buffer->data_ready = false;

        release_spinlock(&ipc_lock);
        g_ipc_message_received = false;
    }
}

This example illustrates all the key concepts: a shared data structure, a lock for mutual exclusion, a memory barrier for ordering, and an IPI for efficient signaling.

Frameworks for IPC: OpenAMP and RPMsg

As you can imagine, building a robust, multi-channel, bi-directional IPC mechanism from scratch is complex. To avoid this, standardized frameworks have been developed. For Asymmetric Multiprocessing (AMP) systems, the most common is OpenAMP.

OpenAMP is an open-source framework that standardizes IPC. It uses a messaging protocol called RPMsg (Remote Processor Messaging), which in turn is built on top of VirtIO (a virtualization standard for I/O devices). These frameworks don't invent new concepts; they simply standardize the ones we've just discussed:

• They use shared memory to implement virtqueues (circular buffers for messages).
• They use IPIs (or other hardware signaling mechanisms) to "kick" the other core when a new message is placed in a queue.

Let's see a practical example of setting this up on an STM32 dual-core microcontroller.

To reinforce this, the following application note from NXP provides a similar overview for their i.MX processors. It's a good idea to quickly read through it to see how the same principles apply across different vendors.

Interview Corner

Q1: Describe how you would implement a basic message-passing system between two cores on a bare-metal ARM SoC.
A: I would use a combination of shared memory for the data channel and an Inter-Processor Interrupt (IPI), specifically an SGI on ARM, for the signaling channel.

1. Memory: I'd define a struct in a shared header, containing a data buffer and synchronization flags. I would place an instance of this struct at a fixed memory address accessible to both cores using the linker script.
2. Synchronization: I would use a spinlock (implemented with LDREX/STREX) to protect access to this shared struct, preventing race conditions.
3. Flow: The sending core would acquire the lock, write the data, write a memory barrier (DMB), update a "data ready" flag, release the lock, and then trigger an SGI targeting the receiving core.
4. Reception: The receiving core's SGI handler (ISR) would be triggered. To keep the ISR short, it would simply set a flag or post to a semaphore to notify a background task. This task would then acquire the lock, read the data, and clear the "data ready" flag.

Q2: What are OpenAMP and RPMsg? Why would you use them instead of building your own IPC protocol?
A: OpenAMP is an open-source framework that provides a standardized API for communication in Asymmetric Multiprocessing (AMP) systems. RPMsg is the messaging protocol it uses, which is based on the VirtIO standard. They formalize the use of shared memory for message buffers (virtqueues) and IPIs for notification. You would use them for portability, maintainability, and features. Instead of every project having its own proprietary IPC, OpenAMP provides a standard, tested solution. It offers features beyond simple message passing, like service announcements and lifecycle management, which are complex to implement correctly from scratch.

Conclusion

In this lesson, we have connected the hardware concepts of memory coherency and ordering to the software challenge of inter-processor communication. You now understand the two pillars of IPC—shared memory and IPIs—and how they work together to create efficient, event-driven communication channels.

Key Takeaways:

• IPC is fundamentally about two things: a data channel (shared memory) and a signaling channel (IPIs).
• Using IPIs for signaling is far more efficient than polling, as it allows the receiving core to save power and perform other work.
• In ARM systems, IPIs are implemented as Software Generated Interrupts (SGIs) within the Generic Interrupt Controller (GIC).
• Implementing a robust IPC mechanism requires careful management of shared data structures, synchronization with locks, and ordering with memory barriers.
• Frameworks like OpenAMP and RPMsg standardize these concepts, providing a portable and feature-rich API for complex multi-core applications.

Next Lesson Preview:
We have now covered the core software concepts for multi-core programming: synchronization with spinlocks, memory ordering with barriers, visibility with cache coherency, and now communication with IPC. The final piece of the puzzle is understanding how to see all this in action. In our next and final lesson for this module, we will learn how to Configure and analyze a multi-core debugging session with synchronized execution control using a JTAG debugger.