Multicore Applications in Mbed OS Community Edition

This document lays out a plan for how Mbed CE will be extended to have (much better) support for multicore microcontrollers.

Background and Current Status

Multicore MCUs

Over the last decade, several microcontroller manufacturers have released multicore ARM MCUs. These MCUs integrate two or more CPU cores within a single physical microcontroller. Generally, the cores share the same address space (though there may be some functionality only accessible to one core or the other). On these MCUs, there is generally one "primary" core which starts executing code immediately when the chip comes out of reset. Other cores are "secondary" cores that need to be started by the primary core (by passing them a code address to begin executing).

Multicore MCUs generally come in two different flavors: Symmetric MultiProcessing (SMP), and Asymmetric MultiProcessing (AMP). In SMP microcontrollers, you have multiple cores which are (near) identical and can execute the same machine instructions. An example would be the RP2040, with its two identical Cortex-M0+ cores. AMP microcontrollers, on the other hand, have multiple cores of different types whose images need to be compiled with different compiler settings. This is common when microcontroller designers add additional smaller cores intended for the main core to delegate smaller tasks to.

Current Mbed Support

Currently Mbed supports the following multicore MCUs:

• Raspberry Pi RP2040: SMP, 2x Cortex-M0+
• Raspberry Pi RP235X: SMP, 2x Cortex-M33
• NXP MIMXRT1176: AMP, 1x Cortex-M7 + 1x Cortex-M4
• STM32 STM32H745 and H747: AMP, 1x Cortex-M7 + 1x Cortex-M4

On three out of these four, Mbed has no facilities for developing for the secondary CPU at all and exclusively builds code for the primary CPU. On the STM32H745/747, some infrastructure (linker script, build system targets, etc) does exist for developing for the secondary CPU, but it is not hooked in properly with the build system (appears to have been created only for Arduino IDE support). This infrastructure does not work together with Mbed CE's build system features at all and would likely require flashing code to each core separately.

Note: I will also be looking at the MCX N54 in this document, as it's another example of a multicore MCU and it is one I'm interested in supporting in Mbed in the future.

Multithreading or Multiprocessing?

Generally, there are two different software approaches one can take to multicore support: either having multiple threads running within one application (like a desktop OS), or having multiple, mostly-independent applications running on each core (each with their own code binary and memory space). This is similar to "multiprocessing" on a desktop OS, where you have multiple processes which may communicate but can't directly access each other's memory.

A multithreading-based approach is much more efficient of flash and RAM space, but comes at the cost of more complexity in the OS and in the user application. On the OS side, your Real-Time OS (RTOS) library needs to know how to activate the secondary cores and have them correctly share state with the primary core. And in both the OS and the application, code now needs to account for the fact that multiple cores will be executing multiple functions at the same exact time -- things like cache coherency and reentrancy enter the chat in a way that they don't really otherwise. And of course, multithreading only works with symmetric multiprocessing, where each CPU can execute the same exact code.

Unfortunately, the RTOS used by Mbed, Keil RTX, does not support multithreading, and Mbed OS isn't really set up for it in other ways either. When you add in the fact that the significant majority of multicore MCUs supported by Mbed (in fact, all of them not made by Raspberry Pi) are asymmetric multiprocessors, it doesn't really seem promising for a multithreading-based approach with Mbed. It would be a great deal of work, and only a few targets would benefit from it.

Based on this, I believe it is worth deciding that, at least for the foreseeable future, Mbed will operate in an asymmetric multiprocessing mode only. This means that you will always have n different application images in flash and RAM for n CPU cores, and these applications will largely operate independently of each other except for specific multiprocessing functionality. This comes at a cost of memory efficiency (you'll pay around 20-50k code size and 10ish k of RAM to hold Mbed and the C library for each core you use), but it is the much more feasible alternative for the Mbed CE project at this time.

Literature Review: How do other projects solve these problems?

RPi Pico SDK

Shared Memory: Raspberry Pi Pico uses a SMP design, where both CPU cores execute out of the same image. This means memory is shared by default. If you want your memory to not be shared, you generally have to declare it on the stack, or just declare different variables for different cores to use.

Synchronization: RP2xx MCUs have hardware spinlocks that provide mutex-type functionality.

Arduino

• Appears to use a high-level wrapper that integrates rpclib (for serializing C++ method calls into binary with msgpack) and OpenAMP (for implementing a queue between the two cores using shared memory).
• Seems like a fairly heavy approach for us to use, especially the need to integrate msgpack

OpenAMP

OpenAMP seems to be the biggest (and only) standard in town for communications among multi-core processors. This white-paper provides an overview, but here's my rough summary (plus other stuff I've learned):

• OpenAMP provides a standardized interface for starting and communicating with baremetal coprocessor CPUs from a "master" CPU (which can be either Linux or bare-metal)
• It's based on work that TI contributed to the Linux kernel (presumably for using the coprocessors on chips like the AM335x in the BeagleBone) but has been spun off into its own project
• It is split into two layers: a low level "libmetal" that provides abstraction for stuff like the RTOS and atomics, and a high level open-amp library
• The high level library has two major APIs: the remoteproc_xxx family of functions, which deals with loading code into coprocessors, and the rpmsg_xxx family of functions, which provide communication queues between processors.
  • For Mbed purposes we don't really care about remoteproc because all the code is executed out of flash, there is no runtime loading
• For cores to exchange messages, you must create a channel between two cores, then each core can declare one or more "endpoints" (which are identified by unique string names).
• Each message consists of simply a data buffer and a length field. OpenAMP makes no attempt to structure your messages aside from routing them to the correct endpoint.
• OpenAMP does maintain a queue of messages going to each core. In our case, this would be a vring, which is a sort of descriptor queue living in shared memory.
• When messages arrive at a core, they trigger the callback you set up for your endpoint. You must create a thread that polls OpenAMP to dispatch these callbacks.

OpenAMP appears to rely on two low-level abstractions: notifications and shared memory. For example, here's a notification implementation for STM32H745/747 using HSEM (the MAILBOX_Poll() function is called whenever a thread flag is set on the event handler thread). Shared memory generally takes the form of a preallocated region with an agreed-upon size (example).

From my perspective, OpenAMP brings a fair amount of complexity without a lot of advantages. It's a big codebase, with lots of settings to configure, many layers of function pointer callbacks to get lost in, and multiple libraries that would need to each be integrated. However, the functionality it brings to the table is pretty limited: it's just a system for inter-processor queues, with little other functionality. In particular, it does not provide synchronization primitives like multicore mutexes (at least, not ones that work on Cortex-M0) and it does not provide "bulk" shared memory that would be optimized for shoveling lots of data from core to core (each message copies the entire payload when sent). Ultimately, I think this library's Linux roots work against it, in that they make it unimaginative: It feels like it is focused on providing the bare minimum multicore functionality without considering how ergonomic it is to use or what functionality other than queues would be commmonly needed.

In my opinion, rolling our own multicore framework would provide not only more features, but a better user experience as well. As you will see from reading the proposal below, we can leverage features of the linker and of the C++ language to ease some of the common pain points of multicore programming and accomplish much of what OpenAMP does in a simple and efficient way.

By not using OpenAMP, the biggest thing we lose out on is the ability for Mbed to interoperate with other CPUs in the same chip that are running another OS, such as Zephyr or Linux. This is a legitimate downside, however running in such applications (e.g. on smaller Cortex-M cores inside a Linux SoC) is not currently a supported use case of Mbed OS. If and when it is added, we could consider creating a library to allow an Mbed OS core to operate as an OpenAMP slave device.

Build System Design

Requirements

• Ability to define cores via target JSON
• Ability to configure allocations to each core for each RAM and flash bank
• Ability to build and flash the application (for both cores!) with one click.
  • This will work by building first for the highest-numbered core, then the next lowest core, then the next until we build core 0 which builds the complete image
• Ability to attach a debugger to either core (TODO is it possible to debug both at the same time?)

Compile Definitions

• DEVICE_MULTICORE - Defined if this CPU has multiple cores
• MBED_THIS_CORE - Gives the current CPU core that is being compiled for.
• MBED_CORE_COUNT - Gives the number of cores on the device.
• MBED_CORE_xxx - Gives the core index of the core with the given name. Note that core indexes have to be sequential and start from 0.
  • On RP2040, we would have MBED_CORE_0=0 and MBED_CORE_1=1
  • On STM32H745, we would have MBED_CORE_CM7=0 and MBED_CORE_CM4=1

Basic Multicore API

HAL API

• int hal_secondary_core_start(uint8_t coreIdx, uint32_t vectorTableAddr)
  • Starts a secondary core (given by coreIdx) executing at the given vector table address (usually the beginning of the flash image).
  • Upper layer code ensure that this can only be called if the given core is ready to be started and that this is only called on the primary core
  • Returns error code or 0 on success
• void hal_secondary_core_kill(uint8_t coreIdx)
  • Stops the given secondary core, keeping it held in reset until it is started again.
  • Upper layer code ensure that this is only called on the primary core

Upper Level API

• C++ namespace called multicore
  • multicore::start_secondary(uint8_t coreIdx) - See secondary_core_start(). Ifdef'd out when not on core 0. If the secondary core is not currently in NOT_STARTED state, this calls kill_secondary() first.
  • multicore::kill_secondary(uint8_t coreIdx) - See secondary_core_kill(). Ifdef'd out when not on core 0.
  • multicore::get_state(uint8_t coreIdx) - Returns most recent known state of the given core. This is tracked at the software level via shared memory.
    • Possible states:
      • NOT_STARTED: Not doing anything, waiting to be started
      • STARTING: Enabled via start_secondary(). Application has not finished booting yet though.
      • RUNNING: Actively executing code.
      • DONE: Completed its program and has returned from main().

Shared Memory and Cache Coherency API

On multicore MCUs, it's important to have a way to create shared memory which is accessible to all the CPU cores. This allows sharing data and, to some extent, synchronization via atomic variables. However, most approaches for sharing memory between cores rely on defining specific regions with fixed size (that has to be managed and resized over time), or on defining data structures at absolute addresses (that, again, have to be managed in the linker script). Here at Mbed, I think we can do better. I'd like to propose a somewhat different architecture for shared memory that allows the OS to manage populating and sizing the shared memory region automatically. This can be implemented almost entirely using standard C++ features, but a small amount of additional scripting is required.

To let users create shared memory data, we first need to set up a system where the shared memory data items are declared in specific sections, and on only one core. We can achieve this by defining the following macros in a header:

#if MBED_THIS_CORE == MBED_CORE_COUNT - 1
// Define variable in shared memory, will be zero initialized
#define MBED_MULTICORE_SHARED(type, name) __attribute__((section(".mbed_shared_mem.bss"))) type name{}

// Define variable in shared memory, will be initialized to the provided value at startup
#define MBED_MULTICORE_SHARED_INIT(type, name, value) __attribute__((section(".mbed_shared_mem.data"))) type name{value}
#else
#define MBED_MULTICORE_SHARED(type, name) // empty
#define MBED_MULTICORE_SHARED_INIT(type, name, value) // empty

This can be used like:

foo.h:

namespace myapp {

// Shared memory variables
extern bool my_shared_flag;
extern uint8_t my_shared_mem_buffer[256];

}

foo.cpp:

namespace myapp {

MBED_MULTICORE_SHARED_INIT(bool, my_shared_flag, true);
MBED_MULTICORE_SHARED(uint8_t, my_shared_mem_buffer[256]); // will be zero initialized

}

This way, we will end up with my_shared_flag and my_shared_mem_buffer as symbols in the .mbed-shared-mem section when compiling code for the highest-numbered core, and not defined at all (just externed) on all other cores.

With just this setup, we would end up with the shared symbols defined on the highest-numbered core only, and would get undefined references when trying to use them on any other core. How do we fix that? This is where we need a bit of custom scripting. We would write a script that processes the highest-numbered core's linker map file and finds all symbols inside the .mbed_shared_mem section. (we could do this largely using the existing infrastructure developed for memap.py). Once we find these symbols, we would write out a file that associates their (mangled) names with their addresses:

_ZN5myapp14my_shared_flagE = 0x20001234;
_ZN5myapp20my_shared_mem_bufferE = 0x20005678;

We'd also add some definitions that provide info about the .mbed_shared_mem section:

/* Where the .mbed_shared_mem.bss section lives in RAM */
__MBED_SHARED_MEM_BSS_VMA = 0x20005678;
__MBED_SHARED_MEM_BSS_SIZE = 256;

/* Where the .mbed_shared_mem.data section lives in RAM */
__MBED_SHARED_MEM_DATA_VMA = 0x20001234;
__MBED_SHARED_MEM_DATA_SIZE = 1;

/* Initialization data for .mbed_shared_mem.data is loaded from this location in flash */
__MBED_SHARED_MEM_DATA_LMA = 0x08001234;

This file would be saved as something like mbed-multicore-shared-mem.ld.h and included from the target's linker script, via something like

#if TARGET_MULTICORE

#if MBED_THIS_CORE < MBED_CORE_COUNT - 1
/* Include shared memory size info and symbol list */
#include "mbed-multicore-shared-mem.ld.h"
#endif

.mbed_shared_mem.data : ALIGN(8)
{
#if MBED_THIS_CORE == MBED_CORE_COUNT - 1
  /* Include everything in shared memory section */
  *(.mbed_shared_mem.data)
#else
  /* Reserve space for shared memory */
  . += MBED_SHARED_MEM_DATA_SIZE;
#endif
} > ram AT > flash

.mbed_shared_mem.bss (NOLOAD): ALIGN(8)
{
#if MBED_THIS_CORE == MBED_CORE_COUNT - 1
  /* Include everything in shared memory section */
  *(.mbed_shared_mem.bss)
#else
  /* Reserve space for shared memory */
  . += MBED_SHARED_MEM_BSS_SIZE;
#endif
} > ram AT > flash
#endif

With this linker script, on lower-numbered cores, we include the linker script fragment we generated, giving us the information about how big the shared memory sections are and where they live in memory. Also, each of the variable names we wrote out earlier is processed as a linker script symbol, essentially telling the linker "when you see this name, it lives at this address". This SHOULD (if I am understanding linker scripts correctly) make it so we can refer to the variables in shared memory on other cores without any issues.

Initialization

For variables in the shared memory bss, initialization is easy: we just zero the entire section when core 0 boots. Similarly, for the shared memory data section, we do a copy from the __MBED_SHARED_MEM_DATA_LMA address into the .data section to pick up all the initialization values. Note that __MBED_SHARED_MEM_DATA_LMA will actually be in the highest-numbered core's flash image, but that's OK as it's all the same flash bank.

Where this gets a little annoying is objects in shared memory with C++ constructors. Since the objects are declared on the highest-numbered core as far as the compiler is concerned, they will not be constructed until the code boots up on that core. So, if you try to use such an object from core 0 before starting the other cores' applications, you will access the object before its constructor has executed, yielding undefined behavior.

This is weird, but I don't think it's a dealbreaker, especially because you already have to be quite careful about which objects you put into shared memory: things could get very weird if the object holds any pointers to things outside the shared memory section. So, I think we can solve this at the documentation level by telling people to be very careful with this and to use std::optional if they need the constructor called at a deterministic time.

Caching

The shared memory section should be cache-coherent, that is, if I do a write into shared memory on one core, that write should be seen by the other core in real time without needing to do any cache maintenance. On some multicore MCUs (RP2xxx) this is the default behavior, but on others (STM32H7, MIMXRT117x), this will require additional MPU configuration. We might have to get a little creative with this: since MPU blocks have to be power-of-two sized, I think we will need to add something in the linker script that dynamically expands the .mbed_shared_mem.bss section so that the total size of the shared memory region is a power of two.

With cache-coherent memory, ARM atomic operations (on MCUs where they exist, i.e. not cortex-M0) will "automatically" work across cores as well. This gives us a cheap and easy way to implement multicore flags and counters using this shared memory functionality.

Multicore Spinlock API

Our first multicore primitive is a very basic one: a spinlock. This is similar to a mutex in that it can be unlocked or locked, and only one core may hold it at one time. However, spinlocks are basic in that if a core does not hold a spinlock but wants to, it cannot do a blocking wait: it has to keep "spinning" and checking it over and over. This means that spinlocks are often used as simple building blocks for more complex threading primitives that allow such waits.

Spinlock Peripherals

First, let's examine the spinlock peripherals available on each MCU.

Also note that many MCUs can support a software implementation of spinlocks via LDREX/STREX instructions, but not all cores (e.g. Cortex-M0) support this.

Target Family	Peripheral used for spinlocks	Number of HW spinlocks	CPU core supports SW spinlocks?
RP2040	SIO	32, but 13 reserved by SDK (might be possible to reduce)	No
RP234x	SIO	19 due to errata, 13 reserved by SDK	Yes
MIMXRT117x	SEMA4	16	Yes
MCX N54	SEMA42	16	Yes
STM32H745/747	HSEM	32, 6 reserved by SDK	Yes

Looking at the options on each MCU, it appears that the RP2040 is the biggest limiting factor, as its core does not support SW spinlocks. Meanwhile, the RP2350 has a very small number of available spinlocks, but supports SW spinlocks.

This leads us to the following requirements for the spinlock API:

• Must support HW spinlocks for the case where there are no SW spinlocks
• Should support SW spinlocks if the HW spinlocks are exhausted, giving us the ability to support MCUs like the RP2350 more cleanly
• Every MCU has the ability to check if a spinlock is locked, but not all (RP2xxx) have the ability to see what core is holding the spinlock.

With this strategy, we are guaranteed at least 19 spinlocks for Mbed + the application to use on all supported MCUs.

HAL API

Note that some MCUs we want to support only have a small number of hardware spinlocks. It's desirable for the application to be able to use more than the number supported by HW. So, we implement an indirection mechanism.

• MBED_NUM_HW_SPINLOCKS: Defined to the number of available hardware spinlocks on this platform. This includes the number available both to Mbed OS and to user code. It does not include any spinlocks reserved by the processor SDK, if any.
• JSON setting multicore.num-sw-spinlocks: Defined to the number of software spinlocks which will be made available to each core. Only used on CPU cores with software spinlock support, otherwise is 0.
• int hal_multicore_spinlock_try_lock(uint32_t lock_number)
  • Tries to lock the given spinlock. Returns 1 if locked, 0 otherwise.
  • lock_number may be any integer between 0 and MBED_NUM_HW_SPINLOCKS - 1
  • Locking is not recursive: If the spinlock is already held by the current core, a best-effort attempt will be made to return an indicative return code (2), but a 0 return with no other indication is also possible.
• void hal_multicore_spinlock_unlock(uint32_t lock_number)
  • Unlocks a spinlock.
• void hal_multicore_spinlock_check(uint32_t lock_number)
  • Check whether the spinlock is locked without trying to lock it.

C++ API

This API provides access to spinlock functionality. It has a lot of common functionality with rtos::Mutex intentionally, so that a ScopedLock can be used to lock and release the spinlock.

namespace multicore
{
class Spinlock : mbed::NonCopyable {
public:
    /// Construct a Spinlock instance.
    /// Must pass the lock number. A software lock will be used if this is greater than or equal to MBED_NUM_HW_SPINLOCKS.
    /// If the lock number is greater than the number of available HW and SW spinlocks, this will assert fail.
    /// Multiple Spinlock class instances may be constructed for the same lock number (on any core); they will all refer to the same underlying lock.
    Spinlock(uint32_t lock_number);

    /// Check whether the spinlock is locked without trying to lock it.
    bool is_locked() const;

    /// Locks the spinlock, holding the CPU in a busy loop until it can be locked.
    /// Also enters a critical section to remove that chance that an ISR will execute while holding the spinlock.
    /// Locking is not recursive: If the spinlock is already held by the current core, a best-effort attempt will be made to assert fail,
    /// but a deadlock is also possible.
    void lock();

    // Remainder of API same as rtos::Mutex
    bool trylock();
    bool trylock_for(Kernel::Clock::duration_u32 rel_time);
    bool trylock_until(Kernel::Clock::time_point abs_time);
    void unlock();
};
}

To implement software spinlocks, an array of boolean flags will be declared in shared memory (with its size based on multicore.num-sw-spinlocks).

Multicore Event API

The second basic feature we need is the ability to send events from one core to the other in a way that will generate an interrupt. Ideally, we want to be able to send both the interrupt, and some kind of basic "event code" that tells the other core what sort of event has happened.

Notification Peripherals

Target Family	Peripheral	Description	Notes
RP2xxx	SIO FIFO	8-entry (RP2040) / 4-entry (RP235x) FIFO for each core, each entry is a uint32.	Used by the SDK to implement starting core 1 from core 0. Also note that the datasheet seems to imply that the interrupt for this is constantly active if the fifo is not full, but looking at the SDK code this appears to not be the case. Seems like a doc error.
MCX N54	MAILBOX	32-bit register, where any bit being set generates an interrupt in the selected core	The datasheet implies that this is only usable by cortex M33 0 and the CoolFlux DSP, but looking at NXP example code it seems that "CPU 1" in the docs also applies to the second M33 core
MIMXRT117x	MU	4x 32-bit registers in each direction, where a write to any of the registers (with any value) sets a flag on the other core that can trigger an interrupt. When the corresponding register is read, an "empty" bit is cleared for the transmitter (which can also trigger an interrupt)
STM32H745/747	HSEM	32x semaphores (same peripheral as above) with support for getting an interrupt when one or more chosen semaphores become free.

Looking at these peripherals, we can make a few conclusions:

• All the MCU families have a way to deliver an interrupt to the other core
• Three out of the four have a way to attach a 32-bit integer to that interrupt, while the fourth (STM32H745/747) does not but does support atomic variables shared by both cores.
• All MCUs have some method of flow control, i.e. a method to prevent enqueuing another event before the receiver core has processed outstanding event(s)
  • On RP2xxx, there is a multi-entry FIFO containing a RDY flag indicating there is at least one free entry for writing (which can deliver an interrupt)
  • On MCX N54, the sender core can check that the receiver core cleared the IRQ register back to 0 before writing another event
  • MIMXRT117x is similar to RP2xxx: there is a "transmitter empty" bit which sets for the sender when the receiver core reads a value (and can deliver an interrupt)
  • On STM32H745/747, we can do this by just atomically reading the event code word to see if it has been cleared by the other core.
• Note that some MCUs have a built-in way to get an interrupt when it becomes possible to send an event, but not all. Also, this becomes much more complex if you consider MCUs with more than 2 cores. So, I don't think it makes sense to support this for now.

HAL API

• Config setting multicore.max-event-domains: Maximum number of supported event domains that the user application may register (NOT including Mbed internal ones). A sparse array is used, so no need to have the domains be numbered sequentially.
• MBED_MULTICORE_EVENT_QUEUE_LEN define - Indicates the number of events that a core can queue at one time. For example, this would be set to 8 on RP2040, 4 on MIMXRT117x, and 1 on MCX N54.
• void hal_multicore_event_attach_handler(void (*handler)(uint32_t event_code)) - Attach a global function as the handler for incoming events on the current core. It will be called in an ISR whenever another core delivers an event to this core.
• bool hal_multicore_event_send(uint8_t core_idx, uint32_t event_code) - Sends an event with the given event code to the specified core. If the specified core's mailbox is currently full, this does not deliver the event and returns false. Otherwise it returns true.

C++ API

The C++ API will provide a fairly thin wrapper around the above code, with the primary difference being splitting the 32-bit event code into a 16-bit domain and a 16-bit code.

namespace multicore {

enum EventDomain : uint16_t {
    // Users may use the domain values 0 through 0xEFFF freely as long as
    // the number of domains in use does not exceed `multicore.max-event-domains`

    MBED_OS = 0xF000,
    MBED_MUTEX = 0xF001,
    MBED_QUEUE = 0xF002,
    _MBED_LAST_EVENT_DOMAIN
};

/// Try to send an event to another core. This will call the handler registered for the given domain.
/// If the other core cannot accept the event (e.g. due to not having dequeued the previous sent event(s) yet),
/// this will return false, otherwise it will return true.
/// The number of events that can be buffered by a core is given by MBED_MULTICORE_EVENT_QUEUE_LEN.
bool try_send_event(uint8_t core_idx, EventDomain domain, uint16_t code);

/// Same as above but blocks until the event can be sent.
void send_event(uint8_t core_idx, EventDomain domain, uint16_t code);

/// Attach a callback that will be called (from within an ISR) when an event is delivered to this core on the given domain.
bool attach_event_handler(EventDomain domain, mbed::Callback<void(uint16_t)> callback);

}

Mbed would attach a callback handler at boot via hal_multicore_event_attach_handler() which would dispatch events based on the event domain.

Queuing API

Standard Output

Flash Lockout

Generally, if other core(s) are executing from flash, it is not safe to use the flash API as the other cores might end up trying to read the flash while it's inaccessible for writing.

Pico SDK takes the approach of forcing the other core into an interrupt handler when using the flash API. I think this might be a good longer term project, but for the initial implementation it's likely sufficient to simply only allow using the flash API from core 0, and assert fail if the API is used while any other cores are in the active state.

Procedure Calls