RFC: Memory Barriers API Design

[KurtWu10]

Authors(s)	@KurtWu10
Date of last update	27 Feb 2026

This RFC is a work in progress.

Overview

Programmers write sddf code in imperative languages like C, pancake or Rust. However, the actual runtime behaviour on hardware could be different from a simple sequential execution of the program text: for example, both the compiler and the hardware may reorder accesses to memory and device registers. These are optimisations performed by both the compiler and hardware to try to improve the performance of program executions, simplifying hardware implementation, among others.

A common case where explicit barriers are NOT required is for ordering accesses to device registers only: sddf's uncached memory maps to strongly-ordered device memory of the corresponding architectures, which prevents reordering by the hardware; sddf also uses volatile accesses to prevent reordering by the compiler.

However, barriers are sometimes required.

Example

Consider a snippet in tx_provide() of the imx Ethernet driver's transmit path:

sddf/drivers/network/imx/ethernet.c

Lines 151 to 153 in 544bc7c

	update_ring_slot(&tx, idx, buffer.io_or_offset, buffer.len, stat);
	tx.tail++;
	eth->tdar = TDAR_TDAR;

The update_ring_slot() function first writes the address and packet length to a descriptor (in device memory), then updates the 'own' bit of the descriptor to transfer the ownership of the descriptor to device.

After that, L153 triggers the device by writing to a device doorbell register. Since there is no barrier between the update to the 'own' bit inside the update_ring_slot() function and the update at L153 to the device doorbell register, they could be reordered on Arm and RISC-V, such that when the device is awakened, it finds that the descriptor is not owned by it and cannot start transmission immediately.

(I have discussed this issue with Courtney, and we agree that we accept this relaxed behaviour, because the device is guaranteed to see the update to the own bit when a second packet occurs.)

To prevent these sometimes undesired executions, programmers are required to explicitly express the requirements in sddf to both the compiler and the hardware. This could be achieved using fence operations (a.k.a. barriers).

Current Implementation

On C, sddf currently uses fence operations provided by the C standard library (for the C11 standard¹). On pancake, Foreign Function Interface (FFI) calls are required (if necessary). Similar fence operations exist on Rust.

Issues

The major issue with using these operations is that they are not designed for device interaction in general. These operations are designed for shared-memory communications between processors (on normal memory), and they could be insufficient for processors communicating with devices:

• On RISC-V, for example, FENCE instructions with the I and O bits are used for ordering device operations, but the compiler will not use these bits for fence operations, as they are irrelevant to thread synchronisations. These means that using these fence operations does not provide the required ordering for devices.

On other architectures, while the compilation on today's compiler is usually correct, the inherent incompatibility of processor communication and device communication complicates reasoning.

Linux Implementation

Linux provides various Linux kernel memory barriers for the same purpose. These operations provide a common API that abstracts the differences among architectures.

For the purpose of this RFC, the relevant operations are barriers for DMA devices dma_wmb()/dma_rmb() and accessors for MMIO registers readX()/writeX()/readX_relaxed()/writeX_relaxed().

Proposal

Solution 1

A simple solution is to use an architecture-independent API for barriers (dma_wmb()/...), which is coarse-grained and may be implemented by more than one instructions. This simplifies reasoning about their correctness². The current implementation for cache_clean/cache_invalidate is an example of this.

For this solution, we need to understand whether the memory in sddf use the same attributes as Linux (e.g. Device-nGnRnE on Armv8), because the Linux barriers are designed only for memory with default attributes. (In fact it doesn't: sddf's uncached memory is Device-nGnRnE, while Linux uses Device-nGnRE)

Strictly following the Linux implementation may incur some additional overhead. For example,

• the Linux specification for readX() and writeX() operations explicitly mentions that

  A writeX() issued by a CPU thread holding a spinlock is ordered before a writeX() to the same peripheral from another CPU thread issued after a later acquisition of the same spinlock.

  while the PDs in sddf are always single-threaded. On RISC-V, the corresponding mmiowb() can be removed.

• Another case is that the readX() operation in Linux is required to complete before any subsequent delay() loop. If sddf does not rely on it,

  • on Arm64, the extra code in Linux's read I/O barrier can be removed
  • on RISC-V, implementation of the I/O read barrier could be simplified.

A drawback is that, on pancake, using an architecture-independent API may introduce additional overhead for fence operations that are compiler barriers, as the language has already enforced the ordering between shared-memory operations.

Such a common interface may be complex, and platform-specific routines could be required.

Example: I/O accessors

Assuming that load tearing and store tearing do not occur with volatile accesses, the current implementation naturally maps to readX_relaxed() and writeX_relaxed() operations. In cases where ordering between memory and I/O accesses, I/O barriers including __io_ar()/__io_bw()/__io_aw()/__io_bw() could be used.

Draft Implementation

macro	x86	arm64	risc-v
dma_rmb()	(compiler barrier)	dmb ld	fence r, r
dma_wmb()	(compiler barrier)	dmb st	fence w, w
iormb()	(compiler barrier)	dmb ld	fence i, r
iowmb()	(compiler barrier)	dmb st	fence w, o

Interaction with Cache Maintenance Operations

Cache maintenance operations (cache_clean() and cache_invalidate()) should be stronger than the corresponding memory barriers (dma_wmb()+iowmb() and dma_rmb()+iormb() resp.), even on architectures with coherent DMA. This is to prevent redundant barrier operations like this.

Solution 2

Another potential implementation is to explicitly use architecture-specific inline assembly, like

#if defined(__aarch64__)
#define dmb() asm volatile("dmb sy" ::: "memory")
#endif

without going through the dmb_mb abstraction, i.e. the main difference compared to the previous solution is in macro names. This makes the programmer's requirement even more explicit, but it may be unclear which operation a device driver should use.

TODOs

1. implementation.
2. further explanations.

Footnotes

1. The current sddf actually uses compiler builtins for atomics from gcc, which follow the C++11 memory model that is highly similar to the C11 memory model. ↩

2. To the best of my knowledge, there is no formal semantics for device interactions on any architecture that is supported by sddf. The Linux kernel memory consistency model, for example, explicitly excludes DMA. Arm's architecture reference manual (section B2.3, version M.a.a) and RISC-V's RVWMO memory consistency model similarly exclude device interactions. We can at best follow the Linux implementation and assume that it is correct. ↩

[Indanz]

Some comments.

The update_ring_slot function first writes the address and packet length to a descriptor (in normal memory), then updates the 'own' bit of the descriptor to transfer the ownership of the descriptor to device.

After that, L153 triggers the device by writing to a device doorbell register. Since there is no barrier between the update to the 'own' bit inside the update_ring_slot function and the update at L153 to the device doorbell register, they could be reordered on Arm and RISC-V, such that when the device is awakened, it finds that the descriptor is not owned by it and cannot start transmission immediately.

I assume this is a mistake and that the descriptor is in uncached memory instead of normal memory. Because if not, then there is no guarantee at all that the device will see the update except on architectures with coherent DMA (x86 except for some PCI memory and RISCV in theory, but not in practice probably).

If the cache is cleaned, then the code doing that should also include the necessary synchronisation barriers.

(I have discussed this issue with Courtney, and we agree that we accept this relaxed behaviour, because the device is guaranteed to see the update to the own bit when a second packet occurs.)

What if there is no second packet, or it comes minutes later? You really don't want non-deterministic delays caused by these kind of problems.

The major issue with using these operations is that they are not designed for device interaction in general. These operations are designed for shared-memory communications between processors (on normal memory), and they could be insufficient for processors communicating with devices:

For this solution, we need to understand whether the memory in sddf use the same attributes as Linux (e.g. Device-nGnRnE on Armv8), because the Linux barriers are designed only for memory with default attributes. (In fact it doesn't: sddf's uncached memory is Device-nGnRnE, while Linux uses Device-nGnRE)

E is for early write acknowledgement. nE means that the write is not cached at all and only finishes when the device acknowledges it. This can have huge latencies, especially on PCI-E (think many microseconds).

Linux allows early write acknowledgement, which means the write is done from a CPU point of view the moment it hits the bus. Still high overhead, but nowhere near as bad. This does require extra form of serialisation (like doing a device memory read) to know for sure a specific write completed though.

Anyway, whatever Linux does would be good enough and sometimes overkill for sDDF.

A drawback is that, on pancake, using an architecture-independent API may introduce additional overhead for fence operations that are compiler barriers, as the language has already enforced the ordering between shared-memory operations.

Does that only prevent compiler re-ordering etc, or does it also add the correct memory barriers? It doesn't seem like it does, and then the hardware is still free to re-order those.

Assuming that load tearing and store tearing do not occur with volatile accesses,

That's implied by the accesses being atomic, so yes, it's guaranteed not to happen.

[KurtWu10]

Thanks for your feedback!

the descriptor is in uncached memory instead of normal memory

Indeed this is the case. In sddf, currently we use seL4's uncached memory (Device-nGnRnE in Arm) for descriptors. There are ongoing discussions on whether we should use 'cacheable' memory for the purpose, following discussions in seL4/seL4#1339.

What if there is no second packet, or it comes minutes later? You really don't want non-deterministic delays caused by these kind of problems.

I'll forward the issue with the system people.

Does that only prevent compiler re-ordering etc, or does it also add the correct memory barriers?

It acts as compiler barriers in programming languages without inserting barrier instructions. It does not prevent hardware re-ordering. Barrier instructions and cache maintenance instructions are programmed using FFIs.

---

That's implied by the accesses being atomic, so yes, it's guaranteed not to happen.

I am not able to derive that for accesses with the volatile keyword, using only the prose text in the C11 standard. I can only find the following text,

A volatile declaration may be used to describe an object corresponding to a memory-mapped
input/output port or an object accessed by an asynchronously interrupting function. Actions on objects so declared shall not be ‘‘optimized out’’ by an implementation or reordered except as permitted by the rules for evaluating expressions.

which corresponds to non-Gathering and non-Reordering in Arm's terminology. However, would 64-bit accesses on 32-bit machines be always treated as one single-copy atomic operation? (of course, current drivers in sddf do not make 64-bit accesses to devices)

At the compiler level, LLVM IR guarantees this for accesses of natively supported data width.

Most of my reasoning here is based on the text on LKMM.

[Indanz]

Does that only prevent compiler re-ordering etc, or does it also add the correct memory barriers?

It acts as compiler barriers in programming languages without inserting barrier instructions. It does not prevent hardware re-ordering. Barrier instructions and cache maintenance instructions are programmed using FFIs.

Might be worth it to add native Pancake support for barriers, assuming that's possible across all platforms. If it translates to either a nop or one instruction it should be easy to add. You want to keep the API simple though.

That's implied by the accesses being atomic, so yes, it's guaranteed not to happen.

I am not able to derive that for accesses with the volatile keyword, using only the prose text in the C11 standard. I can only find the following text,

I wasn't talking about volatile, I was talking about stdatomic operations. Volatile is notoriously badly defined.

A volatile declaration may be used to describe an object corresponding to a memory-mapped
input/output port or an object accessed by an asynchronously interrupting function. Actions on objects so declared shall not be ‘‘optimized out’’ by an implementation or reordered except as permitted by the rules for evaluating expressions.

which corresponds to non-Gathering and non-Reordering in Arm's terminology.

Note that it only says it won't re-order atomic accesses, it can still re-order other memory operations around the atomic accesses (also same as Arm).

However, would 64-bit accesses on 32-bit machines be always treated as one single-copy atomic operation? (of course, current drivers in sddf do not make 64-bit accesses to devices)

Of course not, not with volatile.

At the compiler level, LLVM IR guarantees this for accesses of natively supported data width.

Most of my reasoning here is based on the text on LKMM.

That document should make it clear that none of this is easy and that you should think twice before trying to introduce your own API. I'm not sure compilers are at the stage that they can be trusted to not be buggy when using stdatomics for device memory. Using barriers like atomic_thread_fence(memory_order_acq_rel) should be relatively safe though.

One thing I would like to point out is that the Linux API is barrier based, while the stdatomic API is more focussed on memory accesses. Both approaches have up and downsides. For shared memory interprocess / SMP programming, the stdatomic way works pretty well. For drivers and hardware interaction you rather have barriers.

[Courtney3141]

I think solution 1 is cleaner and easier to use, especially since memory barrier insertion can be prone to errors, and our drivers our often written by people with little experience (at least at first). Abstracting from the platform with a simpler API should make things simpler, and hopefully more correct and optimised in some cases.

@Indanz your point about our hardware ring updating is noted, it will likely be solved by an additional memory barrier after the final insertion into the hardware ring.

[Indanz]

@Indanz your point about our hardware ring updating is noted, it will likely be solved by an additional memory barrier after the final insertion into the hardware ring.

Actually, the current THREAD_MEMORY_RELEASE is overly strong, as Arm already guarantees not to re-order uncached device memory writes and the volatile should stop the compiler from re-ordering the writes already. So I think you only need a barrier before the tdar write.

[KurtWu10]

Actually, the current THREAD_MEMORY_RELEASE is overly strong, as Arm already guarantees not to re-order uncached device memory writes and the volatile should stop the compiler from re-ordering the writes already. So I think you only need a barrier before the tdar write.

As far as I know, descriptors are stored in DRAM, and using the Device-nGnRnE mapping alone does not prevent hardware reordering, according to the Arm manual

The non-Reordering property ... is not required to have an impact on the order of observation of memory accesses to SDRAM.

Or am I not interpreting your comment correctly?

[Courtney3141]

I was referencing your concern around the possibility that the "device kick" register write could possibly be reordered to before the descriptor "ready" bit write, which in the case of the final insertion, could cause processing delays. My understanding is the same as Kurt's in regards to the strength of the barrier, although he has more expertise than me.

[Indanz]

I was referencing your concern around the possibility that the "device kick" register write could possibly be reordered to before the descriptor "ready" bit write, which in the case of the final insertion, could cause processing delays.

I know. But then you mentioned adding a second barrier, which made me have a better look at the existing first one, hence my comment above.

[Indanz]

Actually, the current THREAD_MEMORY_RELEASE is overly strong, as Arm already guarantees not to re-order uncached device memory writes and the volatile should stop the compiler from re-ordering the writes already. So I think you only need a barrier before the tdar write.

As far as I know, descriptors are stored in DRAM, and using the Device-nGnRnE mapping alone does not prevent hardware reordering, according to the Arm manual

The non-Reordering property ... is not required to have an impact on the order of observation of memory accesses to SDRAM.

Or am I not interpreting your comment correctly?

Yes, you are. However, you are misinterpreting Arm's comment. What it means in a somewhat confusing way is:

• The point of coherency is the SDRAM memory controller, not SDRAM itself.
• The memory controller can do its own buffering, coalescing and reordering, as long as it's transparent to the rest of the system. That is, any returned reads should return the same data as if there was no extra buffering.
• This means that the memory accesses as seen by the SDRAM or other outside observers can be reordered in regard to what the CPU did, even for memory mapped with the non-Reordering property.
• None of this reordering applies to the system side: The SDRAM memory controller is not allowed to reorder read and writes coming from the system in an inconsistent way (e.g. returning old data instead of the new data). If it did, the system would break.

This actually applies to all devices that have memory or storage behind them, that's why I think it's confusing: There's nothing much special about SDRAM here. The MMU memory map properties apply to how the CPU cores behave till the memory bus really. But technically, nitpicky, as the memory controller is part of their CPU, for some reason they felt the need to make this explicit. Maybe because this buffering is observable via tracing or external memory probes and they got questions about it in the past.

Edit: And THREAD_MEMORY_RELEASE doesn't stop the memory controller reordering things anyway, nothing can.

[KurtWu10]

The point of coherency is the SDRAM memory controller, not SDRAM itself.

Isn't the PoC included in the home node in the interconnect itself?

ref: AMBA CHI Architecture Specification (issue H) page 32

In a typical CHI-based system, the PoC is the HN-F in the interconnect.

Although most of our platforms use AXI. The AXI spec does not explicitly mention PoC.

---

I can't say I understand the following three bullet points. What I am particularly interested in is whether you are considering reordering to the same location or different locations. And we could ask Arm by writing an email to memory-model@arm.com?

---

And THREAD_MEMORY_RELEASE doesn't stop the memory controller reordering things anyway, nothing can.

Sure, but if the behaviour is observable and violate architecture semantics, how could the IP pass the validation?

[Indanz]

The point of coherency is the SDRAM memory controller, not SDRAM itself.

Isn't the PoC included in the home node, and the home node is in the interconnect itself? And I'm not sure whether the memory controller IP is licensed separately from the interconnect IP, implying that they are separated.

ref: AMBA CHI Architecture Specification (issue H) page 40, HN-F Fully Coherent Home Node. Although most of our platforms only have AXI.

You're missing the point. Details of the system implementation don't matter (e.g. fully coherent or partially, coherent DMA engines or not etc.). The point is that all the memory ordering rules or mapping properties are in regard to PoC, not to physical SDRAM content, because that's behind the memory controller peripheral.

I can't say I understand the following three bullet points. What I am particularly interested in is whether you are considering reordering to the same location or different locations.

That doesn't matter. The point of view for observation matters here.

Sure, but if the behaviour is observable and violate architecture semantics, how could the IP pass the validation?

Because it is observable by SDRAM and when looking at the system from the outside. None of this reordering is visible at the CPU level.

Mind the wording: "observation of memory accesses to SDRAM". To me that means if you observe the actual memory accesses to SDRAM from a device point of view, reordering is allowed.

---

However, after reading the manual and the quote in context I agree things are much more muddy and that I may be wrong about this. It says:

"If the memory accesses are not to a peripheral, then this attribute imposes no restrictions."

I see the memory controller as a peripheral and the SDRAM behind it as part of it. Unclear whether Arm sees it this way. The memory model seems to only have "normal" and "device" memory, it doesn't talk about SDRAM, that's only mentioned twice in the whole document.

The impression I get is that you're not supposed to map SDRAM-backed memory as device memory. But as you tell the CPU what type of memory something is, it will follow the rules for that type. The CPU doesn't magically know that some memory region is SDRAM or not.

The full quote of the note is:

"The non-Reordering property is only required by the architecture to apply the order of arrival of accesses to a
single memory-mapped peripheral of an IMPLEMENTATION DEFINED size, and is not required to have an impact on
the order of observation of memory accesses to SDRAM. For this reason, there is no effect of the non-Reordering
attribute on the ordering relations between accesses to different locations described in Ordering relations as part
of the formal definition of the memory model. It does have an effect on the Peripheral Coherence Order described
in section Completion and endpoint ordering."

I think the main point of this is that other memory accesses outside the non-Reordered mapping are not ordered in regards to the non-reorder accesses. I think they meant "normal memory" when saying SDRAM here. This is about how the non-Reordering property affects the memory model, not about SDRAM being special I think.

Worth asking Arm for clarification though.

What I was talking about before is the previous line:
"The Arm architecture defines only programmer visible behavior. Therefore, reordering can be performed if a programmer cannot tell whether reordering has occurred."

[KurtWu10]

Got a response from Arm. So barriers or other mechanisms are required here.

My question

I am writing to inquire about the interpretation of the Arm Architecture Reference Manual (ARM DDI 0487, version M.a.a) regarding the Reordering attribute (section B.2.10.2.2).

The context is that one uses the Device-nGnRnE memory type for a DRAM memory region. Based on the manual ('If the memory accesses are not to a peripheral, then this attribute imposes no restrictions.'), since the DRAM region is not a device peripheral, the non-Reordering attribute does not apply.

My questions are as follows:

1. ‎How to interpret the allowed reordering in this case? E.g. one interpretation could be that other memory accesses outside the non-reordering region are not ordered with regard to the non-reorder access. Another interpretation is that, similar to Normal memory, all reorderings are allowed (subject to the restrictions in the memory model).

   A concrete example is in a system with a single PE and a DMA device. Both observers have access to a shared memory region. In the message passing test, the PE acts as a producer, and the DMA device acts as a consumer. The memory region is mapped as Device-nGnRnE memory for the PE, and the DMA device accesses the interconnect directly without an SMMU. The PE performs two explicit memory writes in program order with no intermediate memory barriers like a dmb or a dsb. Does the first write ordered-before the second write?

2. In the note section, the manual mentions that 'The non-Reordering property ... is not required to have an impact on the order of observation of memory accesses to SDRAM.' What is special to SDRAM here?

Response

1. The non-Reordering property provides stronger ordering guarantees in relation to the Peripheral arrival order (B2.4.1.1 Peripherals). However the non-Reordering property does not affect the architectural ordering requirements for Processing Elements. In other words, the following litmus test is architecturally allowed when P0 and P1 are CPU cores:

AArch64 MP+nGnRE-po-nGnRE+nGnRE-po-nGnRE
Variant=vmsa
{
 [PTE(x)]=(oa:PA(x),attrs:(Device-nGnRE));
 [PTE(y)]=(oa:PA(y),attrs:(Device-nGnRE));
 0:X1=x; 1:X1=x;
 0:X3=y; 1:X3=y;
}
 P0          | P1          ;
 MOV W0,#1   | LDR W0,[X3] ;
 STR W0,[X1] | LDR W2,[X1] ;
 MOV W2,#1   |             ;
 STR W2,[X3] |             ;
exists(1:X0=1 /\ 1:X2=0)

Similarly, if P0 is a CPU core, P1 is a DMA engine where the two LDRs represent Read Memory Effects generated by the DMA engine and x and y are both mapped to DRAM memory or x is mapped to DRAM and y maps to a register of the DMA, the DMA engine is permitted to observe the two Explicit Memory Write Effects out of order.

However if x and y were Virtual Addresses mapped to the same Peripheral Device, the architecture requires that the Explicit Memory Write Effect to x arrives at the Peripheral Device before the Explicit Memory Write Effect to y.

2. In this note, SDRAM is used to describe an idempotent memory of the system. SDRAM is a special type of DRAM.