Skip to content

support branch hint for AtomicExpandImpl::expandAtomicCmpXchg #152366

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 2, 2025
Merged

support branch hint for AtomicExpandImpl::expandAtomicCmpXchg #152366

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5afd456
add branch hint
diggerlin Jul 16, 2025
b6976bf
modify test case
diggerlin Jul 24, 2025
98d946a
change function name from getTrueBranchHintWeightForAtomicCmpXchgg to…
diggerlin Aug 7, 2025
5d2dcdd
using the target independ code
diggerlin Aug 7, 2025
70c1ccf
change test case
diggerlin Aug 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions llvm/include/llvm/CodeGen/TargetLowering.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2332,6 +2332,12 @@ class LLVM_ABI TargetLoweringBase {
virtual Instruction *emitTrailingFence(IRBuilderBase &Builder,
Instruction *Inst,
AtomicOrdering Ord) const;

virtual MDNode *
getTrueBranchHintWeightForAtomicCmpXchg(LLVMContext &Ctx) const {
return nullptr;
}

/// @}

// Emits code that executes when the comparison result in the ll/sc
Expand Down
13 changes: 9 additions & 4 deletions llvm/lib/CodeGen/AtomicExpandPass.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1454,7 +1454,9 @@ bool AtomicExpandImpl::expandAtomicCmpXchg(AtomicCmpXchgInst *CI) {

// If the cmpxchg doesn't actually need any ordering when it fails, we can
// jump straight past that fence instruction (if it exists).
Builder.CreateCondBr(ShouldStore, ReleasingStoreBB, NoStoreBB);
Builder.CreateCondBr(
ShouldStore, ReleasingStoreBB, NoStoreBB,
TLI->getTrueBranchHintWeightForAtomicCmpXchg(F->getContext()));

Builder.SetInsertPoint(ReleasingStoreBB);
if (ShouldInsertFencesForAtomic && !UseUnconditionalReleaseBarrier)
Expand All @@ -1472,8 +1474,9 @@ bool AtomicExpandImpl::expandAtomicCmpXchg(AtomicCmpXchgInst *CI) {
StoreSuccess = Builder.CreateICmpEQ(
StoreSuccess, ConstantInt::get(Type::getInt32Ty(Ctx), 0), "success");
BasicBlock *RetryBB = HasReleasedLoadBB ? ReleasedLoadBB : StartBB;
Builder.CreateCondBr(StoreSuccess, SuccessBB,
CI->isWeak() ? FailureBB : RetryBB);
Builder.CreateCondBr(
StoreSuccess, SuccessBB, CI->isWeak() ? FailureBB : RetryBB,
TLI->getTrueBranchHintWeightForAtomicCmpXchg(F->getContext()));

Builder.SetInsertPoint(ReleasedLoadBB);
Value *SecondLoad;
Expand All @@ -1486,7 +1489,9 @@ bool AtomicExpandImpl::expandAtomicCmpXchg(AtomicCmpXchgInst *CI) {

// If the cmpxchg doesn't actually need any ordering when it fails, we can
// jump straight past that fence instruction (if it exists).
Builder.CreateCondBr(ShouldStore, TryStoreBB, NoStoreBB);
Builder.CreateCondBr(
ShouldStore, TryStoreBB, NoStoreBB,
TLI->getTrueBranchHintWeightForAtomicCmpXchg(F->getContext()));
// Update PHI node in TryStoreBB.
LoadedTryStore->addIncoming(SecondLoad, ReleasedLoadBB);
} else
Expand Down
6 changes: 6 additions & 0 deletions llvm/lib/Target/PowerPC/PPCISelLowering.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@
#include "llvm/IR/Instructions.h"
#include "llvm/IR/Intrinsics.h"
#include "llvm/IR/IntrinsicsPowerPC.h"
#include "llvm/IR/MDBuilder.h"
#include "llvm/IR/Module.h"
#include "llvm/IR/Type.h"
#include "llvm/IR/Use.h"
Expand Down Expand Up @@ -12816,6 +12817,11 @@ Value *PPCTargetLowering::emitStoreConditional(IRBuilderBase &Builder,
return Builder.CreateXor(Call, Builder.getInt32(1));
}

MDNode *PPCTargetLowering::getTrueBranchHintWeightForAtomicCmpXchg(
LLVMContext &Ctx) const {
return MDBuilder(Ctx).createLikelyBranchWeights();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't target specific. Just remove the new TLI hook and directly do it in AtomicExpand

Copy link
Contributor Author

@diggerlin diggerlin Aug 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not all platforms use the same branch prediction mechanism, so I introduced a new virtual API:

virtual MDNode *
getTrueBranchHintWeightForAtomicCmpXchg(LLVMContext &Ctx) const {
  return nullptr;
}

for example, Target AArch64 do not has branch hint mechanism, it will use default getTrueBranchHintWeightForAtomicCmpXchg(LLVMContext &Ctx) const which return nullptr

Targets that want to support branch hinting for expandAtomicCmpXchg can override this with a target-specific implementation.

according to Power10 Chip manual

The Power10 core normally ignores any software that attempts to override the dynamic branch prediction by setting the “a” bit in the BO field.

When the hardware detects a l*arx instruction near a static prediction, the static prediction is honored.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The details of what the target is going to do with the metadata is unimportant, this is a generic branch likely hint.

}

// The mappings for emitLeading/TrailingFence is taken from
// http://www.cl.cam.ac.uk/~pes20/cpp/cpp0xmappings.html
Instruction *PPCTargetLowering::emitLeadingFence(IRBuilderBase &Builder,
Expand Down
2 changes: 2 additions & 0 deletions llvm/lib/Target/PowerPC/PPCISelLowering.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -938,6 +938,8 @@ namespace llvm {
Instruction *emitTrailingFence(IRBuilderBase &Builder, Instruction *Inst,
AtomicOrdering Ord) const override;

virtual MDNode *
getTrueBranchHintWeightForAtomicCmpXchg(LLVMContext &Ctx) const override;
bool shouldInlineQuadwordAtomics() const;

TargetLowering::AtomicExpansionKind
Expand Down
72 changes: 38 additions & 34 deletions llvm/test/CodeGen/PowerPC/PR35812-neg-cmpxchg.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,51 +19,53 @@ define signext i32 @main() nounwind {
; CHECK-NEXT: addi 3, 1, 46
; CHECK-NEXT: lharx 4, 0, 3
; CHECK-NEXT: cmplwi 4, 33059
; CHECK-NEXT: bne 0, .LBB0_4
; CHECK-NEXT: bne- 0, .LBB0_4
; CHECK-NEXT: # %bb.1: # %cmpxchg.fencedstore
; CHECK-NEXT: sync
; CHECK-NEXT: li 4, 234
; CHECK-NEXT: .p2align 5
; CHECK-NEXT: .LBB0_2: # %cmpxchg.trystore
; CHECK-NEXT: #
; CHECK-NEXT: sthcx. 4, 0, 3
; CHECK-NEXT: beq 0, .LBB0_7
; CHECK-NEXT: beq+ 0, .LBB0_5
; CHECK-NEXT: # %bb.3: # %cmpxchg.releasedload
; CHECK-NEXT: #
; CHECK-NEXT: lharx 5, 0, 3
; CHECK-NEXT: cmplwi 5, 33059
; CHECK-NEXT: beq 0, .LBB0_2
; CHECK-NEXT: beq+ 0, .LBB0_2
; CHECK-NEXT: .LBB0_4: # %cmpxchg.nostore
; CHECK-NEXT: lwsync
; CHECK-NEXT: b .LBB0_8
; CHECK-NEXT: .LBB0_5: # %L.B0000
; CHECK-NEXT: crxor 20, 20, 20
; CHECK-NEXT: b .LBB0_6
; CHECK-NEXT: .LBB0_5: # %cmpxchg.success
; CHECK-NEXT: lwsync
; CHECK-NEXT: creqv 20, 20, 20
; CHECK-NEXT: .LBB0_6: # %cmpxchg.end
; CHECK-NEXT: bc 4, 20, .LBB0_9
; CHECK-NEXT: # %bb.7: # %L.B0000
; CHECK-NEXT: lhz 3, 46(1)
; CHECK-NEXT: cmplwi 3, 234
; CHECK-NEXT: bne 0, .LBB0_9
; CHECK-NEXT: # %bb.6: # %L.B0001
; CHECK-NEXT: bne 0, .LBB0_10
; CHECK-NEXT: # %bb.8: # %L.B0001
; CHECK-NEXT: addis 3, 2, .L_MergedGlobals@toc@ha
; CHECK-NEXT: addi 3, 3, .L_MergedGlobals@toc@l
; CHECK-NEXT: bl puts
; CHECK-NEXT: nop
; CHECK-NEXT: li 3, 0
; CHECK-NEXT: b .LBB0_11
; CHECK-NEXT: .LBB0_7: # %cmpxchg.success
; CHECK-NEXT: lwsync
; CHECK-NEXT: b .LBB0_5
; CHECK-NEXT: .LBB0_8: # %L.B0003
; CHECK-NEXT: b .LBB0_12
; CHECK-NEXT: .LBB0_9: # %L.B0003
; CHECK-NEXT: addis 3, 2, .L_MergedGlobals@toc@ha
; CHECK-NEXT: addi 3, 3, .L_MergedGlobals@toc@l
; CHECK-NEXT: addi 3, 3, 16
; CHECK-NEXT: b .LBB0_10
; CHECK-NEXT: .LBB0_9: # %L.B0005
; CHECK-NEXT: b .LBB0_11
; CHECK-NEXT: .LBB0_10: # %L.B0005
; CHECK-NEXT: addis 3, 2, .L_MergedGlobals@toc@ha
; CHECK-NEXT: addi 3, 3, .L_MergedGlobals@toc@l
; CHECK-NEXT: addi 3, 3, 64
; CHECK-NEXT: .LBB0_10: # %L.B0003
; CHECK-NEXT: .LBB0_11: # %L.B0003
; CHECK-NEXT: bl puts
; CHECK-NEXT: nop
; CHECK-NEXT: li 3, 1
; CHECK-NEXT: .LBB0_11: # %L.B0003
; CHECK-NEXT: .LBB0_12: # %L.B0003
; CHECK-NEXT: addi 1, 1, 48
; CHECK-NEXT: ld 0, 16(1)
; CHECK-NEXT: mtlr 0
Expand All @@ -83,7 +85,7 @@ define signext i32 @main() nounwind {
; CHECK-P7-NEXT: srw 6, 5, 4
; CHECK-P7-NEXT: clrlwi 6, 6, 16
; CHECK-P7-NEXT: cmplwi 6, 33059
; CHECK-P7-NEXT: bne 0, .LBB0_4
; CHECK-P7-NEXT: bne- 0, .LBB0_4
; CHECK-P7-NEXT: # %bb.1: # %cmpxchg.fencedstore
; CHECK-P7-NEXT: lis 6, 0
; CHECK-P7-NEXT: li 7, 234
Expand All @@ -92,51 +94,53 @@ define signext i32 @main() nounwind {
; CHECK-P7-NEXT: slw 7, 7, 4
; CHECK-P7-NEXT: slw 6, 6, 4
; CHECK-P7-NEXT: not 6, 6
; CHECK-P7-NEXT: .p2align 4
; CHECK-P7-NEXT: .LBB0_2: # %cmpxchg.trystore
; CHECK-P7-NEXT: #
; CHECK-P7-NEXT: and 5, 5, 6
; CHECK-P7-NEXT: or 5, 5, 7
; CHECK-P7-NEXT: stwcx. 5, 0, 3
; CHECK-P7-NEXT: beq 0, .LBB0_7
; CHECK-P7-NEXT: beq+ 0, .LBB0_5
; CHECK-P7-NEXT: # %bb.3: # %cmpxchg.releasedload
; CHECK-P7-NEXT: #
; CHECK-P7-NEXT: lwarx 5, 0, 3
; CHECK-P7-NEXT: srw 8, 5, 4
; CHECK-P7-NEXT: clrlwi 8, 8, 16
; CHECK-P7-NEXT: cmplwi 8, 33059
; CHECK-P7-NEXT: beq 0, .LBB0_2
; CHECK-P7-NEXT: beq+ 0, .LBB0_2
; CHECK-P7-NEXT: .LBB0_4: # %cmpxchg.nostore
; CHECK-P7-NEXT: crxor 20, 20, 20
; CHECK-P7-NEXT: lwsync
; CHECK-P7-NEXT: b .LBB0_8
; CHECK-P7-NEXT: .LBB0_5: # %L.B0000
; CHECK-P7-NEXT: b .LBB0_6
; CHECK-P7-NEXT: .LBB0_5: # %cmpxchg.success
; CHECK-P7-NEXT: lwsync
; CHECK-P7-NEXT: creqv 20, 20, 20
; CHECK-P7-NEXT: .LBB0_6: # %cmpxchg.end
; CHECK-P7-NEXT: bc 4, 20, .LBB0_9
; CHECK-P7-NEXT: # %bb.7: # %L.B0000
; CHECK-P7-NEXT: lhz 3, 46(1)
; CHECK-P7-NEXT: cmplwi 3, 234
; CHECK-P7-NEXT: bne 0, .LBB0_9
; CHECK-P7-NEXT: # %bb.6: # %L.B0001
; CHECK-P7-NEXT: bne 0, .LBB0_10
; CHECK-P7-NEXT: # %bb.8: # %L.B0001
; CHECK-P7-NEXT: addis 3, 2, .L_MergedGlobals@toc@ha
; CHECK-P7-NEXT: addi 3, 3, .L_MergedGlobals@toc@l
; CHECK-P7-NEXT: bl puts
; CHECK-P7-NEXT: nop
; CHECK-P7-NEXT: li 3, 0
; CHECK-P7-NEXT: b .LBB0_11
; CHECK-P7-NEXT: .LBB0_7: # %cmpxchg.success
; CHECK-P7-NEXT: lwsync
; CHECK-P7-NEXT: b .LBB0_5
; CHECK-P7-NEXT: .LBB0_8: # %L.B0003
; CHECK-P7-NEXT: b .LBB0_12
; CHECK-P7-NEXT: .LBB0_9: # %L.B0003
; CHECK-P7-NEXT: addis 3, 2, .L_MergedGlobals@toc@ha
; CHECK-P7-NEXT: addi 3, 3, .L_MergedGlobals@toc@l
; CHECK-P7-NEXT: addi 3, 3, 16
; CHECK-P7-NEXT: b .LBB0_10
; CHECK-P7-NEXT: .LBB0_9: # %L.B0005
; CHECK-P7-NEXT: b .LBB0_11
; CHECK-P7-NEXT: .LBB0_10: # %L.B0005
; CHECK-P7-NEXT: addis 3, 2, .L_MergedGlobals@toc@ha
; CHECK-P7-NEXT: addi 3, 3, .L_MergedGlobals@toc@l
; CHECK-P7-NEXT: addi 3, 3, 64
; CHECK-P7-NEXT: .LBB0_10: # %L.B0003
; CHECK-P7-NEXT: .LBB0_11: # %L.B0003
; CHECK-P7-NEXT: bl puts
; CHECK-P7-NEXT: nop
; CHECK-P7-NEXT: li 3, 1
; CHECK-P7-NEXT: .LBB0_11: # %L.B0003
; CHECK-P7-NEXT: .LBB0_12: # %L.B0003
; CHECK-P7-NEXT: addi 1, 1, 48
; CHECK-P7-NEXT: ld 0, 16(1)
; CHECK-P7-NEXT: mtlr 0
Expand Down
Loading