Skip to content

Commit 214a8c2

Browse files
bernhardkaindlbernhardkaindl
committed
Hugo docs: Add NUMA design docs
Signed-off-by: Bernhard Kaindl <bernhard.kaindl@cloud.com>
1 parent b86bbe8 commit 214a8c2

4 files changed

Lines changed: 158 additions & 0 deletions

File tree

‎doc/content/toolstack/features/NUMA/node-fallbacks.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
---
2+
title: VM.build Use neighbouring NUMA nodes
3+
mermaid:
4+
force: true
5+
theme: forest
6+
mermaidInitialize: '{ "theme": "base" }'
7+
---
8+
9+
{{%include "topologies/2x2.md" %}}
10+
11+
This shows that the distance to remote memory on the same socket
12+
is far lower than the distance to the other socket, which is twice
13+
the local memory distance.
14+
15+
As a result, this means that using remote memory on the same socket
16+
increases the distance by only by 1/10th than using the other socket.
17+
18+
19+
Hence, for VMs builds where might not be enough NUMA memory on one
20+
NUMA node, using same-socket memory would have only about 50% of the
21+
than remote-socket memory would have.
22+
23+
At same time, if the memory is (roughly) equally spread over two
24+
NUMA nodes on the same socket, it could make sense to move the
25+
vCPU affinity between the two NUMA nodes depending on their CPU load.
26+
27+
28+
29+
30+
31+
In the simplest case, the vCPU affinity could be set to e.g. two
32+
NUMA nodes on the same socket (specified as having low distance),
33+
which would cause Xen to allocate the memory from both NUMA nodes.
34+
35+
If this is not done,
36+
37+
38+
##
39+
| Node | RAM | used | free |
40+
| ----:| ---:| ----:| ----:|
41+
| 1 | 50 | 35 | 15 |
42+
| 2 | 50 | 45 | 5 |
43+
| 3 | 50 | 35 | 15 |
44+
| 4 | 50 | 35 | 15 |
45+
| all | 200 | 150 | 50 |
46+
47+
<!---
48+
This shows that the distance to remote memory on the same socket
49+
is far lower than the distance to the other socket, which is twice
50+
the local memory distance.
51+
52+
As a result, this means that using remote memory on the same socket
53+
increases the distance by only by 1/10th than using the other socket.
54+
55+
Hence, for VMs builds where might not be enough NUMA memory on one
56+
NUMA node, using same-socket memory would have only about 50% of the
57+
than remote-socket memory would have.
58+
59+
At same time, if the memory is (roughly) equally spread over two
60+
NUMA nodes on the same socket, it could make sense to move the
61+
vCPU affinity between the two NUMA nodes depending on their CPU load.
62+
63+
In the simplest case, the vCPU affinity could be set to e.g. two
64+
NUMA nodes on the same socket (specified as having low distance),
65+
which would cause Xen to allocate the memory from both NUMA nodes.
66+
67+
If this is not done,
68+
69+
##
70+
| Node | RAM | used | free |
71+
| ----:| ---:| ----:| ----:|
72+
| 1 | 50 | 35 | 15 |
73+
| 2 | 50 | 45 | 5 |
74+
| 3 | 50 | 35 | 15 |
75+
| 4 | 50 | 35 | 15 |
76+
| all | 200 | 150 | 50 |
77+
78+
--->

‎doc/content/toolstack/features/NUMA/parallel-boot.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
title: "Parallel VM.build"
3+
---
4+
5+
Summary of the xenopsd architecture when running micro-ops
6+
in parallel:
7+
8+
See: [VM.build: Architecture](architecture.md)
9+
10+
Running multiple `VM.build` micro-ops in parallel can
11+
run into two kinds of race conditions for NUMA placement:
12+
13+
- [Lazy scrubbing by the Xen hypervisor](lazy-scrubbing.md)

‎doc/content/toolstack/features/NUMA/topologies/2x2.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
---
2+
title: 2 sockets, 4 nodes
3+
description: NUMA toplogy with 2 sockets, 4 nodes
4+
---
5+
6+
### Example NUMA topology with 2 sockets, 4 nodes:
7+
8+
A topology with 2 sockets and 4 nodes results
9+
in this NUMA memory distance matrix:
10+
11+
|node| 0| 2| 1| 3|
12+
|---:|-:|-:|-:|-:|
13+
| 0 |10|21|11|21|
14+
| 2 |21|10|21|11|
15+
| 1 |11|21|10|21|
16+
| 3 |21|11|21|10|
17+
18+
19+
The distance values in this diagram describes in a normalized way how large
20+
the distance from a NUMA node's CPU to the memory of another node is:
21+
22+
- 10: This is (by convention) the distance to local memory of the NUMA node
23+
- 11: Relative to 10, the distance to the remote memory on the same socket
24+
- 21: Relative to 10, the distance to the remote memory on the other socket
25+
26+
This NUMA distance matrix could be visualized using this block diagram:
27+
28+
{{< mermaid >}}
29+
block-beta
30+
columns 3
31+
%% 1st row, left column
32+
block columns 1
33+
Mem0[/"Memory of Node 0"/]
34+
Dist0<["Distance: 10"]>(up)
35+
Node0{{"CPU of Node 0"}}
36+
end
37+
%% 1st row, middle column
38+
space
39+
%% 1st row, right column
40+
block columns 1
41+
Mem2[/"Memory of Node 2"/]
42+
Dist2<["Distance: 10"]>(up)
43+
Node2{{"CPU of Node 2"}}
44+
end
45+
%% 2nd row
46+
Socket_1<["Distance: 1"]>(y)
47+
x<["Distance: 10"]>(x)
48+
Socket_2<["Distance: 1"]>(y)
49+
%% 3rd row
50+
block columns 1
51+
Node1{{"CPU of Node 1"}}
52+
Dist1<["Distance: 10"]>(down)
53+
Mem1[/"Memory of Node 1"/]
54+
end
55+
space
56+
block columns 1
57+
Node3{{"CPU of Node 3"}}
58+
Dist3<["Distance: 10"]>(down)
59+
Mem3[/"Memory of Node 3"/]
60+
end
61+
{{< /mermaid >}}

‎doc/content/toolstack/features/NUMA/topologies/_index.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
+++
2+
title = "NUMA topologies"
3+
weight = 20
4+
+++
5+
6+
{{% children description=true %}}

0 commit comments

Comments
 (0)