Preview PR: Hugo docs: Add NUMA design docs #16
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bernhardkaindl
temporarily deployed
to
docs-add-NUMA-infos - bernhard-xapi.onrender.com PR #16
Render
Destroyed
Taken from https://github.com/qemu/qemu/blob/a9cd5bc6399a80fcf233ed0fffe6067b731227d8/scripts/qcow2-to-stdout.py > This tool converts a disk image to qcow2, writing the result directly > to stdout. This can be used for example to send the generated file > over the network. > > This is equivalent to using qemu-img to convert a file to qcow2 and > then writing the result to stdout, with the difference that this tool > does not need to create this temporary qcow2 file and therefore does > not need any additional disk space. Signed-off-by: Andrii Sultanov <[email protected]>
We are only ever going to use this script with "raw" files. Signed-off-by: Andrii Sultanov <[email protected]>
The original script is designed to only work on regular files. Adapt it slightly to work on special block devices (since that's what we want to export from). Aside from dropping some unnecessary checks and replacing os.path.getsize with a functional equivalent that works on block special files, these changes remove detection of "holes" in the raw file, since SEEK_DATA and SEEK_HOLE are unimplemented in this case. Signed-off-by: Andrii Sultanov <[email protected]>
…sters When specified, only the clusters that have changed between diff_file_name and input_file will be allocated and exported in a sparse manner. This is analogous to vhd-tool's --relative-to option. Example usage: ``` Create an empty raw file: $ dd if=/dev/zero of=disk.before.img bs=1M count=100 Fill the first three clusters with random data $ dd if=/dev/random conv=notrunc of=disk.before.img bs=65536 count=3 $ python3/libexec/qcow2-to-stdout.py disk.before.img > disk.before.qcow2 Check that only the first three clusters are allocated (each is 0x10000 in length) $ qemu-img map disk.before.qcow2 Offset Length Mapped to File 0 0x30000 0x50000 disk.before.qcow2 Overwrite the 2nd and 3rd clusters with new data $ cp disk.before.img disk.after.img $ dd if=/dev/random conv=notrunc of=disk.after.img bs=65536 count=2 oseek=1 Export the difference, verifying that only the 2nd and 3rd clusters are allocated $ python3/libexec/qcow2-to-stdout.py disk.after.img --diff disk.before.img > disk.diff.qcow2 $ qemu-img map disk.diff.qcow2 Offset Length Mapped to File 0x10000 0x20000 0x50000 disk.diff.qcow2 The image can be recreated if the base is imported first, with the difference overwriting it $ qemu-img convert -f qcow2 -O raw disk.before.qcow2 disk.test.img $ qemu-img convert -f qcow2 -O raw disk.diff.qcow2 disk.test.img --target-is-zero -n $ diff disk.test.img disk.after.img <the same> ``` Signed-off-by: Andrii Sultanov <[email protected]>
Most of these have been unused for almost 10 years since c4ccc56 ("CA-217842: Replace instances of vm_lacks_feature_x with vm_lacks_feature") Drop them.
When shutdown a VM, xapi will check SM API version to decide if to call `post_deactivate_hook`. But if the SR has already been unplugged, the checking will fail. Solution: Check if the plugin of the SR still exists. If not, skip the SM API checking.
Previously if only some of the values was missing, the arithmetic operations where done and they were taken into account into the host total. When a domain's values are missing, they are not taken into the sum. Also prepare the code to add per-VM metrics by using shared infrastructure and using a record. Signed-off-by: Pau Ruiz Safont <[email protected]>
The daemon was already watching the vm memory targets, so it makes sense to make it generate the datasources directly as well. This uses the shared xenctrl infrastructure that's needed to fetch the domains' uuids. This means that xcp-rrdd does not need to collect this anymore, and xenopsd does not need to send it to xcp-rrdd. The function in the idl is safe to delete because the only user was xenopsd, and it was well-protected against errors. This means that if an old version of xenopsd tries to call the function while xcp-rrdd has already been updated, it won't interrupt xneopsd functionality. Signed-off-by: Pau Ruiz Safont <[email protected]>
Since this daemon already uses xenstore to watch other memory keys in xenstore, move another one in here, allows to delete quite a bit of code from xcp-rrdd and drop dependencies as well. Signed-off-by: Pau Ruiz Safont <[email protected]>
… file Xapi currently reads directly from memory-mapped metrics files. Since we want to move all the memory metrics to rrdp_squeezed, and this will output all the metrics into a single memory-mapped file, move the host and VM memory metrics into a single module, and make them share a bit of code. Signed-off-by: Pau Ruiz Safont <[email protected]>
This allows to xcp-rrdd to stop writing to memory-mapped files This needed xapi to be adapted since it read the contents of the memory-mapped files. Now it can get those metric from a single memory-mapped file. Signed-off-by: Pau Ruiz Safont <[email protected]>
Signed-off-by: Pau Ruiz Safont <[email protected]>
…i-project#6561) The former should not collect any metrics, and instead only consolidate them, while the latter was already collecting memory metrics using xenctrl and xenstore, but only 2 of them. Xapi had to be modified to be able to collect the metrics from the new source, since the file where they reside has changed. The memory_target metric was also associated with xenopsd and an RPC call. These are no longer needed and code has been deleted. The only functional change is that now memory_target is not reported when it's not available. The new behaviour makes it work the same as other guest-agent-reported metrics. Previously it was reported as 0 instead, which was incorrect. Since the other metrics are reported in the same way, clients already know how to cope with this behaviour. Tests done have compared an unpatched host with a patched one: - Upgrade using yum update: * no changes apparent in xencenter, both on host and vm tabs * none of the metrics disappeared, and they have the same metadata. Checked with rrd2csv, xe vm-data-sources-list and xe host-data-sources-list * memory-target now is reported as N/A in rrd2csv, it was previously reported as 0 (expected change) * Host metrics are still shown in the database with correct numbers ``` # xe host-param-list uuid=5efc236c-f137-4df7-9902-ba8fbf9a2ac2 | grep mem memory-overhead ( RO): 639635456 memory-total ( RO): 33849733120 memory-free ( RO): 26117525504 memory-free-computed ( RO): 26077683712 ``` - New host installation: * xencenter performance tabs look fine * memory-target is now missing (expected change, because this metric was never collected) * Host metrics are shown and total memory matches update case (same host was used, with the VMs using less memory) ``` # xe host-param-list uuid=b9c0e6d5-c362-469f-8147-38184125e9be | grep mem memory-overhead ( RO): 639635456 memory-total ( RO): 33849733120 memory-free ( RO): 21789081600 memory-free-computed ( RO): 21749161984 ```
- Move `set_ssh_expiry` outside of the `schedule_disable_ssh_job` to avoid restart XAPI reset host.ssh_expiry - Ensure the SSH service is enabled during auto-mode Signed-off-by: Lunfan Zhang[Lunfan.Zhang] <[email protected]>
Signed-off-by: Steven Woods <[email protected]>
…t#6489) VBDs can be attached to multiple VMs, so now that VBD_plug has been split into VBD_attach and VBD_activate, the attach can happen outside of the VM migrate downtime. This doesn't change the overall duration of the migration but does reduce the downtime by several seconds. Opening as a draft PR to get some eyes on it early as it will need extensive testing.
According to https://www.freedesktop.org/software/systemd/man/latest/hostname.html Systemd set hostname with following sequence - kernel parameter, systemd.hostname - static hostname in /etc/hostname - transient hostname like DHCP - localhost at systemd compile time Once the host is configured with static IP and reboot, it would just lost its hostname as no DHCP or static IP available. However, the hostname is critical to AD function as it construct the machine account. The hostname should be persisted as static name during joining AD, this is also what PBIS does. Note: the static hostname is not cleaned during domain leave. This is by intention to avoid losing hostname after reboot with static IP cba2f1d tried to resovle the issue and update /etc/resolv.conf However, /etc/resolv.conf does not help and conflict with xcp-networkd, as networkd override the configure every 5 minutes Here we just revert the resolv.conf update. Other parts of that commit can still benifit as it push the hostname to DNS Signed-off-by: Lin Liu <[email protected]>
…api-project#6546) According to https://www.freedesktop.org/software/systemd/man/latest/hostname.html Systemd set hostname with following sequence - kernel parameter, systemd.hostname - static hostname in /etc/hostname - transient hostname like DHCP - localhost at systemd compile time Once the host is configured with static IP and reboot, it would just lost its hostname as no DHCP or static IP available. However, the hostname is critical to AD function as it construct the machine account. The hostname should be persisted as static name during joining AD, this is also what PBIS does. Note: the static hostname is not cleaned during domain leave. This is by intention to avoid losing hostname after reboot with static IP cba2f1d tried to resovle the issue and update /etc/resolv.conf However, /etc/resolv.conf does not help and conflict with xcp-networkd, as networkd override the configure every 5 minutes Here we just revert the resolv.conf update
Signed-off-by: Steven Woods <[email protected]>
For deprivileged qemu, following ops are performed - bind mount /dev/ to qemu chroot, so qemu can access it - cgroup controller deny all devices, except the target usb device However, new XS updated to cgroup v2 and the devices controller available anymore. Instead of bind mount all /dev folder, only the permitted usb devices are created into the chroot. Thus, the cgroup controller is no longer necessary. Besides, there are following updates accordingly - qemu pid is no longer necessary as command line args, as cgroup is dropped. - save and restore system /etc/ devices file ownership is no longer necessary. New file is cloned into chroot instead of bind mount system device file, so only need to set ownership of chroot file directly Signed-off-by: Lin Liu <[email protected]>
This patch add a new parameter named 'xen-platform-pci-bar-uc' in xenopsd config file who has a default value of 'true' to keep the default behavior of hvmloader. Putting 'false' to this parameter will tell xenopsd to add a xenstore key of '0' in: '/local/domain/<domid>/hvmloader/pci/xen-platform-pci-bar-uc'. Only this key set to 0 will change the behavior of hvmloader. This changeset is link to this xen commit: https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=22650d6054625be10172fe0c78b9cadd1a39bd63 Signed-off-by: Anthoine Bourgeois <[email protected]>
Add a new API call for VM sysprep and the corresponding XE implementation. This is mostly scaffolding. Signed-off-by: Christian Lindig <[email protected]>
We want to create a temporary directory that will be used to hold files for creating an ISO. There is no existing function that creates all necessary directories in a predicatble way. Signed-off-by: Christian Lindig <[email protected]>
Implement creating an ISO from a temporary directory Signed-off-by: Christian Lindig <[email protected]>
Signed-off-by: Christian Lindig <[email protected]>
Create a local SR unless it exists. Signed-off-by: Christian Lindig <[email protected]>
We need to locate the CD drive of the VM. Signed-off-by: Christian Lindig <[email protected]>
The VM is notified to perform a sysprep by writing to XenStore. The VM picks this up via its guest agent. Signed-off-by: Christian Lindig <[email protected]>
Signed-off-by: Christian Lindig <[email protected]>
* Use the decoder from the OCaml standard library instead of our own implementation, which this patch removes. * Validate UTF-8/XML conformance for maps and sets, in addition to strings. This is XSA-474 / CVE-2025-58146. Reviewed-by: Edwin Török <[email protected]> Patch from: https://xenbits.xen.org/xsa/advisory-474.html
This adds the monitor service required for the SSH auto-mode, as described in `doc/content/toolstack/features/SSH`. Signed-off-by: Lunfan Zhang <[email protected]> Signed-off-by: Rob Hoes <[email protected]>
Signed-off-by: Bengang Yuan <[email protected]> Signed-off-by: Rob Hoes <[email protected]>
Original code by @LunfanZhang and @BengangY.
…ject#6607) Adds a new span.depth key to the trace context baggage, and a configurable max_span_depth. This defaults to 100 and so will not limit traces (the traces I've seen with the most depth are ~40 depth e.g. https://jaeger.kfd.eng.citrite.net/trace/ea5ddca5509b3ae1102bc7279092652d), but is useful when wanting to analyse large traces which can often become slow if all the spans are recorded in a trace. This isn't perfect, the span.depth seems to get lost sometimes between xapi and xenops, resulting in a greater depth than that listed, but I have created ticket CP-308999 for this and this works well enough to greatly reduce the number of spans in a trace when needed, which is the intention. As an example, a host evacuate trace with max_span_depth 10 goes down to ~1000 spans rather than the 34k+ withou a depth limit.
Previously both xapi and networkd had to inspect the IP configuration to decide whether the DNS values should be persistend into /etc/resolv.conf. This actually lead to a mismatch in them. Instead use an option value for DNS that simply means that if there's a value, it must be persisted. Now xapi decides the instances where these values are written. Treat a couple of empty lists as a lack of value to avoid writing empty resolv.conf files. This can happen when updating a host from previous versions, which use empty lists when using DHCP. Tested manually by installing a version with this change and restarting the toolstack. The file is kept intact, unlike the previous version of the change that did not take into account the update behaviour. Signed-off-by: Pau Ruiz Safont <[email protected]>
Signed-off-by: Gabriel Buica <[email protected]>
…lugging When unplugging a pbd, enabling a host, or adding a vbd, the shared SR constraint violation could be violated, but the error used in these cases was that the operation blocked the failover planning. This was confusing because the main reason was not mentioned in the error. Instead use the SR constraint violation error, and log a more descriptive message in the logs as info, because these can happen during normal operation and there's nothing dodgy going on. Signed-off-by: Pau Ruiz Safont <[email protected]>
…d unplugging When unplugging a PIF, enabling a host, or adding a VIF, the shared network constraint violation could be violated, but the error used in these cases was that the operation blocked the failover planning. This was confusing because the main reason was not mentioned in the error. Instead use the network constraint violation error, and log a more descriptive message in the logs as info, because these can happen during normal operation and there's nothing dodgy going on. Signed-off-by: Pau Ruiz Safont <[email protected]>
It didn't add any useful information to the error. Also cleaned up the formatting of some comments found during the patch series. Signed-off-by: Pau Ruiz Safont <[email protected]>
…xapi-project#6666) Now the HA shared SR and network constraint violations are used when plugging and unplugging. When unplugging a pbd or pif, enabling a host, or adding a vbd or vif, the shared SR or network constraint violations could be violated, but the error used in these cases was that the operation blocked the failover planning. This was confusing because the main reason was not mentioned in the error. Instead use the shared constraint violation error, and log a more descriptive message in the logs as info, because these can happen during normal operation and there's nothing dodgy going on. I previously wanted to know Tina's opinion on how we change the reason in a way that can be better treated by clients and internationalized, but saw that the error used was simply not the right one.
…i-project#6664) Because Autoconf is not DHCP, networkd uses the dns value to write to resolv.conf. This is done on ocaml/networkd/bin/network_server.ml line 745 This allows to have non-empty resolv.conf when using IPv6 autoconf. xapi-idl/network: Remove code duplication for DNS persistence decisions: Previously both xapi and networkd had to inspect the IP configuration to decide whether the DNS values should be persistend into /etc/resolv.conf. This actually lead to a mismatch in them. Instead use an option value for DNS that simply means that if there's a value, it must be persisted. Now xapi decides the instances where these values are written. Treat a couple of empty lists as a lack of value to avoid writing empty resolv.conf files. This can happen when updating a host from previous versions, which use empty lists when using DHCP. Tested manually by installing a version with this change and restarting the toolstack. The file is kept intact, unlike the previous version of the change that did not take into account the update behaviour. This is PR fixed version of xapi-project#6586
Some of these were passed through several layers of functions only to be unused in the end. Drop them, improving the legibility of the code. Signed-off-by: Andrii Sultanov <[email protected]>
Some of these were passed through several layers of functions only to be unused in the end. Drop them, improving the legibility of the code.
When performing the changes described in add-function.md for adding a host-price-of function to xapi, a type error would arise from the message-forwarding.ml file - this is fixed by explicitly giving the remote_fn named argument. Signed-off-by: Christian Pardillo Laursen <[email protected]>
When performing the changes described in add-function.md for adding a host-price-of function to xapi, a type error would arise from the message-forwarding.ml file - this is fixed by explicitly giving the remote_fn named argument.
This removes using effect as a label because now it's a reserved word, and changes how the memprof is used to be compatible both with 4.14 and 5.3, by ignoring the value it returns. Signed-off-by: Pau Ruiz Safont <[email protected]>
This removes using effect as a label because now it's a reserved word, and changes how the memprof is used to be compatible both with 4.14 and 5.3, by ignoring the value it returns. I also see a storm of warning about not linking against the unix library explicitly. We'll deal with that later on
Rolling pool upgrades should only allow VMs to be evacuated to hosts that have the same or higher versions of xapi build or xen. Previously, the check was done only for platform version. Signed-off-by: Gabriel Buica <[email protected]>
The "vgpu_map" can be empty for an intra-pool migration. This was missed previously because XenCenter indeed prepares non-empty "vgpu_map" for intra-pool migration with shared storage. But it prepares empty "vgpu_map" for intra-pool migration with local storage. Signed-off-by: Ming Lu <[email protected]>
This changes comments as well as the location of some comments,otherwise ocamlformat is unable to process files Signed-off-by: Pau Ruiz Safont <[email protected]>
Signed-off-by: Pau Ruiz Safont <[email protected]>
Some comments have had its location changed because this version of ocamlformat got confused with them. Ignoring whitespace changes is highly recommended, there's a button with a square as its icon on top of each commit page to do enable this mode.
…ject#6655) Rolling pool upgrades should only allow VMs to be evacuated to hosts that have the same or higher versions of xapi build or xen. Previously, the check was done only for platform vision. I will run a BST again after I squash the fixup to be sure. Otherwise, I think this is ready.
…project#6680) The "vgpu_map" can be empty for an intra-pool migration. This was missed previously because XenCenter indeed prepares non-empty "vgpu_map" for intra-pool migration with shared storage. But it prepares empty "vgpu_map" for intra-pool migration with local storage.
networkd had code to call ovs-vlan-bug-workaround, but this has been made obsolete as the bug in question only applies to kernels before 2.6.37, which is long out of support. We can safely remove the functionality associated with applying the workaround. Signed-off-by: Christian Pardillo Laursen <[email protected]>
Now that we have removed handle_vlan_bug_workaround, has_vlan_accel is no longer used anywhere, and it is no longer relevant. Therefore, we can safely remove it, and the function get_features that it calls. Signed-off-by: Christian Pardillo Laursen <[email protected]>
networkd had code to call ovs-vlan-bug-workaround, but this has been made obsolete as the bug in question only applies to kernels before 2.6.37, which is long out of support. We can safely remove the functionality associated with applying the workaround.
Signed-off-by: Bernhard Kaindl <[email protected]>
bernhardkaindl
force-pushed
the
docs-add-NUMA-infos
branch
from
9f46a3d to
c93e957
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.