Add resident device change call #1517

lslusarczyk · 2025-09-11T12:30:55Z

UMF part of https://github.com/intel/llvm/blob/sycl/sycl/doc/extensions/supported/sycl_ext_oneapi_peer_access.asciidoc feature.

UR/llvm part is in intel/llvm#19257

include/umf/memory_pool_ops.h

src/libumf.map

bratpiorka · 2025-09-11T12:39:25Z

src/memory_provider.c

+        hProvider->provider_priv, deviceIndex, isAdding);
+    checkErrorAndSetLastProvider(res, hProvider);
+    return res;
+}


add empty line

src/provider/provider_level_zero.c

bratpiorka · 2025-09-11T12:48:49Z

src/provider/provider_cuda.c

    return UMF_RESULT_ERROR_INVALID_ARGUMENT;
 }

+static umf_result_t


add a "default" handler in memory_provider.c (see umfDefaultCtlHandle) and do not modify CUDA provider

Where exactly in the code you think "default" is missing?

do not modify CUDA provid

Rejected. umf_memory_provider_ops_t type received a new field and it should be somehow initialized in UMF_CUDA_MEMORY_PROVIDER_OPS as all other fields. Please explain if you still want me to remove this change.

Please initialize cuda ops .ext_resident_device_change to NULL.
Next, define the umfDefaultResidentDeviceChange in src/memory_provider.c that returns UMF_RESULT_ERROR_NOT_SUPPORTED and set this handler for each created provider if the ext_resident_device_change is set to NULL (see how this is done for other default handlers like "umfDefaultCloseIPCHandle")

ok, applied

bratpiorka · 2025-09-11T12:51:31Z

src/provider/provider_tracking.c

        p->hUpstream, memory_property_id, size);
 }

+static umf_result_t trackingResidentDeviceChange(void *provider,


Rejected. umf_memory_provider_ops_t type received a new field and it should be somehow initialized in UMF_TRACKING_MEMORY_PROVIDER_OPS as all other fields. Please explain if you still want me to remove this change.

applied in the same way as in cuda provider

bratpiorka · 2025-09-11T12:51:53Z

test/common/provider_null.c

    return UMF_RESULT_SUCCESS;
 }

+static umf_result_t nullResidentDeviceChange(void *provider,


rejected, same reason as above

bratpiorka · 2025-09-12T08:15:51Z

src/provider/provider_level_zero.c


-    if (deviceCount && !hDevices) {
-        LOG_ERR("Resident devices array is NULL, but deviceCount is not zero");
+    if (residentDevicesCount && !residentDevicesIndices) {


please also check if residentDevicesCount == 0 but residentDevicesIndices != NULL

additionally, should indices be unique?

ok, added checking for uniqueness

bratpiorka · 2025-09-12T08:18:39Z

src/provider/provider_level_zero.c

 }

+struct ze_memory_provider_resident_device_change_data {
+    bool isAdding;


bratpiorka · 2025-09-12T08:19:39Z

src/provider/provider_level_zero.c

+        change_data->source_memory_provider
+            ->device_handles[change_data->peer_device_index];
+
+    // TODO: add assertions to UMF and change it to be an assertion


use ASSERT() macros from src/utils/utils_assert.h

Postponed to some future PR. I need here a permanent ASSERT, not a one that compiles on in debug config like ASSERT does.

you assert only things which are can be an artefact of some internal umf error - if it is true debug only assert is fine. If an error can be a result of an user input you just handle the issue and return a error to the user

This case it is artifact of some internal umf error.

I believe, what Łukasz was saying, since it is artifact of some internal umf error a simple debug assert is ok - pls add assert() and remove the TODO

bratpiorka · 2025-09-12T08:21:28Z

src/provider/provider_level_zero.c

+            info->props.base, info->props.base_size);
+    } else {
+        result = ZE_RESULT_SUCCESS;
+        // TODO: currently not implemented call evict here


do you plan to add the missing code here?

Not in this PR.

bratpiorka · 2025-09-12T08:24:18Z

src/provider/provider_level_zero.c

+    if (ze_provider->resident_device_count == 0 ||
+        existing_peer_index == ze_provider->resident_device_count) {
+        // not found
+        if (!isAdding) { // impossible for UR, should be an assertion


if this should be an assertion, please use the ASSERT() macro from src/utils/utils_assert.h
Also, move the comment to the next line

To be changed in future PR when umf will have permanent assertions. Comments moved.

bratpiorka · 2025-09-12T08:24:27Z

src/provider/provider_level_zero.c

+
+static umf_result_t
+ze_memory_provider_resident_device_change(void *provider, uint32_t device_index,
+                                          bool isAdding) {


bratpiorka · 2025-09-12T08:24:46Z

src/provider/provider_level_zero.c

+        // adding case
+        if (ze_provider->device_count <=
+            ze_provider
+                ->resident_device_count) { // impossible for UR, should be an assertion


if this should be an assertion, please use the ASSERT() macro from src/utils/utils_assert.h
Also, move the comment to the next line

postponed when UMF have permanent assertions, comments moved

bratpiorka · 2025-09-12T08:24:54Z

src/provider/provider_level_zero.c

+        }
+
+        if (ze_provider->device_count <=
+            device_index) { // impossible for UR, should be an assertion


applied as above

bratpiorka · 2025-09-12T08:25:32Z

src/provider/provider_level_zero.c

+
+    } else {
+        // found
+        if (isAdding) { // impossible for UR, should be an assertion


applied as above

bratpiorka · 2025-09-12T08:27:28Z

src/provider/provider_level_zero.c

+        LOG_ERR("umfMemoryTrackerIterateAll did not manage to do some change "
+                "numFailed:%d, numSuccess:%d",
+                privData.success_changes, privData.failed_changes);
+        return UMF_RESULT_ERROR_INVALID_ARGUMENT; // probably some other result is better, best just change into assertion


you could use UMF_RESULT_ERROR_MEMORY_PROVIDER_SPECIFIC
or change to assert

src/critnib/critnib.c

src/libumf.map

lukaszstolarczuk · 2025-09-15T09:02:58Z

src/libumf.def

    umfMemoryProviderPurgeForce
    umfMemoryProviderPurgeLazy
    umfMemoryProviderPutIPCHandle
+    umfMemoryProviderResidentDeviceChange


pls move down in the file (after a comment ; Added in UMF_1.0)

first of all, wrong function was moved - umfMemoryProviderResidentDeviceChange is the new one

mea culpa, it should be already in ; Added in UMF_1.1 section

Moved down in 01d74ba

umfLevelZeroMemoryProviderResidentDeviceChange still not in the 1.1 section (it should be right after umfLevelZeroMemoryProviderParamsSetName)

src/memory_provider.c

lplewa · 2025-09-15T10:18:31Z

include/umf/memory_provider_ops.h

+    /// @brief Adds or removes devices on which allocations should be made
+    ///        resident.
+    /// @param provider handle to the memory provider
+    /// @param device_index identifier of device
+    /// @param is_adding Boolean indicating if peer is to be removed or added
+    /// @return UMF_RESULT_SUCCESS on success or appropriate error code on
+    ///         failure.
+    umf_result_t (*ext_resident_device_change)(void *provider,
+                                               uint32_t device_index,
+                                               bool is_adding);
+


This function should not be in OPS. This is something very specific to L0 provider. Resident devices are passed to L0 provider thru provider specific params, so control of them should be olso be done through provider specific API, this is why i think with should be implemented through CTL

Rejected. We discussed this in teams (4th Aug, 25) we do it as ops (Piotr, sorry in Polish): co do tego API co Ty robisz (w stylu daj mi wszystkie zaalokowane page'e) ja bym chyba sugerował zrobić API a nie robić przez CTLa, Rafał: no to faktycznie chyba dedykowane API lepiej pasuje niż CTL.

CTL should be for statistics, not a universal, hard-to-read tool to implement any API, Łukasz (me): Być może będziecie chcieli uprościć ext_ctl by nie był maszynką, którą mozna zaimplementować wszystko, a jedynie służył do statystyk. Ale to już ja się na tym nie znam - zostawiam do przemyśleń.

We talked about API to iterate over all allocations, not about API to modify some internal settings of the L0

I can see it both ways. My comment during that discussion was about a generic functionality implementable for all providers. This isn't that.
I also remember making a point about CTL being useful for provider/pool-specific functionality.

On the other hand, adding an API function is simpler.

On the other hand, adding an API function is simpler.

Thank you for the comment. Chosen simpler way.

lplewa · 2025-09-15T10:18:33Z

include/umf/providers/provider_level_zero.h

 /// @brief  Set the resident devices in the parameters struct.
 /// @param  hParams handle to the parameters of the Level Zero Memory Provider.
-/// @param  hDevices array of devices for which the memory should be made resident.
-/// @param  deviceCount number of devices for which the memory should be made resident.
+/// @param  hDevices array of all devices for which the memory can be made resident.
+/// @param  deviceCount number of devices for which the memory can be made resident.
+/// @param  residentDevicesIndices array of indices in all devices array to devices for which the memory should be made resident.
+/// @param  residentDevicesCount number of items in indices array.
 /// @return UMF_RESULT_SUCCESS on success or appropriate error code on failure.
 umf_result_t umfLevelZeroMemoryProviderParamsSetResidentDevices(
    umf_level_zero_memory_provider_params_handle_t hParams,
-    ze_device_handle_t *hDevices, uint32_t deviceCount);
+    ze_device_handle_t *hDevices, uint32_t deviceCount,
+    uint32_t *residentDevicesIndices, uint32_t residentDevicesCount);


This is an API(and ABI) break - we are post 1.0 release so you cannot do changes like this.

Shall I bump some number? Only UR uses this API and I am changing UR right now as well.

Even if UR uses only this API, old UR should work with new umf. And we will not do 2.0 release just after 1.0.

You cannot brake API. If changes are needed we have to keep old function working correctly and add new one.

We can't break backwards compatibility with SYCL / UR. Changing a major version is a significant undertaking that involves updating the components across all the different layers (UMF is nearly the lowest-most component in the stack). We've been burned on this in the past, and it's very disruptive.

In this case, my suggestion would be to simply create a new function that sets the indices.

OK, applied, I rewrote code in 53ec753 to be backward compatible

ldorau

14 / 16 files reviewed (will be continued)

pbalcer · 2025-09-16T08:54:51Z

include/umf/memory_provider_ops.h

+    ///         failure.
+    umf_result_t (*ext_resident_device_change)(void *provider,
+                                               uint32_t device_index,
+                                               bool is_adding);


I'd rather we have two functions - add/remove.

I will not apply this, please, because this will introduce too much copy-pasted code.

lplewa · 2025-09-19T13:16:18Z

I still oppose against adding provider specific code to ops interface.
If you don't want to do it thru CTL just implement it as umfLevelZeroMemoryProviderResidentDeviceChange() in provider_level_zero.h which do not go throught OPS. This function will act as an "level zero specific api extension".

note: remember that you will have to call umfMemoryProviderGetPriv() in this function to retrieve ze_memory_provider_t from provider handle

bratpiorka · 2025-09-19T13:23:54Z

src/libumf.def

    umfCUDAMemoryProviderParamsSetContext
    umfCUDAMemoryProviderParamsSetDevice
    umfCUDAMemoryProviderParamsSetMemoryType
+    umfLevelZeroMemoryProviderParamsSetResidentDevices


please keep alphabetical order

ok, applied

bratpiorka · 2025-09-19T13:25:00Z

src/provider/provider_level_zero.c


 static void init_ze_global_state(void) {
+
+    char *lib_name = getenv("UMF_ZE_LOADER_LIB_NAME");


why we need this code? will this code path be used by UR? if not please remove this

see MockedLevelZeroTestEnvironment::SetUp This variable is needed by tests which mock L0.

lplewa · 2025-09-19T13:21:25Z

src/provider/provider_level_zero.c


 static void init_ze_global_state(void) {
+
+    char *lib_name = getenv("UMF_ZE_LOADER_LIB_NAME");


Can you do this thru CTL please.

We created CTL to limit new envvariables.

This is for tests only. What is wrong with env variables? They are simple to use in the code and to understand what they do there. Anyway I may do it by CTL. Please write me how to set in a way that it will be effective in this early place, that is global state initialization and how to read it.

lplewa · 2025-09-19T13:26:46Z

src/provider/provider_level_zero.c

    hParams->resident_device_handles = hDevices;
    hParams->resident_device_count = deviceCount;



Shouldn't we copy this array? - user after call to this function can do anything with this array.

This is how it worked before. UR code is aware of it and keeps params until they are used. hParams is copied in ze_memory_provider_initialize. Decided not to fix this in this PR.

lplewa · 2025-09-19T13:29:52Z

src/provider/provider_level_zero.c

            ze_provider->context, ze_provider->resident_device_handles[i],
            *resultPtr, size);
        if (ze_result != ZE_RESULT_SUCCESS) {
+            utils_read_unlock(&ze_provider->resident_device_rwlock);


If we return an failure then it's means allocation failed, but allocation is still "done". IMHO we should free it here.

good point, fixed

lplewa · 2025-09-19T13:33:02Z

src/provider/provider_level_zero.c

+        change_data->source_memory_provider
+            ->device_handles[change_data->peer_device_index];
+
+    // TODO: add assertions to UMF and change it to be an assertion


you assert only things which are can be an artefact of some internal umf error - if it is true debug only assert is fine. If an error can be a result of an user input you just handle the issue and return a error to the user

lplewa · 2025-09-19T13:33:09Z

src/provider/provider_level_zero.c

+
+    // TODO: add assertions to UMF and change it to be an assertion
+    if (info->props.base != (void *)key) {
+        LOG_ERR("key:%p is different than base:%p", (void *)key,


ok, applied

lplewa · 2025-09-19T13:37:40Z

src/utils/utils_log.h

-#define LOG_DEBUG(...) utils_log(LOG_DEBUG, __func__, __VA_ARGS__);
-#define LOG_INFO(...) utils_log(LOG_INFO, __func__, __VA_ARGS__);
-#define LOG_WARN(...) utils_log(LOG_WARNING, __func__, __VA_ARGS__);
-#define LOG_ERR(...) utils_log(LOG_ERROR, __func__, __VA_ARGS__);
-#define LOG_FATAL(...) utils_log(LOG_FATAL, __func__, __VA_ARGS__);
-
-#define LOG_PDEBUG(...) utils_plog(LOG_DEBUG, __func__, __VA_ARGS__);
-#define LOG_PINFO(...) utils_plog(LOG_INFO, __func__, __VA_ARGS__);
-#define LOG_PWARN(...) utils_plog(LOG_WARNING, __func__, __VA_ARGS__);
-#define LOG_PERR(...) utils_plog(LOG_ERROR, __func__, __VA_ARGS__);
-#define LOG_PFATAL(...) utils_plog(LOG_FATAL, __func__, __VA_ARGS__);
+#ifdef UMF_DEVELOPER_MODE
+#define UMF_STRINGIFY(x) #x
+#define UMF_TOSTRING(x) UMF_STRINGIFY(x)
+#define UMF_FUNC_DESC() __FILE__ ":" UMF_TOSTRING(__LINE__)
+#else
+#define UMF_FUNC_DESC() __func__
+#endif
+
+#define LOG_DEBUG(...) utils_log(LOG_DEBUG, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_INFO(...) utils_log(LOG_INFO, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_WARN(...) utils_log(LOG_WARNING, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_ERR(...) utils_log(LOG_ERROR, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_FATAL(...) utils_log(LOG_FATAL, UMF_FUNC_DESC(), __VA_ARGS__);
+
+#define LOG_PDEBUG(...) utils_plog(LOG_DEBUG, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_PINFO(...) utils_plog(LOG_INFO, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_PWARN(...) utils_plog(LOG_WARNING, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_PERR(...) utils_plog(LOG_ERROR, UMF_FUNC_DESC(), __VA_ARGS__);
+#define LOG_PFATAL(...) utils_plog(LOG_FATAL, UMF_FUNC_DESC(), __VA_ARGS__);


This should be a separate PR.

There are comments which i whould make - but i strongly believe that this discussion should happen in the separate PR to do not block this PR

lplewa · 2025-09-19T14:05:12Z

src/provider/provider_level_zero.c

+        // found
+        if (is_adding) {
+            utils_write_unlock(&ze_provider->resident_device_rwlock);
+            // impossible for UR, should be an assertion


imho we should not assume that user will not do something then is should be an assertion. If it comes it for any kind of user it should be validated an correct error should be returned

ok, comment removed

lplewa · 2025-09-19T14:09:47Z

src/provider/provider_level_zero.c

+
+    uint32_t existing_peer_index = 0;
+    utils_write_lock(&ze_provider->resident_device_rwlock);
+    while (existing_peer_index < ze_provider->resident_device_count &&


NIT: this code whould be much easier to read if it whould be a for loop

for (index = 0; index < count; index++) if (device[index] == index) break;

I need for loop index to be available outside the loop. I don't think it is more readable. Left as is.

lplewa · 2025-09-19T14:36:06Z

src/provider/provider_level_zero.c

+            const uint32_t new_capacity =
+                ze_provider->resident_device_capacity * 2 +
+                1; // +1 to work also with old capacity == 0
+            ze_device_handle_t *new_handles =


Do this code is performance critical, or will be called multiple times

If not i would prefer to have this array just always reallocated by one,
If yes can we abstract this vector like structure to separate module, or even better just use list structure with is included in umf.

ok, not critical, applied

lplewa · 2025-09-19T14:39:34Z

src/provider/provider_level_zero.c

+        .failed_changes = 0,
+    };
+
+    umf_result_t result = umfMemoryTrackerIterateAll(


This is "hacky" and it will not work if someone uses pool without tracker or just use provider without pool.

We allready solved this issue in other providers were we keep track allocation by the provider (for example please check os_provider)

I was told by you, that I can use tracker. Since it is not a case now that l0 provider is used without tracker, so I am only leaving a comment here that warns about what you noticed.

lslusarczyk · 2025-09-22T13:45:24Z

I still oppose against adding provider specific code to ops interface. If you don't want to do it thru CTL just implement it as umfLevelZeroMemoryProviderResidentDeviceChange() in provider_level_zero.h which do not go throught OPS. This function will act as an "level zero specific api extension".

note: remember that you will have to call umfMemoryProviderGetPriv() in this function to retrieve ze_memory_provider_t from provider handle

Great! Using umfLevelZeroMemoryProviderResidentDeviceChange simplified code even more than using ops. Applied with pleasure.

lukaszstolarczuk · 2025-09-22T14:46:32Z

test/common/ze_loopback.cpp

+
+// defined in both libze_ops from src/utils/utils_level_zero.cpp and
+// ze_ops_t operations from src/provider/provider_level_zero.c
+// zeDeviceGetProperties, zeMemFree, zeMemGetAllocProperties, zeMemFree, zeMemAllocDevice, find them in the former ones


missing EOF

lukaszstolarczuk · 2025-09-22T14:47:55Z

test/common/level_zero_mocks.h

+ *
+ */
+
+#ifndef UMF_PROVIDER_LEVEL_ZERO_MOCKS_H


UMF_PROVIDER_LEVEL_ZERO_MOCKS_H -> UMF_TEST_PROVIDER_LEVEL_ZERO_MOCKS_H

lukaszstolarczuk · 2025-09-22T14:48:26Z

test/common/level_zero_mocks.h

+    MOCK_METHOD2(zeMemFree,
+                 ze_result_t(ze_context_handle_t hContext, void *ptr));
+
+    // helper setting all expects related to successful l0 provider creation & initialization and calling its creation & initialization


pls break a long line

lukaszstolarczuk · 2025-09-22T14:48:50Z

test/common/ze_loopback.h

+// Under the Apache License v2.0 with LLVM Exceptions. See LICENSE.TXT.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+#ifndef UMF_ZE_LOOPBACK_H


UMF_ZE_LOOPBACK_H -> UMF_TEST_ZE_LOOPBACK_H

lukaszstolarczuk · 2025-09-22T14:49:26Z

test/common/level_zero_mocks.cpp

@@ -0,0 +1,96 @@
+/*
+ *
+ * Copyright (C) 2023-2025 Intel Corporation


simply 2025, as it's a new file

// apply to all new files

lukaszstolarczuk · 2025-09-22T14:49:50Z

test/pools/pool_residency.cpp

+    InitGoogleTest(&argc, argv);
+    AddGlobalTestEnvironment(new MockedLevelZeroTestEnvironment);
+    return RUN_ALL_TESTS();
+}


missing EOF

lukaszstolarczuk · 2025-09-22T14:53:37Z

src/provider/provider_level_zero.c

+        change_data->source_memory_provider
+            ->device_handles[change_data->peer_device_index];
+
+    // TODO: add assertions to UMF and change it to be an assertion


I believe, what Łukasz was saying, since it is artifact of some internal umf error a simple debug assert is ok - pls add assert() and remove the TODO

lukaszstolarczuk · 2025-09-22T15:04:09Z

src/provider/provider_level_zero.c


 static void init_ze_global_state(void) {
+
+    const char *lib_name = getenv("UMF_ZE_LOADER_LIB_NAME");


should we document this new env var? is there a use case to set it outside of testing...?

lslusarczyk requested a review from a team as a code owner September 11, 2025 12:30

bratpiorka requested changes Sep 11, 2025

View reviewed changes

bratpiorka requested review from lplewa and ldorau September 11, 2025 12:53

bratpiorka reviewed Sep 12, 2025

View reviewed changes

lslusarczyk force-pushed the memresident branch from d53a23f to 9d7e12c Compare September 15, 2025 08:16

lukaszstolarczuk reviewed Sep 15, 2025

View reviewed changes

lplewa requested changes Sep 15, 2025

View reviewed changes

lslusarczyk mentioned this pull request Sep 15, 2025

memory resident limit to enabled peers intel/llvm#19257

Draft

ldorau reviewed Sep 15, 2025

View reviewed changes

pbalcer reviewed Sep 16, 2025

View reviewed changes

lslusarczyk added 10 commits September 19, 2025 13:56

Add resident device change call

e5d8739

added missing doxygen documentation

8c54695

fix documentation spelling

064d1ac

applied Rafal's comments

c4a7499

fixed compilation error

588d1af

applied second round of Rafal's comments

b24574c

applied Lukasz's comments

5983cfa

one more missed Lukasz comment applied

5cbbad2

added some tests for resident device feature

c27407a

add unittests and refactor to be backward compatible

53ec753

lslusarczyk force-pushed the memresident branch from 88f672b to 53ec753 Compare September 19, 2025 12:00

lslusarczyk added 3 commits September 19, 2025 14:07

moved umfMemoryProviderResidentDeviceChange down

01d74ba

added empty line for Rafal :)

0bc9a29

cmake format fixed

d13a3ef

bratpiorka reviewed Sep 19, 2025

View reviewed changes

lplewa requested changes Sep 19, 2025

View reviewed changes

lslusarczyk added 7 commits September 22, 2025 12:10

Lukasz and Rafal comments applied

272bd66

fix log test

cabd5e7

removed usless ext_resident_device_change from memory_pool

7de5b15

use umfLevelZeroMemoryProviderResidentDeviceChange instead of ops

32ebea7

reverted log change as it was moved to different PR

50db5c5

removed accidental changes

780794c

fix doc

4f75bca

lukaszstolarczuk reviewed Sep 22, 2025

View reviewed changes


		static void init_ze_global_state(void) {

		char *lib_name = getenv("UMF_ZE_LOADER_LIB_NAME");

		hParams->resident_device_handles = hDevices;
		hParams->resident_device_count = deviceCount;


		static void init_ze_global_state(void) {

		const char *lib_name = getenv("UMF_ZE_LOADER_LIB_NAME");

Add resident device change call #1517

Are you sure you want to change the base?

Add resident device change call #1517

Uh oh!

Conversation

lslusarczyk commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lslusarczyk Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lslusarczyk Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lslusarczyk commented Sep 11, 2025 •

edited

Loading

lslusarczyk Sep 12, 2025 •

edited

Loading

lslusarczyk Sep 15, 2025 •

edited

Loading

lplewa Sep 15, 2025 •

edited

Loading