CpuBoundWork#CpuBoundWork(): don't spin on atomic int to acquire slot #9990

Al2Klimov · 2024-02-07T10:42:13Z

This is inefficient and implies
possible bad surprises regarding waiting durations on busy nodes. Instead,
use AsioConditionVariable#Wait() if there are no free slots. It's notified
by others' CpuBoundWork#~CpuBoundWork() once finished.

fixes #9988

Also, the current implementation is a spin-lock. 🙈 #10117 (comment)

Al2Klimov · 2024-02-07T11:18:27Z

Low load test

[2024-02-07 12:16:44 +0100] information/ApiListener: New client connection from [::ffff:127.0.0.1]:51732 (no client certificate)
[2024-02-07 12:16:44 +0100] information/CpuBoundWork: Using one free slot, free: 12 => 11
[2024-02-07 12:16:44 +0100] information/CpuBoundWork: Releasing one used slot, free: 11 => 12
[2024-02-07 12:16:44 +0100] information/CpuBoundWork: Using one free slot, free: 12 => 11
[2024-02-07 12:16:44 +0100] information/CpuBoundWork: Releasing one used slot, free: 11 => 12
[2024-02-07 12:16:44 +0100] information/CpuBoundWork: Using one free slot, free: 12 => 11
[2024-02-07 12:16:44 +0100] information/CpuBoundWork: Releasing one used slot, free: 11 => 12
[2024-02-07 12:16:44 +0100] information/HttpServerConnection: Request: GET /v1/objects/services/3d722963ae43!4272 (from [::ffff:127.0.0.1]:51732), user: root, agent: curl/8.4.0, status: Not Found).
[2024-02-07 12:16:44 +0100] information/HttpServerConnection: HTTP client disconnected (from [::ffff:127.0.0.1]:51732)
[2024-02-07 12:16:44 +0100] information/CpuBoundWork: Using one free slot, free: 12 => 11
[2024-02-07 12:16:44 +0100] information/CpuBoundWork: Releasing one used slot, free: 11 => 12

Just a few increments/decrements. 👍

Al2Klimov · 2024-02-07T11:24:13Z

High load test

If I literally DoS Icinga with https://github.com/Al2Klimov/i2all.tf/tree/master/i2dos, I get a few of these:

[2024-02-07 12:19:37 +0100] information/CpuBoundWork: Handing over one used slot, free: 0 => 0

After I stop that program and fire one curl as in my low load test above, I get the same picture: still 12 free slots. 👍

Logs

--- lib/base/io-engine.cpp
+++ lib/base/io-engine.cpp
@@ -24,6 +24,7 @@ CpuBoundWork::CpuBoundWork(boost::asio::yield_context yc)
        std::unique_lock<std::mutex> lock (sem.Mutex);

        if (sem.FreeSlots) {
+               Log(LogInformation, "CpuBoundWork") << "Using one free slot, free: " << sem.FreeSlots << " => " << sem.FreeSlots - 1u;
                --sem.FreeSlots;
                return;
        }
@@ -32,7 +33,9 @@ CpuBoundWork::CpuBoundWork(boost::asio::yield_context yc)

        sem.Waiting.emplace(&cv);
        lock.unlock();
+       Log(LogInformation, "CpuBoundWork") << "Waiting...";
        cv.Wait(yc);
+       Log(LogInformation, "CpuBoundWork") << "Waited!";
 }

 void CpuBoundWork::Done()
@@ -42,8 +45,10 @@ void CpuBoundWork::Done()
                std::unique_lock<std::mutex> lock (sem.Mutex);

                if (sem.Waiting.empty()) {
+                       Log(LogInformation, "CpuBoundWork") << "Releasing one used slot, free: " << sem.FreeSlots << " => " << sem.FreeSlots + 1u;
                        ++sem.FreeSlots;
                } else {
+                       Log(LogInformation, "CpuBoundWork") << "Handing over one used slot, free: " << sem.FreeSlots << " => " << sem.FreeSlots;
                        sem.Waiting.front()->Set();
                        sem.Waiting.pop();
                }

lib/base/io-engine.cpp

julianbrost · 2024-02-19T16:07:37Z

Is the way boost::asio::deadline_timer is used by AsioConditionVariable here actually safe? It's documentation says that shared objects are not thread-safe.

lib/base/io-engine.cpp

Al2Klimov · 2024-11-06T15:00:39Z

In addition, v2.14.2 could theoretically misbehave once the free slot amount falls "temporarily" noticeably below zero. Like, three requestors achieve an ioEngine.m_CpuBoundSemaphore.fetch_sub(1) while it's zero (0 - 3 x 1 = -3). Now, requestor A realizes that it's not allowed to take that slot and adds 1 again (-2). And so does requestor B (-1). But A subtracts again (-2) before C also adds 1 again (-1). And so on.

https://github.com/Icinga/icinga2/blob/v2.14.2/lib/base/io-engine.cpp#L24-L31

So that spinlock blocks not only CPU time, but also slots from legit requestors. The father of all spinlocks, so to say. 🙈 #10117 (comment)

Al2Klimov · 2024-11-28T13:51:34Z

lib/remote/eventshandler.cpp

 	response.set(http::field::content_type, "application/json");

-	IoBoundWorkSlot dontLockTheIoThread (yc);
+	handlingRequest.Done();


I've figured out and pushed 75271fe how to get rid of this and the 38 changed files.

$ git diff --stat 74009f0fc..a663f98b2 lib/base/io-engine.cpp | 77 +++++++++++++++++++++++++++++++++++++++-------------------------------------- lib/base/io-engine.hpp | 64 +++++++++++++++++++++++++++++++++------------------------------- lib/remote/eventshandler.cpp | 2 -- lib/remote/httpserverconnection.cpp | 19 +++++++++++++------ lib/remote/httpserverconnection.hpp | 2 ++ lib/remote/jsonrpcconnection.cpp | 2 +- 6 files changed, 88 insertions(+), 78 deletions(-) $

Please let me know whether you consider this better.

CC @julianbrost @yhabteab @oxzi

I've figured out and pushed 75271fe how to get rid of this and the 38 changed files.

424e1bc (#9990) is an annoying commit, but not something that would have to block this PR.

On the change suggested in 75271fe: that sounds like moving into the direction I suggested in #10142:

Related: when doing bigger changes to the interface there, one other improvement that comes to mind is how HttpServerConnection::StartStreaming() works: currently, to take control over the whole connection, this has to be called, but the underlying ASIO stream is still passed to every handler but it must not be used without calling StartStreaming(), otherwise, there's a good chance the connection ends up in a broken state. This could be improved by only exposing the underlying stream as a return value of the StartStreaming() method, similar to how it works in Go's net/http package.

So if the point of StartStreaming() is to transfer ownership of the connection to the caller, for me it makes sense to release other resources related to the connection like the CpuBoundWork.

Thus, overall I'd say this is a sane change for StartStreaming(), so feel free to keep it in (but the commit history should be cleaned up, that doesn't need that revert commit in there).

Al2Klimov · 2024-11-28T14:22:35Z

lib/base/io-engine.cpp

-}
+	try {
+		cv->Wait(yc);
+	} catch (...) {


Unfortunately, yes – this can throw.

--- test/base-shellescape.cpp +++ test/base-shellescape.cpp @@ -1,6 +1,8 @@ /* Icinga 2 | (c) 2012 Icinga GmbH | GPLv2+ */ #include "base/utility.hpp" +#include <boost/asio.hpp> +#include <boost/asio/spawn.hpp> #include <BoostTestTargetConfig.h> #include <iostream> @@ -16,6 +18,26 @@ BOOST_AUTO_TEST_CASE(escape_basic) BOOST_CHECK(Utility::EscapeShellCmd("$PATH") == "\\$PATH"); BOOST_CHECK(Utility::EscapeShellCmd("\\$PATH") == "\\\\\\$PATH"); #endif /* _WIN32 */ + + auto io = new boost::asio::io_context; + boost::asio::spawn(*io, [io](boost::asio::yield_context yc) { + boost::asio::deadline_timer timer(*io, boost::posix_time::seconds(3)); + boost::system::error_code ec; + + try { + timer.async_wait(yc[ec]); + } catch (const boost::coroutines::detail::forced_unwind&) { + BOOST_CHECK(false); // error: in "base_shellescape/escape_basic": check false has failed + throw; + } + }); + boost::asio::deadline_timer timer(*io, boost::posix_time::seconds(1)); + timer.async_wait([io](boost::system::error_code ec) { + if (!ec) { + delete io; + } + }); + io->run(); } BOOST_AUTO_TEST_CASE(escape_quoted)

Unfortunately, yes – this can throw.

You can't be serious :), you deliberately deleted the I/O object with delete io;, what do you expect to happen in that case then? This is not a realistic test case, I mean where do we perform a questionable operation like this in Icinga 2 code? @julianbrost and I tried to trigger this exception last week but weren't able to, and reading the detailed 🙃 boost docs about it didn't help to understand this either.

If for some reason the I/O context gets deleted in Icinga 2, do you think you can ever recover from it? If something like this happens in Icinga 2, then you have far more severe problems than unreleased CPU semaphores.

If something like this happens in Icinga 2, then you have far more severe problems than unreleased CPU semaphores.

So you'd not catch it at all?

So you'd not catch it at all?

No, I didn't say that! I just want to understand under what normal circumstances such an exception would be triggered and not by deleting the global io object. For instance, how can you explicitly trigger a stack unwinding of a coroutine? If one can force the destruction of a coroutine, then you can also verify that it enters into that new catch-all handler as intended, but none of us is able to do that, and that is the puzzle that needs to be solved.

the puzzle that needs to be solved

Does it need to be resolved? To me, the bare lack of noexcept in async_wait is enough. Yes, I'm serious! If something could be thrown, I catch it and clean up after myself. As I said to Julian, if you want to test it, add a throw 1;. The most often used function that theoretically can throw an exception is operator new, but I don't know yet how to temporarily override malloc(3) locally. But I think I don't even need that, as #9990 (comment) already shows enough. (Also, I didn't test what happens across fork(2), I hope on_before_fork() (or whatever it's called in ASIO) preserves coroutine stacks.)

Though if even deleting the io object triggers this exception (also tested with IoEngine::SpawnCoroutine()), I guess it's an indicator for that if the coroutine gets destroyed for whatever reason, we'll be able to intercept it and handle it accordingly.

Didn't see your intermediate comment above while sending my previous comment!

As I said to Julian, if you want to test it, add a throw 1;.

Where should I put that throw expression? Throwing an exception within the coroutine handler (the provided callback that is called within the coroutine) does not cause the coroutine to be destroyed.

Does it need to be resolved? To me, the bare lack of noexcept in async_wait is enough

If you don't want to make decisions just based on assumptions, then yes you need to understand when this exception could be triggered. I'm not talking about just any exception, but the specific force_unwind exception. As I said before, if the user-supplied callback throws an exception, it will never hit that new catch-all handler here, nor will it destroy the coroutine.

As I said to Julian, if you want to test it, add a throw 1;.

Where should I put that throw expression?

Just inside the try catch instead of cv->Wait(yc);. Seriously. Actually I don't worry especially about a particular exception type. I just easily got forced_unwind in my comment above. Who knows, maybe a malloc(3) fails? It's just: theoretically, the method could throw – I handle it. Especially in this case the code above in CpuBoundWork#CpuBoundWork() has already deployed some pointers to stack variables to IoEngine#m_CpuBoundSemaphore.Waiting. If our super unlikely exception hits someone w/o my try/catch once in 10y, happy debugging!

Now "better", colleagues?

--- a/lib/base/io-engine.cpp +++ b/lib/base/io-engine.cpp @@ -30,2 +30,5 @@ CpuBoundWork::CpuBoundWork(boost::asio::yield_context yc, boost::asio::io_contex if (ie.m_CpuBoundSemaphore.compare_exchange_weak(freeSlots, freeSlots - 1)) { + if (cv) { + Log(LogCritical, "LOLCAT", "Got slot via ASIO!"); + } return; @@ -36,3 +39,3 @@ CpuBoundWork::CpuBoundWork(boost::asio::yield_context yc, boost::asio::io_contex cv = Shared<AsioConditionVariable>::Make(ie.GetIoContext()); - continue; + //continue; }

[2024-12-05 17:37:35 +0100] critical/LOLCAT: Got slot via ASIO! [2024-12-05 17:37:35 +0100] critical/LOLCAT: Got slot via ASIO! [2024-12-05 17:37:35 +0100] critical/LOLCAT: Got slot via ASIO!

I mean, it works well, a DoS even triggers the above logs, but the fair scheduling is unsurprisingly gone. :(

Also, FWIW, I gave up on df63a78 (boost::asio::async_result).

lib/base/io-engine.hpp

yhabteab

Apart from the horrible interface design of the HttpServerConnection class, looks fine for me now! Tested it with this script!

$ cat post-checkresults.sh
#!/bin/zsh
for x in $(seq "$1"); do
	curl -sSk \
		-u root:icinga \
		-o /dev/null \
		-H 'Content-Type: application/json' \
		-H 'Accept: application/json' \
		-X POST 'https://localhost:5667/v1/actions/process-check-result' -d@- <<EOF &
{
	"type": "Service",
	"filter": "host.name==\"big-switch-server-155\" || host.name==\"big-switch-server-156\"",
	"exit_status": $(( RANDOM % 4 )),
	"plugin_output": "$(( RANDOM ))",
	"pretty": true
}
EOF
done

wait $(jobs -p)
---
$ while :; do ./post-checkresults.sh 100; printf .; done

Al2Klimov · 2025-07-22T09:06:39Z

I've rebased to resolve merge conflicts.

Al2Klimov · 2025-08-04T11:00:49Z

@yhabteab As you've already approved this, can we merge this already?

julianbrost · 2025-08-04T11:30:30Z

Please don't merge this without me having a look at it.

Al2Klimov · 2025-08-22T08:29:33Z

Then please have a look at it, so we can merge this. :)

lib/base/io-engine.cpp

lib/base/io-engine.hpp

lib/base/io-engine.cpp

Al2Klimov · 2025-08-29T13:30:49Z

~~Actually, in contrast to GHA,~~ it builds on Windows. ~~(Trust me, bro.^TM)~~

@jschmidt-icinga I have resolved the conflicts with #10516. The only major change is cc46ab0.

jschmidt-icinga · 2025-09-01T07:27:50Z

lib/remote/httpmessage.cpp

 void HttpResponse::StartStreaming(bool checkForDisconnect)
 {
+	auto work (m_CpuBoundWork.lock());
+
+	if (work) {
+		work->Done();
+	}


If this was done here, it should at least be conditional on the checkForDisconnect bool, because otherwise this will also be done at the start of ObjectQueryHandler, which does do CPU-bound work in between sending chunks.

In that case it could then be done up in HttpServerConnection::StartDetectClientSideShutdown, same as before, with no need to pass it down to the message.

However, there isn't really any good way for HttpServerConnection to guess what the handler is doing. Even releasing it in StartDetectClientSideShutdown is an assumption HttpServerConneciton can only make because the EventsHandler is the only place using it at the moment.

So personally, what I'd actually prefer, would be to either hand it to the message, and add a method to manually stop the CpuBoundWork, which currently only the EventsHandler would call. Or, as I said before in #10046, just lock it as needed in each of the handlers. HttpResponse could have a factory method that makes a CpuBoundWork object, similar to the one that makes a JsonEncoder. The yield_context is already there anyway, but then we would need to additionally keep the strand in the message object, which granted, is a bit ugly.

If this was done here, it should at least be conditional on the checkForDisconnect bool, because otherwise this will also be done at the start of ObjectQueryHandler, which does do CPU-bound work in between sending chunks.

Done.

In that case it could then be done up in HttpServerConnection::StartDetectClientSideShutdown, same as before, with no need to pass it down to the message.

I'm afraid a pointer to CpuBoundWork will end up in one of the HTTP classes anyway.

HttpResponse seems better to me as every HttpResponse is cleaned up after being sent – together with the CpuBoundWork pointer.

However, there isn't really any good way for HttpServerConnection to guess what the handler is doing. Even releasing it in StartDetectClientSideShutdown is an assumption HttpServerConneciton can only make because the EventsHandler is the only place using it at the moment.

Exactly what #9990 did before #10516. 🤷‍♂️

So personally, what I'd actually prefer, would be to either hand it to the message, and add a method to manually stop the CpuBoundWork, which currently only the EventsHandler would call.

Well, also only the EventsHandler calls StartStreaming with true.

…nect is true so that /v1/events doesn't have to use IoBoundWorkSlot. IoBoundWorkSlot#~IoBoundWorkSlot() will wait for a free semaphore slot which will be almost immediately released by CpuBoundWork#~CpuBoundWork(). Just releasing the already aquired slot in HttpServerConnection#StartStreaming() is more efficient.

This is inefficient and involves possible bad surprises regarding waiting durations on busy nodes. Instead, use AsioConditionVariable#Wait() if there are no free slots. It's notified by others' CpuBoundWork#~CpuBoundWork() once finished.

Al2Klimov requested a review from yhabteab February 7, 2024 10:42

Al2Klimov self-assigned this Feb 7, 2024

cla-bot bot added the cla/signed label Feb 7, 2024

icinga-probot bot added area/api REST API area/distributed Distributed monitoring (master, satellites, clients) core/quality Improve code, libraries, algorithms, inline docs ref/IP labels Feb 7, 2024

Al2Klimov removed their assignment Feb 7, 2024

Al2Klimov marked this pull request as ready for review February 7, 2024 11:24

Al2Klimov force-pushed the re-think-cpuboundwork-implementation-9988 branch from c11989f to 8d24525 Compare February 9, 2024 13:03

julianbrost reviewed Feb 9, 2024

View reviewed changes

lib/base/io-engine.cpp Outdated Show resolved Hide resolved

Al2Klimov force-pushed the re-think-cpuboundwork-implementation-9988 branch from 8d24525 to bf74280 Compare February 20, 2024 12:21

julianbrost requested changes Feb 20, 2024

View reviewed changes

lib/base/io-engine.cpp Outdated Show resolved Hide resolved

Al2Klimov force-pushed the re-think-cpuboundwork-implementation-9988 branch from bf74280 to 9062934 Compare February 21, 2024 11:13

yhabteab reviewed Feb 22, 2024

View reviewed changes

lib/base/io-engine.cpp Outdated Show resolved Hide resolved

lib/base/io-engine.cpp Outdated Show resolved Hide resolved

lib/base/io-engine.cpp Outdated Show resolved Hide resolved

lib/base/io-engine.cpp Outdated Show resolved Hide resolved

Al2Klimov force-pushed the re-think-cpuboundwork-implementation-9988 branch from 9062934 to a00262f Compare February 22, 2024 09:58

Al2Klimov requested a review from julianbrost February 22, 2024 09:58

Al2Klimov added this to the 2.15.0 milestone Apr 5, 2024

Al2Klimov mentioned this pull request Jun 20, 2024

Two separate CpuBoundWork semaphores – HTTP | JSONRPC #10046

Open

yhabteab mentioned this pull request Sep 27, 2024

Log HTTP/RPC message processing stats #10141

Merged

Al2Klimov force-pushed the re-think-cpuboundwork-implementation-9988 branch from a00262f to 26ef66e Compare September 27, 2024 09:51

Al2Klimov mentioned this pull request Nov 27, 2024

Add a dedicated method for disconnecting TLS connections #10005

Merged

Al2Klimov commented Nov 28, 2024

View reviewed changes

Al2Klimov requested review from yhabteab and oxzi November 29, 2024 09:19

Al2Klimov commented May 14, 2025

View reviewed changes

lib/base/io-engine.hpp Show resolved Hide resolved

yhabteab previously approved these changes May 15, 2025

View reviewed changes

Al2Klimov mentioned this pull request May 20, 2025

Checkable#ProcessCheckResult(): discard🗑️ CR or delay its producers shutdown #10397

Merged

julianbrost removed this from the 2.15.0 milestone Jun 10, 2025

Al2Klimov dismissed yhabteab’s stale review via e4b73f3 July 8, 2025 15:55

Al2Klimov force-pushed the re-think-cpuboundwork-implementation-9988 branch from 1cccf84 to e4b73f3 Compare July 8, 2025 15:55

Al2Klimov requested a review from yhabteab July 22, 2025 09:06

oxzi removed their request for review July 22, 2025 09:32

jschmidt-icinga reviewed Aug 25, 2025

View reviewed changes

lib/base/io-engine.cpp Outdated Show resolved Hide resolved

lib/base/io-engine.hpp Outdated Show resolved Hide resolved

lib/base/io-engine.cpp Show resolved Hide resolved

lib/base/io-engine.cpp Show resolved Hide resolved

Al2Klimov force-pushed the re-think-cpuboundwork-implementation-9988 branch from e4b73f3 to ea642eb Compare August 25, 2025 16:03

Al2Klimov requested a review from jschmidt-icinga August 27, 2025 10:28

Al2Klimov force-pushed the re-think-cpuboundwork-implementation-9988 branch from ea642eb to 725e868 Compare August 29, 2025 13:04

jschmidt-icinga reviewed Sep 1, 2025

View reviewed changes

Al2Klimov added 9 commits September 15, 2025 10:41

Remove unused IoBoundWorkSlot

fa143c6

CpuBoundWork#~CpuBoundWork(): just call #Done() to deduplicate code

da79b0d

[Refactor] CpuBoundWork#CpuBoundWork(): require an io_context::strand

5324f1c

Introduce AsioConditionVariable

7f04881

Make IoEngine#m_CpuBoundSemaphore unsigned

e07459c

Remove unused IoEngine::YieldCurrentCoroutine()

756999d

Remove unused IoEngine#m_AlreadyExpiredTimer

1cd9347

Al2Klimov force-pushed the re-think-cpuboundwork-implementation-9988 branch from 725e868 to 1cd9347 Compare September 15, 2025 08:47

Al2Klimov requested a review from jschmidt-icinga September 15, 2025 08:48

Al2Klimov mentioned this pull request Sep 30, 2025

Replace CpuBoundWork #10572

Draft

CpuBoundWork#CpuBoundWork(): don't spin on atomic int to acquire slot #9990

Are you sure you want to change the base?

CpuBoundWork#CpuBoundWork(): don't spin on atomic int to acquire slot #9990

Uh oh!

Conversation

Al2Klimov commented Feb 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Al2Klimov commented Feb 7, 2024

Low load test

Uh oh!

Al2Klimov commented Feb 7, 2024

High load test

Logs

Uh oh!

Uh oh!

julianbrost commented Feb 19, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Al2Klimov commented Nov 6, 2024

Uh oh!

Al2Klimov Nov 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yhabteab Nov 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yhabteab left a comment

Choose a reason for hiding this comment

Uh oh!

Al2Klimov commented Jul 22, 2025

Uh oh!

Al2Klimov commented Aug 4, 2025

Uh oh!

julianbrost commented Aug 4, 2025

Uh oh!

Al2Klimov commented Aug 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Al2Klimov commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Al2Klimov commented Feb 7, 2024 •

edited

Loading

Al2Klimov Nov 28, 2024 •

edited

Loading

yhabteab Nov 29, 2024 •

edited

Loading

Al2Klimov commented Aug 29, 2025 •

edited

Loading