Feat/singleton preloading #3
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The sub process "./flow job:execute" now starts with its "queue" argument but without its "messageCacheIdentifier" argument in an interactive process object. The messaeg identifier gets passed to this process when available, which might be at a later time. This allows the sub process to boot its fraemwork while still waiting for messaegs to come in. While this will increae memory consumption by a bit, it will decrease the wait time for executing a message.
The worker object starst the JobCommandController without having a message to execute and idles while waiting for th emessage. This change loads all available singletons during the idle phase so that they are present once a message is ready for execution.
The previous setting only reacted to data sent from the child process to the parent process via STDOUT and STDERR but had no timeout based callback, so during long-running child processees with no data interaction, the parent could not e.g. keep its database alive.
paxuclus
reviewed
Oct 10, 2025
The react/child-process Process implementation can work on an event loop, which also supports periodic timers.
This allows for a poolSize of 0, meaning there is no idling worker.
stephanschuler
force-pushed
the
feat/singleton-preloading
branch
from
f8b60d6 to
3ecd4a7
Compare
Not loading dependencies with expiring connections is hard, so default to not loading any singletons at all in this package.
When a child process is assigned a task, the pool is restocked to this amount. So this is the number of idle processes at any time. The total number of processes can be higher if there are busy ones.
stephanschuler
force-pushed
the
feat/singleton-preloading
branch
from
3ecd4a7 to
1f90b55
Compare
Our FastRabbit loop will retart every 6 hours. This results in child processes loosing their STDIN, so they will throw an exceptions. Since this exception is expected, we don't need to log it.
stephanschuler
force-pushed
the
feat/singleton-preloading
branch
from
bcdc480 to
64a09db
Compare
In a previous commit, this package used a pool of React/Process objects for executing jobs, which allowed the children to preload singletons, even if the parent has no Job payload to pass on, yet. This chnage moves the Pool object with its "prefork strategy" to a separate library package, which allows for som code to be reused.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.