system store - endpoints load from core instead of db #9150
Conversation
## Walkthrough
A new dual-source loading mechanism was introduced for the system store, allowing data to be loaded either from the database or via core RPC. Supporting API methods and exports were added, including a new `get_system_store` API endpoint and related server-side logic. The system store's initialization and loading logic were updated accordingly.
## Changes
| File(s) | Change Summary |
|-------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------|
| src/api/system_api.js | Added `get_system_store` API method with GET method and no system-level authentication required. |
| src/endpoint/endpoint.js | Modified call to `system_store.get_instance()` to include `{source: system_store.SOURCE.CORE}` as an argument for md_server. |
| src/server/system_services/system_server.js | Added and exported async `get_system_store` function returning recent DB data as a buffer keyed by `RPC_BUFFERS`. |
| src/server/system_services/system_store.js | Enhanced `SystemStore` to support dual-source loading (DB/CORE), added `SOURCE` constant, new methods, and conditional logic. |
| src/util/postgres_client.js | Added export for `decode_json` function. |
## Sequence Diagram(s)
```mermaid
sequenceDiagram
participant Client
participant API as system_api.js
participant Server as system_server.js
participant Store as system_store.js
participant DB
Client->>API: GET /get_system_store
API->>Server: get_system_store()
Server->>Store: recent_db_data()
Store->>Store: Acquire semaphore lock
Store->>DB: (If source is DB) Fetch latest data
Store-->>Server: Return recent DB data
Server-->>API: Return data buffer
API-->>Client: Respond with system store dataEstimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Suggested labels
Suggested reviewers
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
src/api/system_api.js(1 hunks)src/endpoint/endpoint.js(1 hunks)src/server/system_services/system_server.js(3 hunks)src/server/system_services/system_store.js(8 hunks)src/util/postgres_client.js(1 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
src/server/system_services/system_server.js (1)
src/server/system_services/system_store.js (2)
require(41-41)require(43-43)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: warp-tests / Warp Tests
- GitHub Check: ceph-s3-tests / Ceph S3 Tests
- GitHub Check: run-unit-tests / Unit Tests
🔇 Additional comments (2)
src/util/postgres_client.js (1)
1954-1954: LGTM! Clean export addition for system store RPC support.This export enables the
decode_jsonfunction to be used by the system store module for processing JSON data received via RPC from the core system, which aligns perfectly with the PR objectives to reduce database load by centralizing data access.src/server/system_services/system_store.js (1)
545-565: Review _read_new_data_from_core: adjust error handling and confirm no circular RPCWe’ve confirmed that passing
this.datainto_read_new_data_from_coreis correct, so no change is needed there. However, the method currently has no guard against RPC failures, and it’s worth double-checking that theget_system_storehandler doesn’t itself call back intosystem_store.load()(which would create a circular dependency).Please address the following:
• Remove the parameter-fix suggestion at line 438—
this.datais already the newSystemStoreDatainstance.
• Add a try/catch around the RPC call to surface failures clearly:async _read_new_data_from_core(target) { dbg.log3("_read_new_data_from_core begins"); + try { const res = await server_rpc.client.system.get_system_store(); const ss = JSON.parse(res[RPC_BUFFERS].data.toString()); dbg.log3("_read_new_data_from_core new system store", ss); for (const key of Object.keys(ss)) { … } + } catch (err) { + dbg.error("Failed to read system store from core:", err); + throw err; + } }• Manually verify that the
get_system_storeRPC handler (in your core/service code) doesn’t invokesystem_store.load()or_read_new_data_from_core, to avoid circular calls.
| get_system_store: { | ||
| method: 'GET', | ||
| auth: { | ||
| system: false | ||
| } | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add reply schema to define the expected response format.
The new get_system_store API method lacks a reply schema, which makes it unclear what data structure this endpoint returns. This could lead to integration issues and runtime errors.
Consider adding a reply schema similar to other methods in this file:
get_system_store: {
method: 'GET',
+ reply: {
+ type: 'object',
+ // Define the structure of system store data
+ properties: {
+ // Add expected properties here
+ }
+ },
auth: {
system: false
}
}Alternatively, if the response is a binary buffer or serialized data, specify the appropriate type (e.g., type: 'string' for base64-encoded data).
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| get_system_store: { | |
| method: 'GET', | |
| auth: { | |
| system: false | |
| } | |
| } | |
| get_system_store: { | |
| method: 'GET', | |
| reply: { | |
| type: 'object', | |
| // Define the structure of system store data | |
| properties: { | |
| // Add expected properties here | |
| } | |
| }, | |
| auth: { | |
| system: false | |
| } | |
| } |
🤖 Prompt for AI Agents
In src/api/system_api.js around lines 464 to 469, the get_system_store method is
missing a reply schema that defines the expected response format. To fix this,
add a reply schema property to the method specifying the structure of the
returned data, similar to other API methods in this file. If the response is a
JSON object, define its properties and types; if it returns binary or serialized
data, specify the appropriate type such as 'string' for base64-encoded content.
| // this point (and should not kill the process); | ||
| system_store.get_instance().load(); | ||
| system_store.get_instance({source: system_store.SOURCE.CORE}).load(); |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Consider handling the promise rejection to avoid unhandled promise warnings.
The comment states that errors are not relevant, but the .load() promise might still reject and cause unhandled promise warnings. Consider adding a .catch() handler.
- system_store.get_instance({source: system_store.SOURCE.CORE}).load();
+ system_store.get_instance({source: system_store.SOURCE.CORE}).load().catch(err => {
+ dbg.warn('Failed to load system store from core:', err);
+ });Also, verify that this change applies to all endpoint scenarios, not just when LOCAL_MD_SERVER is true.
🏁 Script executed:
#!/bin/bash
# Description: Check if there are other places where endpoints might need to use CORE source
# Look for other system_store.get_instance() calls in endpoint-related files
rg -A 5 'system_store\.get_instance\(' --glob 'src/endpoint/**/*.js'Length of output: 902
Handle unhandled promise for system_store.load()
The .load() call returns a promise and may reject, triggering unhandled‐promise warnings. Add a .catch() handler to swallow or log errors as intended.
- File:
src/endpoint/endpoint.js
Lines ~173–174 in theLOCAL_MD_SERVERblock
Suggested diff:
- system_store.get_instance({source: system_store.SOURCE.CORE}).load();
+ system_store.get_instance({source: system_store.SOURCE.CORE})
+ .load()
+ .catch(err => {
+ dbg.warn('Failed to load system store from core:', err);
+ });I ran a search for other system_store.get_instance( calls under src/endpoint/ and found only this one use of .load() with CORE—no additional endpoints require the same change.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // this point (and should not kill the process); | |
| system_store.get_instance().load(); | |
| system_store.get_instance({source: system_store.SOURCE.CORE}).load(); | |
| // this point (and should not kill the process); | |
| system_store.get_instance({source: system_store.SOURCE.CORE}) | |
| .load() | |
| .catch(err => { | |
| dbg.warn('Failed to load system store from core:', err); | |
| }); |
🤖 Prompt for AI Agents
In src/endpoint/endpoint.js around lines 173 to 174, the call to
system_store.get_instance(...).load() returns a promise that may reject, causing
unhandled promise warnings. Fix this by adding a .catch() handler to the load()
call to handle any rejection, either by logging the error or swallowing it as
appropriate to prevent unhandled promise rejections.
| async function get_system_store() { | ||
| return { | ||
| [RPC_BUFFERS]: {data: Buffer.from(JSON.stringify(await system_store.recent_db_data()))}, | ||
| }; | ||
| } |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Add error handling and consider security implications of exposing system store data.
This function exposes the entire system store data without any visible authorization checks or data filtering. Additionally, it lacks error handling.
Consider the following improvements:
- Add error handling:
async function get_system_store() {
+ try {
return {
[RPC_BUFFERS]: {data: Buffer.from(JSON.stringify(await system_store.recent_db_data()))},
};
+ } catch (err) {
+ dbg.error('Failed to get system store data:', err);
+ throw new RpcError('INTERNAL', 'Failed to retrieve system store data');
+ }
}-
Verify authorization requirements:
-
Consider implementing data filtering to avoid exposing sensitive information like credentials or keys.
-
For large datasets, consider implementing pagination or streaming to avoid memory issues with JSON serialization.
🏁 Script executed:
#!/bin/bash
# Description: Check how this RPC method is exposed and what authorization it requires
# Look for the API definition and any auth requirements
# Search for get_system_store in API definitions
rg -A 10 'get_system_store' src/api/
# Check if there are any auth decorators or middleware
ast-grep --pattern 'get_system_store: {
$$$
}'Length of output: 509
Enforce authorization and add error handling in get_system_store
The get_system_store RPC is currently exposed via HTTP GET with auth: { system: false } and no error handling or data filtering. This allows unauthenticated clients to fetch the entire store and may crash the server on large payloads.
Please address the following:
-
Require system-level auth
Insrc/api/system_api.js, changeget_system_store: { method: 'GET', auth: { system: false }, }
to
auth: { system: true }
or integrate your authorization middleware so only permitted clients can call this RPC.
-
Add error handling
Wrap the data fetch/serialize intry/catchand return a proper RPC error:async function get_system_store() {
- try {
return {
[RPC_BUFFERS]: {
data: Buffer.from(JSON.stringify(await system_store.recent_db_data())),
},
}; - } catch (err) {
-
dbg.error('get_system_store failed:', err); -
throw new RpcError('INTERNAL', 'Unable to retrieve system store data'); - }
}
- **Filter sensitive data**
Only include non-sensitive fields in the payload (e.g. omit credentials, tokens, or PII).
- **Handle large datasets**
Consider implementing pagination or streaming (e.g. chunked responses) to avoid high memory/CPU usage.
<details>
<summary>🤖 Prompt for AI Agents</summary>
In src/server/system_services/system_server.js around lines 306 to 310 and in
src/api/system_api.js where get_system_store is defined, update the auth setting
to require system-level authorization by setting auth: { system: true }. In the
get_system_store function, wrap the data retrieval and serialization in a
try/catch block to handle errors gracefully and return an appropriate RPC error
on failure. Modify the data returned to exclude sensitive information such as
credentials, tokens, or PII by filtering the fields before serialization. To
handle large datasets, implement pagination or chunked responses to limit memory
and CPU usage during data transmission.
</details>
<!-- fingerprinting:phantom:medusa:lion -->
<!-- This is an auto-generated comment by CodeRabbit -->
| this.source = (process.env.HOSTNAME && process.env.HOSTNAME.indexOf("endpoint") === -1) ? SOURCE.DB : SOURCE.CORE; | ||
| dbg.log0("system store source is", this.source); |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Improve the source determination logic for better clarity and maintainability.
The current logic for determining the source relies on checking if HOSTNAME contains "endpoint", which could be fragile and unclear.
Consider using a more explicit approach:
- this.source = (process.env.HOSTNAME && process.env.HOSTNAME.indexOf("endpoint") === -1) ? SOURCE.DB : SOURCE.CORE;
+ // Explicitly determine source based on process type
+ // Endpoints should load from CORE to reduce DB load, others use DB directly
+ const isEndpointProcess = process.env.HOSTNAME && process.env.HOSTNAME.includes("endpoint");
+ this.source = options.source || (isEndpointProcess ? SOURCE.CORE : SOURCE.DB);This also respects the options.source parameter that's already being passed in some cases.
🏁 Script executed:
#!/bin/bash
# Description: Verify how HOSTNAME is set for different process types
# Check for HOSTNAME usage patterns
# Search for HOSTNAME environment variable usage
rg 'process\.env\.HOSTNAME' --type js
# Look for how different services set their process names/titles
rg 'process\.title\s*=' --type jsLength of output: 325
🏁 Script executed:
#!/bin/bash
# Search for options.source usage
rg 'options\.source' -n src
# Locate the SystemStore class and its constructor around the source determination logic
rg -n 'class SystemStore' -A20 -B10 src/server/system_services/system_store.jsLength of output: 2118
Clarify and Simplify Source Determination
Extract a clear isEndpoint boolean and use it (or process.title) rather than an inline ternary on HOSTNAME. Optionally add support for a future options.source override.
• File: src/server/system_services/system_store.js (around lines 360–361)
• Replace this block:
- this.source = (process.env.HOSTNAME && process.env.HOSTNAME.indexOf("endpoint") === -1) ? SOURCE.DB : SOURCE.CORE;
+ // Determine if this is the endpoint process by hostname
+ const hostName = process.env.HOSTNAME || '';
+ const isEndpoint = hostName.includes('endpoint');
+ this.source = isEndpoint ? SOURCE.CORE : SOURCE.DB;• (Optional) If you want to honor a passed–in override:
constructor(options = {}) {
// …
+ if (options.source) {
+ this.source = options.source;
+ } else {
+ const hostName = process.env.HOSTNAME || '';
+ const isEndpoint = hostName.includes('endpoint');
+ this.source = isEndpoint ? SOURCE.CORE : SOURCE.DB;
+ }• Alternatively, detect via process.title (set in src/endpoint/endpoint.js):
const isEndpoint = process.title === config.ENDPOINT_PROCESS_TITLE;
this.source = isEndpoint ? SOURCE.CORE : SOURCE.DB;This makes the intent explicit and easier to maintain.
🤖 Prompt for AI Agents
In src/server/system_services/system_store.js around lines 360 to 361, replace
the inline ternary that sets this.source based on process.env.HOSTNAME with a
clearer approach: first define a boolean isEndpoint by checking if the hostname
includes "endpoint" or by comparing process.title to a known endpoint process
title, then set this.source to SOURCE.CORE if isEndpoint is true or SOURCE.DB
otherwise. Optionally, add support for an override via options.source if
applicable. This refactor clarifies intent and improves maintainability.
| //return the latest copy of in-memory data | ||
| async recent_db_data() { | ||
| return this._load_serial.surround(async () => this.old_db_data); | ||
| } |
There was a problem hiding this comment.
Handle the case when recent_db_data is called on a CORE source instance.
This method returns old_db_data which is only populated when the source is DB. For CORE source instances, this will return undefined.
Consider this safer implementation:
//return the latest copy of in-memory data
async recent_db_data() {
- return this._load_serial.surround(async () => this.old_db_data);
+ return this._load_serial.surround(async () => {
+ if (this.source === SOURCE.CORE) {
+ throw new RpcError('BAD_REQUEST', 'recent_db_data is not available for CORE source');
+ }
+ return this.old_db_data || {};
+ });
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| //return the latest copy of in-memory data | |
| async recent_db_data() { | |
| return this._load_serial.surround(async () => this.old_db_data); | |
| } | |
| //return the latest copy of in-memory data | |
| async recent_db_data() { | |
| return this._load_serial.surround(async () => { | |
| if (this.source === SOURCE.CORE) { | |
| throw new RpcError( | |
| 'BAD_REQUEST', | |
| 'recent_db_data is not available for CORE source' | |
| ); | |
| } | |
| return this.old_db_data || {}; | |
| }); | |
| } |
🤖 Prompt for AI Agents
In src/server/system_services/system_store.js around lines 475 to 478, the
recent_db_data method returns old_db_data which is undefined for CORE source
instances. Modify the method to check if the instance is a CORE source and
handle that case safely, for example by returning null or an empty object
instead of old_db_data, to avoid returning undefined.
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (2)
src/util/postgres_client.js (1)
1954-1954: Exportingdecode_jsonwidens the public API – consider symmetry & documentation
This is the first time an internal helper is exposed from this module. Down-stream code may now rely on the exact traversal semantics (wrapper unwrapping, recursive decode, etc.), making future refactors harder.
encode_jsonis still internal. External callers that usedecode_jsonalmost always need the reverse operation. Exporting only one direction invites accidental misuse or duplicated logic elsewhere.
decode_jsonsilently unwrapsSensitiveString, which leaks plain-text secrets to any caller. If that’s intentional, call-sites should at least opt-in explicitly or a comment should warn about it.Action:
exports.PostgresClient = PostgresClient; exports.instance = PostgresClient.instance; +// NOTE: exporting both helpers keeps the API symmetrical and explicit. +exports.encode_json = encode_json; exports.decode_json = decode_json;…and add JSDoc explaining the security implication.
src/server/system_services/system_server.js (1)
306-310: Consider performance implications for large system storesFor systems with many buckets, accounts, or other entities, the system store data could be quite large. Serializing and buffering the entire dataset might impact memory usage and RPC transfer times.
Consider implementing:
- Data size limits or pagination
- Compression before sending over RPC
- Monitoring the size of data being transferred
- Caching mechanism to avoid repeated serialization
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
src/api/system_api.js(1 hunks)src/endpoint/endpoint.js(1 hunks)src/server/system_services/system_server.js(3 hunks)src/server/system_services/system_store.js(8 hunks)src/util/postgres_client.js(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: ceph-s3-tests / Ceph S3 Tests
🔇 Additional comments (2)
src/endpoint/endpoint.js (1)
174-174: Approved –SOURCE.COREis properly definedVerified in
src/server/system_services/system_store.jsthat theSOURCEenum includesCORE: 'CORE',and is exported (
exports.SOURCE = SOURCE). No further action needed.src/server/system_services/system_store.js (1)
360-361: Improve source selection logic robustnessThe current logic for determining the source has potential issues:
- It assumes endpoints have "endpoint" in their hostname
- The logic is inverted - it should check if hostname contains "endpoint" to set source to CORE
The condition should be:
-this.source = (process.env.HOSTNAME && process.env.HOSTNAME.indexOf("endpoint") === -1) ? SOURCE.DB : SOURCE.CORE; +this.source = (process.env.HOSTNAME && process.env.HOSTNAME.indexOf("endpoint") !== -1) ? SOURCE.CORE : SOURCE.DB;Additionally, consider making this more configurable:
+const source_override = process.env.SYSTEM_STORE_SOURCE; +this.source = source_override || ((process.env.HOSTNAME && process.env.HOSTNAME.indexOf("endpoint") !== -1) ? SOURCE.CORE : SOURCE.DB);Likely an incorrect or invalid review comment.
| get_system_store: { | ||
| method: 'GET', | ||
| auth: { | ||
| system: false | ||
| } | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add reply schema and documentation for the new API method.
The new get_system_store API method is missing essential documentation and schema definition:
- Missing reply schema: All other methods in this API define a proper reply schema for validation and documentation purposes.
- Missing documentation: The
docfield should describe the purpose of this endpoint. - Verify authentication requirements: Confirm that
system: falseis the correct authentication level for this endpoint.
get_system_store: {
+ doc: 'Get system store data from core',
method: 'GET',
+ reply: {
+ type: 'object',
+ // Define the expected structure of system store data
+ additionalProperties: true
+ },
auth: {
system: false
}
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| get_system_store: { | |
| method: 'GET', | |
| auth: { | |
| system: false | |
| } | |
| } | |
| get_system_store: { | |
| doc: 'Get system store data from core', | |
| method: 'GET', | |
| reply: { | |
| type: 'object', | |
| // Define the expected structure of system store data | |
| additionalProperties: true | |
| }, | |
| auth: { | |
| system: false | |
| } | |
| } |
🤖 Prompt for AI Agents
In src/api/system_api.js around lines 464 to 469, the get_system_store API
method lacks a reply schema and documentation. Add a reply schema that defines
the expected response structure for validation and documentation. Include a doc
field describing the endpoint's purpose clearly. Also, review and confirm if the
auth setting system: false is appropriate for this method's security
requirements, adjusting if necessary.
| async function get_system_store() { | ||
| return { | ||
| [RPC_BUFFERS]: {data: Buffer.from(JSON.stringify(await system_store.recent_db_data()))}, | ||
| }; | ||
| } |
There was a problem hiding this comment.
Add error handling for potential failures
The function should handle potential errors from recent_db_data() or JSON.stringify() operations to prevent unhandled exceptions.
Consider wrapping the operations in a try-catch block:
async function get_system_store() {
+ try {
return {
[RPC_BUFFERS]: {data: Buffer.from(JSON.stringify(await system_store.recent_db_data()))},
};
+ } catch (err) {
+ dbg.error('get_system_store: Failed to get system store data', err);
+ throw new RpcError('INTERNAL_ERROR', 'Failed to retrieve system store data');
+ }
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| async function get_system_store() { | |
| return { | |
| [RPC_BUFFERS]: {data: Buffer.from(JSON.stringify(await system_store.recent_db_data()))}, | |
| }; | |
| } | |
| async function get_system_store() { | |
| try { | |
| return { | |
| [RPC_BUFFERS]: {data: Buffer.from(JSON.stringify(await system_store.recent_db_data()))}, | |
| }; | |
| } catch (err) { | |
| dbg.error('get_system_store: Failed to get system store data', err); | |
| throw new RpcError('INTERNAL_ERROR', 'Failed to retrieve system store data'); | |
| } | |
| } |
🤖 Prompt for AI Agents
In src/server/system_services/system_server.js around lines 306 to 310, the
async function get_system_store does not handle errors that may occur during the
call to recent_db_data() or the JSON.stringify operation. To fix this, wrap the
entire operation inside a try-catch block, returning a meaningful error or
fallback value in the catch block to prevent unhandled exceptions and ensure the
function handles failures gracefully.
| //return the latest copy of in-memory data | ||
| async recent_db_data() { | ||
| return this._load_serial.surround(async () => this.old_db_data); | ||
| } |
There was a problem hiding this comment.
Handle undefined old_db_data when source is CORE
When the source is CORE, old_db_data will be undefined, which could cause issues in get_system_store().
//return the latest copy of in-memory data
async recent_db_data() {
- return this._load_serial.surround(async () => this.old_db_data);
+ return this._load_serial.surround(async () => {
+ if (this.source !== SOURCE.DB) {
+ throw new RpcError('BAD_REQUEST', 'recent_db_data is only available when source is DB');
+ }
+ return this.old_db_data || {};
+ });
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| //return the latest copy of in-memory data | |
| async recent_db_data() { | |
| return this._load_serial.surround(async () => this.old_db_data); | |
| } | |
| //return the latest copy of in-memory data | |
| async recent_db_data() { | |
| return this._load_serial.surround(async () => { | |
| if (this.source !== SOURCE.DB) { | |
| throw new RpcError('BAD_REQUEST', 'recent_db_data is only available when source is DB'); | |
| } | |
| return this.old_db_data || {}; | |
| }); | |
| } |
🤖 Prompt for AI Agents
In src/server/system_services/system_store.js around lines 475 to 478, the
method recent_db_data returns old_db_data which can be undefined when the source
is CORE, potentially causing issues in get_system_store(). Modify recent_db_data
to check if old_db_data is undefined and handle that case appropriately, such as
returning a default value or throwing a clear error, to prevent downstream
errors.
| async _read_new_data_from_core(target) { | ||
| dbg.log3("_read_new_data_from_core begins"); | ||
| const res = await server_rpc.client.system.get_system_store(); | ||
| const ss = JSON.parse(res[RPC_BUFFERS].data.toString()); | ||
| dbg.log3("_read_new_data_from_core new system store", ss); | ||
| for (const key of Object.keys(ss)) { | ||
| const collection = COLLECTIONS_BY_NAME[key]; | ||
| if (collection) { | ||
| target[key] = []; | ||
| _.each(ss[key], item => { | ||
| //these two lines will transform string values into appropriately typed objects | ||
| //(SensitiveString, ObjectId) according to schema | ||
| const after = decode_json(collection.schema, item); | ||
| db_client.instance().validate(key, after); | ||
| target[key].push(after); | ||
| }); | ||
| } else { | ||
| target[key] = ss[key]; | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
💡 Verification agent
❓ Verification inconclusive
Monitor RPC data transfer and implement operational safeguards
Since endpoints will now fetch system store data via RPC from core, consider implementing:
- Metrics to monitor the frequency and size of these RPC calls
- Rate limiting to prevent endpoints from overwhelming the core
- Circuit breaker pattern for RPC failures
- Alerts for unusually large data transfers
Let me verify if there are any existing rate limiting or monitoring mechanisms:
🏁 Script executed:
#!/bin/bash
# Search for rate limiting or monitoring in RPC client code
rg -A 5 "rate.*limit|throttle|circuit.*breaker" --glob "src/rpc/**/*.js"
# Check if there are any metrics collection for RPC calls
rg -A 5 "metrics|monitor|stats.*rpc" --glob "src/rpc/**/*.js"Length of output: 138
Implement RPC Monitoring, Rate Limiting, and Resilience Patterns
No existing rate-limit, metrics, or circuit-breaker logic was found around your RPC client (verification via rg returned no matches), so let’s add operational safeguards:
• In src/server/system_services/system_store.js (around the _read_new_data_from_core call at lines 545–565):
– Instrument a counter and histogram (e.g. Prometheus) for calls to server_rpc.client.system.get_system_store() (call count, payload size, latency).
– Emit alerts if payload size or latency exceeds thresholds.
• In your RPC client layer (e.g. src/rpc/client/*.js):
– Implement rate limiting or token bucket to cap calls per second.
– Wrap the get_system_store invocation in a circuit-breaker (e.g. with a library like opossum) to fail fast on repeated errors.
• Add configuration knobs for thresholds and integrate with your observability stack (alerts, dashboards).
These changes will help you detect abnormal patterns, protect the core from overload, and degrade gracefully on failures.
🤖 Prompt for AI Agents
In src/server/system_services/system_store.js around lines 545 to 565, add
Prometheus metrics instrumentation for the call to
server_rpc.client.system.get_system_store(), including a counter for call count,
a histogram for payload size and latency, and emit alerts if thresholds are
exceeded. Additionally, in the RPC client layer (e.g., src/rpc/client/*.js),
implement rate limiting (such as a token bucket) to cap calls per second and
wrap the get_system_store call in a circuit breaker (using a library like
opossum) to fail fast on repeated errors. Add configuration options for these
thresholds and integrate the metrics and alerts with the observability stack for
monitoring and graceful degradation.
Add comprehensive error handling and improve validation
The method needs error handling for RPC calls and JSON parsing. Also, validation should happen before data transformation.
async _read_new_data_from_core(target) {
dbg.log3("_read_new_data_from_core begins");
+ try {
const res = await server_rpc.client.system.get_system_store();
+ if (!res || !res[RPC_BUFFERS] || !res[RPC_BUFFERS].data) {
+ throw new Error('Invalid response from core system store');
+ }
const ss = JSON.parse(res[RPC_BUFFERS].data.toString());
dbg.log3("_read_new_data_from_core new system store", ss);
for (const key of Object.keys(ss)) {
const collection = COLLECTIONS_BY_NAME[key];
if (collection) {
target[key] = [];
_.each(ss[key], item => {
+ // Validate before transformation
+ db_client.instance().validate(key, item, 'warn');
//these two lines will transform string values into appropriately typed objects
//(SensitiveString, ObjectId) according to schema
const after = decode_json(collection.schema, item);
- db_client.instance().validate(key, after);
+ // Validate again after transformation to ensure correctness
+ db_client.instance().validate(key, after, 'error');
target[key].push(after);
});
} else {
target[key] = ss[key];
}
}
+ } catch (err) {
+ dbg.error('_read_new_data_from_core: Failed to read data from core', err);
+ throw new RpcError('INTERNAL_ERROR', `Failed to load system store from core: ${err.message}`);
+ }
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| async _read_new_data_from_core(target) { | |
| dbg.log3("_read_new_data_from_core begins"); | |
| const res = await server_rpc.client.system.get_system_store(); | |
| const ss = JSON.parse(res[RPC_BUFFERS].data.toString()); | |
| dbg.log3("_read_new_data_from_core new system store", ss); | |
| for (const key of Object.keys(ss)) { | |
| const collection = COLLECTIONS_BY_NAME[key]; | |
| if (collection) { | |
| target[key] = []; | |
| _.each(ss[key], item => { | |
| //these two lines will transform string values into appropriately typed objects | |
| //(SensitiveString, ObjectId) according to schema | |
| const after = decode_json(collection.schema, item); | |
| db_client.instance().validate(key, after); | |
| target[key].push(after); | |
| }); | |
| } else { | |
| target[key] = ss[key]; | |
| } | |
| } | |
| } | |
| async _read_new_data_from_core(target) { | |
| dbg.log3("_read_new_data_from_core begins"); | |
| try { | |
| const res = await server_rpc.client.system.get_system_store(); | |
| if (!res || !res[RPC_BUFFERS] || !res[RPC_BUFFERS].data) { | |
| throw new Error('Invalid response from core system store'); | |
| } | |
| const ss = JSON.parse(res[RPC_BUFFERS].data.toString()); | |
| dbg.log3("_read_new_data_from_core new system store", ss); | |
| for (const key of Object.keys(ss)) { | |
| const collection = COLLECTIONS_BY_NAME[key]; | |
| if (collection) { | |
| target[key] = []; | |
| _.each(ss[key], item => { | |
| // Validate before transformation | |
| db_client.instance().validate(key, item, 'warn'); | |
| // these two lines will transform string values into appropriately typed objects | |
| // (SensitiveString, ObjectId) according to schema | |
| const after = decode_json(collection.schema, item); | |
| // Validate again after transformation to ensure correctness | |
| db_client.instance().validate(key, after, 'error'); | |
| target[key].push(after); | |
| }); | |
| } else { | |
| target[key] = ss[key]; | |
| } | |
| } | |
| } catch (err) { | |
| dbg.error('_read_new_data_from_core: Failed to read data from core', err); | |
| throw new RpcError('INTERNAL_ERROR', `Failed to load system store from core: ${err.message}`); | |
| } | |
| } |
🤖 Prompt for AI Agents
In src/server/system_services/system_store.js around lines 545 to 565, add
try-catch blocks to handle errors from the RPC call and JSON parsing to prevent
crashes. Move the validation step to occur before the decode_json transformation
to ensure data integrity early. Log or handle any errors appropriately within
the catch block to maintain robustness.
Signed-off-by: Amit Prinz Setter <[email protected]>
alphaprinz
force-pushed
the
system_store_phase2
branch
from
d880a87 to
766302a
Compare
Describe the Problem
Reduce db load spikes when all endpoints load from db simultaneously by having only core load from db and endpoints load from core instead.
Explain the Changes
Issues: Fixed #xxx / Gap #xxx
Testing Instructions:
Summary by CodeRabbit
New Features
Improvements