Skip to content

Conversation

lucab
Copy link
Contributor

@lucab lucab commented Jun 13, 2025

This introduces some new (internal-only and lazy) static entities in order to shard meta-schemas according to the draft that they belong to. Sharded meta-schemas allow to reduce the amount of registry resources that need to be constructed and statically cached.

@Stranger6667 Stranger6667 mentioned this pull request Jun 13, 2025
Copy link

codecov bot commented Jun 13, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.79%. Comparing base (44520ad) to head (db184aa).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #763      +/-   ##
==========================================
+ Coverage   85.71%   85.79%   +0.08%     
==========================================
  Files          95       95              
  Lines       15531    15620      +89     
==========================================
+ Hits        13312    13401      +89     
  Misses       2219     2219              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Stranger6667
Copy link
Owner

Amazing! Thank you so much! Do you have some numbers on the memory usage change?

Copy link

codspeed-hq bot commented Jun 13, 2025

CodSpeed Performance Report

Merging #763 will not alter performance

Comparing lucab:lucab/meta-shard-draft-schemas (db184aa) with master (44520ad)

Summary

✅ 47 untouched benchmarks

This introduces some new (internal-only and lazy) static entities in
order to shard meta-schemas according to the draft that they belong to.
Sharded meta-schemas allow to reduce the amount of registry resources
that need to be constructed and statically cached.
@lucab lucab force-pushed the lucab/meta-shard-draft-schemas branch from 11e4ee0 to db184aa Compare June 13, 2025 07:29
@lucab
Copy link
Contributor Author

lucab commented Jun 13, 2025

I have some numbers from a private application, but unfortunately I can't share the code nor the JSON schema/docs.

As a single data-point, performing validation of a single doc and a single schema under draft-202012, this seems to be saving ~300kB of lazy-cached memory. I'm checking via https://github.com/KDE/heaptrack.

I'm honestly a bit disappointed as I was hoping for more, but at this point I think most of the memory usage is coming from the lazy/recursive $ref handling.

@Stranger6667
Copy link
Owner

Aha, I see.

300kb is not nothing, and the change seems to be relatively straightforward :)

Yesterday, I got at least some stuff working in #749, and I believe that approach should decrease the memory usage significantly, as it allows using an id to refer to a subschema, instead of building that lazy validator, which currently is implemented for $ref

@lucab
Copy link
Contributor Author

lucab commented Jun 13, 2025

Thanks, I'll try to have a look at #749 if I manage to (I'm currently traveling).
If there are no major conflicts and this PR seems overall fine, I think you can land this first then rebase your ongoing work.

@Stranger6667 Stranger6667 merged commit 958b430 into Stranger6667:master Jun 13, 2025
39 checks passed
@Stranger6667
Copy link
Owner

Thank you!

@lucab lucab deleted the lucab/meta-shard-draft-schemas branch June 13, 2025 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants