From 87ddf0af10c8ab8fb93da0c949feb73c335ac04c Mon Sep 17 00:00:00 2001 From: Noah Stapp Date: Fri, 8 Aug 2025 16:17:45 -0400 Subject: [PATCH 1/8] DRIVERS-2917 - Standardized Performance Testing of ODMs and Integrations --- source/benchmarking/odm-benchmarking.md | 377 ++++++++++++++++++++++++ source/index.md | 1 + 2 files changed, 378 insertions(+) create mode 100644 source/benchmarking/odm-benchmarking.md diff --git a/source/benchmarking/odm-benchmarking.md b/source/benchmarking/odm-benchmarking.md new file mode 100644 index 0000000000..e73c033dfa --- /dev/null +++ b/source/benchmarking/odm-benchmarking.md @@ -0,0 +1,377 @@ +# ODM Performance Benchmarking + +- Status: In progress +- Minimum Server Version: N/A + +## Abstract + +This document describes a standard benchmarking suite for MongoDB ODMs (Object Document Mappers). Much of the structure +is taken from the existing MongoDB driver benchmarking suite for consistency and readability. + +## Overview + +### Name and purpose + +ODM performance will be measured by the MongoDB ODM Performance Benchmark. It will provide both "horizontal" insights +into how individual ODM performance evolves over time and two types of "vertical" insights: the relative performance of +different ODMs, and the relative performance of ODMs and their associated language drivers. + +We expect substantial performance differences between ODMs based on both their language families (e.g. static vs. +dynamic or compiled vs. virtual-machine-based) as well as their inherent design (e.g. web frameworks such as Django vs. +application-agnostic such as Mongoose). However we still expect "vertical" comparison within families of ODMs to expose +outlier behavior that can be optimized away. + +### Task Hierarchy + +The benchmark suite consists of multiple groups of small, independent benchmarks. This allows us to better isolate areas +within ODMs that are faster or slower. + +- Flat models -- reading and writing flat models of various sizes, to explore basic operation efficiency +- Nested models -- reading and writing nested models of various sizes, to explore basic operation efficiency for complex + data +- Joins -- performing operations that involve joins (if supported), to explore $lookup efficiency + +### Measurement + +In addition to latency data, all benchmark tasks will be measured in terms of "megabytes/second" (MB/s) of documents +processed, with higher scores being better. (In this document, "megabyte" refers to the SI decimal unit, i.e. 1,000,000 +bytes.) This makes cross-benchmark comparisons easier. + +To avoid various types of measurement skew, tasks will be measured over numerous iterations. Each iteration will have a +number of operations performed per iteration that depends on the task being benchmarked. The final score for a task will +be the median score of the iterations. A range of percentiles will also be recorded for diagnostic analysis. + +### Data sets + +Data sets will vary by task. In some cases, they will be synthetically generated models inserted repeatedly to construct +an overall corpus of models. In other cases, data sets will be synthetic line-delimited JSON files or mock binary files +to be constructed by the ODM into the appropriate model. + +### Versioning + +The MongoDB ODM Performance Benchmark will have vX.Y versioning. Minor updates and clarifications will increment "Y" and +should have little impact on score comparison. Major changes, such as task modifications, MongoDB version tested +against, or hardware used, will increment "X" to indicate that older version scores are unlikely to be comparable. + +## Benchmark execution phases and measurement + +All benchmark tasks will be conducted via a number of iterations. Each iteration will be timed and will generally +include a large number of individual ODM operations. + +The measurement is broken up this way to better isolate the benchmark from external volatility. If we consider the +problem of benchmarking an operation over many iterations, such as 100,000 model insertions, we want to avoid two +extreme forms of measurement: + +- measuring a single insertion 100,000 times -- in this case, the timing code is likely to be a greater proportion of + executed code, which could routinely evict the insertion code from CPU caches or mislead a JIT optimizer and throw + off results +- measuring 100,000 insertions one time -- in this case, the longer the timer runs, the higher the likelihood that an + external event occurs that affects the time of the run + +Therefore, we choose a middle ground: + +- measuring the same 1000 insertions over 100 iterations -- each timing run includes enough operations that insertion + code dominates timing code; unusual system events are likely to affect only a fraction of the 100 timing + measurements + +With 100 timings of inserting the same 1000 models, we build up a statistical distribution of the operation timing, +allowing a more robust estimate of performance than a single measurement. (In practice, the number of iterations could +exceed 100, but 100 is a reasonable minimum goal.) + +Because a timing distribution is bounded by zero on one side, taking the mean would allow large positive outlier +measurements to skew the result substantially. Therefore, for the benchmark score, we use the median timing measurement, +which is robust in the face of outliers. + +Each benchmark is structured into discrete setup/execute/teardown phases. Phases are as follows, with specific details +given in a subsequent section: + +- setup -- (ONCE PER TASK) something to do once before any benchmarking, e.g. construct a client object, load test data, + insert data into a collection, etc. +- before operation -- (ONCE PER ITERATION) something to do before every task iteration, e.g. drop a collection, or + reload test data (if the test run modifies it), etc. +- do operation -- (ONCE PER ITERATION) smallest amount of code necessary to execute the task; e.g. insert 1000 models + one by one into the database, or retrieve 1000 models of test data from the database, etc. +- after operation -- (ONCE PER ITERATION) something to do after every task iteration (if necessary) +- teardown -- (ONCE PER TASK) something done once after all benchmarking is complete (if necessary); e.g. drop the test + database + +The wall-clock execution time of each "do operation" phase will be recorded. We use wall clock time to model user +experience and as a lowest-common denominator across ODMs. Iteration timing should be done with a high-resolution +monotonic timer (or best language approximation). + +Unless otherwise specified, the number of iterations to measure per task is variable: + +- iterations should loop for at least 1 minute cumulative execution time +- iterations should stop after 100 iterations or 5 minutes cumulative execution time, whichever is shorter + +This balances measurement stability with a timing cap to ensure all tasks can complete in a reasonable time. + +For each task, the 10th, 25th, 50th, 75th, 90th, 95th, 98th and 99th percentiles will be recorded using the following +algorithm: + +- Given a 0-indexed array A of N iteration wall clock times +- Sort the array into ascending order (i.e. shortest time first) +- Let the index i for percentile p in the range [1,100] be defined as: `i = int(N * p / 100) - 1` + +*N.B. This is the [Nearest Rank](https://en.wikipedia.org/wiki/Percentile#The_Nearest_Rank_method) algorithm, chosen for +its utter simplicity given that it needs to be implemented identically across a wide variety of ODMs and languages.* + +The 50th percentile (i.e. the median) will be used for score composition. Other percentiles will be stored for +visualizations and analysis. + +Each task will have defined for it an associated size in megabytes (MB). The benchmarking score for each task will be +the task size in MB divided by the median wall clock time. + +## Benchmark task definitions + +Datasets are available in the `odm-data` directory adjacent to this spec. + +Note: The term "LDJSON" means "line-delimited JSON", which should be understood to mean a collection of UTF-8 encoded +JSON documents (without embedded CR or LF characters), separated by a single LF character. (Some Internet definition of +line-delimited JSON use CRLF delimiters, but this benchmark uses only LF.) + +### Flat models + +Datasets are in the `flat-models` tarball. + +Flat model tests focus on flatly-structured model reads and writes across data sizes. They are designed to give insights +into the efficiency of the ODM's implementation of basic data operations. + +The data will be stored as strict JSON with no extended types. These JSON representations must be converted into +equivalent models as part of the benchmark's setup. + +Flat model benchmark tasks include:s + +- Small model creation +- Small model update +- Small model find by filter +- Small model find foreign key by filter (if joins are supported) +- Large model creation +- Large model update + +#### Small model creation + +Summary: This benchmark tests ODM performance creating a single small model. + +Dataset: The dataset (SMALL_DOC) is contained within `small-document.json` and consists of a sample document stored as +strict JSON with an encoded length of approximately X bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `small-document` source file (X bytes) times +10,000 operations, which equals X,XXX,XXX bytes or X MB. + +| Phase | Description | +| ----------- | -------------------------------------------------------------------------------------- | +| Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. | +| Before task | Drop the collection associated with the SMALL_DOC model. | +| Do task | Save the model to the database in an ODM-appropriate manner. Repeat this 10,000 times. | +| After task | n/a | +| Teardown | n/a. | + +#### Large model update + +Summary: This benchmark tests ODM performance updating fields on a single small model. + +Dataset: The dataset (SMALL_DOC) is contained within `small-document.json` and consists of a sample document stored as +strict JSON with an encoded length of approximately X bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `small-document` source file (X bytes) times +10,000 operations, which equals X,XXX,XXX bytes or X MB. + +| Phase | Description | +| ----------- | ------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. Save 10,000 instances into the database. | +| Before task | n/a. | +| Do task | Update the `FIELD_NAME` field for each instance of the model in an ODM-appropriate manner. | +| After task | Drop the collection associated with the SMALL_DOC model. | +| Teardown | n/a. | + +#### Small model find by filter + +Summary: This benchmark tests ODM performance finding documents using a basic filter. + +Dataset: The dataset (SMALL_DOC) is contained within `small-document.json` and consists of a sample document stored as +strict JSON with an encoded length of approximately X bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `small-document` source file (X bytes) times +10,000 operations, which equals X,XXX,XXX bytes or X MB. + +| Phase | Description | +| ----- | -------------------------------------------------------------------------- | +| Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. | + +``` + Insert 10,000 instances into the database, saving the inserted _id field for each into a list. | +``` + +| Before task | n/a. | | Do task | For each of the 10,000 \_id values, perform a filter operation to search for the +corresponding model document. | | After task | n/a. | | Teardown | Drop the collection associated with the SMALL_DOC +model. | + +#### Small model find foreign key by filter + +Summary: This benchmark tests ODM performance finding documents by foreign keys. This benchmark must only be run by ODMs +that support join ($lookup) operations. + +Dataset: The dataset (SMALL_DOC_FK) is contained within `small-document-foreign-key.json` and consists of two sample +documents, both stored as strict JSON: the main document with an encoded length of approximately X bytes, and the +associated foreign key document with an encoded length of approximately Y bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `small-document-foreign-key.json` source file (X +\+ Y bytes) times 10,000 operations, which equals X,XXX,XXX bytes or X + Y MB. + +| Phase | Description | +| ----- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| Setup | Load the SMALL_DOC_FK dataset into memory as two ODM-appropriate model objects: one for the main model, and one for the foreign key model. | + +``` + Insert 10,000 instances of each into the database, saving the inserted _id field for each foreign key model into a list. | +``` + +| Before task | n/a. | | Do task | For each of the 10,000 foreign key \_id values, perform a filter operation on the +main model. | | After task | n/a. | | Teardown | Drop the collection associated with the SMALL_DOC_FK model. | + +#### Large model creation + +Summary: This benchmark tests ODM performance creating a single large model. + +Dataset: The dataset (LARGE_DOC) is contained within `large-document.json` and consists of a sample document stored as +strict JSON with an encoded length of approximately X bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `large-document` source file (X bytes) times +10,000 operations, which equals X,XXX,XXX bytes or X MB. + +| Phase | Description | +| ----------- | ----------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC dataset into memory as an ODM-appropriate model object. | +| Before task | Drop the collection associated with the LARGE_DOC model. | +| Do task | Save the document to the database in an ODM-appropriate manner. Repeat this 10,000 times. | +| After task | n/a | +| Teardown | n/a. | + +#### Large model update + +Summary: This benchmark tests ODM performance updating fields on a single large model. + +Dataset: The dataset (LARGE_DOC) is contained within `large-document.json` and consists of a sample document stored as +strict JSON with an encoded length of approximately X bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `large-document` source file (X bytes) times +10,000 operations, which equals X,XXX,XXX bytes or X MB. + +| Phase | Description | +| ----------- | ------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC dataset into memory as an ODM-appropriate model object. Save 10,000 instances into the database. | +| Before task | n/a. | +| Do task | Update the `FIELD_NAME` field for each instance of the model in an ODM-appropriate manner. | +| After task | Drop the collection associated with the LARGE_DOC model. | +| Teardown | n/a. | + +### Nested models + +Datasets are in the `nested-models` tarball. + +Nested model tests focus performing reads and writes on models containing nested (embedded) documents. They are designed +to give insights into the efficiency of the ODM's implementation of a core advantage of the document model. + +The data will be stored as strict JSON with no extended types. These JSON representations must be converted into +equivalent models as part of the benchmark's setup. + +Nested model benchmark tasks include:s + +- Small model creation +- Small model update +- Small model find nested field by filter +- Large model creation +- Large model update nested field + +#### Small model insert + +Summary: This benchmark tests ODM performance inserting a single small nested model. + +Dataset: The dataset (SMALL_DOC_NESTED) is contained within `small-document-nested.json` and consists of a sample +document stored as strict JSON with an encoded length of approximately X bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `small-document-nested` source file (X bytes) +times 10,000 operations, which equals X,XXX,XXX bytes or X MB. + +| Phase | Description | +| ----------- | ----------------------------------------------------------------------------------------- | +| Setup | Load the SMALL_DOC_NESTED dataset into memory as an ODM-appropriate model object. | +| Before task | Drop the collection associated with the SMALL_DOC_NESTED model. | +| Do task | Save the document to the database in an ODM-appropriate manner. Repeat this 10,000 times. | +| After task | n/a | +| Teardown | n/a. | + +#### Small model find nested field + +Summary: This benchmark tests ODM performance finding documents by fields within nested documents. + +Dataset: The dataset (SMALL_DOC_NESTED) is contained within `small-document-nested.json` and consists of a sample +document stored as strict JSON with an encoded length of approximately X bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `small-document-nested` source file (X bytes) +times 10,000 operations, which equals X,XXX,XXX bytes or X MB. + +| Phase | Description | +| ----- | --------------------------------------------------------------------------------- | +| Setup | Load the SMALL_DOC_NESTED dataset into memory as an ODM-appropriate model object. | + +``` + Insert 10,000 instances into the database, saving the inserted _id field for each model into a list. | +``` + +| Before task | n/a. | | Do task | For each of the 10,000 \_id values, perform a filter operation on the nested model's +field and the main model's \_id. | | After task | n/a. | | Teardown | Drop the collection associated with the +SMALL_DOC_NESTED model. | + +#### Large model creation + +Summary: This benchmark tests ODM performance creating a single large nested model. + +Dataset: The dataset (LARGE_DOC_NESTED) is contained within `large-document-nested.json` and consists of a sample +document stored as strict JSON with an encoded length of approximately X bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `large-document-nested` source file (X bytes) +times 10,000 operations, which equals X,XXX,XXX bytes or X MB. + +| Phase | Description | +| ----------- | -------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. | +| Before task | Drop the collection associated with the LARGE_DOC model. | +| Do task | Save the model to the database in an ODM-appropriate manner. Repeat this 10,000 times. | +| After task | n/a | +| Teardown | n/a. | + +#### Large model update + +Summary: This benchmark tests ODM performance updating nested fields on a single large model. + +Dataset: The dataset (LARGE_DOC_NESTED) is contained within `large-document-nested.json` and consists of a sample +document stored as strict JSON with an encoded length of approximately X bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `large-document-nested` source file (X bytes) +times 10,000 operations, which equals X,XXX,XXX bytes or X MB. + +| Phase | Description | +| ----------- | -------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Save 10,000 instances into the database. | +| Before task | n/a. | +| Do task | Update the `NESTED_FIELD_NAME` field for each instance of the model in an ODM-appropriate manner. | +| After task | Drop the collection associated with the LARGE_DOC_NESTED model. | +| Teardown | n/a. | + +## Benchmark platform, configuration and environments + +### Benchmark Client + +Benchmarks should be run with the most recent stable version of the ODM and the newest version of the driver it +supports. + +All operations must be run with write concern "w:1". + +### Benchmark Server + +TBD: spec Amazon instance size; describe configuration (e.g. no auth, journal, pre-alloc sizes?, WT with compression to +minimize disk I/O impact?); same AWS zone as client + +## Changelog diff --git a/source/index.md b/source/index.md index fcf33d1933..0dcbd2815e 100644 --- a/source/index.md +++ b/source/index.md @@ -35,6 +35,7 @@ - [Max Staleness Tests](max-staleness/max-staleness-tests.md) - [MongoDB Handshake](mongodb-handshake/handshake.md) - [OCSP Support](ocsp-support/ocsp-support.md) +- [ODM Performance Benchmarking](benchmarking/odm-benchmarking.md) - [OP_MSG](message/OP_MSG.md) - [Performance Benchmarking](benchmarking/benchmarking.md) - [Polling SRV Records for mongos Discovery](polling-srv-records-for-mongos-discovery/polling-srv-records-for-mongos-discovery.md) From 05203724bea7472ea6030a0d6260b9e34422f56b Mon Sep 17 00:00:00 2001 From: Noah Stapp Date: Tue, 12 Aug 2025 16:47:41 -0400 Subject: [PATCH 2/8] Note direct comparisons for model creation tests --- source/benchmarking/odm-benchmarking.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/source/benchmarking/odm-benchmarking.md b/source/benchmarking/odm-benchmarking.md index e73c033dfa..c2ac85b20e 100644 --- a/source/benchmarking/odm-benchmarking.md +++ b/source/benchmarking/odm-benchmarking.md @@ -159,6 +159,8 @@ strict JSON with an encoded length of approximately X bytes. Dataset size: For scoring purposes, the dataset size is the size of the `small-document` source file (X bytes) times 10,000 operations, which equals X,XXX,XXX bytes or X MB. +This benchmark uses the same dataset as the driver `small doc insertOne` benchmark, allowing for direct comparisons. + | Phase | Description | | ----------- | -------------------------------------------------------------------------------------- | | Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. | @@ -240,6 +242,8 @@ strict JSON with an encoded length of approximately X bytes. Dataset size: For scoring purposes, the dataset size is the size of the `large-document` source file (X bytes) times 10,000 operations, which equals X,XXX,XXX bytes or X MB. +This benchmark uses the same dataset as the driver `large doc insertOne` benchmark, allowing for direct comparisons. + | Phase | Description | | ----------- | ----------------------------------------------------------------------------------------- | | Setup | Load the LARGE_DOC dataset into memory as an ODM-appropriate model object. | From f44d5a90226e3e73a00992cfc47b41ff0221bffa Mon Sep 17 00:00:00 2001 From: Noah Stapp Date: Thu, 14 Aug 2025 15:59:23 -0400 Subject: [PATCH 3/8] Remove small nested benchmarks --- source/benchmarking/odm-benchmarking.md | 182 +++++++++++------------- 1 file changed, 84 insertions(+), 98 deletions(-) diff --git a/source/benchmarking/odm-benchmarking.md b/source/benchmarking/odm-benchmarking.md index c2ac85b20e..248463fe5f 100644 --- a/source/benchmarking/odm-benchmarking.md +++ b/source/benchmarking/odm-benchmarking.md @@ -153,11 +153,11 @@ Flat model benchmark tasks include:s Summary: This benchmark tests ODM performance creating a single small model. -Dataset: The dataset (SMALL_DOC) is contained within `small-document.json` and consists of a sample document stored as -strict JSON with an encoded length of approximately X bytes. +Dataset: The dataset (SMALL_DOC) is contained within `small_doc.json` and consists of a sample document stored as strict +JSON with an encoded length of approximately 250 bytes. -Dataset size: For scoring purposes, the dataset size is the size of the `small-document` source file (X bytes) times -10,000 operations, which equals X,XXX,XXX bytes or X MB. +Dataset size: For scoring purposes, the dataset size is the size of the `small_doc` source file (250 bytes) times 10,000 +operations, which equals 2,250,000 bytes or 2.5 MB. This benchmark uses the same dataset as the driver `small doc insertOne` benchmark, allowing for direct comparisons. @@ -169,15 +169,15 @@ This benchmark uses the same dataset as the driver `small doc insertOne` benchma | After task | n/a | | Teardown | n/a. | -#### Large model update +#### Small model update Summary: This benchmark tests ODM performance updating fields on a single small model. -Dataset: The dataset (SMALL_DOC) is contained within `small-document.json` and consists of a sample document stored as -strict JSON with an encoded length of approximately X bytes. +Dataset: The dataset (SMALL_DOC) is contained within `small_doc.json` and consists of a sample document stored as strict +JSON with an encoded length of approximately 250 bytes. -Dataset size: For scoring purposes, the dataset size is the size of the `small-document` source file (X bytes) times -10,000 operations, which equals X,XXX,XXX bytes or X MB. +Dataset size: For scoring purposes, the dataset size is the size of the `small_doc` source file (250 bytes) times 10,000 +operations, which equals 2,250,000 bytes or 2.5 MB. | Phase | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------- | @@ -191,58 +191,49 @@ Dataset size: For scoring purposes, the dataset size is the size of the `small-d Summary: This benchmark tests ODM performance finding documents using a basic filter. -Dataset: The dataset (SMALL_DOC) is contained within `small-document.json` and consists of a sample document stored as -strict JSON with an encoded length of approximately X bytes. - -Dataset size: For scoring purposes, the dataset size is the size of the `small-document` source file (X bytes) times -10,000 operations, which equals X,XXX,XXX bytes or X MB. - -| Phase | Description | -| ----- | -------------------------------------------------------------------------- | -| Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. | +Dataset: The dataset (SMALL_DOC) is contained within `small_doc.json` and consists of a sample document stored as strict +JSON with an encoded length of approximately 250 bytes. -``` - Insert 10,000 instances into the database, saving the inserted _id field for each into a list. | -``` +Dataset size: For scoring purposes, the dataset size is the size of the `small_doc` source file (250 bytes) times 10,000 +operations, which equals 2,250,000 bytes or 2.5 MB. -| Before task | n/a. | | Do task | For each of the 10,000 \_id values, perform a filter operation to search for the -corresponding model document. | | After task | n/a. | | Teardown | Drop the collection associated with the SMALL_DOC -model. | +| Phase | Description | +| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the inserted \_id field for each into a list. | +| Before task | n/a. | +| Do task | For each of the 10,000 \_id values, perform a filter operation to search for the corresponding model document. | +| After task | n/a. | +| Teardown | Drop the collection associated with the SMALL_DOC model. | #### Small model find foreign key by filter Summary: This benchmark tests ODM performance finding documents by foreign keys. This benchmark must only be run by ODMs that support join ($lookup) operations. -Dataset: The dataset (SMALL_DOC_FK) is contained within `small-document-foreign-key.json` and consists of two sample +Dataset: The dataset (SMALL_DOC_FK) is contained within `small_doc-foreign-key.json` and consists of two sample documents, both stored as strict JSON: the main document with an encoded length of approximately X bytes, and the associated foreign key document with an encoded length of approximately Y bytes. -Dataset size: For scoring purposes, the dataset size is the size of the `small-document-foreign-key.json` source file (X -\+ Y bytes) times 10,000 operations, which equals X,XXX,XXX bytes or X + Y MB. - -| Phase | Description | -| ----- | ------------------------------------------------------------------------------------------------------------------------------------------ | -| Setup | Load the SMALL_DOC_FK dataset into memory as two ODM-appropriate model objects: one for the main model, and one for the foreign key model. | - -``` - Insert 10,000 instances of each into the database, saving the inserted _id field for each foreign key model into a list. | -``` +Dataset size: For scoring purposes, the dataset size is the size of the `small_doc-foreign-key.json` source file (X + Y +bytes) times 10,000 operations, which equals X,XXX,XXX bytes or X + Y MB. -| Before task | n/a. | | Do task | For each of the 10,000 foreign key \_id values, perform a filter operation on the -main model. | | After task | n/a. | | Teardown | Drop the collection associated with the SMALL_DOC_FK model. | +| Phase | Description | +| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the SMALL_DOC_FK dataset into memory as two ODM-appropriate model objects: one for the main model, and one for the foreign key model. Insert 10,000 instances of each into the database, saving the inserted \_id field for each foreign key model into a list. | +| Before task | n/a. | +| Do task | For each of the 10,000 foreign key \_id values, perform a filter operation on the main model. | +| After task | n/a. | +| Teardown | Drop the collection associated with the SMALL_DOC_FK model. | #### Large model creation Summary: This benchmark tests ODM performance creating a single large model. -Dataset: The dataset (LARGE_DOC) is contained within `large-document.json` and consists of a sample document stored as -strict JSON with an encoded length of approximately X bytes. +Dataset: The dataset (LARGE_DOC) is contained within `large_doc.json` and consists of a sample document stored as strict +JSON with an encoded length of approximately 8,000 bytes. -Dataset size: For scoring purposes, the dataset size is the size of the `large-document` source file (X bytes) times -10,000 operations, which equals X,XXX,XXX bytes or X MB. - -This benchmark uses the same dataset as the driver `large doc insertOne` benchmark, allowing for direct comparisons. +Dataset size: For scoring purposes, the dataset size is the size of the `large_doc` source file (X bytes) times 10,000 +operations, which equals 80,000,000 bytes or 80 MB. | Phase | Description | | ----------- | ----------------------------------------------------------------------------------------- | @@ -256,11 +247,11 @@ This benchmark uses the same dataset as the driver `large doc insertOne` benchma Summary: This benchmark tests ODM performance updating fields on a single large model. -Dataset: The dataset (LARGE_DOC) is contained within `large-document.json` and consists of a sample document stored as -strict JSON with an encoded length of approximately X bytes. +Dataset: The dataset (LARGE_DOC) is contained within `large_doc.json` and consists of a sample document stored as strict +JSON with an encoded length of approximately 8,000 bytes. -Dataset size: For scoring purposes, the dataset size is the size of the `large-document` source file (X bytes) times -10,000 operations, which equals X,XXX,XXX bytes or X MB. +Dataset size: For scoring purposes, the dataset size is the size of the `large_doc` source file (X bytes) times 10,000 +operations, which equals 80,000,000 bytes or 80 MB. | Phase | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------- | @@ -282,61 +273,20 @@ equivalent models as part of the benchmark's setup. Nested model benchmark tasks include:s -- Small model creation -- Small model update -- Small model find nested field by filter - Large model creation - Large model update nested field - -#### Small model insert - -Summary: This benchmark tests ODM performance inserting a single small nested model. - -Dataset: The dataset (SMALL_DOC_NESTED) is contained within `small-document-nested.json` and consists of a sample -document stored as strict JSON with an encoded length of approximately X bytes. - -Dataset size: For scoring purposes, the dataset size is the size of the `small-document-nested` source file (X bytes) -times 10,000 operations, which equals X,XXX,XXX bytes or X MB. - -| Phase | Description | -| ----------- | ----------------------------------------------------------------------------------------- | -| Setup | Load the SMALL_DOC_NESTED dataset into memory as an ODM-appropriate model object. | -| Before task | Drop the collection associated with the SMALL_DOC_NESTED model. | -| Do task | Save the document to the database in an ODM-appropriate manner. Repeat this 10,000 times. | -| After task | n/a | -| Teardown | n/a. | - -#### Small model find nested field - -Summary: This benchmark tests ODM performance finding documents by fields within nested documents. - -Dataset: The dataset (SMALL_DOC_NESTED) is contained within `small-document-nested.json` and consists of a sample -document stored as strict JSON with an encoded length of approximately X bytes. - -Dataset size: For scoring purposes, the dataset size is the size of the `small-document-nested` source file (X bytes) -times 10,000 operations, which equals X,XXX,XXX bytes or X MB. - -| Phase | Description | -| ----- | --------------------------------------------------------------------------------- | -| Setup | Load the SMALL_DOC_NESTED dataset into memory as an ODM-appropriate model object. | - -``` - Insert 10,000 instances into the database, saving the inserted _id field for each model into a list. | -``` - -| Before task | n/a. | | Do task | For each of the 10,000 \_id values, perform a filter operation on the nested model's -field and the main model's \_id. | | After task | n/a. | | Teardown | Drop the collection associated with the -SMALL_DOC_NESTED model. | +- Large model find nested field by filter +- Large model find nested array field by filter #### Large model creation Summary: This benchmark tests ODM performance creating a single large nested model. -Dataset: The dataset (LARGE_DOC_NESTED) is contained within `large-document-nested.json` and consists of a sample -document stored as strict JSON with an encoded length of approximately X bytes. +Dataset: The dataset (LARGE_DOC_NESTED) is contained within `large_doc_nested.json` and consists of a sample document +stored as strict JSON with an encoded length of approximately 8,000 bytes. -Dataset size: For scoring purposes, the dataset size is the size of the `large-document-nested` source file (X bytes) -times 10,000 operations, which equals X,XXX,XXX bytes or X MB. +Dataset size: For scoring purposes, the dataset size is the size of the `large_doc_nested` source file (8,000 bytes) +times 10,000 operations, which equals 80,000,000 bytes or 80 MB. | Phase | Description | | ----------- | -------------------------------------------------------------------------------------- | @@ -350,11 +300,11 @@ times 10,000 operations, which equals X,XXX,XXX bytes or X MB. Summary: This benchmark tests ODM performance updating nested fields on a single large model. -Dataset: The dataset (LARGE_DOC_NESTED) is contained within `large-document-nested.json` and consists of a sample -document stored as strict JSON with an encoded length of approximately X bytes. +Dataset: The dataset (LARGE_DOC_NESTED) is contained within `large_doc_nested.json` and consists of a sample document +stored as strict JSON with an encoded length of approximately 8,000 bytes. -Dataset size: For scoring purposes, the dataset size is the size of the `large-document-nested` source file (X bytes) -times 10,000 operations, which equals X,XXX,XXX bytes or X MB. +Dataset size: For scoring purposes, the dataset size is the size of the `large_doc_nested` source file (8,000 bytes) +times 10,000 operations, which equals 80,000,000 bytes or 80 MB. | Phase | Description | | ----------- | -------------------------------------------------------------------------------------------------------------------------- | @@ -364,6 +314,42 @@ times 10,000 operations, which equals X,XXX,XXX bytes or X MB. | After task | Drop the collection associated with the LARGE_DOC_NESTED model. | | Teardown | n/a. | +#### Large nested model find by filter + +Summary: This benchmark tests ODM performance finding nested documents using a basic filter. + +Dataset: The dataset (LARGE_DOC_NESTED) is contained within `large_doc_nested.json` and consists of a sample document +stored as strict JSON with an encoded length of approximately 8,000 bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `large_doc_nested` source file (8,000 bytes) +times 10,000 operations, which equals 80,000,000 bytes or 80 MB. + +| Phase | Description | +| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the value of the unique_id field for each model's `embedded_str_doc_1` nested model into a list. | +| Before task | n/a. | +| Do task | For each of the 10,000 unique_id values, perform a filter operation to search for the corresponding parent model. | +| After task | n/a. | +| Teardown | Drop the collection associated with the LARGE_DOC_NESTED model. | + +#### Large nested model find array by filter + +Summary: This benchmark tests ODM performance finding nested document arrays using a basic filter. + +Dataset: The dataset (LARGE_DOC_NESTED) is contained within `large_doc_nested.json` and consists of a sample document +stored as strict JSON with an encoded length of approximately 8,000 bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `large_doc_nested` source file (8,000 bytes) +times 10,000 operations, which equals 80,000,000 bytes or 80 MB. + +| Phase | Description | +| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the value of the unique_id field for the first item in each model's `embedded_str_doc_array` nested model into a list. | +| Before task | n/a. | +| Do task | For each of the 10,000 unique_id values, perform a filter operation to search for the corresponding parent model. | +| After task | n/a. | +| Teardown | Drop the collection associated with the LARGE_DOC_NESTED model. | + ## Benchmark platform, configuration and environments ### Benchmark Client From 293443a7cec68ecd025ae9f9072b989e223e35ff Mon Sep 17 00:00:00 2001 From: Noah Stapp Date: Fri, 15 Aug 2025 09:53:47 -0400 Subject: [PATCH 4/8] Update instructions for foreign key test --- source/benchmarking/odm-benchmarking.md | 86 ++++++++++++------------- 1 file changed, 42 insertions(+), 44 deletions(-) diff --git a/source/benchmarking/odm-benchmarking.md b/source/benchmarking/odm-benchmarking.md index 248463fe5f..1337579fb5 100644 --- a/source/benchmarking/odm-benchmarking.md +++ b/source/benchmarking/odm-benchmarking.md @@ -23,13 +23,12 @@ outlier behavior that can be optimized away. ### Task Hierarchy -The benchmark suite consists of multiple groups of small, independent benchmarks. This allows us to better isolate areas +The benchmark suite consists of two groups of small, independent benchmarks. This allows us to better isolate areas within ODMs that are faster or slower. - Flat models -- reading and writing flat models of various sizes, to explore basic operation efficiency - Nested models -- reading and writing nested models of various sizes, to explore basic operation efficiency for complex data -- Joins -- performing operations that involve joins (if supported), to explore $lookup efficiency ### Measurement @@ -43,9 +42,9 @@ be the median score of the iterations. A range of percentiles will also be recor ### Data sets -Data sets will vary by task. In some cases, they will be synthetically generated models inserted repeatedly to construct -an overall corpus of models. In other cases, data sets will be synthetic line-delimited JSON files or mock binary files -to be constructed by the ODM into the appropriate model. +Data sets will vary by task. In most cases, data sets will be synthetic line-delimited JSON files or mock binary files +to be constructed by the ODM into the appropriate model. Some tasks will require additional modifications to these +constructed models. ### Versioning @@ -70,13 +69,12 @@ extreme forms of measurement: Therefore, we choose a middle ground: -- measuring the same 1000 insertions over 100 iterations -- each timing run includes enough operations that insertion - code dominates timing code; unusual system events are likely to affect only a fraction of the 100 timing - measurements +- measuring the same 10000 insertions over 10 iterations -- each timing run includes enough operations that insertion + code dominates timing code; unusual system events are likely to affect only a fraction of the 10 timing measurements -With 100 timings of inserting the same 1000 models, we build up a statistical distribution of the operation timing, +With 10 timings of inserting the same 10000 models, we build up a statistical distribution of the operation timing, allowing a more robust estimate of performance than a single measurement. (In practice, the number of iterations could -exceed 100, but 100 is a reasonable minimum goal.) +exceed 10, but 10 is a reasonable minimum goal.) Because a timing distribution is bounded by zero on one side, taking the mean would allow large positive outlier measurements to skew the result substantially. Therefore, for the benchmark score, we use the median timing measurement, @@ -85,12 +83,12 @@ which is robust in the face of outliers. Each benchmark is structured into discrete setup/execute/teardown phases. Phases are as follows, with specific details given in a subsequent section: -- setup -- (ONCE PER TASK) something to do once before any benchmarking, e.g. construct a client object, load test data, +- setup -- (ONCE PER TASK) something to do once before any benchmarking, e.g. construct a model object, load test data, insert data into a collection, etc. - before operation -- (ONCE PER ITERATION) something to do before every task iteration, e.g. drop a collection, or reload test data (if the test run modifies it), etc. -- do operation -- (ONCE PER ITERATION) smallest amount of code necessary to execute the task; e.g. insert 1000 models - one by one into the database, or retrieve 1000 models of test data from the database, etc. +- do operation -- (ONCE PER ITERATION) smallest amount of code necessary to execute the task; e.g. insert 10000 models + one by one into the database, or retrieve 10000 models of test data from the database, etc. - after operation -- (ONCE PER ITERATION) something to do after every task iteration (if necessary) - teardown -- (ONCE PER TASK) something done once after all benchmarking is complete (if necessary); e.g. drop the test database @@ -102,7 +100,7 @@ monotonic timer (or best language approximation). Unless otherwise specified, the number of iterations to measure per task is variable: - iterations should loop for at least 1 minute cumulative execution time -- iterations should stop after 100 iterations or 5 minutes cumulative execution time, whichever is shorter +- iterations should stop after 10 iterations or 5 minutes cumulative execution time, whichever is shorter This balances measurement stability with a timing cap to ensure all tasks can complete in a reasonable time. @@ -183,7 +181,7 @@ operations, which equals 2,250,000 bytes or 2.5 MB. | ----------- | ------------------------------------------------------------------------------------------------------------------- | | Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. Save 10,000 instances into the database. | | Before task | n/a. | -| Do task | Update the `FIELD_NAME` field for each instance of the model in an ODM-appropriate manner. | +| Do task | Update the `field1` field for each instance of the model to equal `updated_value` in an ODM-appropriate manner. | | After task | Drop the collection associated with the SMALL_DOC model. | | Teardown | n/a. | @@ -199,9 +197,9 @@ operations, which equals 2,250,000 bytes or 2.5 MB. | Phase | Description | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the inserted \_id field for each into a list. | +| Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the inserted `id` field for each into a list. | | Before task | n/a. | -| Do task | For each of the 10,000 \_id values, perform a filter operation to search for the corresponding model document. | +| Do task | For each of the 10,000 `id` values, perform a filter operation to find the corresponding SMALL_DOC model. | | After task | n/a. | | Teardown | Drop the collection associated with the SMALL_DOC model. | @@ -210,20 +208,20 @@ operations, which equals 2,250,000 bytes or 2.5 MB. Summary: This benchmark tests ODM performance finding documents by foreign keys. This benchmark must only be run by ODMs that support join ($lookup) operations. -Dataset: The dataset (SMALL_DOC_FK) is contained within `small_doc-foreign-key.json` and consists of two sample -documents, both stored as strict JSON: the main document with an encoded length of approximately X bytes, and the -associated foreign key document with an encoded length of approximately Y bytes. +Dataset: The dataset (SMALL_DOC) is contained within `small_doc.json` and consists of a sample document stored as strict +JSON with an encoded length of approximately 250 bytes. An additional model (FOREIGN_KEY) representing the foreign key, +consisting of only a string field called `name`, must also be created. -Dataset size: For scoring purposes, the dataset size is the size of the `small_doc-foreign-key.json` source file (X + Y -bytes) times 10,000 operations, which equals X,XXX,XXX bytes or X + Y MB. +Dataset size: For scoring purposes, the dataset size is the size of the `small_doc` source file (250 bytes) times 10,000 +operations, which equals 2,250,000 bytes or 2.5 MB. -| Phase | Description | -| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Setup | Load the SMALL_DOC_FK dataset into memory as two ODM-appropriate model objects: one for the main model, and one for the foreign key model. Insert 10,000 instances of each into the database, saving the inserted \_id field for each foreign key model into a list. | -| Before task | n/a. | -| Do task | For each of the 10,000 foreign key \_id values, perform a filter operation on the main model. | -| After task | n/a. | -| Teardown | Drop the collection associated with the SMALL_DOC_FK model. | +| Phase | Description | +| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. For each SMALL_DOC model, create and assign a FOREIGN_KEY instance to the `field_fk` field. Insert 10,000 instances of both models into the database, saving the inserted `id` field for each FOREIGN_KEY into a list. | +| Before task | n/a. | +| Do task | For each of the 10,000 FOREIGN_KEY `id` values, perform a filter operation to find the corresponding SMALL_DOC model. | +| After task | n/a. | +| Teardown | Drop the collections associated with the SMALL_DOC and FOREIGN_KEY models. | #### Large model creation @@ -235,13 +233,13 @@ JSON with an encoded length of approximately 8,000 bytes. Dataset size: For scoring purposes, the dataset size is the size of the `large_doc` source file (X bytes) times 10,000 operations, which equals 80,000,000 bytes or 80 MB. -| Phase | Description | -| ----------- | ----------------------------------------------------------------------------------------- | -| Setup | Load the LARGE_DOC dataset into memory as an ODM-appropriate model object. | -| Before task | Drop the collection associated with the LARGE_DOC model. | -| Do task | Save the document to the database in an ODM-appropriate manner. Repeat this 10,000 times. | -| After task | n/a | -| Teardown | n/a. | +| Phase | Description | +| ----------- | ------------------------------------------------------------------------------------------------ | +| Setup | Load the LARGE_DOC dataset into memory as an ODM-appropriate model object. | +| Before task | Drop the collection associated with the LARGE_DOC model. | +| Do task | Save the LARGE_DOC model to the database in an ODM-appropriate manner. Repeat this 10,000 times. | +| After task | n/a | +| Teardown | n/a. | #### Large model update @@ -257,7 +255,7 @@ operations, which equals 80,000,000 bytes or 80 MB. | ----------- | ------------------------------------------------------------------------------------------------------------------- | | Setup | Load the LARGE_DOC dataset into memory as an ODM-appropriate model object. Save 10,000 instances into the database. | | Before task | n/a. | -| Do task | Update the `FIELD_NAME` field for each instance of the model in an ODM-appropriate manner. | +| Do task | Update the `field1` field for each instance of the model to `updated_value` in an ODM-appropriate manner. | | After task | Drop the collection associated with the LARGE_DOC model. | | Teardown | n/a. | @@ -324,13 +322,13 @@ stored as strict JSON with an encoded length of approximately 8,000 bytes. Dataset size: For scoring purposes, the dataset size is the size of the `large_doc_nested` source file (8,000 bytes) times 10,000 operations, which equals 80,000,000 bytes or 80 MB. -| Phase | Description | -| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the value of the unique_id field for each model's `embedded_str_doc_1` nested model into a list. | -| Before task | n/a. | -| Do task | For each of the 10,000 unique_id values, perform a filter operation to search for the corresponding parent model. | -| After task | n/a. | -| Teardown | Drop the collection associated with the LARGE_DOC_NESTED model. | +| Phase | Description | +| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the value of the `unique_id` field for each model's `embedded_str_doc_1` nested model into a list. | +| Before task | n/a. | +| Do task | For each of the 10,000 `unique_id` values, perform a filter operation to search for the corresponding parent model. | +| After task | n/a. | +| Teardown | Drop the collection associated with the LARGE_DOC_NESTED model. | #### Large nested model find array by filter From 3b25a2706be3cd42c51873b6828b9c222f04aea7 Mon Sep 17 00:00:00 2001 From: Noah Stapp Date: Fri, 15 Aug 2025 14:52:23 -0400 Subject: [PATCH 5/8] Add datasets --- source/benchmarking/odm-benchmarking.md | 70 ++--- .../odm-data/flat-models/large_doc.json | 252 ++++++++++++++++++ .../odm-data/flat-models/small_doc.json | 1 + source/benchmarking/odm-data/flat_models.tgz | Bin 0 -> 3190 bytes .../nested-models/large_doc_nested.json | 242 +++++++++++++++++ .../benchmarking/odm-data/nested_models.tgz | Bin 0 -> 2475 bytes 6 files changed, 532 insertions(+), 33 deletions(-) create mode 100644 source/benchmarking/odm-data/flat-models/large_doc.json create mode 100644 source/benchmarking/odm-data/flat-models/small_doc.json create mode 100644 source/benchmarking/odm-data/flat_models.tgz create mode 100644 source/benchmarking/odm-data/nested-models/large_doc_nested.json create mode 100644 source/benchmarking/odm-data/nested_models.tgz diff --git a/source/benchmarking/odm-benchmarking.md b/source/benchmarking/odm-benchmarking.md index 1337579fb5..27b6c8676e 100644 --- a/source/benchmarking/odm-benchmarking.md +++ b/source/benchmarking/odm-benchmarking.md @@ -130,7 +130,7 @@ line-delimited JSON use CRLF delimiters, but this benchmark uses only LF.) ### Flat models -Datasets are in the `flat-models` tarball. +Datasets are in the `flat_models` tarball. Flat model tests focus on flatly-structured model reads and writes across data sizes. They are designed to give insights into the efficiency of the ODM's implementation of basic data operations. @@ -174,8 +174,8 @@ Summary: This benchmark tests ODM performance updating fields on a single small Dataset: The dataset (SMALL_DOC) is contained within `small_doc.json` and consists of a sample document stored as strict JSON with an encoded length of approximately 250 bytes. -Dataset size: For scoring purposes, the dataset size is the size of the `small_doc` source file (250 bytes) times 10,000 -operations, which equals 2,250,000 bytes or 2.5 MB. +Dataset size: For scoring purposes, the dataset size is the size of the `updated_value` string file (13 bytes) times +10,000 operations, which equals 130,000 bytes or 130 KB. | Phase | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------- | @@ -230,8 +230,8 @@ Summary: This benchmark tests ODM performance creating a single large model. Dataset: The dataset (LARGE_DOC) is contained within `large_doc.json` and consists of a sample document stored as strict JSON with an encoded length of approximately 8,000 bytes. -Dataset size: For scoring purposes, the dataset size is the size of the `large_doc` source file (X bytes) times 10,000 -operations, which equals 80,000,000 bytes or 80 MB. +Dataset size: For scoring purposes, the dataset size is the size of the `large_doc` source file (8,000 bytes) times +10,000 operations, which equals 80,000,000 bytes or 80 MB. | Phase | Description | | ----------- | ------------------------------------------------------------------------------------------------ | @@ -248,8 +248,8 @@ Summary: This benchmark tests ODM performance updating fields on a single large Dataset: The dataset (LARGE_DOC) is contained within `large_doc.json` and consists of a sample document stored as strict JSON with an encoded length of approximately 8,000 bytes. -Dataset size: For scoring purposes, the dataset size is the size of the `large_doc` source file (X bytes) times 10,000 -operations, which equals 80,000,000 bytes or 80 MB. +Dataset size: For scoring purposes, the dataset size is the size of the `updated_value` string file (13 bytes) times +10,000 operations, which equals 130,000 bytes or 130 KB. | Phase | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------- | @@ -261,7 +261,7 @@ operations, which equals 80,000,000 bytes or 80 MB. ### Nested models -Datasets are in the `nested-models` tarball. +Datasets are in the `nested_models` tarball. Nested model tests focus performing reads and writes on models containing nested (embedded) documents. They are designed to give insights into the efficiency of the ODM's implementation of a core advantage of the document model. @@ -294,25 +294,25 @@ times 10,000 operations, which equals 80,000,000 bytes or 80 MB. | After task | n/a | | Teardown | n/a. | -#### Large model update +#### Large model update nested Summary: This benchmark tests ODM performance updating nested fields on a single large model. Dataset: The dataset (LARGE_DOC_NESTED) is contained within `large_doc_nested.json` and consists of a sample document stored as strict JSON with an encoded length of approximately 8,000 bytes. -Dataset size: For scoring purposes, the dataset size is the size of the `large_doc_nested` source file (8,000 bytes) -times 10,000 operations, which equals 80,000,000 bytes or 80 MB. +Dataset size: For scoring purposes, the dataset size is the size of the `updated_value` string file (13 bytes) times +10,000 operations, which equals 130,000 bytes or 130 KB. -| Phase | Description | -| ----------- | -------------------------------------------------------------------------------------------------------------------------- | -| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Save 10,000 instances into the database. | -| Before task | n/a. | -| Do task | Update the `NESTED_FIELD_NAME` field for each instance of the model in an ODM-appropriate manner. | -| After task | Drop the collection associated with the LARGE_DOC_NESTED model. | -| Teardown | n/a. | +| Phase | Description | +| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Save 10,000 instances into the database. | +| Before task | n/a. | +| Do task | Update the value of the `embedded_str_doc_1.field1` field to `updated_value` for each instance of the model in an ODM-appropriate manner. | +| After task | Drop the collection associated with the LARGE_DOC_NESTED model. | +| Teardown | n/a. | -#### Large nested model find by filter +#### Large nested model find nested by filter Summary: This benchmark tests ODM performance finding nested documents using a basic filter. @@ -326,11 +326,11 @@ times 10,000 operations, which equals 80,000,000 bytes or 80 MB. | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the value of the `unique_id` field for each model's `embedded_str_doc_1` nested model into a list. | | Before task | n/a. | -| Do task | For each of the 10,000 `unique_id` values, perform a filter operation to search for the corresponding parent model. | +| Do task | For each of the 10,000 `embedded_str_doc_1.unique_id` values, perform a filter operation to search for the corresponding parent model. | | After task | n/a. | | Teardown | Drop the collection associated with the LARGE_DOC_NESTED model. | -#### Large nested model find array by filter +#### Large nested model find nested array by filter Summary: This benchmark tests ODM performance finding nested document arrays using a basic filter. @@ -340,26 +340,30 @@ stored as strict JSON with an encoded length of approximately 8,000 bytes. Dataset size: For scoring purposes, the dataset size is the size of the `large_doc_nested` source file (8,000 bytes) times 10,000 operations, which equals 80,000,000 bytes or 80 MB. -| Phase | Description | -| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the value of the unique_id field for the first item in each model's `embedded_str_doc_array` nested model into a list. | -| Before task | n/a. | -| Do task | For each of the 10,000 unique_id values, perform a filter operation to search for the corresponding parent model. | -| After task | n/a. | -| Teardown | Drop the collection associated with the LARGE_DOC_NESTED model. | +| Phase | Description | +| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the value of the `unique_id` field for the first item in each model's `embedded_str_doc_array` nested model into a list. | +| Before task | n/a. | +| Do task | For each of the 10,000 `unique_id` values, perform a filter operation to search for the corresponding parent model. | +| After task | n/a. | +| Teardown | Drop the collection associated with the LARGE_DOC_NESTED model. | ## Benchmark platform, configuration and environments ### Benchmark Client -Benchmarks should be run with the most recent stable version of the ODM and the newest version of the driver it +The benchmarks should be run with the most recent stable version of the ODM and the newest version of the driver it supports. -All operations must be run with write concern "w:1". - ### Benchmark Server -TBD: spec Amazon instance size; describe configuration (e.g. no auth, journal, pre-alloc sizes?, WT with compression to -minimize disk I/O impact?); same AWS zone as client +The MongoDB ODM Performance Benchmark must be run against a standalone MongoDB server running the latest database +version without authentication or SSL enabled. + +### Benchmark Placement + +The MongoDB ODM Performance Benchmark should be placed within the ODM's test directory as an independent test suite. Due +to the relatively long runtime of the benchmarks, including them as part of an automated suite that runs against every +PR is not recommended. ## Changelog diff --git a/source/benchmarking/odm-data/flat-models/large_doc.json b/source/benchmarking/odm-data/flat-models/large_doc.json new file mode 100644 index 0000000000..8a931b89d7 --- /dev/null +++ b/source/benchmarking/odm-data/flat-models/large_doc.json @@ -0,0 +1,252 @@ +{ + "field1": "kj9$mxz#p2qw8r*vn4@h7c&u1s", + "field2": "x3@9zf#mk7w$qp2n8v*r6j4h&c1u5s0a", + "field3": "p9#m2x$k7z@w3q8v*n4r&j6h1c5u0s", + "field4": "z8@x3m#k9w$p2q7v*r4n6j&h1c5u0s", + "field5": "m7#k9x$z3w@p8q2v*n4r6j&h1c5u0s", + "field6": "k6$x9m#z7w3p@q8v2n*r4j6h&c1u5s0", + "field7": "x5@m9k#z6w$p3q7v*n2r8j&h4c1u0s", + "field8": "m4#x8k$z9w6p@q3v*n7r2j&h5c1u0s", + "field9": "k3$m7x#z8w9p@q6v*n2r4j&h1c5u0s", + "field10": "x2@k6m#z7w8p$q9v*n3r1j&h4c5u0s", + "field11": "m1#x5k$z6w7p@q8v*n9r2j&h3c4u0s", + "field12": "k0$x4m#z5w6p@q7v*n8r1j&h2c3u9s", + "field13": "x9@m3k#z4w5p$q6v*n7r0j&h1c2u8s", + "field14": "m8#k2x$z3w4p@q5v*n6r9j&h0c1u7s", + "field15": "k7$x1m#z2w3p@q4v*n5r8j&h9c0u6s", + "field16": "x6@m0k#z1w2p$q3v*n4r7j&h8c9u5s", + "field17": "m5#x9k$z0w1p@q2v*n3r6j&h7c8u4s", + "field18": "k4$m8x#z9w0p@q1v*n2r5j&h6c7u3s", + "field19": "x3@k7m#z8w9p$q0v*n1r4j&h5c6u2s", + "field20": "m2#x6k$z7w8p@q9v*n0r3j&h4c5u1s", + "field21": "k1$x5m#z6w7p@q8v*n9r2j&h3c4u0s", + "field22": "x0@m4k#z5w6p$q7v*n8r1j&h2c3u9s", + "field23": "m9#k3x$z4w5p@q6v*n7r0j&h1c2u8s", + "field24": "k8$x2m#z3w4p@q5v*n6r9j&h0c1u7s", + "field25": "x7@m1k#z2w3p$q4v*n5r8j&h9c0u6s", + "field26": "m6#x0k$z1w2p@q3v*n4r7j&h8c9u5s", + "field27": "k5$m9x#z0w1p@q2v*n3r6j&h7c8u4s", + "field28": "x4@k8m#z9w0p$q1v*n2r5j&h6c7u3s", + "field29": "m3#x7k$z8w9p@q0v*n1r4j&h5c6u2s", + "field30": "k2$x6m#z7w8p@q9v*n0r3j&h4c5u1s", + "field31": "x1@m5k#z6w7p$q8v*n9r2j&h3c4u0s", + "field32": "m0#k4x$z5w6p@q7v*n8r1j&h2c3u9s", + "field33": "k9$x3m#z4w5p@q6v*n7r0j&h1c2u8s", + "field34": "x8@m2k#z3w4p$q5v*n6r9j&h0c1u7s", + "field35": "m7#x1k$z2w3p@q4v*n5r8j&h9c0u6s", + "field36": "k6$m0x#z1w2p@q3v*n4r7j&h8c9u5s", + "field37": "x5@k9m#z0w1p$q2v*n3r6j&h7c8u4s", + "field38": "m4#x8k$z9w0p@q1v*n2r5j&h6c7u3s", + "field39": "k3$x7m#z8w9p@q0v*n1r4j&h5c6u2s", + "field40": "x2@m6k#z7w8p$q9v*n0r3j&h4c5u1s", + "field41": "m1#k5x$z6w7p@q8v*n9r2j&h3c4u0s", + "field42": "k0$x4m#z5w6p@q7v*n8r1j&h2c3u9s", + "field43": "x9@m3k#z4w5p$q6v*n7r0j&h1c2u8s", + "field44": "m8#x2k$z3w4p@q5v*n6r9j&h0c1u7s", + "field45": "k7$m1x#z2w3p@q4v*n5r8j&h9c0u6s", + "field46": "x6@k0m#z1w2p$q3v*n4r7j&h8c9u5s", + "field47": "m5#x9k$z0w1p@q2v*n3r6j&h7c8u4s", + "field48": "k4$m8x#z9w0p@q1v*n2r5j&h6c7u3s", + "field49": "x3@k7m#z8w9p$q0v*n1r4j&h5c6u2s", + "field50": "m2#x6k$z7w8p@q9v*n0r3j&h4c5u1s", + "field51": "k1$x5m#z6w7p@q8v*n9r2j&h3c4u0s9f8d", + "field52": "x0@m4k#z5w6p$q7v*n8r1j&h2c3u9s7g6e", + "field53": "m9#k3x$z4w5p@q6v*n7r0j&h1c2u8s5b4a", + "field54": "k8$x2m#z3w4p@q5v*n6r9j&h0c1u7s3v2t", + "field55": "x7@m1k#z2w3p$q4v*n5r8j&h9c0u6s1y0w", + "field56": "m6#x0k$z1w2p@q3v*n4r7j&h8c9u5s9i8o", + "field57": "k5$m9x#z0w1p@q2v*n3r6j&h7c8u4s7l6p", + "field58": "x4@k8m#z9w0p$q1v*n2r5j&h6c7u3s5m4n", + "field59": "m3#x7k$z8w9p@q0v*n1r4j&h5c6u2s3q2r", + "field60": "k2$x6m#z7w8p@q9v*n0r3j&h4c5u1s1x0z", + "field61": "p7#q8r$s9t@u0v*w1x&y2z3a4b5c6d", + "field62": "f3$g4h#i5j@k6l*m7n&o8p9q0r1s2t", + "field63": "v5@w6x#y7z$a8b*c9d&e0f1g2h3i4j", + "field64": "n2#o3p$q4r@s5t*u6v&w7x8y9z0a1b", + "field65": "j8$k9l#m0n@o1p*q2r&s3t4u5v6w7x", + "field66": "b4@c5d#e6f$g7h*i8j&k9l0m1n2o3p", + "field67": "x6#y7z$a8b@c9d*e0f&g1h2i3j4k5l", + "field68": "t0$u1v#w2x@y3z*a4b&c5d6e7f8g9h", + "field69": "l9@m0n#o1p$q2r*s3t&u4v5w6x7y8z", + "field70": "h5#i6j$k7l@m8n*o9p&q0r1s2t3u4v", + "field71": "d1@e2f#g3h$i4j*k5l&m6n7o8p9q0r", + "field72": "z7$a8b#c9d@e0f*g1h&i2j3k4l5m6n", + "field73": "v3@w4x#y5z$a6b*c7d&e8f9g0h1i2j", + "field74": "r9#s0t$u1v@w2x*y3z&a4b5c6d7e8f", + "field75": "n5$o6p#q7r@s8t*u9v&w0x1y2z3a4b", + "field76": "j1@k2l#m3n$o4p*q5r&s6t7u8v9w0x", + "field77": "f7#g8h$i9j@k0l*m1n&o2p3q4r5s6t", + "field78": "b3@c4d#e5f$g6h*i7j&k8l9m0n1o2p", + "field79": "x9$y0z#a1b@c2d*e3f&g4h5i6j7k8l", + "field80": "t5#u6v$w7x@y8z*a9b&c0d1e2f3g4h", + "field81": "p1@q2r#s3t$u4v*w5x&y6z7a8b9c0d", + "field82": "l7#m8n$o9p@q0r*s1t&u2v3w4x5y6z", + "field83": "h3@i4j#k5l$m6n*o7p&q8r9s0t1u2v", + "field84": "d9$e0f#g1h@i2j*k3l&m4n5o6p7q8r", + "field85": "z5#a6b$c7d@e8f*g9h&i0j1k2l3m4n", + "field86": "v1@w2x#y3z$a4b*c5d&e6f7g8h9i0j", + "field87": "r7#s8t$u9v@w0x*y1z&a2b3c4d5e6f", + "field88": "n3@o4p#q5r$s6t*u7v&w8x9y0z1a2b", + "field89": "j9$k0l#m1n@o2p*q3r&s4t5u6v7w8x", + "field90": "f5#g6h$i7j@k8l*m9n&o0p1q2r3s4t", + "field91": "b1@c2d#e3f$g4h*i5j&k6l7m8n9o0p", + "field92": "x7#y8z$a9b@c0d*e1f&g2h3i4j5k6l", + "field93": "t3@u4v#w5x$y6z*a7b&c8d9e0f1g2h", + "field94": "p9$q0r#s1t@u2v*w3x&y4z5a6b7c8d", + "field95": "l5#m6n$o7p@q8r*s9t&u0v1w2x3y4z", + "field96": "h1@i2j#k3l$m4n*o5p&q6r7s8t9u0v", + "field97": "d7$e8f#g9h@i0j*k1l&m2n3o4p5q6r", + "field98": "z3#a4b$c5d@e6f*g7h&i8j9k0l1m2n", + "field99": "v9@w0x#y1z$a2b*c3d&e4f5g6h7i8j", + "field100": "r5#s6t$u7v@w8x*y9z&a0b1c2d3e4f", + "field101": "n1@o2p#q3r$s4t*u5v&w6x7y8z9a0b", + "field102": "j7$k8l#m9n@o0p*q1r&s2t3u4v5w6x", + "field103": "f3#g4h$i5j@k6l*m7n&o8p9q0r1s2t", + "field104": "b9@c0d#e1f$g2h*i3j&k4l5m6n7o8p", + "field105": "x5$y6z#a7b@c8d*e9f&g0h1i2j3k4l", + "field106": "t1#u2v$w3x@y4z*a5b&c6d7e8f9g0h", + "field107": "p7@q8r#s9t$u0v*w1x&y2z3a4b5c6d", + "field108": "l3#m4n$o5p@q6r*s7t&u8v9w0x1y2z", + "field109": "h9$i0j#k1l@m2n*o3p&q4r5s6t7u8v", + "field110": "d5@e6f#g7h$i8j*k9l&m0n1o2p3q4r", + "field111": "z1#a2b$c3d@e4f*g5h&i6j7k8l9m0n", + "field112": "v7$w8x#y9z@a0b*c1d&e2f3g4h5i6j", + "field113": "r3@s4t#u5v$w6x*y7z&a8b9c0d1e2f", + "field114": "n9#o0p$q1r@s2t*u3v&w4x5y6z7a8b", + "field115": "j5$k6l#m7n@o8p*q9r&s0t1u2v3w4x", + "field116": "f1@g2h#i3j$k4l*m5n&o6p7q8r9s0t", + "field117": "b7#c8d$e9f@g0h*i1j&k2l3m4n5o6p", + "field118": "x3@y4z#a5b$c6d*e7f&g8h9i0j1k2l", + "field119": "t9$u0v#w1x@y2z*a3b&c4d5e6f7g8h", + "field120": "p5#q6r$s7t@u8v*w9x&y0z1a2b3c4d", + "field121": "l1@m2n#o3p$q4r*s5t&u6v7w8x9y0z", + "field122": "h7$i8j#k9l@m0n*o1p&q2r3s4t5u6v", + "field123": "d3@e4f#g5h$i6j*k7l&m8n9o0p1q2r", + "field124": "z9#a0b$c1d@e2f*g3h&i4j5k6l7m8n", + "field125": "v5$w6x#y7z@a8b*c9d&e0f1g2h3i4j", + "field126": 42, + "field127": 789, + "field128": 156, + "field129": 923, + "field130": 347, + "field131": 681, + "field132": 294, + "field133": 835, + "field134": 167, + "field135": 459, + "field136": 672, + "field137": 381, + "field138": 928, + "field139": 514, + "field140": 760, + "field141": 293, + "field142": 846, + "field143": 107, + "field144": 582, + "field145": 734, + "field146": 395, + "field147": 861, + "field148": 248, + "field149": 657, + "field150": 419, + "field151": 703, + "field152": 186, + "field153": 542, + "field154": 698, + "field155": 375, + "field156": 829, + "field157": 164, + "field158": 517, + "field159": 743, + "field160": 286, + "field161": 904, + "field162": 358, + "field163": 621, + "field164": 795, + "field165": 432, + "field166": 876, + "field167": 189, + "field168": 653, + "field169": 407, + "field170": 728, + "field171": 564, + "field172": 391, + "field173": 817, + "field174": 275, + "field175": 938, + "field176": 102, + "field177": 586, + "field178": 749, + "field179": 423, + "field180": 867, + "field181": 234, + "field182": 695, + "field183": 458, + "field184": 701, + "field185": 316, + "field186": 872, + "field187": 539, + "field188": 684, + "field189": 297, + "field190": 815, + "field191": 463, + "field192": 728, + "field193": 156, + "field194": 934, + "field195": 507, + "field196": 682, + "field197": 349, + "field198": 876, + "field199": 214, + "field200": 758, + "field201": 436, + "field202": 691, + "field203": 325, + "field204": 809, + "field205": 582, + "field206": 147, + "field207": 763, + "field208": 498, + "field209": 625, + "field210": 384, + "field211": 917, + "field212": 256, + "field213": 743, + "field214": 469, + "field215": 836, + "field216": 172, + "field217": 594, + "field218": 721, + "field219": 358, + "field220": 907, + "field221": 283, + "field222": 649, + "field223": 416, + "field224": 785, + "field225": 532, + "field226": 698, + "field227": 245, + "field228": 871, + "field229": 409, + "field230": 756, + "field231": 183, + "field232": 627, + "field233": 394, + "field234": 819, + "field235": 568, + "field236": 742, + "field237": 125, + "field238": 896, + "field239": 453, + "field240": 607, + "field241": 784, + "field242": 329, + "field243": 856, + "field244": 291, + "field245": 673, + "field246": 418, + "field247": 905, + "field248": 562, + "field249": 739, + "field250": 864 +} \ No newline at end of file diff --git a/source/benchmarking/odm-data/flat-models/small_doc.json b/source/benchmarking/odm-data/flat-models/small_doc.json new file mode 100644 index 0000000000..d0727271fd --- /dev/null +++ b/source/benchmarking/odm-data/flat-models/small_doc.json @@ -0,0 +1 @@ +{"field1":"miNVpaKW","field2":"CS5VwrwN","field3":"Oq5Csk1w","field4":"ZPm57dhu","field5":"gxUpzIjg","field6":"Smo9whci","field7":"TW34kfzq","field8":55336395,"field9":41992681,"field10":72188733,"field11":46660880,"field12":3527055,"field13":74094448} \ No newline at end of file diff --git a/source/benchmarking/odm-data/flat_models.tgz b/source/benchmarking/odm-data/flat_models.tgz new file mode 100644 index 0000000000000000000000000000000000000000..9c550f40e771d255026da7cd18086fd1766e689a GIT binary patch literal 3190 zcmXw0c{~&RA4iVxM2OWE3cprD7dFF^V-_KgNzJj)6S>0X9+8|&Dv8`zY;!&C+?6@f z0E|CR^P`dNF3LSi>{EN``_>T82*zt@r)uPYy&IRHeLr*M-Y&GDW z-ho$!zAx2>T&c%s?@bUe463S$=R)LupxGmkdwX-zc$|0K5x75fGZpdX+b#gIkYp5n z@XRdglMJbd?N`4y3*DXB+D)9X`NE!9Jm`4Xy?J#lVThSo(Y3nqxu9!%CP-kt1yen3 zxRrYM>QW6NyxEYW($$mn(d=NqXY0UI^HBq)Ymh~)*_)~vcLzV(-d`{Dy5TsV+FZA2 zx-(Eiy%b({b#V-0mJYLRS8Cg5C($E@C;g?d$AgaVH)$VuZFx$s9N%$mzFyF}%7#f@ zT(4P-V@=P{e3a-Rth7sX1-}#5YcSS{FW1d&5h(1APp@Lz-G6)zyZ$ZH8B z+xu=V;h5A>w}hf9Hs*lUc%ZzyxrH%V7u{&y-l9Smrl(dGcV|`(3U@;`m#3%0`Nqb4 zKP&7^C{+jg#qMkexQ`kn-!fdo?F>aB)8z#Q@>gtDZK0oNw@Rn5jT^@G#C#?rBSnw^ z(FjS^FBKS$zy@8`)GuWMa>Kt1#J+P}9)#0-SNv7f?UairRz4GlI=cL@7(2i>v zT_jptdcMw7s=f7s>Pd)*%oDGyc!GbzLJk_i@f=7K3L${z_) zhDNQir+t^xOl3qN(IP={&Kq919MseKmQ&sAS4Ua1!|UzhJt0vj+-N|jmd6j3(_{0J zZ2S~PQJK=zA_l7i{vSWg~UUNz(^Ajlip zHedwmw3T%fGu7WY3p07Dl(BGhVMB1@i29jh6Hh9Iy>B8B5()%Q!}^T;u0Fs71-Ilr ze(HozKeFvP56{Ob)v$A-;QU4jy^r0h;D`d(kq)Kgi*bh5Pb3DrJK;PQFtx*0Pd=R< zGi&KPHZkk1TGtzo`cZQ2ogCSN8D9rQ>Q_-6mXK#G++$oSAOP}@T2MUkW%yj{C~D2$ zG_|^hhO}%#SV|1RW4gI4@<)E{RXVT9zk!fka$rxYt$qXOgzOL9_sA?)l$tp!ZC>JH z<8==0L$&R^A$eJioI+ej{K5U{snePyIg;p$7NJ&bD_ZKI1cug2K%`D85tuUq$*Fk1 zI(*ty;n0a(GtzA>a5!zl(D>UZ`)Jzl;#H^-6>#wg@F< z2lhE#Df9Ce1uw?zj&%ssAC32jp}Sjd*xR~Ksn$1#xx=3$tT42Bk&8BcIRKSBeK$ctBjN9SzA~ z?B9kPe<+J;r6I?MeVO+s={-O6ZV$#E<c}AB*=^auq9<-lJx1t7A<%lpD4KQ{5#tUDG-@w2N+;wxbyUo+LNS?OMG(m zo|AnSyvA==1=~s5uSH9z40cQ28|-SZ`tbUGkS| zRRXA4vYty!Kl;=)-mWYa^7jdSAJT}a9`Ai+h?Jb?(Xgve9Pw>;rsNLm`^ZrAC?yJq z--U%r@)mhXlm|sU&`KlH-}t&@pAhKscjoWzZMEcqMLf{LN5z>`_T`Z76q~adZ{mYT(PPH}~^HyYB);WqZhvN9Xw}<+vN+8flh~M05lmO*yHF z4_O}VjU9U)tm7agKG5Si@WCD^3MLB==$XdrDR7@M?!U_H=N8mr` zw!t-rZ4pVg>3{2a@Kct9jpr68=>3Y8MpQ|7a*MaJpC_>K%JXDxXjXWicg+8CV17L4 z7rjDSvf{?iZ&z6AIQl%nd#z})9wiS?hXyLCsI z$K}O)yIe&}fS*~i6-DCfU(cTtLT%8VrG7$+AeOdOgdF|p+uJ%9mRWxm0d<3t%J1^> z%3o_jNEfQ`XU^%0U0wfE?l5#I~Lr@na4F^QZWqgx)6?#uWVc>DFw%5%OXLjx7 zV{H3*t@8W4s`8jcEtHpB0>}1TM+;t*CNA2DIH8*|-kS!b# zHMt_rPOnu0aStPxO8bB)Kx)rWb~L!j>3>-uL0aWE%vETI`|6gB3@6LZNd=sIR;twQ z9O*F~k=wKO6Bsn03*o7I@ePv;AgP#U3hRqqRX4ltMy%DR4fdvn(Gi6`k~z_`^H;(t9&aZdWdA$~Kj7`~4Im|%lLXc3m(B5_3QgAF|DK)6aRG*u zNN`+iNkyrydg$EK#VXG-g$=~a@PN~E2pvdmMPc8tCE#;UVh$(Al>{Go4tHM-UC*4{ zBG0piHocf>yg2#n)UnYa28&FXok5#Z>dQ7>C(-MJ8#ivP26Vc>UkXMhZ{Cc|OPkSq zqt_q#$hgbe-R+xR0A!`vr!2d{9(D)*YC{3o9<^Macp)Rv>b`Ujq3Pr1`-*}K%K3Q5 zJe0g_=dfmLLRP8V2M8W%PR7Tr`pBc3Bv$zjHMK=~ud6IKFpTvtwt}$hTbcaBH@o41 z8QQ^Wy|10jF#HWu5A#NwE%8`Jq%1H)oA%A}`|!XKnj}va=$+Ya9Iiz@tiTAuAQB+_ zkK3q7$lZci6n6asN0Psg(7=dH0q!f+MF87Za#HiD_-2OsiCEjWpjg%uUI2|^Qzn5My&S_4Cf{}wp{3x;`fyi_^8FjYpV)kTN7^N= zku}M8w;nU{9S9c2H7{N4Bw-`CPW?g@z#~tDf-GhDZwwY zz=&2CZy+AiEpL`sw?Oo_8@yxPLq*cC*P~(I`}>K)rmf|q0Uthh=KWx=YLcT`E5gDG z%_P9gy0i@~uqQU9T$r-OY`pVWP%Ec4geLq|MZC$R-hv~jf3W1-LSG83{QZWp#lT+{ zF8~3iY=letAK*Narjbow3h-1H1Z@#-0S(PpZ{yVKqgeJ;&;@bK>REnB%Ue0Yer*V! ziLIW!bP6z>(&s|ngDp1@8-kwQvf!)X&|`7Ycq~ywb3r4;$bF(P#ih8qm13@;6{H;~ zV)pTA=fw4!#g+!^DA&em*F3aq>|PEUWXbmLq_3bzU-<3Lst%zoogcfn-!QOyr42DR zsrmv4GOJQQ=8%JS^^fR7xjL4)k*n&U(kR#9vvj2Uvq|9`bSme+htM-yK#s)1jgh!# d51My(n`!v{;9qM`D&&BltLw!ymkJlx{{ed2O0xg} literal 0 HcmV?d00001 diff --git a/source/benchmarking/odm-data/nested-models/large_doc_nested.json b/source/benchmarking/odm-data/nested-models/large_doc_nested.json new file mode 100644 index 0000000000..497a6e2551 --- /dev/null +++ b/source/benchmarking/odm-data/nested-models/large_doc_nested.json @@ -0,0 +1,242 @@ +{ + "embedded_str_doc_1": { + "field1": "kj9$mxz#p2qw8r*vn4@h7c&u1s", + "field2": "x3@9zf#mk7w$qp2n8v*r6j4h&c1u5s0a", + "field3": "p9#m2x$k7z@w3q8v*n4r&j6h1c5u0s", + "field4": "z8@x3m#k9w$p2q7v*r4n6j&h1c5u0s", + "field5": "m7#k9x$z3w@p8q2v*n4r6j&h1c5u0s", + "field6": "k6$x9m#z7w3p@q8v2n*r4j6h&c1u5s0", + "field7": "x5@m9k#z6w$p3q7v*n2r8j&h4c1u0s", + "field8": "m4#x8k$z9w6p@q3v*n7r2j&h5c1u0s", + "field9": "k3$m7x#z8w9p@q6v*n2r4j&h1c5u0s", + "field10": "x2@k6m#z7w8p$q9v*n3r1j&h4c5u0s", + "field11": "m1#x5k$z6w7p@q8v*n9r2j&h3c4u0s", + "field12": "k0$x4m#z5w6p@q7v*n8r1j&h2c3u9s", + "field13": "x9@m3k#z4w5p$q6v*n7r0j&h1c2u8s", + "field14": "m8#k2x$z3w4p@q5v*n6r9j&h0c1u7s", + "field15": "k7$x1m#z2w3p@q4v*n5r8j&h9c0u6s" + }, + "embedded_str_doc_2": { + "field1": "x6@m0k#z1w2p$q3v*n4r7j&h8c9u5s", + "field2": "m5#x9k$z0w1p@q2v*n3r6j&h7c8u4s", + "field3": "k4$m8x#z9w0p@q1v*n2r5j&h6c7u3s", + "field4": "x3@k7m#z8w9p$q0v*n1r4j&h5c6u2s", + "field5": "m2#x6k$z7w8p@q9v*n0r3j&h4c5u1s", + "field6": "k1$x5m#z6w7p@q8v*n9r2j&h3c4u0s", + "field7": "x0@m4k#z5w6p$q7v*n8r1j&h2c3u9s", + "field8": "m9#k3x$z4w5p@q6v*n7r0j&h1c2u8s", + "field9": "k8$x2m#z3w4p@q5v*n6r9j&h0c1u7s", + "field10": "x7@m1k#z2w3p$q4v*n5r8j&h9c0u6s", + "field11": "m6#x0k$z1w2p@q3v*n4r7j&h8c9u5s", + "field12": "k5$m9x#z0w1p@q2v*n3r6j&h7c8u4s", + "field13": "x4@k8m#z9w0p$q1v*n2r5j&h6c7u3s", + "field14": "m3#x7k$z8w9p@q0v*n1r4j&h5c6u2s", + "field15": "k2$x6m#z7w8p@q9v*n0r3j&h4c5u1s" + }, + "embedded_str_doc_3": { + "field1": "k9$x3m#z4w5p@q6v*n7r0j&h1c2u8s", + "field2": "x8@m2k#z3w4p$q5v*n6r9j&h0c1u7s", + "field3": "m7#x1k$z2w3p@q4v*n5r8j&h9c0u6s", + "field4": "k6$m0x#z1w2p@q3v*n4r7j&h8c9u5s", + "field5": "x5@k9m#z0w1p$q2v*n3r6j&h7c8u4s", + "field6": "m4#x8k$z9w0p@q1v*n2r5j&h6c7u3s", + "field7": "k3$x7m#z8w9p@q0v*n1r4j&h5c6u2s", + "field8": "x2@m6k#z7w8p$q9v*n0r3j&h4c5u1s", + "field9": "m1#k5x$z6w7p@q8v*n9r2j&h3c4u0s", + "field10": "k0$x4m#z5w6p@q7v*n8r1j&h2c3u9s", + "field11": "x9@m3k#z4w5p$q6v*n7r0j&h1c2u8s", + "field12": "m8#x2k$z3w4p@q5v*n6r9j&h0c1u7s", + "field13": "k7$m1x#z2w3p@q4v*n5r8j&h9c0u6s", + "field14": "x6@k0m#z1w2p$q3v*n4r7j&h8c9u5s", + "field15": "m5#x9k$z0w1p@q2v*n3r6j&h7c8u4s" + }, + "embedded_str_doc_4": { + "field1": "k1$x5m#z6w7p@q8v*n9r2j&h3c4u0s9f8d", + "field2": "x0@m4k#z5w6p$q7v*n8r1j&h2c3u9s7g6e", + "field3": "m9#k3x$z4w5p@q6v*n7r0j&h1c2u8s5b4a", + "field4": "k8$x2m#z3w4p@q5v*n6r9j&h0c1u7s3v2t", + "field5": "x7@m1k#z2w3p$q4v*n5r8j&h9c0u6s1y0w", + "field6": "m6#x0k$z1w2p@q3v*n4r7j&h8c9u5s9i8o", + "field7": "k5$m9x#z0w1p@q2v*n3r6j&h7c8u4s7l6p", + "field8": "x4@k8m#z9w0p$q1v*n2r5j&h6c7u3s5m4n", + "field9": "m3#x7k$z8w9p@q0v*n1r4j&h5c6u2s3q2r", + "field10": "k2$x6m#z7w8p@q9v*n0r3j&h4c5u1s1x0z", + "field11": "p7#q8r$s9t@u0v*w1x&y2z3a4b5c6d", + "field12": "f3$g4h#i5j@k6l*m7n&o8p9q0r1s2t", + "field13": "v5@w6x#y7z$a8b*c9d&e0f1g2h3i4j", + "field14": "n2#o3p$q4r@s5t*u6v&w7x8y9z0a1b", + "field15": "j8$k9l#m0n@o1p*q2r&s3t4u5v6w7x" + }, + "embedded_str_doc_5": { + "field1": "d1@e2f#g3h$i4j*k5l&m6n7o8p9q0r", + "field2": "z7$a8b#c9d@e0f*g1h&i2j3k4l5m6n", + "field3": "v3@w4x#y5z$a6b*c7d&e8f9g0h1i2j", + "field4": "r9#s0t$u1v@w2x*y3z&a4b5c6d7e8f", + "field5": "n5$o6p#q7r@s8t*u9v&w0x1y2z3a4b", + "field6": "j1@k2l#m3n$o4p*q5r&s6t7u8v9w0x", + "field7": "f7#g8h$i9j@k0l*m1n&o2p3q4r5s6t", + "field8": "b3@c4d#e5f$g6h*i7j&k8l9m0n1o2p", + "field9": "x9$y0z#a1b@c2d*e3f&g4h5i6j7k8l", + "field10": "t5#u6v$w7x@y8z*a9b&c0d1e2f3g4h", + "field11": "p1@q2r#s3t$u4v*w5x&y6z7a8b9c0d", + "field12": "l7#m8n$o9p@q0r*s1t&u2v3w4x5y6z", + "field13": "h3@i4j#k5l$m6n*o7p&q8r9s0t1u2v", + "field14": "d9$e0f#g1h@i2j*k3l&m4n5o6p7q8r", + "field15": "z5#a6b$c7d@e8f*g9h&i0j1k2l3m4n" + }, + "embedded_str_doc_array": [ + { + "field1": "n3@o4p#q5r$s6t*u7v&w8x9y0z1a2b", + "field2": "j9$k0l#m1n@o2p*q3r&s4t5u6v7w8x", + "field3": "f5#g6h$i7j@k8l*m9n&o0p1q2r3s4t", + "field4": "b1@c2d#e3f$g4h*i5j&k6l7m8n9o0p", + "field5": "x7#y8z$a9b@c0d*e1f&g2h3i4j5k6l", + "field6": "t3@u4v#w5x$y6z*a7b&c8d9e0f1g2h", + "field7": "p9$q0r#s1t@u2v*w3x&y4z5a6b7c8d", + "field8": "l5#m6n$o7p@q8r*s9t&u0v1w2x3y4z", + "field9": "h1@i2j#k3l$m4n*o5p&q6r7s8t9u0v", + "field10": "d7$e8f#g9h@i0j*k1l&m2n3o4p5q6r", + "field11": "z3#a4b$c5d@e6f*g7h&i8j9k0l1m2n", + "field12": "v9@w0x#y1z$a2b*c3d&e4f5g6h7i8j", + "field13": "r5#s6t$u7v@w8x*y9z&a0b1c2d3e4f", + "field14": "n1@o2p#q3r$s4t*u5v&w6x7y8z9a0b", + "field15": "j7$k8l#m9n@o0p*q1r&s2t3u4v5w6x" + }, + { + "field1": "t1#u2v$w3x@y4z*a5b&c6d7e8f9g0h", + "field2": "p7@q8r#s9t$u0v*w1x&y2z3a4b5c6d", + "field3": "l3#m4n$o5p@q6r*s7t&u8v9w0x1y2z", + "field4": "h9$i0j#k1l@m2n*o3p&q4r5s6t7u8v", + "field5": "d5@e6f#g7h$i8j*k9l&m0n1o2p3q4r", + "field6": "z1#a2b$c3d@e4f*g5h&i6j7k8l9m0n", + "field7": "v7$w8x#y9z@a0b*c1d&e2f3g4h5i6j", + "field8": "r3@s4t#u5v$w6x*y7z&a8b9c0d1e2f", + "field9": "n9#o0p$q1r@s2t*u3v&w4x5y6z7a8b", + "field10": "j5$k6l#m7n@o8p*q9r&s0t1u2v3w4x", + "field11": "f1@g2h#i3j$k4l*m5n&o6p7q8r9s0t", + "field12": "b7#c8d$e9f@g0h*i1j&k2l3m4n5o6p", + "field13": "x3@y4z#a5b$c6d*e7f&g8h9i0j1k2l", + "field14": "t9$u0v#w1x@y2z*a3b&c4d5e6f7g8h", + "field15": "p5#q6r$s7t@u8v*w9x&y0z1a2b3c4d" + } + ], + "embedded_int_doc_8": { + "field1": 42, + "field2": 789, + "field3": 156, + "field4": 923, + "field5": 347, + "field6": 681, + "field7": 294, + "field8": 835, + "field9": 167, + "field10": 459, + "field11": 672, + "field12": 381, + "field13": 928, + "field14": 514, + "field15": 760 + }, + "embedded_int_doc_9": { + "field1": 846, + "field2": 107, + "field3": 582, + "field4": 734, + "field5": 395, + "field6": 861, + "field7": 248, + "field8": 657, + "field9": 419, + "field10": 703, + "field11": 186, + "field12": 542, + "field13": 698, + "field14": 375, + "field15": 829 + }, + "embedded_int_doc_10": { + "field1": 904, + "field2": 358, + "field3": 621, + "field4": 795, + "field5": 432, + "field6": 876, + "field7": 189, + "field8": 653, + "field9": 407, + "field10": 728, + "field11": 564, + "field12": 391, + "field13": 817, + "field14": 275, + "field15": 938 + }, + "embedded_int_doc_11": { + "field1": 867, + "field2": 234, + "field3": 695, + "field4": 458, + "field5": 701, + "field6": 316, + "field7": 872, + "field8": 539, + "field9": 684, + "field10": 297, + "field11": 815, + "field12": 463, + "field13": 728, + "field14": 156, + "field15": 934 + }, + "embedded_int_doc_12": { + "field1": 214, + "field2": 758, + "field3": 436, + "field4": 691, + "field5": 325, + "field6": 809, + "field7": 582, + "field8": 147, + "field9": 763, + "field10": 498, + "field11": 625, + "field12": 384, + "field13": 917, + "field14": 256, + "field15": 743 + }, + "embedded_int_doc_13": { + "field1": 721, + "field2": 358, + "field3": 907, + "field4": 283, + "field5": 649, + "field6": 416, + "field7": 785, + "field8": 532, + "field9": 698, + "field10": 245, + "field11": 871, + "field12": 409, + "field13": 756, + "field14": 183, + "field15": 627 + }, + "embedded_int_doc_14": { + "field1": 896, + "field2": 453, + "field3": 607, + "field4": 784, + "field5": 329, + "field6": 856, + "field7": 291, + "field8": 673, + "field9": 418, + "field10": 905, + "field11": 562, + "field12": 739, + "field13": 864, + "field14": 285, + "field15": 928 + } +} diff --git a/source/benchmarking/odm-data/nested_models.tgz b/source/benchmarking/odm-data/nested_models.tgz new file mode 100644 index 0000000000000000000000000000000000000000..390c66ff21083f76e6fb89196423315b25250d41 GIT binary patch literal 2475 zcmV;c2~_qUiwFRZeV=Fm1MQjFjw81jg?;U(XbcHm6r6olsh;iPbY9H=)a>Sq&FL>c{2>ox%xe7-KaGi> zOsM{IcwV?>${K655HE}?$=MGt`j1)8__>_U(<7F$#pizicpQGbTt0cvu#aCq|3{wf z$p6js__p|YzMK8r|NWoWr``5no&$Oqr5^dujg|6O|CQn|OmzSK!wdU2?C+b;|GEGF z`0?fC%c6N*%;$^w&!_V-y3fD-^`-j>eztsHZ06nnf2r4=Htn74#o^|T)pe`F%FZSi ze){s$Z^nvlY%7DmFG*9|8$ImB)?KTkS*z7##xMHBrlShkRqQ=!L`!RX4>x(h_^mol z)@H?L`ohLDsBZk-g;qAC_BV_rx!M_bjl z=^3C0hVeWypl9YHO3(|^8my;7AVdV%Q4R?367-z5x*?w-pl6@M^Pbca zc6R=U=l=K1JK74Y=QH;A%-hiLy0@R^_7|8pq-9X3b2vPAcwk=B#>28tSUfPVLhYK~ zWBT0Vfq6+??S!QB}|pBrSxIsDBYv_lTJUwiNy_P8O28&m%_h9CBzA5ys1 z?dMXM{oWMjzc+=&kixB~KbOLCNa2RJ4^ud29zk2P=2Jo2GyjdGeayTnGXFI9{L;+} z+&|4e`(_pe_MhgTepS-~|4%beUd6f415aa+f3RDj51uBWe|Nh=FFXxHyD|GhKRnGt z-KcG$C!R*4JcwhdFP_FCZ`r-j8+%I*?nqDm94?sW#@oqW_7JGb`-cI2nTn!Y!#I^0Z4KJ1-N-D@@T^T~oO`I}hDceO6`##WGBwDA~D z`dnSib#k+<`|x)*<*$X_SUXz#4Qbdm?D)QdI+K$;tBbyZ@AgS==u&UYd02=gd6O%O z^;NarOd7Mb!zS~-xZ7@D1p5luSM|nMlXtO}wc2RZhgX#dU9fCdg0e_6K(Y&?~}nDi@LlQwwmtDo*XRXbCAzNKGyQV)w~bZJk$c}$t~R} zsHdTxIopf7g4j0ifu$vH4%&DCzyJUT00Q3t)KQ}@=l@lPnVOS@UeY(Ss@~CKwcB{O z#Zi~D-+KDN?gUncnV44#xtstv{chG4bp`Ivnt%}stHX!8S5yBwnXx&Ci4uh+<~e%z zNWciaAS-Sf%$vJ~i^z)-^BYSV2YF&M9;*{SPcGt$fNC`g3(T)%06RF>HWUn0JG-Bt z%RGGKC@e5P_Z09Dz!LyLRZBppwi-rS6c(7jYXW*G=n0^pdh?);tvQ^OU7|c@etJAk zKk$aX#aH(9kNDrp01gN^pzyQ0Sn%Uo4^Q}1eC21<0r;K_5(r2LaI`u|5Two;rXY&i zq$29G)C4S0uz+j?LmmvVJ%=k2l}SYu=#_Ug5YW)YPlfoIAbu=t@u*BHhBJWx2MQbl zT&)%y9QB!_QJGZ4PB_Z|9te1#u(g_6faKSfuHg($ zgn3dCyJ4dVh@iW8?gAwbl-Lz#P%BZHRK$K*@eT=`qhLvaB@LF$(W19KDwB%7Bj%O@ z3IZqspr~pNn#5MZR*llMB)Wc=1nEF$8ax^BWWkeLd-%#xnwCV{uRb6k$OnhW0zoDb zWNN8ltVL;B5`8~v0;VXK3Lv9Gq)(>o6}N%8L}^;mp9Z{xiU2AKG%AGp1jmVmwH~Eu zNq-(#3aSXG3Sgr;aPTP365itO63>I7JBs|DIf0iHUIsW)O*JU!v$o6V690SKc}d_U zeSC(F_}@qZFX=Acia>z{1%2LjIbGs^3Yee8C5UE=>J18gL) zk;01#w}Q!#yIo+H_}_Z6i#I2nX95670MG|-m)J%A*P5cm3GOBV$98q_0HBZFF0+gL zFL^+ZlXtnMIJv7vBa!;>?NYnQ|Encva=P%#05Ga|T!x4E?Q*-we=9qt2$-TEqgq&G zl3V#9xr_XteFqf*Q~_vIQwbc^T*F$6(jxzRO+Xa|RRL^NHxC~B^v3;uKJbq}|2{q; z-nZv?Lb$@MNQv~`W1Ua0He8$Z_Tq(1FDO#k^bTSiPp=umtMn$}q)smj+>G88-B)_F zuxPKbqxDDs8P;BuZ=z%TL&^`uT!>PMK6d&TJ~6b|MwfN|g!dbt<4U zxq!;00vc^o0TuU|fR@QjK<-Q?pdLdJ1vEYvP+FY{=!AbPU|dD2pl2#oP->SdXavs` z9Lv?suB6Bmv?f)Mj|uM;l$nAoN6;o1b?etglah)lsjLZ~F)mM2m4GO9! z3YLC9E_lYE%^6fOi$OCsI(D(hyO{Y@!6Xqn1-Z()*d{8(81y-VM&t_0MEw|pB?b!$ zS|y(-So&?N9rx#hF81S|iqpI7ATi3P3aXqz>vYbb$QVr2iVP~9GiZ6v;MlgF!9=ad z;CKsBK_l!F1xvrXb3Qvs)Hs!57v~CEm!BjeQ_$&5K{19m=Ag}Ukmm||mMf@DeiU1o pE9jEE>>()Rpols6<1at{r~hgB`}6(z{`{9e{{)-DbtC{L005ZI*R}uv literal 0 HcmV?d00001 From c1207dd87cfaa4faf6030ce7d487fe3ffa3c1d8f Mon Sep 17 00:00:00 2001 From: Noah Stapp Date: Mon, 18 Aug 2025 08:30:46 -0400 Subject: [PATCH 6/8] More specific wording and benchmark phases --- source/benchmarking/odm-benchmarking.md | 100 +++++++++++++----------- 1 file changed, 54 insertions(+), 46 deletions(-) diff --git a/source/benchmarking/odm-benchmarking.md b/source/benchmarking/odm-benchmarking.md index 27b6c8676e..197d04f223 100644 --- a/source/benchmarking/odm-benchmarking.md +++ b/source/benchmarking/odm-benchmarking.md @@ -5,8 +5,8 @@ ## Abstract -This document describes a standard benchmarking suite for MongoDB ODMs (Object Document Mappers). Much of the structure -is taken from the existing MongoDB driver benchmarking suite for consistency and readability. +This document describes a standard benchmarking suite for MongoDB ODMs (Object Document Mappers). Much of this +document's structure and content is taken from the existing MongoDB driver benchmarking suite for consistency. ## Overview @@ -30,21 +30,27 @@ within ODMs that are faster or slower. - Nested models -- reading and writing nested models of various sizes, to explore basic operation efficiency for complex data +The suite is intentionally kept small for several reasons. One, ODM feature sets vary significantly across libraries. +This limits the number of benchmarks that can be run across the entire collection of extant ODMs. Two, several popular +MongoDB ODMs are actively maintained by third-parties, such as Mongoose. By limiting the benchmarking suite to a minimal +set of representative tests that are easy to implement, we encourage adoption of the suite by these third-party +maintainers. + ### Measurement In addition to latency data, all benchmark tasks will be measured in terms of "megabytes/second" (MB/s) of documents processed, with higher scores being better. (In this document, "megabyte" refers to the SI decimal unit, i.e. 1,000,000 bytes.) This makes cross-benchmark comparisons easier. -To avoid various types of measurement skew, tasks will be measured over numerous iterations. Each iteration will have a +To avoid various types of measurement skew, tasks will be measured over several iterations. Each iteration will have a number of operations performed per iteration that depends on the task being benchmarked. The final score for a task will be the median score of the iterations. A range of percentiles will also be recorded for diagnostic analysis. ### Data sets -Data sets will vary by task. In most cases, data sets will be synthetic line-delimited JSON files or mock binary files -to be constructed by the ODM into the appropriate model. Some tasks will require additional modifications to these -constructed models. +Data sets will vary by task. In most cases, data sets will be synthetic line-delimited JSON files to be constructed by +the ODM being benchmarked into the appropriate model. Some tasks will require additional modifications to these +constructed models, such as adding generated ObjectIds. ### Versioning @@ -69,10 +75,10 @@ extreme forms of measurement: Therefore, we choose a middle ground: -- measuring the same 10000 insertions over 10 iterations -- each timing run includes enough operations that insertion +- measuring the same 10,000 insertions over 10 iterations -- each timing run includes enough operations that insertion code dominates timing code; unusual system events are likely to affect only a fraction of the 10 timing measurements -With 10 timings of inserting the same 10000 models, we build up a statistical distribution of the operation timing, +With 10 timings of inserting the same 10,000 models, we build up a statistical distribution of the operation timing, allowing a more robust estimate of performance than a single measurement. (In practice, the number of iterations could exceed 10, but 10 is a reasonable minimum goal.) @@ -87,8 +93,8 @@ given in a subsequent section: insert data into a collection, etc. - before operation -- (ONCE PER ITERATION) something to do before every task iteration, e.g. drop a collection, or reload test data (if the test run modifies it), etc. -- do operation -- (ONCE PER ITERATION) smallest amount of code necessary to execute the task; e.g. insert 10000 models - one by one into the database, or retrieve 10000 models of test data from the database, etc. +- do operation -- (ONCE PER ITERATION) smallest amount of code necessary to execute the task; e.g. insert 10,000 models + one by one into the database, or retrieve 10,000 models of test data from the database, etc. - after operation -- (ONCE PER ITERATION) something to do after every task iteration (if necessary) - teardown -- (ONCE PER TASK) something done once after all benchmarking is complete (if necessary); e.g. drop the test database @@ -99,8 +105,8 @@ monotonic timer (or best language approximation). Unless otherwise specified, the number of iterations to measure per task is variable: -- iterations should loop for at least 1 minute cumulative execution time -- iterations should stop after 10 iterations or 5 minutes cumulative execution time, whichever is shorter +- iterations should loop for at least 30 seconds cumulative execution time +- iterations should stop after 10 iterations or 1 minute cumulative execution time, whichever is shorter This balances measurement stability with a timing cap to ensure all tasks can complete in a reasonable time. @@ -117,8 +123,9 @@ its utter simplicity given that it needs to be implemented identically across a The 50th percentile (i.e. the median) will be used for score composition. Other percentiles will be stored for visualizations and analysis. -Each task will have defined for it an associated size in megabytes (MB). The benchmarking score for each task will be -the task size in MB divided by the median wall clock time. +Each task will have defined for it an associated size in megabytes (MB). This size will be calculated using the task's +dataset size and the number of documents processed per iteration. The benchmarking score for each task will be the task +size in MB divided by the median wall clock time. ## Benchmark task definitions @@ -136,7 +143,7 @@ Flat model tests focus on flatly-structured model reads and writes across data s into the efficiency of the ODM's implementation of basic data operations. The data will be stored as strict JSON with no extended types. These JSON representations must be converted into -equivalent models as part of the benchmark's setup. +equivalent models as part of each benchmark task. Flat model benchmark tasks include:s @@ -157,15 +164,15 @@ JSON with an encoded length of approximately 250 bytes. Dataset size: For scoring purposes, the dataset size is the size of the `small_doc` source file (250 bytes) times 10,000 operations, which equals 2,250,000 bytes or 2.5 MB. -This benchmark uses the same dataset as the driver `small doc insertOne` benchmark, allowing for direct comparisons. +This benchmark uses a comparable dataset to the driver `small doc insertOne` benchmark, allowing for direct comparisons. -| Phase | Description | -| ----------- | -------------------------------------------------------------------------------------- | -| Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. | -| Before task | Drop the collection associated with the SMALL_DOC model. | -| Do task | Save the model to the database in an ODM-appropriate manner. Repeat this 10,000 times. | -| After task | n/a | -| Teardown | n/a. | +| Phase | Description | +| ----------- | -------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the SMALL_DOC dataset into memory. | +| Before task | n/a. | +| Do task | Create an ODM-appropriate model instance for the SMALL_DOC document and save it to the database. Repeat this 10,000 times. | +| After task | Drop the collection associated with the SMALL_DOC model. | +| Teardown | n/a. | #### Small model update @@ -219,7 +226,7 @@ operations, which equals 2,250,000 bytes or 2.5 MB. | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. For each SMALL_DOC model, create and assign a FOREIGN_KEY instance to the `field_fk` field. Insert 10,000 instances of both models into the database, saving the inserted `id` field for each FOREIGN_KEY into a list. | | Before task | n/a. | -| Do task | For each of the 10,000 FOREIGN_KEY `id` values, perform a filter operation to find the corresponding SMALL_DOC model. | +| Do task | For each of the 10,000 FOREIGN_KEY `id` values, perform a filter operation in an ODM-appropriate manner to find the corresponding SMALL_DOC model. | | After task | n/a. | | Teardown | Drop the collections associated with the SMALL_DOC and FOREIGN_KEY models. | @@ -233,13 +240,13 @@ JSON with an encoded length of approximately 8,000 bytes. Dataset size: For scoring purposes, the dataset size is the size of the `large_doc` source file (8,000 bytes) times 10,000 operations, which equals 80,000,000 bytes or 80 MB. -| Phase | Description | -| ----------- | ------------------------------------------------------------------------------------------------ | -| Setup | Load the LARGE_DOC dataset into memory as an ODM-appropriate model object. | -| Before task | Drop the collection associated with the LARGE_DOC model. | -| Do task | Save the LARGE_DOC model to the database in an ODM-appropriate manner. Repeat this 10,000 times. | -| After task | n/a | -| Teardown | n/a. | +| Phase | Description | +| ----------- | -------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC dataset into memory. | +| Before task | n/a. | +| Do task | Create an ODM-appropriate model instance for the LARGE_DOC document and save it to the database. Repeat this 10,000 times. | +| After task | Drop the collection associated with the LARGE_DOC model. | +| Teardown | n/a. | #### Large model update @@ -264,10 +271,10 @@ Dataset size: For scoring purposes, the dataset size is the size of the `updated Datasets are in the `nested_models` tarball. Nested model tests focus performing reads and writes on models containing nested (embedded) documents. They are designed -to give insights into the efficiency of the ODM's implementation of a core advantage of the document model. +to give insights into the efficiency of operations on the more complex data structures enabled by the document model. The data will be stored as strict JSON with no extended types. These JSON representations must be converted into -equivalent models as part of the benchmark's setup. +equivalent ODM models as part of each benchmark task. Nested model benchmark tasks include:s @@ -286,13 +293,13 @@ stored as strict JSON with an encoded length of approximately 8,000 bytes. Dataset size: For scoring purposes, the dataset size is the size of the `large_doc_nested` source file (8,000 bytes) times 10,000 operations, which equals 80,000,000 bytes or 80 MB. -| Phase | Description | -| ----------- | -------------------------------------------------------------------------------------- | -| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. | -| Before task | Drop the collection associated with the LARGE_DOC model. | -| Do task | Save the model to the database in an ODM-appropriate manner. Repeat this 10,000 times. | -| After task | n/a | -| Teardown | n/a. | +| Phase | Description | +| ----------- | --------------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC_NESTED dataset into memory. | +| Before task | n/a. | +| Do task | Create an ODM-appropriate model instance for the LARGE_DOC_NESTED document and save it to the database. Repeat this 10,000 times. | +| After task | Drop the collection associated with the LARGE_DOC_NESTED model. | +| Teardown | n/a. | #### Large model update nested @@ -308,7 +315,7 @@ Dataset size: For scoring purposes, the dataset size is the size of the `updated | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Save 10,000 instances into the database. | | Before task | n/a. | -| Do task | Update the value of the `embedded_str_doc_1.field1` field to `updated_value` for each instance of the model in an ODM-appropriate manner. | +| Do task | Update the value of the `embedded_str_doc_1.field1` field to `updated_value` in an ODM-appropriate manner for each instance of the model. | | After task | Drop the collection associated with the LARGE_DOC_NESTED model. | | Teardown | n/a. | @@ -326,7 +333,7 @@ times 10,000 operations, which equals 80,000,000 bytes or 80 MB. | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the value of the `unique_id` field for each model's `embedded_str_doc_1` nested model into a list. | | Before task | n/a. | -| Do task | For each of the 10,000 `embedded_str_doc_1.unique_id` values, perform a filter operation to search for the corresponding parent model. | +| Do task | For each of the 10,000 `embedded_str_doc_1.unique_id` values, perform a filter operation to search for the parent LARGE_DOC_NESTED model. | | After task | n/a. | | Teardown | Drop the collection associated with the LARGE_DOC_NESTED model. | @@ -344,7 +351,7 @@ times 10,000 operations, which equals 80,000,000 bytes or 80 MB. | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the value of the `unique_id` field for the first item in each model's `embedded_str_doc_array` nested model into a list. | | Before task | n/a. | -| Do task | For each of the 10,000 `unique_id` values, perform a filter operation to search for the corresponding parent model. | +| Do task | For each of the 10,000 `unique_id` values, perform a filter operation to search for the parent LARGE_DOC_NESTED model. | | After task | n/a. | | Teardown | Drop the collection associated with the LARGE_DOC_NESTED model. | @@ -357,13 +364,14 @@ supports. ### Benchmark Server -The MongoDB ODM Performance Benchmark must be run against a standalone MongoDB server running the latest database +The MongoDB ODM Performance Benchmark must be run against a standalone MongoDB server running the latest stable database version without authentication or SSL enabled. -### Benchmark Placement +### Benchmark placement and scheduling The MongoDB ODM Performance Benchmark should be placed within the ODM's test directory as an independent test suite. Due to the relatively long runtime of the benchmarks, including them as part of an automated suite that runs against every -PR is not recommended. +PR is not recommended. Instead, scheduling benchmark runs on a regular cadence is the recommended method of automating +this suite of tests. ## Changelog From 38b26bf40713980c00d1d341249642a800613a24 Mon Sep 17 00:00:00 2001 From: Noah Stapp Date: Thu, 21 Aug 2025 14:17:37 -0700 Subject: [PATCH 7/8] Add ODM-specific testing blurb --- source/benchmarking/odm-benchmarking.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/source/benchmarking/odm-benchmarking.md b/source/benchmarking/odm-benchmarking.md index 197d04f223..65adc999d5 100644 --- a/source/benchmarking/odm-benchmarking.md +++ b/source/benchmarking/odm-benchmarking.md @@ -189,8 +189,8 @@ Dataset size: For scoring purposes, the dataset size is the size of the `updated | Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. Save 10,000 instances into the database. | | Before task | n/a. | | Do task | Update the `field1` field for each instance of the model to equal `updated_value` in an ODM-appropriate manner. | -| After task | Drop the collection associated with the SMALL_DOC model. | -| Teardown | n/a. | +| After task | n/a. | +| Teardown | Drop the collection associated with the SMALL_DOC model. | #### Small model find by filter @@ -374,4 +374,11 @@ to the relatively long runtime of the benchmarks, including them as part of an a PR is not recommended. Instead, scheduling benchmark runs on a regular cadence is the recommended method of automating this suite of tests. +## ODM-specific benchmarking + +As discussed earlier in this document, ODM feature sets vary significantly across libraries. Many ODMs have features +unique to them or their niche in the wider ecosystem, which makes specifying concrete benchmark test cases for every +possible API unfeasible. Instead, ODM authors should determine what mainline use cases of their library are not covered +by the benchmarks specified above and expand this testing suite with additional benchmarks to cover those areas. + ## Changelog From c07cd0884a82aca2d615cc48b641b1e7ffe48a6a Mon Sep 17 00:00:00 2001 From: Noah Stapp Date: Thu, 21 Aug 2025 14:19:50 -0700 Subject: [PATCH 8/8] Use replica set of size 1 --- source/benchmarking/odm-benchmarking.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/source/benchmarking/odm-benchmarking.md b/source/benchmarking/odm-benchmarking.md index 65adc999d5..4a8605681a 100644 --- a/source/benchmarking/odm-benchmarking.md +++ b/source/benchmarking/odm-benchmarking.md @@ -364,8 +364,8 @@ supports. ### Benchmark Server -The MongoDB ODM Performance Benchmark must be run against a standalone MongoDB server running the latest stable database -version without authentication or SSL enabled. +The MongoDB ODM Performance Benchmark must be run against a MongoDB replica set of size 1 running the latest stable +database version without authentication or SSL enabled. ### Benchmark placement and scheduling