DRIVERS-2917 - Standardized Performance Testing of ODMs and Integrations #1828
Conversation
| @@ -0,0 +1 @@ | |||
| {"field1":"miNVpaKW","field2":"CS5VwrwN","field3":"Oq5Csk1w","field4":"ZPm57dhu","field5":"gxUpzIjg","field6":"Smo9whci","field7":"TW34kfzq","field8":55336395,"field9":41992681,"field10":72188733,"field11":46660880,"field12":3527055,"field13":74094448} | |||
|
|
||
| ### Benchmark Server | ||
|
|
||
| The MongoDB ODM Performance Benchmark must be run against a standalone MongoDB server running the latest stable database |
There was a problem hiding this comment.
I think we can open up this to be a standalone or a replica set with a size of 1. (This is because some ODMs leverage transactions)
There was a problem hiding this comment.
Using a replica set of size 1 makes more sense here, agreed.
|
|
||
| ### Benchmark placement and scheduling | ||
|
|
||
| The MongoDB ODM Performance Benchmark should be placed within the ODM's test directory as an independent test suite. Due |
There was a problem hiding this comment.
I still think we should leave an option for folks to create their own benchmarking repo if that helps out. I'm open to others take on this one seeing as I worry about maintainers not wanting a benchmark repo.
| to the relatively long runtime of the benchmarks, including them as part of an automated suite that runs against every | ||
| PR is not recommended. Instead, scheduling benchmark runs on a regular cadence is the recommended method of automating | ||
| this suite of tests. | ||
|
|
There was a problem hiding this comment.
Per your suggestion earlier, we should include some new information about testing mainline usecases.
| As discussed earlier in this document, ODM feature sets vary significantly across libraries. Many ODMs have features | ||
| unique to them or their niche in the wider ecosystem, which makes specifying concrete benchmark test cases for every | ||
| possible API unfeasible. Instead, ODM authors should determine what mainline use cases of their library are not covered | ||
| by the benchmarks specified above and expand this testing suite with additional benchmarks to cover those areas. |
There was a problem hiding this comment.
This section is attempting to specify that ODMs should implement additional benchmark tests to cover mainline use cases that do not fall into those included in this specification. One example would be the use of Django's in filter operator: Model.objects.filter(field__in=["some_val"]).
NoahStapp
requested review from
JamesKovacs,
alexbevi,
aclark4life,
ajcvickers,
rozza,
damieng and
R-shubham
and removed request for
a team
Please complete the following before merging:
clusters).
Python Django implementation: mongodb/django-mongodb-backend#366.