-
Notifications
You must be signed in to change notification settings - Fork 26
INTPYTHON-527 Add Queryable Encryption support #329
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
django_mongodb_backend/management/commands/get_encrypted_fields_map.py
Outdated
Show resolved
Hide resolved
django_mongodb_backend/management/commands/get_encrypted_fields_map.py
Outdated
Show resolved
Hide resolved
django_mongodb_backend/management/commands/get_encrypted_fields_map.py
Outdated
Show resolved
Hide resolved
django_mongodb_backend/management/commands/get_encrypted_fields_map.py
Outdated
Show resolved
Hide resolved
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
django_mongodb_backend/management/commands/get_encrypted_fields_map.py
Outdated
Show resolved
Hide resolved
2d672f2
to
10ae20e
Compare
4acc5ab
to
796855e
Compare
affa499
to
29a1d97
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added my first round of comments. I haven't looked over the tests yet but plan to do so next week. A big area I'll dig more into is around the discussion on KMS_PROVIDERS and KMS_CREDENTIALS
# Avoid using PyMongo to check the database version or require | ||
# pymongocrypt>=1.14.2 which will contain a fix for the `buildInfo` | ||
# command. https://jira.mongodb.org/browse/PYTHON-5429 | ||
return tuple(self.connection.admin.command("buildInfo")["versionArray"]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One caveat is depending on auth status, the admin and commands may not be available.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would apply to the PyMongo
API as well. This change only addresses the libmongocrypt
issue that we have not worked around in PyMongo.
django_mongodb_backend/features.py
Outdated
self.connection.ensure_connection() | ||
client = self.connection.connection.admin | ||
build_info = client.command("buildInfo") | ||
is_enterprise = "enterprise" in build_info.get("modules") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cc: @ShaneHarvey is this guaranteed to capture enterprise connections?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe @JamesKovacs confirmed this
Enterprise 07895885-9f55-4ee3-a362-4a06a321aefa [direct: secondary] test> db.adminCommand("buildInfo")
{
version: '8.1.1',
versionArray: [ 8, 1, 1, 0 ],
gitVersion: 'c441b67d7260844c1422bf259e23c054a33ee7d8',
modules: [ 'enterprise' ],
…
django_mongodb_backend/management/commands/get_encrypted_fields_map.py
Outdated
Show resolved
Hide resolved
data_key = ce.create_data_key( | ||
kms_provider=kms_provider, | ||
master_key=master_key, | ||
) |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
89205af
to
293656a
Compare
270727c
to
f0d4e92
Compare
f0d4e92
to
4ef6b84
Compare
6e1b8ed
to
3815a70
Compare
3815a70
to
a6a12e7
Compare
The encryption tests are passing locally for me on Enterprise and on the Atlas VM. On GitHub actions, this first issue was solved by adding
But this issue remains:
|
self.assertEqual( | ||
PatientRecord.objects.get(ssn="123-45-6789").profile_picture, b"image data" | ||
) | ||
with self.assertRaises(AssertionError): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's your thinking about the usefulness of this assertion? If assertEqual(profile_picture, b"image data")
passed, then of course asserting it's not equal to something else is going to work? (Incidentally, assertNotEqual()
is more natural than assertEqual() + assertRaises()
.)
More generally, it seems like you weren't sure exactly what to test here, so you wrote various things that came to mind. Maybe we need to define the test conditions so we can have some more standardized testing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we're relying on the encryption algorithm to detect changes I wanted to see it pass and fail, and yes the plan was to add fields to the patient-themed test suite, expanding on the tutorial example.
# FIXME: pymongo.errors.EncryptionError: Cannot encrypt element of type int | ||
# because schema requires that type is one of: [ long ] | ||
# pos_int=1, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The data type for each field is determined by this mapping:
django-mongodb-backend/django_mongodb_backend/base.py
Lines 36 to 63 in d5aa1a7
data_types = { | |
"AutoField": "int", | |
"BigAutoField": "long", | |
"BinaryField": "binData", | |
"BooleanField": "bool", | |
"CharField": "string", | |
"DateField": "date", | |
"DateTimeField": "date", | |
"DecimalField": "decimal", | |
"DurationField": "long", | |
"FileField": "string", | |
"FilePathField": "string", | |
"FloatField": "double", | |
"IntegerField": "int", | |
"BigIntegerField": "long", | |
"GenericIPAddressField": "string", | |
"JSONField": "object", | |
"OneToOneField": "int", | |
"PositiveBigIntegerField": "int", | |
"PositiveIntegerField": "long", | |
"PositiveSmallIntegerField": "int", | |
"SlugField": "string", | |
"SmallAutoField": "int", | |
"SmallIntegerField": "int", | |
"TextField": "string", | |
"TimeField": "date", | |
"UUIDField": "string", | |
} |
It seems the mapping has some mistakes. For example, PositiveBigIntegerField
should be long
(64-bit) [I think]. That said, this is an issue that should be correct in a separate PR.
The question remains how to send the value to the database as a long to avoid this error. Frankly, I wouldn't expect any special handling to be needed, but maybe Jib has some idea. (Was this already discussed when you ran into the error for DurationField?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It has not been discussed and now that you mention it, DurationField
may have been similar.
Previous attempts and additional context here: