INTPYTHON-527 Add Queryable Encryption support #329

aclark4life · 2025-06-27T02:16:28Z

Previous attempts and additional context here:

INTPYTHON-527 Add Queryable Encryption config #318
- Branched from transaction support and includes only helpers
INTPYTHON-527 Add queryable encryption support #319
- In which this bug in libmongocrypt was discovered
- A fix for that bug in PyMongo is being discussed here.
INTPYTHON-527 Add Queryable Encryption support #323
- Before I understood that the shared lib for mongocrypt was included in the pymongocrypt wheel.

tests/encryption_/tests.py

django_mongodb_backend/models.py

django_mongodb_backend/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/features.py

django_mongodb_backend/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

tests/encryption_/routers.py

tests/encryption_/models.py

tests/backend_/test_features.py

django_mongodb_backend/schema.py

tests/encryption_/models.py

django_mongodb_backend/schema.py

django_mongodb_backend/encryption.py

tests/encryption_/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/models.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

docs/source/ref/encryption.rst

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

django_mongodb_backend/encryption.py

django_mongodb_backend/utils.py

docs/source/howto/encryption.rst

django_mongodb_backend/schema.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

docs/source/ref/models/fields.rst

django_mongodb_backend/utils.py

django_mongodb_backend/features.py

docs/source/ref/models/fields.rst

tests/backend_/test_features.py

tests/encryption_/models.py

tests/backend_/test_features.py

Jibola

I've added my first round of comments. I haven't looked over the tests yet but plan to do so next week. A big area I'll dig more into is around the discussion on KMS_PROVIDERS and KMS_CREDENTIALS

Jibola · 2025-07-28T17:02:32Z

django_mongodb_backend/base.py

+        # Avoid using PyMongo to check the database version or require
+        # pymongocrypt>=1.14.2 which will contain a fix for the `buildInfo`
+        # command. https://jira.mongodb.org/browse/PYTHON-5429
+        return tuple(self.connection.admin.command("buildInfo")["versionArray"])


One caveat is depending on auth status, the admin and commands may not be available.

That would apply to the PyMongo API as well. This change only addresses the libmongocrypt issue that we have not worked around in PyMongo.

django_mongodb_backend/routers.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/encryption.py

Jibola · 2025-08-01T20:06:32Z

django_mongodb_backend/features.py

+        self.connection.ensure_connection()
+        client = self.connection.connection.admin
+        build_info = client.command("buildInfo")
+        is_enterprise = "enterprise" in build_info.get("modules")


cc: @ShaneHarvey is this guaranteed to capture enterprise connections?

I believe @JamesKovacs confirmed this

Enterprise 07895885-9f55-4ee3-a362-4a06a321aefa [direct: secondary] test> db.adminCommand("buildInfo") { version: '8.1.1', versionArray: [ 8, 1, 1, 0 ], gitVersion: 'c441b67d7260844c1422bf259e23c054a33ee7d8', modules: [ 'enterprise' ], …

django_mongodb_backend/schema.py

docs/source/howto/queryable-encryption.rst

tests/backend_/test_features.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

django_mongodb_backend/routers.py

django_mongodb_backend/encryption.py

docs/source/howto/queryable-encryption.rst

tests/backend_/test_features.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

+                        data_key = ce.create_data_key(
+                            kms_provider=kms_provider,
+                            master_key=master_key,
+                        )


docs/source/intro/install.rst

docs/source/topics/known-issues.rst

tests/encryption_/tests.py

docs/source/topics/known-issues.rst

timgraham · 2025-08-02T23:39:06Z

The encryption tests are passing locally for me on Enterprise and on the Atlas VM.

On GitHub actions, this first issue was solved by adding "directConnection": True in DATABASES:

  File "/home/runner/work/django-mongodb-backend/django-mongodb-backend/django_repo/django/db/backends/base/base.py", line 197, in check_database_version_supported
    and self.get_database_version() < self.features.minimum_database_version
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/django-mongodb-backend/django-mongodb-backend/django_mongodb_backend/base.py", line 235, in get_database_version
    return tuple(self.connection.admin.command("buildInfo")["versionArray"])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/_csot.py", line 125, in csot_wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/database.py", line 926, in command
    with self._client._conn_for_reads(read_preference, session, operation=command_name) as (
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1864, in _conn_for_reads
    server = self._select_server(read_preference, session, operation)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1812, in _select_server
    server = topology.select_server(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 409, in select_server
    server = self._select_server(
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 387, in _select_server
    servers = self.select_servers(
              ^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 294, in select_servers
    server_descriptions = self._select_servers_loop(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 344, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: Could not reach any servers in [('3ff0eef351ff', 27017)]. Replica set is configured with internal hostnames or IPs?, Timeout: 30s, Topology Description: <TopologyDescription id: 688e9488ca8c88d98365da45, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('3ff0eef351ff', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('3ff0eef351ff:27017: [Errno -3] Temporary failure in name resolution (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)')>]>

But this issue remains:

Creating test database for alias 'encrypted' ('test_djangotests-encrypted')...
/home/runner/.local/lib/python3.12/site-packages/pymongo/daemon.py:147: RuntimeWarning: Failed to start mongocryptd: is it on your $PATH?
Original exception: [Errno 2] No such file or directory: 'mongocryptd'
  _silence_resource_warning(_spawn(sys.argv[1:]))
/home/runner/.local/lib/python3.12/site-packages/pymongo/daemon.py:147: RuntimeWarning: Failed to start mongocryptd: is it on your $PATH?
Original exception: [Errno 2] No such file or directory: 'mongocryptd'
  _silence_resource_warning(_spawn(sys.argv[1:]))
  Applying sites.0002_alter_domain_unique... OK
Operations to perform:
  Synchronize unmigrated apps: auth, contenttypes, encryption_, messages, sessions, staticfiles
  Apply all migrations: admin, sites
Synchronizing apps without migrations:
  Creating tables...
    Creating table encryption__appointment
Traceback (most recent call last):
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/encryption.py", line 286, in mark_command
    res = self.mongocryptd_client[database].command(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/_csot.py", line 125, in csot_wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/database.py", line 926, in command
    with self._client._conn_for_reads(read_preference, session, operation=command_name) as (
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1864, in _conn_for_reads
    server = self._select_server(read_preference, session, operation)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1812, in _select_server
    server = topology.select_server(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 409, in select_server
    server = self._select_server(
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 387, in _select_server
    servers = self.select_servers(
              ^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 294, in select_servers
    server_descriptions = self._select_servers_loop(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 344, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:27020: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 10.0s, Topology Description: <TopologyDescription id: 688e9f76697ba6965a378048, topology_type: Unknown, servers: [<ServerDescription ('localhost', 27020) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27020: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)')>]>

timgraham · 2025-08-02T21:38:47Z

tests/encryption_/tests.py

+        self.assertEqual(
+            PatientRecord.objects.get(ssn="123-45-6789").profile_picture, b"image data"
+        )
+        with self.assertRaises(AssertionError):


What's your thinking about the usefulness of this assertion? If assertEqual(profile_picture, b"image data") passed, then of course asserting it's not equal to something else is going to work? (Incidentally, assertNotEqual() is more natural than assertEqual() + assertRaises().)

More generally, it seems like you weren't sure exactly what to test here, so you wrote various things that came to mind. Maybe we need to define the test conditions so we can have some more standardized testing.

Since we're relying on the encryption algorithm to detect changes I wanted to see it pass and fail, and yes the plan was to add fields to the patient-themed test suite, expanding on the tutorial example.

timgraham · 2025-08-02T23:03:00Z

tests/encryption_/tests.py

+            # FIXME: pymongo.errors.EncryptionError: Cannot encrypt element of type int
+            # because schema requires that type is one of: [ long ]
+            # pos_int=1,


The data type for each field is determined by this mapping:

django-mongodb-backend/django_mongodb_backend/base.py

Lines 36 to 63 in d5aa1a7

data_types = {

"AutoField": "int",

"BigAutoField": "long",

"BinaryField": "binData",

"BooleanField": "bool",

"CharField": "string",

"DateField": "date",

"DateTimeField": "date",

"DecimalField": "decimal",

"DurationField": "long",

"FileField": "string",

"FilePathField": "string",

"FloatField": "double",

"IntegerField": "int",

"BigIntegerField": "long",

"GenericIPAddressField": "string",

"JSONField": "object",

"OneToOneField": "int",

"PositiveBigIntegerField": "int",

"PositiveIntegerField": "long",

"PositiveSmallIntegerField": "int",

"SlugField": "string",

"SmallAutoField": "int",

"SmallIntegerField": "int",

"TextField": "string",

"TimeField": "date",

"UUIDField": "string",

}

It seems the mapping has some mistakes. For example, PositiveBigIntegerField should be long (64-bit) [I think]. That said, this is an issue that should be correct in a separate PR.

The question remains how to send the value to the database as a long to avoid this error. Frankly, I wouldn't expect any special handling to be needed, but maybe Jib has some idea. (Was this already discussed when you ran into the error for DurationField?)

It has not been discussed and now that you mention it, DurationField may have been similar.

aclark4life commented Jun 27, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/models.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/encryption.py Outdated Show resolved Hide resolved

timgraham reviewed Jun 27, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jun 30, 2025

View reviewed changes

django_mongodb_backend/features.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/routers.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 4, 2025

View reviewed changes

tests/encryption_/models.py Show resolved Hide resolved

timgraham reviewed Jul 7, 2025

View reviewed changes

tests/encryption_/models.py Outdated Show resolved Hide resolved

aclark4life commented Jul 7, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 9, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life commented Jul 11, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 22, 2025

View reviewed changes

addaleax reviewed Jul 25, 2025

View reviewed changes

django_mongodb_backend/features.py Show resolved Hide resolved

django_mongodb_backend/features.py Outdated Show resolved Hide resolved

aclark4life requested review from a team, Copilot, timgraham and WaVEV July 25, 2025 20:08

This comment was marked as resolved.

Sign in to view

aclark4life force-pushed the INTPYTHON-527 branch from 2d672f2 to 10ae20e Compare July 31, 2025 16:10

timgraham reviewed Jul 31, 2025

View reviewed changes

docs/source/ref/models/fields.rst Outdated Show resolved Hide resolved

docs/source/ref/models/fields.rst Outdated Show resolved Hide resolved

tests/backend_/test_features.py Outdated Show resolved Hide resolved

tests/encryption_/models.py Outdated Show resolved Hide resolved

aclark4life force-pushed the INTPYTHON-527 branch from 4acc5ab to 796855e Compare August 1, 2025 19:37

timgraham removed request for a team and WaVEV August 1, 2025 21:42

timgraham force-pushed the INTPYTHON-527 branch from affa499 to 29a1d97 Compare August 1, 2025 21:59

aclark4life commented Aug 1, 2025

View reviewed changes

tests/backend_/test_features.py Show resolved Hide resolved

Jibola reviewed Aug 1, 2025

View reviewed changes

timgraham reviewed Aug 1, 2025

View reviewed changes

aclark4life force-pushed the INTPYTHON-527 branch from 89205af to 293656a Compare August 1, 2025 23:16

timgraham reviewed Aug 2, 2025

View reviewed changes

docs/source/intro/install.rst Outdated Show resolved Hide resolved

docs/source/topics/known-issues.rst Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

docs/source/topics/known-issues.rst Outdated Show resolved Hide resolved

aclark4life force-pushed the INTPYTHON-527 branch from 270727c to f0d4e92 Compare August 2, 2025 03:04

INTPYTHON-527 Add Queryable Encryption support

4ef6b84

aclark4life force-pushed the INTPYTHON-527 branch from f0d4e92 to 4ef6b84 Compare August 2, 2025 15:02

timgraham force-pushed the INTPYTHON-527 branch 3 times, most recently from 6e1b8ed to 3815a70 Compare August 2, 2025 22:41

timgraham added 2 commits August 2, 2025 19:27

fix encrypted tests

9954a7c

[debug] skip most tests for now

a6a12e7

timgraham force-pushed the INTPYTHON-527 branch from 3815a70 to a6a12e7 Compare August 2, 2025 23:28

timgraham reviewed Aug 2, 2025

View reviewed changes

Fix db name on two encryption tests + doc updates

6ab0a86

	data_types = {
	"AutoField": "int",
	"BigAutoField": "long",
	"BinaryField": "binData",
	"BooleanField": "bool",
	"CharField": "string",
	"DateField": "date",
	"DateTimeField": "date",
	"DecimalField": "decimal",
	"DurationField": "long",
	"FileField": "string",
	"FilePathField": "string",
	"FloatField": "double",
	"IntegerField": "int",
	"BigIntegerField": "long",
	"GenericIPAddressField": "string",
	"JSONField": "object",
	"OneToOneField": "int",
	"PositiveBigIntegerField": "int",
	"PositiveIntegerField": "long",
	"PositiveSmallIntegerField": "int",
	"SlugField": "string",
	"SmallAutoField": "int",
	"SmallIntegerField": "int",
	"TextField": "string",
	"TimeField": "date",
	"UUIDField": "string",
	}

INTPYTHON-527 Add Queryable Encryption support #329

Are you sure you want to change the base?

INTPYTHON-527 Add Queryable Encryption support #329

Uh oh!

Conversation

aclark4life commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jibola left a comment

aclark4life commented Jun 27, 2025 •

edited

Loading