PHPORM-13 Feature Queryable Encryption #2779

GromNaN · 2025-06-12T13:17:56Z

Q	A
Type	feature
BC Break	no
Fixed issues	PHPORM-13

Add Automatic Queryable Encryption to the MongoDB Documents. This feature requires a MongoDB Enterprise license or a MongoDB Atlas cluster.

Bundle integration: doctrine/DoctrineMongoDBBundle#897

GromNaN · 2025-08-08T08:17:15Z

lib/Doctrine/ODM/MongoDB/Mapping/ClassMetadata.php

@@ -2176,6 +2183,15 @@ public function isView(): bool
        return $this->isView;
    }

+    public function isDocument(): bool


Note to leverage this new method in the SchemaManager.

mongodb-odm/lib/Doctrine/ODM/MongoDB/SchemaManager.php

Line 75 in b0909ca

if ($class->isMappedSuperclass || $class->isEmbeddedDocument || $class->isQueryResultDocument || $class->isView()) {

Do you intend to make this change in the current PR, or create a ticket to do so later? No objections to doing it now.

I don't think it's that simple. There are details to work out with File-type collections, which are sometimes Documents and sometimes not. I'll keep that for later.

alcaeus

Requesting a Copilot for additional review since I already reviewed a number of these PRs and authored some of them, but LGTM so far.

alcaeus

LGTM with phpstan errors fixed.

Copilot

Pull Request Overview

This PR implements Automatic Queryable Encryption support for the MongoDB ODM, enabling client-side encryption of document fields with the ability to perform queries on encrypted data while maintaining security. The feature requires MongoDB Enterprise 8.0+ or MongoDB Atlas.

Key changes include:

Added comprehensive queryable encryption mapping support with new #[Encrypt] attribute and EncryptQuery enum
Enhanced configuration to support KMS providers and auto-encryption options
Extended schema management to automatically create encrypted collections with proper metadata

Reviewed Changes

Copilot reviewed 36 out of 36 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`lib/Doctrine/ODM/MongoDB/Mapping/Annotations/Encrypt.php`	New annotation for marking fields/documents as encrypted with query type configuration
`lib/Doctrine/ODM/MongoDB/Mapping/Annotations/EncryptQuery.php`	Enum defining supported query types (Equality, Range) for encrypted fields
`lib/Doctrine/ODM/MongoDB/Configuration.php`	Added KMS provider configuration and auto-encryption options support
`lib/Doctrine/ODM/MongoDB/Utility/EncryptedFieldsMapGenerator.php`	New utility for generating MongoDB encrypted fields maps from ODM metadata
`tests/Documents/Encryption/*.php`	Test document classes demonstrating various encryption scenarios
`tests/Doctrine/ODM/MongoDB/Tests/Functional/QueryableEncryptionTest.php`	Functional tests for encrypted document operations
`docs/en/cookbook/queryable-encryption.rst`	Comprehensive documentation and tutorial for using queryable encryption

lib/Doctrine/ODM/MongoDB/Mapping/Driver/AttributeDriver.php

* Add test document classes from tutorial https://github.com/mongodb/docs/blob/master/source/includes/qe-tutorials/node/queryable-encryption-tutorial.js * Create encryption field map * Create encrypted collection * Use promoted properties for encrypted document tests * Revert changes in BaseTestCase * Rename EncryptionFieldMapTest * Create Configuration::getDriverOptions() to create the client * Support class-level #[Encrypt] attribute * Skip QE tests on non-supported server configuration * Add documentation on #[Encrypt] attribute * Add XML mapping for "encrypt" tag * Create specific xsd type for embedded documents to enable <encrypt> tag * Fix import class Throwable * Fix access to $version property before initialization * Use an enum for query type * Make getClientEncryption internal * Improve type of min/max bounds for range queries * Use getWriteOptions with createEncryptedCollection * Use random local master key * Add assertion on non-decrypted data * Ignore all DOCTRINE_MONGODB_DATABASE phpstan errors * Baseline phpstan * Fix CS and skip phpstan issues

)

The driver already detects encrypted collections

…ction inheritance (#2790)

Co-authored-by: Copilot <[email protected]>

lib/Doctrine/ODM/MongoDB/Mapping/Annotations/Encrypt.php

jmikola · 2025-08-15T02:56:59Z

lib/Doctrine/ODM/MongoDB/Mapping/ClassMetadata.php

@@ -2176,6 +2183,15 @@ public function isView(): bool
        return $this->isView;
    }

+    public function isDocument(): bool


Do you intend to make this change in the current PR, or create a ticket to do so later? No objections to doing it now.

lib/Doctrine/ODM/MongoDB/Mapping/ClassMetadata.php

lib/Doctrine/ODM/MongoDB/Mapping/Driver/XmlDriver.php

lib/Doctrine/ODM/MongoDB/Mapping/Driver/AttributeDriver.php

lib/Doctrine/ODM/MongoDB/Utility/EncryptedFieldsMapGenerator.php

jmikola · 2025-08-15T03:41:03Z

lib/Doctrine/ODM/MongoDB/Utility/EncryptedFieldsMapGenerator.php

+                'bsonType' => match ($mapping['type']) {
+                    ClassMetadata::ONE, Type::HASH => 'object',
+                    ClassMetadata::MANY, Type::COLLECTION => 'array',
+                    Type::INT, Type::INTEGER => 'int',


The BSON type mappings for encryption differentiates between 32-bit and 64-bit integers (int and long, respectively). This could be problematic, as the match here is going to limit all integer-mapped fields to 32-bit. Defaulting to 64-bit isn't necessarily a workaround, as the server is going to expect matching types for any equality queries.

I believe this is being tracked by SERVER-75897.

So, do you suggest we should change it to long?

Suggested change

Type::INT, Type::INTEGER => 'int',

Type::INT, Type::INTEGER => 'long',

According to the CSFLE Encryption Schemas: encrypt docs, we can specify an array of types for bsonType. Can we use ['int', 'long'] here to remain flexible?

I'll also share this question in the CSFLE Slack for a sanity check.

@kevinAlbs replied in Slack:

Notably: though CSFLE schema (JSON Schema) supports multiple BSON types, a QE schema (encryptedFields) only supports a single valued bsonType.

So I don't think we have a user-friendly solution here unless we prefer "long" (since it can contain any integer value) and have ODM always encodes the encrypted field as a MongoDB\BSON\Int64 to ensure even values in the 32-bit range are stored as int64 and thus satisfy the QE bsonType restriction.

When sending a int64 to a field encoded with the int type, we get this error:

mongocryptd error: Cannot encrypt element of type long because schema requires that type is one of: [ int ]

Also, I cannot set multiple types:

mongocryptd error: BSON field 'create.encryptedFields.fields.bsonType' is the wrong type 'array', expected type 'string'

If I change the bson type to long, then it doesn't accept an int.

It seems that its a limitation until we add the Int64 type in the ODM.

New Int64 type: #2805

Well, that was a fun little detour ⚠️

Consider this resolved but I'd like to leave the thread unresolved for posterity (and future accessibility).

lib/Doctrine/ODM/MongoDB/Utility/EncryptedFieldsMapGenerator.php

composer.json

doctrine-mongo-mapping.xsd

lib/Doctrine/ODM/MongoDB/Configuration.php

lib/Doctrine/ODM/MongoDB/Utility/EncryptedFieldsMapGenerator.php

tests/Doctrine/ODM/MongoDB/Tests/ConfigurationTest.php

tests/Doctrine/ODM/MongoDB/Tests/Tools/EncryptedFieldsMapGeneratorTest.php

jmikola · 2025-08-22T02:57:04Z

tests/Doctrine/ODM/MongoDB/Tests/Tools/EncryptedFieldsMapGeneratorTest.php

+            ],
+            [
+                'path' => 'clientCards',
+                'bsonType' => 'array',


I noted that the EmbedMany relation on Client isn't mapped as encrypted. The objects within the array are encrypted. Would ODM produce the same EncryptedFields object if the relationship on Client was annotated with #[Encrypt]?

I'll note that it doesn't seem possible to map all array elements as encrypted (based on server manual docs). I suppose one could specify each array index as a separate path, but that wouldn't be flexible for dynamic arrays.

Therefore, I'm not suggesting the current behavior to encrypt the entire array is incorrect. But I am curious about whether users have two ways to define a mapping for the same effect.

The clientCards field is encrypted as a whole, into a single Binary value.

QE doesn't allow querying objects, so having an array of documents would not be useful for querying. It it could for partial updates, like pushing a new element.

Meanwhile, we could have a list of int/long/double/date that could be queryable. But this results in this exception:

MongoDB\Driver\Exception\RuntimeException: mongocryptd error: Cannot encrypt element of type: array

QE doesn't allow querying objects, so having an array of documents would not be useful for querying. It it could for partial updates, like pushing a new element.

Would we expect this to conflict with ODM's collection strategies? I expect they could end up pushing or pulling elements from the array.

Yes, since arrays are stored as a single binary value, they need to be encrypted and saved globally client-side.

mongocryptd error: $addToSet not allowed on encrypted values mongocryptd error: $push not allowed on encrypted values

I added the restriction to the documentation.

* Add Int64 type * Convert min/max options in the metadata class * Add type assertions on Type tests

jmikola

A few comments on docs but I think you can address those yourself.

docs/en/cookbook/queryable-encryption.rst

jmikola · 2025-08-28T05:18:44Z

lib/Doctrine/ODM/MongoDB/Configuration.php

+     *
+     * @param array{ keyVaultClient?: Client|Manager, keyVaultNamespace?: string, tlsOptions?: array<string, mixed>, schemaMap?: array<string, mixed>, bypassAutoEncryption?: bool, bypassQueryAnalysis?: bool, encryptedFieldsMap?: array<string,mixed>, extraOptions?: array<string, mixed>} $options
+     */
+    public function setAutoEncryption(array $options): void


Based on the logic in getAutoEncryptionOptions() below, which merged this in using the spread operator (...), keys here may overwrite the kmsProviders and keyVaultNamespace options created by ODM. On the off chance that could lead to unexpected behavior (or a hypothetical security issue down the line), it may be prudent to document that these options will take precedence.

I don't think there could be a problem:

In setAutoEncryption, kmsProvider is not allowed to be set (exception). This option is not in the array shape of the param.

In getAutoEncryptionOptions, keyVaultNamespace uses the default db.datakeys but can be overridden by the value set with setAutoEncryption.

Co-authored-by: Jeremy Mikola <[email protected]>

GromNaN changed the base branch from 2.11.x to 2.12.x June 12, 2025 13:18

This was referenced Jun 19, 2025

Enable auto encryption doctrine/DoctrineMongoDBBundle#897

Merged

[Encryption] Fix getDefaultKmsProvider method name #2784

Merged

[Encryption] Fix BSON types and query option name #2785

Merged

GromNaN marked this pull request as ready for review June 30, 2025 16:39

GromNaN requested a review from alcaeus July 3, 2025 14:23

GromNaN mentioned this pull request Jul 7, 2025

Add purge mode to MongoDB Purger doctrine/data-fixtures#544

Merged

GromNaN commented Aug 8, 2025

View reviewed changes

alcaeus reviewed Aug 8, 2025

View reviewed changes

alcaeus requested a review from Copilot August 8, 2025 09:50

This comment was marked as outdated.

Sign in to view

alcaeus approved these changes Aug 8, 2025

View reviewed changes

GromNaN changed the title ~~Feature Queryable Encryption~~ PHPORM-13 Feature Queryable Encryption Aug 11, 2025

jmikola requested review from jmikola and paulinevos August 11, 2025 13:58

GromNaN requested a review from Copilot August 11, 2025 16:03

Copilot AI reviewed Aug 11, 2025

View reviewed changes

lib/Doctrine/ODM/MongoDB/Mapping/Driver/AttributeDriver.php Show resolved Hide resolved

GromNaN and others added 13 commits August 11, 2025 18:05

[Encryption] Set master key when creating an encrypted collection (#2780

59ddb05

)

[Encryption] Refactor encrypted fields map generator (#2783)

8fc9d01

Drop metadata collection when dropping encrypted collection (#2781)

4776dcf

Fix getDefaultKmsProvider method name (#2784)

9d7cdbc

[Encryption] Fix BSON types and query option name (#2785)

5f42562

Remove encryptedFields option when dropping collection (#2788)

f852bd2

The driver already detects encrypted collections

[Encryption] PHPORM-360 Document limitations of encryption with colle…

097c4d5

…ction inheritance (#2790)

[Encryption] Add cookbook for queryable encryption (#2793)

bbb9134

Fix typo in precision mapping option

6c797fa

Co-authored-by: Copilot <[email protected]>

Fix typos

20241f1

Co-authored-by: Copilot <[email protected]>

Fix wrong namespace in documentation

40c8b8e

Co-authored-by: Copilot <[email protected]>

Remove duplicate field mapping in test document

ae66c28

Co-authored-by: Copilot <[email protected]>

jmikola reviewed Aug 15, 2025

View reviewed changes

Update test on dropping encrypted collection (#2803)

c46fb02

GromNaN commented Aug 19, 2025

View reviewed changes

composer.json Outdated Show resolved Hide resolved

GromNaN added 6 commits August 19, 2025 16:47

Amend cf3c0b7 to cast only [] to an empty stdClass document

8f05ac1

Remove maxOccurs=1

f859c06

Update docs

8299e1f

Add comment about EncryptConfig psalm type

44fb5e0

Rename loop variables

c183333

Add fields property to encryptedFieldsMap

492c723

GromNaN force-pushed the feature/queryable-encryption branch from d3049fd to 492c723 Compare August 20, 2025 14:02

GromNaN added 3 commits August 20, 2025 16:05

Rename $path to $parentPath

d9e9a75

Fix phpstan baseline

a0b8c69

Update composer version constraint

dad3800

GromNaN mentioned this pull request Aug 21, 2025

[Encryption] Add Int64 type #2805

Merged

jmikola reviewed Aug 22, 2025

View reviewed changes

GromNaN added 5 commits August 22, 2025 18:40

[Encryption] Add Int64 type (#2805)

b7a7409

* Add Int64 type * Convert min/max options in the metadata class * Add type assertions on Type tests

Document default keyVaultNamespace

5736515

Rename getEncryptedFieldsForClass

ea87a8d

Use base64 local key in tests

bd6834a

Document limitation on collection strategies

c5f919b

GromNaN requested a review from jmikola August 26, 2025 18:00

Update doc for unsupported inheritence

5746309

GromNaN force-pushed the feature/queryable-encryption branch from b415d41 to 5746309 Compare August 27, 2025 11:34

jmikola approved these changes Aug 28, 2025

View reviewed changes

Docs updates

010ea2f

Co-authored-by: Jeremy Mikola <[email protected]>

GromNaN merged commit 7c0fbe7 into 2.12.x Aug 28, 2025
43 checks passed

GromNaN deleted the feature/queryable-encryption branch August 28, 2025 12:15

GromNaN added this to the 2.12.0 milestone Aug 28, 2025

GromNaN added the Feature label Aug 28, 2025

github-actions bot mentioned this pull request Aug 29, 2025

Merge release 2.12.0 into 3.0.x #2809

Merged

	Type::INT, Type::INTEGER => 'int',
	Type::INT, Type::INTEGER => 'long',

Uh oh!

PHPORM-13 Feature Queryable Encryption #2779

PHPORM-13 Feature Queryable Encryption #2779

Uh oh!

Conversation

GromNaN commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alcaeus left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

alcaeus left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmikola Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmikola left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

GromNaN commented Jun 12, 2025 •

edited

Loading

jmikola Aug 28, 2025 •

edited

Loading