Upsert implementation - #3940
Conversation
|
This PR introduces
I applied the same separation for SQL Server through @dstepanov - If this approach looks good to you, we can continue moving Oracle-specific code out of |
There was a problem hiding this comment.
Pull request overview
This PR introduces native SQL upsert support across Micronaut Data SQL repositories by adding an @Upsert lifecycle annotation, a compile-time method matcher, dialect-specific SQL generation, and runtime execution paths (including generated-id returning for Oracle and SQL Server). It also adds extensive TCK entities/repositories and JDBC/R2DBC test coverage, plus new user-guide documentation.
Changes:
- Add
@UpsertAPI + processor support (UpsertMethodMatcher) and wire operation types through the model/runtime. - Implement SQL upsert SQL generation in
SqlQueryBuilderand add specialized JDBC/R2DBC operations for Oracle/SQL Server generated-id returning. - Add documentation (“Upserting”) and introduce JDBC/R2DBC/SQLite test suites plus new TCK entities/repositories for upsert scenarios.
Reviewed changes
Copilot reviewed 106 out of 106 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| test-suite-data-jdbc-sqlite/src/test/java/io/micronaut/data/jdbc/sqlite/SQLiteWarehouseInventoryRepository.java | Adds SQLite repository bean for upsert TCK + finder used by tests. |
| test-suite-data-jdbc-sqlite/src/test/java/io/micronaut/data/jdbc/sqlite/SQLiteUpsertTest.java | Adds SQLite JDBC tests validating insert/update upsert behavior. |
| test-suite-data-jdbc-sqlite/src/test/java/io/micronaut/data/jdbc/sqlite/SQLiteProductReviewRepository.java | Adds SQLite repository bean for ProductReview upsert tests. |
| test-suite-data-jdbc-sqlite/src/test/java/io/micronaut/data/jdbc/sqlite/SQLiteCustomerProfileUuidRepository.java | Adds SQLite repository bean for UUID upsert coverage (not directly exercised in shown test). |
| test-suite-data-jdbc-sqlite/src/test/java/io/micronaut/data/jdbc/sqlite/SQLiteCustomerProfileRepository.java | Adds SQLite repository bean for CustomerProfile upsert tests + finder used by tests. |
| src/main/docs/guide/toc.yml | Adds “Upserting” to docs TOC under data updates. |
| src/main/docs/guide/shared/dataUpdates/upserts.adoc | New guide section describing upsert API, constraints, and dialect behavior. |
| data-tck/src/main/java/io/micronaut/data/tck/repositories/upsert/WarehouseInventoryRepository.java | Adds TCK repository contract for conflict-column upsert. |
| data-tck/src/main/java/io/micronaut/data/tck/repositories/upsert/ProductReviewRepository.java | Adds TCK repository contract for identity-based upsert + @Upsert alternate method names. |
| data-tck/src/main/java/io/micronaut/data/tck/repositories/upsert/CustomerProfileUuidRepository.java | Adds TCK repository contract for upsert with generated UUID identity and conflict column. |
| data-tck/src/main/java/io/micronaut/data/tck/repositories/upsert/CustomerProfileRepository.java | Adds TCK repository contract for sync/async/reactive upsert return shapes. |
| data-tck/src/main/java/io/micronaut/data/tck/jdbc/entities/upsert/WarehouseInventory.java | New TCK entity with unique index conflict target (sku, warehouse). |
| data-tck/src/main/java/io/micronaut/data/tck/jdbc/entities/upsert/ProductReview.java | New TCK entity with assigned identity for default-conflict upsert tests. |
| data-tck/src/main/java/io/micronaut/data/tck/jdbc/entities/upsert/CustomerProfileUuid.java | New TCK entity with generated UUID identity + unique email conflict target. |
| data-tck/src/main/java/io/micronaut/data/tck/jdbc/entities/upsert/CustomerProfile.java | New TCK entity with generated identity + unique email conflict target. |
| data-runtime/src/main/java/io/micronaut/data/runtime/intercept/AbstractQueryInterceptor.java | Wires UPSERT into stored-query operation-type resolution. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/sqlserver/upsert/MSWarehouseInventoryRepository.java | SQL Server R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/sqlserver/upsert/MSProductReviewRepository.java | SQL Server R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/sqlserver/upsert/MSCustomerProfileUuidRepository.java | SQL Server R2DBC test repository binding for TCK UUID upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/sqlserver/upsert/MSCustomerProfileSequenceRepository.java | SQL Server R2DBC test repository for sequence-generated identity upsert returning. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/sqlserver/upsert/MSCustomerProfileRepository.java | SQL Server R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/sqlserver/upsert/CustomerProfileSequence.java | SQL Server R2DBC test entity for sequence-generated identity. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/postgres/upsert/PostgresWarehouseInventoryRepository.java | Postgres R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/postgres/upsert/PostgresProductReviewRepository.java | Postgres R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/postgres/upsert/PostgresCustomerProfileUuidRepository.java | Postgres R2DBC test repository binding for TCK UUID upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/postgres/upsert/PostgresCustomerProfileSequenceRepository.java | Postgres R2DBC test repository for sequence-generated identity upsert returning. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/postgres/upsert/PostgresCustomerProfileRepository.java | Postgres R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/postgres/upsert/CustomerProfileSequence.java | Postgres R2DBC test entity for sequence-generated identity. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/oraclexe/upsert/OracleXEWarehouseInventoryRepository.java | Oracle XE R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/oraclexe/upsert/OracleXEProductReviewRepository.java | Oracle XE R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/oraclexe/upsert/OracleXECustomerProfileUuidRepository.java | Oracle XE R2DBC test repository binding for TCK UUID upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/oraclexe/upsert/OracleXECustomerProfileSequenceRepository.java | Oracle XE R2DBC test repository for sequence-generated identity upsert returning. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/oraclexe/upsert/OracleXECustomerProfileRepository.java | Oracle XE R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/oraclexe/upsert/CustomerProfileSequence.java | Oracle XE R2DBC test entity for sequence-generated identity. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/mysql/upsert/MySqlWarehouseInventoryRepository.java | MySQL R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/mysql/upsert/MySqlProductReviewRepository.java | MySQL R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/mysql/upsert/MySqlCustomerProfileUuidRepository.java | MySQL R2DBC test repository binding for TCK UUID upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/mysql/upsert/MySqlCustomerProfileRepository.java | MySQL R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/h2/upsert/H2WarehouseInventoryRepository.java | H2 R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/h2/upsert/H2ProductReviewRepository.java | H2 R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/h2/upsert/H2CustomerProfileUuidRepository.java | H2 R2DBC test repository binding for TCK UUID upsert contract. |
| data-r2dbc/src/test/java/io/micronaut/data/r2dbc/h2/upsert/H2CustomerProfileRepository.java | H2 R2DBC test repository binding for TCK upsert contract. |
| data-r2dbc/src/test/groovy/io/micronaut/data/r2dbc/sqlserver/SqlServerUpsertSpec.groovy | Adds SQL Server R2DBC upsert spec exercising TCK + sequence-id returning cases. |
| data-r2dbc/src/test/groovy/io/micronaut/data/r2dbc/postgres/vector/PostgresDbInit.java | Removes old vector-specific init hook (merged into shared Postgres init). |
| data-r2dbc/src/test/groovy/io/micronaut/data/r2dbc/postgres/PostgresUpsertSpec.groovy | Adds Postgres R2DBC upsert spec and wires spec-name properties. |
| data-r2dbc/src/test/groovy/io/micronaut/data/r2dbc/postgres/PostgresDbInit.java | Enhances Postgres R2DBC init: readiness wait (local only) + extension creation for specific specs. |
| data-r2dbc/src/test/groovy/io/micronaut/data/r2dbc/oraclexe/OracleXEUpsertSpec.groovy | Adds Oracle XE R2DBC upsert spec covering generated-id returning cases. |
| data-r2dbc/src/test/groovy/io/micronaut/data/r2dbc/operations/R2dbcRepositoryOperationsConditionsSpec.groovy | Tests per-dialect R2DBC operations bean selection logic. |
| data-r2dbc/src/test/groovy/io/micronaut/data/r2dbc/mysql/MySqlUpsertSpec.groovy | Adds MySQL R2DBC upsert spec and flags unsupported UUID returning. |
| data-r2dbc/src/test/groovy/io/micronaut/data/r2dbc/mariadb/MariaDbUpsertSpec.groovy | Adds MariaDB R2DBC upsert spec and flags unsupported UUID returning. |
| data-r2dbc/src/test/groovy/io/micronaut/data/r2dbc/h2/H2UpsertSpec.groovy | Adds H2 R2DBC upsert spec. |
| data-r2dbc/src/main/java/io/micronaut/data/r2dbc/operations/SqlServerR2dbcRepositoryOperations.java | New SQL Server R2DBC operations implementation handling MERGE OUTPUT id reads for upsert. |
| data-r2dbc/src/main/java/io/micronaut/data/r2dbc/operations/R2dbcRepositoryOperationsConditions.java | New shared R2DBC operations conditions for Oracle/SQL Server/default selection. |
| data-r2dbc/src/main/java/io/micronaut/data/r2dbc/operations/OracleR2dbcRepositoryOperations.java | New Oracle R2DBC operations implementation handling RETURNING INTO for upsert generated-id. |
| data-processor/src/test/groovy/io/micronaut/data/processor/sql/BuildTableSpec.groovy | Adds SQL Server sequence generation DDL test coverage. |
| data-processor/src/main/resources/META-INF/services/io.micronaut.data.processor.visitors.finders.MethodMatcher | Registers UpsertMethodMatcher with the processor. |
| data-processor/src/main/java/io/micronaut/data/processor/visitors/RepositoryTypeElementVisitor.java | Includes @Upsert in lifecycle method entity-type inference. |
| data-processor/src/main/java/io/micronaut/data/processor/visitors/finders/UpsertMethodMatcher.java | New processor matcher generating upsert stored queries + validating constraints. |
| data-processor/src/main/java/io/micronaut/data/processor/visitors/finders/FindersUtils.java | Routes UPSERT operation type through update-style interceptor selection. |
| data-model/src/main/java/io/micronaut/data/model/runtime/StoredQuery.java | Adds StoredQuery.OperationType.UPSERT. |
| data-model/src/main/java/io/micronaut/data/model/query/builder/sql/SqlQueryBuilder.java | Adds SQL Server sequence DDL tweaks + upsert query building entry point. |
| data-model/src/main/java/io/micronaut/data/model/query/builder/QueryBuilder.java | Adds buildUpsert API and UpsertQueryDefinition. |
| data-model/src/main/java/io/micronaut/data/intercept/annotation/DataMethodQuery.java | Adds UPSERT operation type. |
| data-model/src/main/java/io/micronaut/data/intercept/annotation/DataMethod.java | Adds UPSERT operation type. |
| data-model/src/main/java/io/micronaut/data/annotation/Upsert.java | New public @Upsert lifecycle annotation. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/sqlserver/upsert/MSWarehouseInventoryRepository.java | SQL Server JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/sqlserver/upsert/MSProductReviewRepository.java | SQL Server JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/sqlserver/upsert/MSCustomerProfileUuidRepository.java | SQL Server JDBC test repository binding for TCK UUID upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/sqlserver/upsert/MSCustomerProfileSequenceRepository.java | SQL Server JDBC test repository for sequence-generated identity upsert returning. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/sqlserver/upsert/MSCustomerProfileRepository.java | SQL Server JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/sqlserver/upsert/CustomerProfileSequence.java | SQL Server JDBC test entity for sequence-generated identity. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/postgres/upsert/PostgresWarehouseInventoryRepository.java | Postgres JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/postgres/upsert/PostgresProductReviewRepository.java | Postgres JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/postgres/upsert/PostgresCustomerProfileUuidRepository.java | Postgres JDBC test repository binding for TCK UUID upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/postgres/upsert/PostgresCustomerProfileSequenceRepository.java | Postgres JDBC test repository for sequence-generated identity upsert returning. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/postgres/upsert/PostgresCustomerProfileRepository.java | Postgres JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/postgres/upsert/CustomerProfileSequence.java | Postgres JDBC test entity for sequence-generated identity. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/oraclexe/upsert/OracleXEWarehouseInventoryRepository.java | Oracle XE JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/oraclexe/upsert/OracleXEProductReviewRepository.java | Oracle XE JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/oraclexe/upsert/OracleXECustomerProfileUuidRepository.java | Oracle XE JDBC test repository binding for TCK UUID upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/oraclexe/upsert/OracleXECustomerProfileSequenceRepository.java | Oracle XE JDBC test repository for sequence-generated identity upsert returning. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/oraclexe/upsert/OracleXECustomerProfileRepository.java | Oracle XE JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/oraclexe/upsert/CustomerProfileSequence.java | Oracle XE JDBC test entity for sequence-generated identity. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/mysql/upsert/MySqlWarehouseInventoryRepository.java | MySQL JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/mysql/upsert/MySqlProductReviewRepository.java | MySQL JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/mysql/upsert/MySqlCustomerProfileUuidRepository.java | MySQL JDBC test repository binding for TCK UUID upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/mysql/upsert/MySqlCustomerProfileRepository.java | MySQL JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/h2/upsert/H2WarehouseInventoryRepository.java | H2 JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/h2/upsert/H2ProductReviewRepository.java | H2 JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/h2/upsert/H2CustomerProfileUuidRepository.java | H2 JDBC test repository binding for TCK UUID upsert contract. |
| data-jdbc/src/test/java/io/micronaut/data/jdbc/h2/upsert/H2CustomerProfileRepository.java | H2 JDBC test repository binding for TCK upsert contract. |
| data-jdbc/src/test/groovy/io/micronaut/data/jdbc/sqlserver/SqlServerUpsertSpec.groovy | Adds SQL Server JDBC upsert spec + sequence-id returning cases. |
| data-jdbc/src/test/groovy/io/micronaut/data/jdbc/postgres/PostgresUpsertSpec.groovy | Adds Postgres JDBC upsert spec + sequence-id returning cases. |
| data-jdbc/src/test/groovy/io/micronaut/data/jdbc/oraclexe/OracleXEUpsertSpec.groovy | Adds Oracle XE JDBC upsert spec + sequence-id returning cases. |
| data-jdbc/src/test/groovy/io/micronaut/data/jdbc/operations/JdbcRepositoryOperationsConditionsSpec.groovy | Tests per-dialect JDBC operations bean selection logic. |
| data-jdbc/src/test/groovy/io/micronaut/data/jdbc/mysql/MySqlUpsertSpec.groovy | Adds MySQL JDBC upsert spec and flags unsupported UUID returning. |
| data-jdbc/src/test/groovy/io/micronaut/data/jdbc/mariadb/MariaUpsertSpec.groovy | Adds MariaDB JDBC upsert spec and flags unsupported UUID returning. |
| data-jdbc/src/test/groovy/io/micronaut/data/jdbc/h2/H2UpsertSpec.groovy | Adds H2 JDBC upsert spec. |
| data-jdbc/src/main/java/io/micronaut/data/jdbc/operations/SqlServerJdbcRepositoryOperations.java | New SQL Server JDBC operations implementation handling MERGE OUTPUT id reads for upsert. |
| data-jdbc/src/main/java/io/micronaut/data/jdbc/operations/JdbcRepositoryOperationsConditions.java | New shared JDBC operations conditions for Oracle/SQL Server/default selection. |
| * @param context The condition context | ||
| * @return The configured datasource names | ||
| */ | ||
| private static List<String> resolveConfiguredDataSourceNames(ConditionContext context) { |
There was a problem hiding this comment.
feels like there should be some utility for this since it is seems like a common requirements to get the configured datasource names. It should also probably cache
There was a problem hiding this comment.
I created DataSourceConfigurationUtils and moved resolveDataSourceName and resolveConfiguredDataSourceNames methods to it. The resolveConfiguredDataSourceNames method caches resolved data source names and keeps the cache in BeanContext attributes
| @Requires(classes = OraclePreparedStatement.class) | ||
| @Requires(condition = OracleJdbcRepositoryOperationsCondition.class) | ||
| @Internal | ||
| public final class OracleJdbcRepositoryOperations extends DefaultJdbcRepositoryOperations { |
There was a problem hiding this comment.
does it need to be public?
There was a problem hiding this comment.
DefaultJdbcRepositoryOperations is public too that was probably the reason why I put OracleJdbcRepositoryOperations and SqlServerJdbcRepositoryOperations to be public too, but I think those two don't have to be public so I modify it.
| import static io.micronaut.data.annotation.GeneratedValue.Type.AUTO; | ||
| import static io.micronaut.data.annotation.GeneratedValue.Type.SEQUENCE; | ||
|
|
||
| final class SqlUpsertQueryBuilder { |
There was a problem hiding this comment.
add maintainer focused javadoc to this class describing its purpose and implementation
| The default conflict target is the entity identity. | ||
| Use ann:data.annotation.Upsert[] with `conflictsOn` to use another persistent property, or a set of persistent properties, as the conflict target. | ||
|
|
||
| [source,java] |
There was a problem hiding this comment.
can we replace these hard coded snippets with the snippet: macro and actual tested source code?
| @Upsert | ||
| Contact put(Contact contact); | ||
|
|
||
| @Upsert(conflictsOn = "email") |
There was a problem hiding this comment.
since we generated JPA metadata at build time would be good to show an example that uses type safe property names instead of strings
There was a problem hiding this comment.
@graemerocher If we want to use type safe property names in the Upsert annoation, the entity must be compiled in another module. Currently Micronaut static metamodel generator is used only in data-tck module, so I would need to create a new entity in data-tck module and add data-tck dependency to a doc-examples project. Is that correct?
There was a problem hiding this comment.
I mean it should work if it is in the same module since javac supports multiple rounds of compilation
There was a problem hiding this comment.
Javac does support multiple processing rounds, but I tried this pattern
@Upsert(conflictsOn = Passenger_.EMAIL)
and Passenger_ is not resolvable when Micronaut Data processes the repository’s Upsert annotation. The metamodel source is generated in a later round, while UpsertMethodMatcher validates conflictsOn in the earlier round and sees an error type rather than the generated "email" constant.
There was a problem hiding this comment.
I will try to modify the code to use postponedRepositories in RepositoryTypeElementVisitor
There was a problem hiding this comment.
yeah this sounds like a bug
There was a problem hiding this comment.
I tried to do that using AI but without success.
The following is AI response:
Postponing RepositoryTypeElementVisitor when conflictsOn is <error> makes Passenger_.EMAIL resolve in the next javac round, but Micronaut’s bean-definition processor has already emitted the repository proxy in the first round. The resulting repository compiles but fails at runtime because its method has no compiled query metadata.
So this cannot be fixed safely in UpsertMethodMatcher or RepositoryTypeElementVisitor alone. It needs Micronaut core annotation-processing support that postpones both repository processing and bean-definition generation, then reprocesses the original repository in the later round.
There was a problem hiding this comment.
hmm ok, probably worth an issue with a reproducer for core
There was a problem hiding this comment.
Ok, once this PR gets merged, I will create an issue and reproducer.
…ourceNames and resolveDataSourceName methods from JdbcRepositoryOperationsConditions and R2dbcRepositoryOperationsConditions to it
…positoryOperations
|
| } | ||
| } | ||
|
|
||
| private void upsert() { |
There was a problem hiding this comment.
I don't like this custom repository operations, can we simply create a query that is a function for oracle or is an ordinary query for other cases


Closes #3351
Upsert implementation overview
This PR adds native SQL upsert support for Micronaut Data SQL repositories.
Repository API
Upsert is supported only through explicitly declared repository methods. No inherited
CrudRepositorymethods were added.Supported declaration styles:
The Upsert annotation also supports custom conflict columns through conflictsOn:
Reactive and async return types are also supported:
Dialect support
Native upsert SQL is generated per dialect: