Skip to content

MySQL 8.0 → PostgreSQL 17 Migration#7

Merged
mikewoo merged 28 commits into
mainfrom
spec/pg17-migration
Jun 6, 2026
Merged

MySQL 8.0 → PostgreSQL 17 Migration#7
mikewoo merged 28 commits into
mainfrom
spec/pg17-migration

Conversation

@mikewoo

@mikewoo mikewoo commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Summary

Complete migration from MySQL 8.0 + pgvector PG16 dual-storage to single PostgreSQL 17 database.

Changes

Database

  • Rewrite V1__init.sql as final-state PostgreSQL DDL (22 tables + seed data + document_chunk vector table)
  • Fold MySQL migration files V4–V7 into new V1
  • Merge document_chunk (pgvector) into main schema
  • Fix TIMESTAMPTZ → TIMESTAMP for MyBatis LocalDateTime compatibility
  • Fix VECTOR → VECTOR(1024) for HNSW index support

Data Access

  • Translate ON DUPLICATE KEY UPDATE → ON CONFLICT ... DO UPDATE (3 locations)
  • Translate JSON_EXTRACT(config, $.key) → (config->>key)::bigint (2 locations)
  • Delete MySQL-only fixDatabase endpoint

Datasource

  • Drop dual-datasource (PgVectorConfig, PgFlywayConfig)
  • Point vector JdbcTemplate at main PostgreSQL datasource
  • Replace Druid with HikariCP across all environments
  • Single Flyway instance managing all tables

Dependencies

  • Remove: mysql-connector-j, flyway-mysql, druid-spring-boot-4-starter
  • Retain: flyway-database-postgresql, pgvector, postgresql

Testing

  • Switch H2 from MySQL mode to PostgreSQL mode
  • Add Testcontainers PG17 integration tests (JSONB, upsert, vector, IDENTITY)

CI/CD

  • GitHub Actions: MySQL service → pgvector/pgvector:pg17
  • Jenkinsfile: MYSQL_*/PGVECTOR_* params → PG_*, credentials 7→5
  • K8s ConfigMap/Secret: MySQL fields → PG fields
  • Add SQL dialect regression guard (scripts/check_sql_dialect.sh)

Documentation

  • New ADR-0013: Single PostgreSQL 17 database decision
  • Supersede ADR-0004: MySQL + pgvector dual storage
  • Update all READMEs, ARCHITECTURE, DEPLOYMENT, CLAUDE.md, DATABASE.md

Cleanup

  • Delete deploy/sql/init_eify_mysql.sql and init_eify_vector_pgsql.sql
  • Remove Druid loggers, deduplicate ContextPropagatingTaskDecorator
  • Remove stale MySQL references from Java Javadoc

Version

  • Bump to v1.1.0

Breaking Changes

Area Before After
Database MySQL 8.0 + pgvector PG16 PostgreSQL 17 single instance
Connection pool Druid HikariCP
Env vars MYSQL_*, PGVECTOR_* PG_URL, PG_USERNAME, PG_PASSWORD
Docker service mysql + pgvector pgvector only
CI service mysql:8.0 pgvector/pgvector:pg17

Migration required: update .env, K8s secrets, and Jenkins credentials. See docs/DEPLOYMENT.md for details.

🤖 Generated with Claude Code

mikewoo and others added 28 commits June 5, 2026 16:52
@mikewoo @claude
@
d577300 
docs: add MySQL 8.0 to PostgreSQL 17 migration design spec

Spec for migrating the business DB from MySQL 8.0 to PG17 and merging
the pgvector store (pg16) into a single PG17 database.

Key decisions captured:
- Clean cutover, drop old data (no data-migration toolchain)
- Single datasource (drop Druid for HikariCP), single Flyway
- Business tables + document_chunk in one PG17 DB (same-tx, JOIN-able)
- Only 4 native SQL annotations need hand-translation (2 ON CONFLICT, 2 JSONB ->>)
- Hybrid test strategy: H2 PG-mode for speed + Testcontainers PG17 for fidelity
- CI grep guard against dialect regression
- IDENTITY/seed-data sequence sync (setval) for fixed-id seed rows

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@
@mikewoo @claude
@
28b3f86 
docs: harden PG17 migration spec with defensive-coding refinements

- setval COALESCE fallback to avoid NULL crash on empty-table seed sync
- CI guard rewritten with exit-code branch (robust on minimal runners)
- new 3.4: clean up MySQL dialect words in comments/logs to avoid CI false-positives

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@
@mikewoo @claude
@
085b69a 
docs: add PG17 migration implementation plan (14 tasks across 7 phases)

Bite-sized TDD tasks with exact files, code blocks, and verify commands.
Key correctness anchors:
- V1__init.sql must encode V1+V4~V7 final state (not per-version translation)
- ChunkRepository already PG-native: repoint datasource, do not rewrite SQL
- 4 dialect annotations translated with exact before/after
- Testcontainers PG17 covers IDENTITY backfill / JSONB / ON CONFLICT / vector

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@
@mikewoo @claude
feat(db): rewrite V1__init.sql as final-state PostgreSQL 17 schema
25b1f95 
Fold MySQL V1+V4~V7 into one PG DDL, merge document_chunk, drop migration-pg.
IDENTITY columns + seed sequence sync with COALESCE fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
docs(db): convert DATABASE.md conventions to PostgreSQL 17
b621e9d 
Field types, general columns, table template, index syntax, idempotency
template to PG native; fix references to deleted V4/V5 migrations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
feat(provider): translate health upsert to PostgreSQL ON CONFLICT
80580b5 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
feat(agent): translate agent_knowledge batch upsert to ON CONFLICT
75a4282 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
feat: translate JSON_EXTRACT to PostgreSQL ->> with ::bigint cast
aaa93f4 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
refactor: drop MySQL-only fixDatabase endpoint, clean dialect words i…
1a79b94 
…n logs

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
refactor: single datasource — point vector JdbcTemplate at main PG DB
b2329de 
Delete PgVectorConfig and PgFlywayConfig; ChunkRepository now injects the
auto-configured JdbcTemplate (single datasource enables Spring Boot's
JdbcTemplateAutoConfiguration).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
build: drop MySQL/Druid/flyway-mysql deps, remove dual-datasource wor…
e86083f 
…karound

Remove PrimaryDataSourceConfig (no longer needed after dual datasource removed);
Spring Boot auto-configures HikariCP DataSource + JdbcTemplate from single source.
Add postgresql + pgvector deps. Clean MySqlFlywayConfig javadoc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
config: main datasource to PostgreSQL + HikariCP, merge vector source
bb3c652 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
config: replace Druid pool with HikariCP across dev/prod/staging
79f6f3b 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
deploy: drop mysql service, pgvector pg17 hosts single DB
d620bcc 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
test: switch H2 to PostgreSQL mode, IDENTITY/TEXT schema adaptations
da8308d 
- application-test.yml: replace Druid+MySQL-mode datasource with HikariCP
  + H2 PostgreSQL mode (MODE=PostgreSQL;DATABASE_TO_LOWER=TRUE); remove
  datasource-pgvector block; drop DruidDataSourceAutoConfigure from exclude
- schema-h2.sql: AUTO_INCREMENT→GENERATED BY DEFAULT AS IDENTITY (16 tables),
  CLOB→TEXT (11 columns), DOUBLE→DOUBLE PRECISION (position_x/y),
  add missing description column to mcp_server

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
test: fix H2 cross-context contamination and IDENTITY sequence collision
ac1cc17 
Two fixes for ProviderControllerIntegrationTest failures in H2 PostgreSQL mode:

1. data-h2.sql: restart IDENTITY sequences to 100 after explicit-id seed inserts
   (ai_user, ai_workspace, provider). H2 PostgreSQL mode does not advance the
   IDENTITY sequence on explicit-id inserts, so app-generated ids collide with
   seed ids (pk=1). Mirrors production V1__init.sql setval sync.

2. schema-h2.sql: prepend DROP TABLE IF EXISTS for all tables before CREATE.
   With DB_CLOSE_DELAY=-1, mem:testdb persists across nested @SpringBootTest
   contexts in the same JVM. Without drops, leftover rows from prior contexts
   (e.g. ai_agent rows causing countAgentReferences to return 1 for a freshly
   created provider) contaminate subsequent test contexts, causing spurious
   PROVIDER_IN_USE and name-uniqueness failures.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
test: add Testcontainers PG17 coverage for JSONB/upsert/vector/identity
46d3440 
Verifies dialect behaviors H2 PostgreSQL mode cannot: JSONB ->> with ::bigint
cast, ON CONFLICT upsert, pgvector cosine search, IDENTITY backfill. Auto-skips
when Docker unavailable (disabledWithoutDocker = true).

Testcontainers 2.x changes applied:
- artifactIds: testcontainers-postgresql, testcontainers-junit-jupiter (not old
  postgresql/junit-jupiter 1.x names)
- PostgreSQLContainer now in org.testcontainers.postgresql (not .containers)
- No generic type param on PostgreSQLContainer (self-bounded in 2.x)
- @AutoConfigureTestDatabase removed in Spring Boot 4.x; @Serviceconnection
  handles datasource injection directly

Versions managed by Spring Boot 4.0.6 BOM via testcontainers-bom 2.0.5 —
no explicit version in pom.xml needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
ci: add SQL dialect regression guard for PostgreSQL migration
9892029 
- Add scripts/check_sql_dialect.sh: CI guard that fails on MySQL-specific
  syntax (ON DUPLICATE KEY / JSON_EXTRACT / INFORMATION_SCHEMA / MODIFY COLUMN)
- Rename MySqlFlywayConfig -> FlywayConfig; update property key from
  eify.flyway.mysql.* to eify.flyway.pg.* throughout all env ymls
- Remove dead Druid connection-pool block from application-test.yml (replaced
  by HikariCP in previous migration tasks)
- Fix test-side application-test.yml: eify.flyway.mysql.enabled -> pg.enabled
  so FlywayConfig @ConditionalOnProperty correctly disables on H2 test context
  (was causing 32 test errors: H2 failing on CREATE EXTENSION vector)

All 931 backend tests pass (4 Skipped = Testcontainers PgIntegrationTest,
no Docker). Frontend: vue-tsc clean, vitest 11/11.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mikewoo @claude
chore: remove Druid/MySQL dependencies, deduplicate ContextPropagatin…
9aeba9e 
…gTaskDecorator

- Remove druid.version, mysql.version and their dependencyManagement entries from parent pom
- Remove Druid logger lines from logback-spring.xml (no longer relevant)
- Wrap FILE_JSON/FILE_JSON_ASYNC appenders in springProfile=prod,staging to fix dev warning
- Delete duplicate ContextPropagatingTaskDecorator from eify-knowledge (use eify-common copy)
- Update AsyncConfig import to eify-common version

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mikewoo @claude
fix(db): clean obsolete SQL files, fix VECTOR/TIMESTAMP types for PG …
6f64659 
…compatibility

- Delete deploy/sql/init_eify_mysql.sql (obsolete MySQL schema dump)
- Delete deploy/sql/init_eify_vector_pgsql.sql (obsolete pgvector-only schema)
- Rewrite deploy/sql/README.md for single PG17 database architecture
- Fix VECTOR -> VECTOR(1024) to allow HNSW index creation on empty table
- Fix TIMESTAMPTZ -> TIMESTAMP for MyBatis LocalDateTime compatibility
- Fix application-pgtest.yml: eify.flyway.mysql.enabled -> eify.flyway.pg.enabled

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mikewoo @claude
ci: switch CI/CD, K8s config, startup scripts from MySQL to PostgreSQL
bdccbe7 
- .github/workflows/ci.yml: MySQL service -> pgvector/pgvector:pg17 container
- Jenkinsfile: MYSQL_HOST/PGVECTOR_HOST params -> PG_HOST, credentials 7->5
- deploy/k8s/configmap.yaml: MYSQL_*/PGVECTOR_* -> PG_URL
- deploy/k8s/secret.yaml: MYSQL_USERNAME/PASSWORD + PGVECTOR_PASSWORD -> PG_USERNAME/PG_PASSWORD
- start.sh: MySQL check -> PostgreSQL, MYSQL_*/PGVECTOR_* env vars -> PG_*
- .env.example: remove MySQL/Druid/PGVECTOR sections, add PG_URL/PG_USERNAME/PG_PASSWORD
- .github/ISSUE_TEMPLATE/bug_report.md: MySQL 8.x -> PostgreSQL 17

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mikewoo @claude
docs: update all documentation to PostgreSQL 17 single-database archi…
61dfd11 
…tecture

- CLAUDE.md: INFORMATION_SCHEMA -> PG native IF NOT EXISTS, MySQL -> PostgreSQL
- README/README.zh-CN: architecture diagram, tech stack table, startup commands
- CHANGELOG.md: MySQL + pgvector -> PostgreSQL 17
- CONTRIBUTING.md: MySQL -> PostgreSQL in setup and CI description
- docs/README.md: MySQL 8.0 -> PostgreSQL 17
- docs/ARCHITECTURE.md: persistent layer, Flyway, knowledge module descriptions
- docs/DEPLOYMENT.md: architecture, env vars table, K8s secret commands, Druid -> HikariCP
- ADR-0004: Status Accepted -> Superseded, reference ADR-0013
- ADR-0013: new — single PostgreSQL 17 database decision

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mikewoo @claude
chore: remove stale MySQL references from Java Javadoc comments
fc8de60 
- Document.java: remove '(MySQL)' from class Javadoc
- ChunkRepository.java: remove 'MySQL 的' from class Javadoc
- DocumentRepository.java: '(MySQL)' -> '(PostgreSQL)'
- KnowledgeRepository.java: '(MySQL)' -> '(PostgreSQL)'

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mikewoo @claude
chore: bump version to 1.1.0
be04c99 
- pom.xml: 1.0.2-SNAPSHOT -> 1.1.0
- eify-web/package.json: 1.0.2-SNAPSHOT -> 1.1.0
- eify-web/package-lock.json: 1.0.2-SNAPSHOT -> 1.1.0

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mikewoo @claude
fix(frontend): resolve vue-tsc build errors, extract component types
57a5905 
- Change vue-tsc -b -> vue-tsc --noEmit (project references mode incompatible
  with export inside <script setup generic>)
- Extract SearchField/SearchCondition types to src/types/eify-search.ts
- Extract TableColumn/PaginationParams/PageResult types to src/types/eify-table.ts
- Fix locale.ts: narrow setLocale param type to 'zh-CN' | 'en-US'
- Fix view files: widen handler signatures to accept Record<string, any>
  from slot props
- Fix AgentList.vue: add description to McpServerWithTools, cast type assertions

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mikewoo @claude
fix(test): add explicit PG driver to pgtest profile for Testcontainers
1ff1637 
@Serviceconnection auto-injects connection details but the driver-class-name
must be set explicitly to prevent Spring Boot from picking up H2 driver
from application-test.yml if the test profile leaks into the context.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mikewoo @claude
fix(test): use custom FlywayConfig in pgtest instead of Spring Boot a…
6a9a709 
…uto-config

Spring Boot FlywayAutoConfiguration doesn't reliably pick up @Serviceconnection
datasource. Switch to programmatic FlywayConfig (eify.flyway.pg.enabled=true)
which directly uses the injected DataSource bean.

- spring.flyway.enabled -> false
- Exclude FlywayAutoConfiguration
- eify.flyway.pg.enabled -> true (programmatic FlywayConfig)
- Explicitly set driver-class-name: org.postgresql.Driver

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mikewoo @claude
fix(test): use 1024-dim vector in PgIntegrationTest to match VECTOR(1…
412639c 
…024) column

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mikewoo mikewoo self-assigned this Jun 6, 2026
@mikewoo mikewoo merged commit 49e41dd into main Jun 6, 2026
3 checks passed
@mikewoo mikewoo deleted the spec/pg17-migration branch June 6, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@mikewoo mikewoo

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mikewoo