Skip to content

fix: Re-importable Bulk Export Closure and Lookup Master Fixes#128

Merged
ilramdhan merged 4 commits into
mutugading:mainfrom
ilramdhan:fix/seed-master-data-product-cost
Jun 23, 2026
Merged

fix: Re-importable Bulk Export Closure and Lookup Master Fixes#128
ilramdhan merged 4 commits into
mutugading:mainfrom
ilramdhan:fix/seed-master-data-product-cost

Conversation

@ilramdhan

Copy link
Copy Markdown
Member

Description

This PR significantly enhances the Bulk Export feature, ensuring that any generated Excel file is fully "re-importable" across different environments without breaking relational dependencies.

It achieves this by implementing a Breadth-First Search (BFS) closure that automatically exports the entire dependency tree of a selected product. Additionally, it transitions the universal cross-sheet key from flex_02 to product_code, ensuring stability for products created within the app rather than imported from Oracle. It also resolves a dynamic SQL issue where lookup dropdowns were returning empty results due to incorrect column name mapping.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 💥 Breaking change (fix or feature that changes existing API)
  • ♻️ Refactor (code change without new feature or bug fix)
  • 📚 Documentation update
  • 🧪 Test update
  • 🔧 Chore (dependencies, config, etc.)

Service(s) Affected

  • Finance Service
  • IAM Service
  • Shared Proto (gen/)
  • Root/Common

Changes Made

📤 Re-importable Bulk Export Closure

  • BFS Product Closure: Added ProductSysIDs filter to ExportRequest. Added resolveProductClosure() which performs a BFS from the seed products, crawling through PRODUCT-type route RMs to collect every intermediate product in the dependency tree. The resulting export includes the full closure, guaranteeing no broken references on re-import.
  • Universal Cross-Sheet Key: Switched the legacy key from flex_02 to product_code. flex_02 is empty for products created directly within the app, causing export/import failures. product_code ensures a stable, non-empty identifier.
  • Cross-Reference Resolution: Fixed route_head, route_sequences, and route_rms sheets to use product_code instead of raw integer IDs (product_sys_id / head_id) so that sheet cross-references resolve correctly during import.
  • Sheet/Column Alignment: Fixed template sheet names (e.g., product_parameters, product_applicable_params, route_sequences, route_rms) and added the data_type column to the product_parameters sheet to strictly match the import validator's expectations.
  • Data Filtering: cpp and capp rows are now strictly filtered to only include products present in the export closure.

🐛 Bug Fixes

  • Lookup Master Columns: Fixed an issue where ListMasterOptions returned empty dropdowns. Migration 000394 originally seeded lm_code_field and lm_label_field with camelCase proto fields (e.g., machineCode). This is now corrected to use actual snake_case DB column names (e.g., mc_code) for dynamic SQL mapping.
  • PostgreSQL 18 Type Inference (SQLSTATE 42P08): Added an explicit $8::text cast in the BulkUpsertByLegacyID subquery to prevent type inference failures when the same parameter appears in both VALUES and a correlated subquery.

Related Issues

Fixes #
Related to #

API Changes (if applicable)

Proto Changes

// No Proto Changes

Breaking Changes

None.

Testing Performed

Unit Tests

  • Existing unit tests pass
  • Coverage maintained/improved

Integration Tests

  • New integration tests added
  • Existing integration tests pass

Manual Testing

# Deployed Finance Service locally
# Triggered Bulk Export for a single product and verified BFS closure gathered all sub-materials
# Verified the exported Excel file could be seamlessly re-imported into a clean database environment
# Verified Lookup Master dropdowns correctly populate data in the UI

Lint & Build

  • golangci-lint run ./... passes
  • go build ./... succeeds
  • go test -race ./... passes

Database (if applicable)

  • Migration added
  • Migration tested (up and down)
  • No breaking schema changes (or documented)

Documentation

  • README.md updated (if needed)
  • RULES.md updated (if needed)
  • Proto comments updated
  • OpenAPI regenerated

Rollback Plan

Revert the finance-service deployment to the previous stable tag. Reverting will cause exports of app-created products to use the old flex_02 key, which may not be re-importable, but will not corrupt any existing database state.

Screenshots/Logs (if applicable)


Pre-merge Checklist

  • I have read and followed RULES.md
  • I have read and followed CONTRIBUTING.md
  • Clean Architecture principles followed
  • All errors are properly handled
  • Context is passed appropriately
  • Structured logging is used
  • No hardcoded secrets
  • PR description is complete and clear
  • CI checks are passing

Reviewer Notes

  • BFS Depth limits: The resolveProductClosure BFS algorithm safely collects the entire tree structure. Circular dependencies are natively avoided by tracking visited product_codes.
  • SQLSTATE 42P08: The cast fix is specific to pgx/PostgreSQL 18 behavior where untyped placeholders in complex ON CONFLICT subqueries fail parser resolution.

ilramdhan and others added 3 commits June 23, 2026 09:30
Add ProductSysIDs filter to ExportRequest. When set, resolveProductClosure()
performs a BFS from the seed products through PRODUCT-type route RMs,
collecting every intermediate product in the dependency tree. The export
includes all products in the closure so the resulting Excel is fully
re-importable without broken references.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migration 000394 seeded lm_code_field/lm_label_field with camelCase proto
field names (e.g. machineCode, pgCode) instead of actual snake_case DB
column names (mc_code, pg_code). ListMasterOptions uses these values
directly in dynamic SQL, so all lookup dropdowns returned empty results.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Use product_code as the universal cross-sheet key (legacy_oracle_sys_id)
  instead of flex_02 (empty for in-app products) so exports from app-created
  products always have a non-empty, stable identifier
- Add data_type column to product_parameters sheet (required by import validator)
- Filter cpp/capp rows to only include products in the export closure
- Fix route_head/route_sequences/route_rms to use product_code instead of
  raw integer product_sys_id / head_id, so sheet cross-references resolve
- Fix SQLSTATE 42P08: add explicit $8::text cast in BulkUpsertByLegacyID
  subquery to prevent PostgreSQL 18 type inference failure when the same
  parameter appears in both VALUES and a correlated subquery
- Align template sheet names with what the import handler expects
  (product_parameters, product_applicable_params, route_sequences, route_rms)
  and fix column headers to match the actual importer field names

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ilramdhan ilramdhan added this to the Costing Release Milestone milestone Jun 23, 2026
@ilramdhan ilramdhan self-assigned this Jun 23, 2026
@ilramdhan ilramdhan added bug Something isn't working fix labels Jun 23, 2026
- Extract loadRoutingData helper from loadExportData (gocyclo 17→11)
- Extract buildExportCodeMaps, filterCPPByProductCode, filterCAPPByProductCode
  helpers from generateExcel (gocyclo 16→10)
- Add nolint:gocognit on resolveProductClosure (BFS traversal is
  inherently nested and cannot be split without losing clarity)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ilramdhan ilramdhan merged commit 8ed5b1c into mutugading:main Jun 23, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working fix

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant