feat/fix: 6-Sheet Bulk Import/Export Engine for Product Routing#127
Merged
ilramdhan merged 22 commits intoJun 22, 2026
Merged
Conversation
…k_cpm_flex02, and regenerated proto code Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…interfaces + postgres implementations - New package costbulkimport: ImportMaps struct (5 maps for sheet-by-sheet ID resolution) and ParseSheet utility for header-validated Excel sheet parsing with empty-row skipping. - costproductmaster.Repository: add ProductUpsertInput/ProductUpsertResult types and BulkUpsertByLegacyID method using cpm_flex_02 as upsert conflict key, batched at 200. - costroute.Repository: add HeadUpsertInput/Result, SeqUpsertInput/Result, RMInput types and BulkUpsertHeads/BulkUpsertSeqs/BulkReplaceRMs methods for import-driven bulk writes. - All three new repo methods implemented in postgres layer; existing fakeRepoForDup test stub extended to satisfy the updated interface. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…terfaces - parser.go: simplify header search loop using slices.Contains (gocritic/intrange lint) - cost_product_master_repository.go: replace manual min-clamp with min() builtin (gocritic/ifElseChain lint) go build + go vet: 0 errors Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…cost_product_code arity, ON CONFLICT predicates, and Skipped logic Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…pp, capp (task-2b) - Add CPPUpsertInput and CAPPUpsertInput types to costproductparameter domain - Add BulkUpsertValues and BulkUpsertApplicable to Repository interface - Implement both bulk upsert methods in postgres repo (batches of 200, xmax insert detection) - Create sheet_product_master.go, sheet_cpp.go, sheet_capp.go processors - Add stub methods to fakeRepo in handlers_test.go Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add sheet_route_head.go, sheet_route_seq.go, and sheet_route_rm.go to the costbulkimport package, implementing processRouteHead, processRouteSeq, and processRouteRM for Sheets 4–6 of the 6-sheet legacy Oracle import format. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move repeated column header literals into a shared constants.go file in the costbulkimport package to satisfy the goconst linter (3+ occurrences). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…and repo additions for bulk import (Task 2d) - Add ListAllParams to costproductparameter.Repository + postgres impl - Add ListAllActive to costproducttype.Repository + postgres impl - Add ExportRouteHead/Seq/RM types + ListAllHeadsForExport/SeqsForExport/RMsForExport to costroute.Repository + postgres impl - Add error_report.go: GenerateErrorReport builds multi-sheet Excel error report - Add handler.go: BulkImportHandler.Handle processes 6-sheet import lifecycle PENDING→RUNNING→DONE/PARTIAL/FAILED - Add validate_handler.go: ValidateHandler.Validate does synchronous dry-run with max 20 sample errors per sheet - Add export_handler.go: ExportHandler.Handle generates 6-sheet Excel export from DB data, uploads to MinIO - Add handler_test.go: tests for GenerateErrorReport, countErrors, appendIfUnderLimit - Update test fakes in costroute, costproducttype, costproductparameter to implement new interface methods Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…unused fileName param - ListAllParams: replace non-existent p.uom_code column reference with LEFT JOIN mst_uom u ON u.uom_id = p.uom_id, consistent with all other queries in the file - ExportHandler: add typeRepo field + param to NewExportHandler; build typeIDToCode reverse map from ListAllActive; write product_type_code string instead of product_type_id int32 in product_master sheet header and data rows - ExportHandler: apply ProductTypeCodes filter using typeCodeToID map when non-empty - BulkImportHandler.Handle: log fileName at start to eliminate unparam CI lint failure - ValidateHandler.validateProductMaster: remove unused _ time.Time parameter and remove time import + now := time.Now() in Validate() to fix unparam Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nto gRPC delivery and worker - cost_import_handler.go: add ImportBulkProductRouting, ValidateBulkProductRoutingFile, ExportBulkProductRouting methods; add validateBulkHandler field; add toProtoBulkSheetResult and marshalExportRequestKey helpers - costing_import_handler.go: add bulkImportHandler/bulkExportHandler fields; add EntityBulkProductRouting and EntityBulkProductRoutingExport dispatch cases; add unmarshalExportRequest helper - cmd/server/main.go: instantiate bulkValidateH and pass to NewCostDataImportHandler - cmd/worker/main.go: instantiate bulkImportHandler, bulkExportHandler and pass to NewCostingImportHandler Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…uards in gRPC handler Dead bulkValidateHandler instantiation in cmd/worker/main.go was created only to be suppressed with _ = — it is wired by the server, not the worker. Nil-guards on validateBulkHandler and importPublisher in cost_import_handler.go masked programming errors that should fail loudly at startup or return a real error. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…emove dead rowNums - Add RouteLevel/RouteSeq fields to ExportRouteRM struct - JOIN cost_route_seq in ListAllRMsForExport to populate those fields - Write rm.RouteLevel/rm.RouteSeq in writeRouteRMSheet instead of empty strings - Add lowercase JSON tags to ExportRequest struct - Remove dead rowNums slice in processProductMaster (built but never used) Fix 2 (partial unique index on crh_product_sys_id): already exists as uk_cost_route_head_active_per_product in migration 000222 — no action needed. Fix 3 (MarkPartial): MarkDone() already handles PARTIAL status internally when j.failed > 0 — no separate MarkPartial method needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…g from unique index
… download Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ities The original constraint (migration 000377) only allowed 5 values and did not include the two bulk routing entities added later — causing INSERT failures when queuing a bulk export or import job. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Export jobs store request parameters as JSON in the file_key column (not a MinIO object path). The worker was calling fetchFile() for all entity types unconditionally, causing a "stat object: key does not exist" error for every bulk export job. Fix: early-dispatch EntityBulkProductRoutingExport via a dedicated handleExport() method before the fetchFile path. Import jobs are unaffected. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…seed - cost_import_handler: presigned URL now uses entity-specific filename; export jobs download as bulk_product_routing_export.xlsx instead of the misleading import_errors.xlsx - IAM migration 000061: seed FINANCE_IMPORT_JOBS menu (level-3 under FINANCE_PRODUCT_COSTING, no permission entries = visible to all authenticated users) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ilramdhan
added
bug
- errcheck: expand all `_ = f.Close()` defers to `if err := f.Close(); err != nil { _ = err }`
- errcheck: propagate SetCellValue/DeleteSheet/CoordinatesToCellName errors instead of blank-assigning
- goconst: add boolTrueStr const and replace 4 occurrences of "true" literal
- gocognit: extract filterProductsByTypeCode helper to reduce loadExportData complexity 25→≤20
- gocognit: add nolint annotations to 3 cohesive row-validation pipelines (21-23)
- unparam: annotate appendIfUnderLimit limit param with nolint
- gofmt: fix formatting in costroute/types.go, cost_route_repository.go, costing_import_handler.go
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- gofmt: fix formatting in sheet_route_rm, sheet_route_head, export_handler, sheet_route_seq, validate_handler - gocyclo: add nolint annotation to processRouteRM (cyclomatic 16 > threshold 15) - gocognit: extract populateTemplateSheet helper to reduce TemplateHandler.Handle complexity from 23 to ≤20 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
github-project-automation
Bot
moved this from In progress
to Done
in Goapps Roadmap [Backend]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces the Bulk Product Routing Engine, a comprehensive, highly scalable feature to handle bulk importing and exporting of complex costing data via a 6-sheet Excel file.
The engine supports asynchronous bulk processing for Product Masters, Applicable Params (CAPP), Parameter Overrides (CPP), Route Heads, Route Sequences, and Route Raw Materials (RM). It includes synchronous dry-run validation, extensive error reporting generation (downloadable as Excel), and integration with the existing RabbitMQ + MinIO worker architecture.
Type of Change
Service(s) Affected
Changes Made
📊 6-Sheet Bulk Import Engine (Finance)
costbulkimportpackage withImportMaps(for cross-sheet ID resolution) and a robustParseSheetutility with header validation and empty-row skipping.UpsertandReplacemethods toCostProductMasterandCostRouterepositories leveragingON CONFLICTclauses for high performance. Batched operations (e.g., 200 rows per batch) are utilized to prevent memory exhaustion.BulkImportHandlerto parse the 6 sheets (Product Master, CAPP, CPP, Route Head, Route Seq, Route RM) asynchronously. Job status transitions throughPENDING → RUNNING → DONE/PARTIAL/FAILED.ValidateHandlerfor synchronous dry-runs with a 20-error-per-sheet limit, andGenerateErrorReportto create downloadable multi-sheet Excel files detailing import failures.fetchFile()for export jobs (which store JSON params instead of object paths). Fixedchk_cij_entityconstraint to accept the new bulk entity constants.📤 Bulk Export & Template Handlers (Finance)
ExportHandlerto dynamically generate the 6-sheet Excel file from current DB state (supporting ProductType filtering), pushing the artifact to MinIO.BulkImportTemplateHandlerfor users to download an empty, correctly formatted 6-sheet template.🐛 SQL, Linting & Domain Fixes
ON CONFLICTpredicates andgenerate_cost_product_codearity bugs.min()/max()(Go 1.21+).goconst,unparam,gocritic/ifElseChain, etc.).ListAllParamsmissinguom_codeby adding a correctLEFT JOINtomst_uom.RouteLevelandRouteSeq.🔐 IAM Service
000061to seed theFINANCE_IMPORT_JOBSmenu underFINANCE_PRODUCT_COSTING. Since it has no permission entries, it is intentionally visible to all authenticated Finance users.Related Issues
Fixes #
Related to #
API Changes (if applicable)
Proto Changes
+ // Added: ImportBulkProductRouting, ValidateBulkProductRoutingFile, ExportBulkProductRouting, DownloadBulkProductRoutingTemplateBreaking Changes
None.
Testing Performed
Unit Tests
GenerateErrorReportand limits).Integration Tests
Manual Testing
Lint & Build
golangci-lint run ./...passesgo build ./...succeedsgo test -race ./...passesDatabase (if applicable)
uk_cpm_flex02adjustments, extendedchk_cij_entityconstraint.000061(Import Jobs Menu).Documentation
Rollback Plan
finance-serviceandfinance-worker.000061to remove the Import Jobs menu.chk_cij_entityenum is non-trivial in Postgres, so any queued export/import jobs in the DB will simply fail to process if the code is rolled back.Screenshots/Logs (if applicable)
Pre-merge Checklist
Reviewer Notes
file_keycolumn rather than pointing to a pre-existing MinIO file. The worker now catches this case early and routes it directly toExportHandler.cpm_flex_02as the legacy ID conflict key. Any manual modifications to this column could disrupt subsequent bulk imports.