Import Data

[kryptodrex]

• Ability to import pay data from a pay stub or paycheck
• Ability to upload a CSV template with recurring expenses, savings, loans, etc. and have the app automatically create the necessary line items

[kryptodrex]

Import Data Plan

Status keys:

• [ ] Planned
• [-] In Progress
• [x] Done

---

Goal

Allow users to import planning data from external files safely and predictably.

Scope Decision for

• In scope (MVP): Build a reusable ingestion engine with source adapters, normalization, validation, and review-before-apply flow.
• In scope (MVP): CSV template import for recurring expenses, savings, loans, accounts, and other future-supported entities.
• In scope (Candidate expanded scope): Automated ingestion from non-CSV structured or semi-structured files where extraction quality is acceptable.
• In scope (Candidate expanded scope): Human review/edit/approval step before imported data is committed.
• Out of scope unless explicitly approved for: OCR-heavy ingestion for scanned documents with weak text extraction.
• Out of scope unless explicitly approved for: Fully generic provider-agnostic paystub parsing with no source-specific heuristics.

Ingestion Engine Architecture

Core Pipeline

• Define a shared ingestion pipeline: source -> extract -> normalize -> validate -> preview -> apply.
• Introduce a normalized intermediate model for imported entities before they are converted into BudgetData structures.
• Add confidence metadata per field and per row so low-confidence values can be highlighted in review UI.
• Support partial imports with explicit warnings and skipped-row reporting.
• Add audit/history description for import operations so users can understand what was created or updated.

Source Adapter Model

• Add adapter interface for csv, json, paystub-pdf, and future sources.
• Keep extraction logic source-specific and normalization logic shared.
• Allow adapters to emit candidate accounts, bills, loans, savings, pay details, and tax line data.
• Require every adapter to provide structured warnings/errors rather than throwing raw parser failures.

Review and Correction Layer

• Add review screen that shows parsed rows grouped by entity type.
• Allow inline correction of key fields before apply.
• Highlight low-confidence values and missing required mappings.
• Require explicit confirm/apply action after review.
• Support deselecting rows or sections the user does not want imported.

---

Workstream A: CSV Template Import

Product Behavior

• Add an Import Data entry point in plan workflow.
• Allow selecting CSV file through Electron file dialog.
• Parse rows into supported entities with validation.
• Show import preview with row-level errors and warnings.
• Let user choose append vs replace behavior per entity category.
• Apply import atomically with clear success/error feedback.

Technical Plan

• Add CSV parser utility/service (schema-driven, testable, no UI coupling).
• Define and version a CSV template schema.
• Add input normalization for frequency labels, booleans, amounts, and account matching.
• Add conflict handling strategy for duplicate names/IDs.
• Add import transaction layer (batch update) for consistency.
• Add optional downloadable CSV template.
• Implement CSV adapter on top of the shared ingestion engine.

Testing Plan

• Unit tests for parser and schema validation.
• Unit tests for normalization and duplicate/conflict handling.
• Integration tests for append/replace outcomes.
• Error-path tests for malformed CSV, missing columns, invalid frequencies.

---

Workstream B: Automated Non-CSV Ingestion (candidate scope)

Supported Candidate Sources

• Paystub PDFs with readable text layers.
• Exported JSON or spreadsheet-like intermediary formats from payroll/banking tools.
• Delimited text variants beyond CSV if they map cleanly into the same schema.

Product Behavior

• Add source selection in Import Data flow.
• Detect source type automatically when possible, with manual override.
• Extract candidate values into the shared review UI.
• Show confidence level and parser notes for ambiguous values.
• Let the user edit, remove, or confirm parsed results before apply.

Technical Plan

• Add extractor abstraction separate from entity normalization.
• For PDF text-layer documents, extract raw text blocks and run rule-based field detection.
• Start with source-specific heuristics rather than claiming universal parsing support.
• Add mapping utilities for labels such as gross pay, net pay, tax lines, deductions, benefits, and employer retirement contributions.
• Separate extraction confidence from validation errors so the UI can distinguish "probably right" from "definitely invalid".

Safety Constraints

• Do not auto-apply non-CSV extracted data without user review.
• Mark low-confidence parsed values as requiring review.
• Skip unsupported/unknown fields rather than fabricating mappings.
• Preserve privacy by keeping parsing local in-app unless a future cloud feature is explicitly approved.

Testing Plan

• Fixture-based tests for each supported non-CSV adapter.
• Confidence-scoring tests for ambiguous label mapping.
• UI tests for review/edit/approve behavior.
• Regression tests ensuring bad extraction does not corrupt plan data.

---

Workstream C: Paystub/PDF Advanced Parsing (stretch / partial defer)

Still Likely Too Large for Full Coverage

• Scanned PDFs requiring OCR.
• Provider-agnostic parsing across widely different paystub formats.
• Deep jurisdiction-specific paycheck semantics without manual review.

If Included At All In

• Limit support to readable text-layer paystubs only.
• Treat support as best-effort with mandatory review.
• Prefer a short supported-source list or heuristic-quality disclaimer.

---

Risks

• Risk: Import can overwrite user data unexpectedly.
• Mitigation: Mandatory preview + explicit append/replace choice + undo/history integration.
• Risk: CSV quality variance from user-edited files.
• Mitigation: Strict schema validation with row-level error messages and partial import policy.
• Risk: Automated extraction may appear trustworthy even when field mapping is wrong.
• Mitigation: Confidence UI, required review for non-CSV sources, and skipped-field transparency.
• Risk: PDF/source-specific logic can sprawl quickly and consume the release.
• Mitigation: Build adapter architecture first and gate each new source behind fixture tests.

Dependencies

• Depends on stable file picker/read APIs in Electron preload/main.
• Depends on update paths in budget context/services to apply batch changes safely.
• Depends on a plan-safe preview/apply transaction flow so imported data can be reviewed before commit.

---

Acceptance Criteria

• Users can import a valid CSV template and see all entries created correctly.
• Users can review and edit imported data before apply.
• Invalid rows are reported with actionable error messages.
• Import operations are reversible or recoverable through existing safety mechanisms.
• If non-CSV ingestion ships, supported sources are explicitly documented and require review before apply.

Done Definition

• Ingestion engine MVP is shipped and validated by tests.
• CSV adapter is complete and stable.
• Non-CSV adapters are either implemented with fixture coverage or explicitly deferred before release cut.