Skip to content

CHANGELOG.md

Latest commit

 

History

History
56 lines (47 loc) · 6.46 KB

CHANGELOG.md

File metadata and controls

56 lines (47 loc) · 6.46 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

  • Mathematical Privacy Evaluator (cloakdb evaluate): Evaluates formal $k$-anonymity, $l$-diversity, and re-identification risk metrics across quasi-identifiers and sensitive attributes with CI/CD pass/fail gates and JSON output.
  • Referential Data Subsetting Engine (cloakdb subset): Graph-based relational subsetting traversing foreign keys upstream and downstream to export compact, consistent staging datasets.
  • Free-Text Semantic PII Redactor (text_redact): In-place unstructured entity redaction for customer notes, support tickets, and chat logs targeting emails, phones, credit cards, TCKN, SSN, and IPv4 addresses.
  • GitHub Actions Step Summaries & PR Annotations: Native $GITHUB_STEP_SUMMARY markdown report publishing and inline PR annotation commands (::error, ::warning) on schema drift and PII leaks.
  • Git Pre-Commit Hook (.pre-commit-hooks.yaml): Seamless pre-commit hook integration running cloakdb lint on SQL migration files before committing.
  • Pluggable KMS & Secret Vault Providers: Extensible SecretProvider interface supporting environment variables, HashiCorp Vault KV v2, and AWS KMS key decryption.
  • DuckDB Connector (DuckDBConnector): Native support for querying, inspecting, and in-place chunked masking of .duckdb databases.
  • Apache Airflow Orchestration Operator (CloakDBOperator): Standard operator for integrating CloakDB directly into Apache Airflow DAGs.

Security

  • Strict Insecure Salt Detection: Added automatic runtime audit detecting default or weak salts (< 32 characters or known defaults like "cloakdb-salt"). Fails CI/CD execution with exit code 1 unless --allow-insecure-salt is explicitly provided.
  • Salt Rotation Fingerprinting: Cryptographic SHA-256 fingerprinting embedded directly into cloakdb.yaml (salt_fingerprint). Prevents silent foreign key inconsistencies across runs and prompts for explicit reconciliation (--ignore-salt-mismatch or --update-salt-fingerprint).
  • Production Guard Protection: Live database connection URLs are analyzed for production heuristics (prod, production, live, rds.amazonaws.com). Prompts for interactive confirmation or --confirm-production flag before executing in-place live database modifications.
  • Zero-PII Leak Post-Masking Verification (cloakdb verify): Audits masked datasets, CSV files, and SQL dumps using data-only multi-layer PII detectors with cryptographic checksums (Luhn Mod-10, TCKN Mod-10/11, IBAN Mod-97) to mathematically assert zero unmasked sensitive values remain.

Added

  • Differential Privacy Strategy (differential_privacy): Implements provable $\epsilon$-Laplace and $(\epsilon, \delta)$-Gaussian privacy mechanisms with deterministic seeded noise and bounds clamping.
  • Apache Parquet Streaming Parser (ParquetStreamParser): Chunked row-group streaming and PII detection for .parquet Big Data pipelines using PyArrow without memory blowup.
  • Reusable GitHub Marketplace Action (action.yml): Composite Action (uses: latryee/CloakDB@v1) enabling one-line database anonymization and verification in CI/CD workflows.
  • Interactive Configuration Wizard (cloakdb wizard): Terminal guided setup with automated PII discovery, salt generation, and instant preview.
  • Automated GHCR & PyPI Container Workflows: Multi-arch Docker builds to GitHub Container Registry (ghcr.io/latryee/cloakdb) and OIDC PyPI publishing on release tags.
  • Automated Foreign Key Inference (cloakdb scan --infer-fks): Introspects live database schemas via SQLAlchemy and parses SQL dump DDL (REFERENCES, FOREIGN KEY, ALTER TABLE ... ADD CONSTRAINT) to auto-populate consistency_groups.
  • Composite (Multi-Column) Foreign Key Support: Native support for composite foreign keys (e.g. orders.(tenant_id, user_id) <-> audit_logs.(tenant_id, user_id)) ensuring multi-column referential integrity across relational schemas.
  • Stateless Deterministic Hashing (--stateless): $O(1)$ memory deterministic mapping without unbounded in-memory cache, enabling infinite streaming on constrained nodes.
  • Config Side-by-Side Diff (cloakdb diff): Side-by-side terminal comparison tool evaluating output differences between two masking policies across datasets.
  • Incremental Masking Mode (--since, --incremental-column): Enables Change Data Capture (CDC) and incremental ETL pipelines by bypassing records older than the target timestamp.
  • JSON Document & MongoDB Export Parser (JSONDocumentStreamParser): Full streaming support for JSON arrays and deep MongoDB documents.
  • High-Scale 1GB+ End-to-End Benchmark (scripts/benchmark_1gb.py): End-to-end multi-table streaming benchmark measuring throughput in MB/s and rows/sec.
  • Constant Memory Profiler (scripts/profile_memory.py): Demonstrates bounded constant heap usage (~4.5MB heap / <40MB RSS) across exponentially scaling row tiers.
  • Property-Based Testing (tests/test_property_based.py): Hypothesis-powered test suite mathematically verifying determinism, collision resistance, and null-safety invariants.
  • Production Docker Image: Minimalist multi-stage Dockerfile with non-root security context (cloakdb:10001) and .dockerignore.

Changed

  • PII Detector Precision & Recall Enhancement: Refined phone, credit card, and TCKN heuristics achieving 100% precision and recall on comprehensive benchmark suite (tests/test_scanner_benchmark.py).
  • Safe Record Immutability: Enforced shallow copying in CloakEngine to prevent in-place record mutation during multi-stage transformations.
  • Extended Test Coverage: Total test suite expanded to 170+ automated tests across unit, integration, property-based, and security scenarios.

[0.1.0] - 2024-03-01

Added

  • Initial release of CloakDB streaming masking engine.
  • CLI commands: scan, preview, apply, init, strategies, and bench.
  • Support for PostgreSQL COPY/INSERT, MySQL, SQLite, CSV, and JSONL streams.
  • Referential integrity manager with LRU caching and HMAC-SHA256 pseudonymization.
  • 17+ masking strategies including Faker, deterministic hashing, date shifting, numeric jitter, redaction, and Turkish TCKN generation.
  • Automated PII detection with Luhn credit card and Mod-10/11 TCKN validation.