OOM Killed when creating FDAs with large amounts of data

[GregorioBlazquez]

When creating FDAs with large data volumes (e.g., windowSize >= 6 months) the container is killed by the OOM Killer (exit code 137) during the creation process, causing 502 errors and service restarts.

Root Cause

The FDA creation pipeline suffers from memory issues at least in two stages:

1. CSV Upload (PostgreSQL → MinIO)

• No backpressure between PostgreSQL stream and AWS SDK Upload
• partSize: 25MB causes memory spikes per part
• Each upload creates its own PostgreSQL connection pool

2. Parquet Conversion (DuckDB)

• DuckDB in :memory: mode processes everything in RAM
• No memory_limit or temp_directory configured
• Partitioning and read_csv_auto() increase memory consumption

Result: Combined issues exceed 2GB limit → OOM Killer → container killed (exit code 137).

Reproduce

reproduce-duckdb-memory.md