Feature Request: Direct PostgreSQL CDC connector without Debezium/Kafka

[Git-Faisal]

Is your feature request related to a problem? Please describe.

Currently, the only way to stream changes from PostgreSQL into Pathway is through pw.io.debezium.read, which requires running Debezium + Kafka as middleware. For many deployments — especially multi-tenant SaaS where each client has its own PostgreSQL instance — this adds significant operational overhead (provisioning and managing Kafka brokers + Debezium connectors per instance).

Describe the solution you'd like

A native PostgreSQL CDC input connector (e.g. pw.io.postgres.read) that uses PostgreSQL's built-in logical replication (logical decoding via replication slots + pgoutput or wal2json plugin) to stream changes directly into Pathway — without requiring Debezium or Kafka.

Pathway already has native Rust-level PostgreSQL connectivity for the output connector (pw.io.postgres.write). Extending this to a read/CDC connector using logical replication seems like a natural next step.

Describe alternatives you've considered

• Using pw.io.debezium.read with full Debezium + Kafka stack — works but adds infrastructure complexity
• Polling PostgreSQL with periodic queries — not real-time, defeats the purpose of streaming
• Using other CDC tools (e.g. Conduit, RisingWave) that already support direct PG logical replication without Kafka

Additional context

PostgreSQL's logical replication protocol is well-documented and stable. Other streaming frameworks (e.g. RisingWave, Materialize) already consume PG WAL directly without Kafka middleware. A direct connector would significantly lower the barrier to adoption for PostgreSQL-heavy deployments.

[zxqfd555]

Hi, @Git-Faisal,

Thank you for your suggestion! Indeed, using the native WAL replication for Postgres is on our backlog, and we can look into it after the current urgent tasks.

In order to prioritize things better, could you tell me a bit more about your specific use case and the table you’re working with? We have somewhat similar asks (#101) that target, however, other types of data storage, namely, Elasticsearch, and have a slightly different setup.

When discussing the replication slots, one thing to keep in mind is that while PostgreSQL replication slots allow streaming changes, they require proper permissions and can create additional load on the database.

On a bit more generalizing (and simplifying) side, quite often the use case involves importing a table that is append-only and has an auto-incrementing column. In this scenario, a common approach is polling the table based on this sequential field. By selecting only rows that haven’t been read since the last poll and limiting the query with the maximum transaction duration, queries remain fast even at tens or hundreds of polls per second, don’t require replication slots, and can scale to many parallel processes without manual configuration. This approach can also be applied to other systems beyond PostgreSQL, which would be nice for us since many DB connectors currently don't have input counterparts.

On the other hand, some use cases might require capturing updates/deletes in PostgreSQL, or the table might not have a sequential/auto-incrementing column to rely on, or it is simply non-append-only, so it's important for us to understand your particular case. Even though the users aren’t fully blocked today, and Debezium is fully supported and works as a fallback for PostgreSQL CDC, the native integration makes sense.

Also, in order to propose something right now, I can recommend Docker Compose usage with all related components (you can refer to the examples) for a simple and compact setup, packing all of the required components.

We would want to prioritize the most impactful use cases first. Hence, I’m curious how your scenario fits into this spectrum so we can better understand the trade-offs and decide the urgency.

[Git-Faisal]

Hi @zxqfd555,

Thanks for the detailed response.

Our use case:

We're building an enterprise platform (HR, CRM, ERP modules) on Django + PostgreSQL. Our tables use UUID primary keys (no auto-incrementing columns) and a soft-delete pattern (is_deleted boolean) — records are updated, never physically deleted.

So the append-only polling approach won't work for us. We need full CDC capturing inserts, updates, and soft-deletes to feed real-time RAG pipelines and semantic search (using pgvector).

Why native matters:

We use per-tenant isolation (separate DB per client), so each instance is a single-source, single-consumer CDC stream. Debezium works as a fallback, but spinning up Kafka/Connect infrastructure for that simple topology feels heavy. A native PG connector would simplify deployment significantly.

Happy to test any early implementation and provide feedback quickly.

[zxqfd555]

The case is clear, thank you.
We'll prioritize it, and I will keep you updated.

[zxqfd555]

Done research on the matter: we would prefer the pgoutput plugin over wal2json, since pgoutput is included in the standard setup and doesn't require additional installations, which may be a blocker for hosted deployments.

Several pgoutput decoders would work for our case; I've tested pg_walstream locally.

We plan to have an early implementation closer to the end of this week.