feat: Support message writing flags

[edgarrmondragon]

Feature scope

Taps (catalog, state, tests, etc.)

Description

When encoding/writing Singer messages to stdout, it'd be good to support certain flags that abstract the specifics of json, simplejson, msgspec, etc.

Some flags could be:

• ascii
• nan as null
• nan passthrough

TDB the specifics and how these interact with one another.

Related:

• feat: Handling of "inf" and similar NaN values #2211
• fix(taps): add argument for nan handling strategy to _flatten_record #2222
• fix(taps): update _flatten_record function by adding allow_nan=True in json.dumps #2214
• fix: Conform Nan, Infinity and -Infinity to null values #2961

[tjvananne]

Bump. I think NaN as null would be a really nice feature or configuration setting. Especially since it's likely that most targets are going to be SQL-based databases or warehouses.

Context: We're migrating from the transferwise variants of tap-postgres and target-snowflake. It doesn't look like it's currently possible to make the switch to the meltanolabs variant because we're getting so many Out of range float values are not JSON compliant errors.

I'm guessing we could use inline stream mappers to convert all these float fields to varchar for ingestion, but it's hundreds (possibly thousands?) of fields. Any other thoughts on workarounds? Thanks!

[edgarrmondragon]

Context: We're migrating from the transferwise variants of tap-postgres and target-snowflake. It doesn't look like it's currently possible to make the switch to the meltanolabs variant because we're getting so many Out of range float values are not JSON compliant errors.

That's interesting, I don't see any indication in the source of the pipelinewise connectors that would suggest they support NaN values. If we know the value causing problems, it may point to a more specific bug in the tap or targets.

[tjvananne]

@edgarrmondragon It looks like pipelinewise's tap-postgres extractor converts NaN, Inf, and -Inf to None/null based on this code.

[edgarrmondragon]

@edgarrmondragon It looks like pipelinewise's tap-postgres extractor converts NaN, Inf, and -Inf to None/null based on this code.

I see. That kind of type conforming is actually fairly easy to patch here, so I'll open a PR.

[edgarrmondragon]

Reinstalling the MeltanoLabs/tap-postgres should get you the latest SDK patch, which handles those values.