BUG: NULL Handling Crash in Non-Nullable ClickHouse Columns (PreparedStatementFieldMapper)

[minguyen9988]

Summary

PreparedStatementFieldMapper.java calls ps.setNull(index, Types.OTHER) without checking whether the target ClickHouse column is nullable. When a MySQL source row has a NULL value for a column that maps to a non-nullable ClickHouse column, this causes a runtime exception that crashes the batch insert.

Location

• File: sink-connector/src/main/java/com/altinity/clickhouse/sink/connector/db/batch/PreparedStatementFieldMapper.java
• Lines: 152-165 and 168-183

Current Code (lines 152-165)

//ToDO: Setting null to a non-nullable field)
// will throw an error.
// If the Received column is not a clickhouse column
try {
    Object value = struct.get(colName);
    boolean nonDefault = config.getBoolean(ClickHouseSinkConnectorConfigVariables.NON_DEFAULT_VALUE.toString());
    if (nonDefault) {
        value = struct.getWithoutDefault(colName);
    }
    if (value == null) {
        ps.setNull(index, Types.OTHER);  // No nullability check!
        continue;
    }
}

Problem

1. At line 163-164, when value == null, the code blindly calls ps.setNull(index, Types.OTHER) without checking if the ClickHouse column is Nullable(...).
2. At lines 176-181, in the DataException catch block, it logs an error about "Setting column to NULL might fail for non-nullable columns" but then still calls ps.setNull(index, Types.OTHER) anyway.
3. The code has a TODO comment acknowledging this exact problem but no fix has been implemented.

Impact

• Severity: Critical (P0)
• Effect: Causes SQLException / batch failure when MySQL rows contain NULL values for columns that are non-nullable in ClickHouse
• Scope: Any table with non-nullable columns receiving NULL data from MySQL
• Data Loss: Entire batch of records is lost when this crash occurs

Reproduction

1. Create a MySQL table with a nullable column
2. Create a corresponding ClickHouse table where that column is NOT nullable (e.g., Int32 instead of Nullable(Int32))
3. Insert a row with NULL in that column in MySQL
4. The connector will crash with an exception when trying to replicate

Suggested Fix

Before calling ps.setNull(), check if the column data type starts with Nullable(. If it does not, use a type-appropriate default value instead (e.g., 0 for integers, empty string for strings, epoch for dates). Add a configuration option like clickhouse.null.handling.behavior with values: fail, default, skip.

References

• doc/issues/DATA-TYPE-BUGS.md (BUG-DATA-1)
• doc/issues/CRASH-SCENARIOS.md (CRASH-1)
• doc/issues/VERIFICATION-REPORT.md (confirmed as the only remaining P0 bug)