Skip to content

Can not import a DateTime64(9) column #38

@aadant

Description

@aadant

Repro : creating a Parquet file with a DateTime64(9) column :

clickhouse-local --query "select toDateTime64(now(9),9) as current_time format Parquet" > test.parquet
./ice insert test.test -p file://$HOME/test.parquet
2025-06-25 11:10:01 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: processing
2025-06-25 11:10:01 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: jvm: heap: init=3008M,used=16M,committed=224M,free=143599M,max=143616M, gc pressure: 100.00%(12ms), off-heap: init=7M,used=52M,committed=54M, open fds: 11
2025-06-25 11:10:01 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: copying to s3://warehouse/gw_logs/test3/data/1750867801084-06a056686f5853ecfc8ab14ec56fc10fee707ecbe9b929a3b53236991e9fee4f.parquet
2025-06-25 11:10:01 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: adding data file (copy took 0s)
2025-06-25 11:10:02 [main] INFO o.a.i.SnapshotProducer > Committed snapshot 3966130372501728897 (MergeAppend)
(.venv) [aadant@fpif1-dbal2 examples]$ ./ice insert gw_logs.test4 -p file:///home/aadant/test.parquet
2025-06-25 11:13:08 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: processing
2025-06-25 11:13:08 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: jvm: heap: init=3008M,used=16M,committed=224M,free=143599M,max=143616M, gc pressure: 100.00%(14ms), off-heap: init=7M,used=51M,committed=53M, open fds: 11
2025-06-25 11:13:08 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: copying to s3://warehouse/test/test/data/1750867988512-06a056686f5853ecfc8ab14ec56fc10fee707ecbe9b929a3b53236991e9fee4f.parquet
2025-06-25 11:13:08 [main] ERROR com.altinity.ice.cli.Main > Fatal
java.io.IOException: Error processing file(s)
	at com.altinity.ice.cli.internal.cmd.Insert.run(Insert.java:264)
	at com.altinity.ice.cli.Main.insert(Main.java:317)
	at com.altinity.ice.cli.Main.lambda$main$1(Main.java:480) [9 skipped]
	at com.altinity.ice.cli.Main.main(Main.java:487) [1 skipped]
Caused by: java.io.IOException: Error processing file:///home/aadant/test.parquet
	at com.altinity.ice.cli.internal.cmd.Insert.lambda$run$3(Insert.java:249)
 [4 skipped]
Caused by: java.lang.UnsupportedOperationException: Unsupported unit for timestamp: NANOS
	at org.apache.iceberg.data.parquet.BaseParquetReaders.timestampReader(BaseParquetReaders.java:128)
	at org.apache.iceberg.data.parquet.BaseParquetReaders$LogicalTypeReadBuilder.visit(BaseParquetReaders.java:211)
	at org.apache.parquet.schema.LogicalTypeAnnotation$TimestampLogicalTypeAnnotation.accept(LogicalTypeAnnotation.java:710)
	at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.primitive(BaseParquetReaders.java:394)
	at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.primitive(BaseParquetReaders.java:253)
	at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visit(TypeWithSchemaVisitor.java:52)
	at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visitField(TypeWithSchemaVisitor.java:183)
	at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visitFields(TypeWithSchemaVisitor.java:198)
	at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visit(TypeWithSchemaVisitor.java:47)
	at org.apache.iceberg.data.parquet.BaseParquetReaders.createReader(BaseParquetReaders.java:69)
	at org.apache.iceberg.data.parquet.BaseParquetReaders.createReader(BaseParquetReaders.java:61)
	at org.apache.iceberg.data.parquet.GenericParquetReaders.buildReader(GenericParquetReaders.java:51)
	at com.altinity.ice.cli.internal.cmd.Insert.lambda$processFile$5(Insert.java:468)
	at org.apache.iceberg.parquet.ReadConf.<init>(ReadConf.java:121)
	at org.apache.iceberg.parquet.ParquetReader.init(ParquetReader.java:71)
	at org.apache.iceberg.parquet.ParquetReader.iterator(ParquetReader.java:91)
	at org.apache.iceberg.parquet.ParquetReader.iterator(ParquetReader.java:37)
	at org.apache.iceberg.io.FileAppender.addAll(FileAppender.java:36)
	at com.altinity.ice.cli.internal.cmd.Insert.copy(Insert.java:713)
	at com.altinity.ice.cli.internal.cmd.Insert.processFile(Insert.java:478)
	at com.altinity.ice.cli.internal.cmd.Insert.lambda$run$3(Insert.java:228)
	... 4 common frames omitted

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions