Problem
When a Z-order column has Spark type TimestampNTZType, rewrite_data_files with strategy => 'sort' and sort_order => 'zorder(...)' fails with:
Cannot use column of type TimestampNTZType in ZOrdering, the type is unsupported
Root Cause
SparkZOrderUDF#sortedLexicographically handles TimestampType (cast to long for ordered bytes) but does not handle TimestampNTZType, so NTZ falls through to the unsupported-type branch.
Steps to Reproduce
Environment:
- Iceberg:
1.10.0
- Spark:
4.0
from datetime import datetime
from pyspark.sql import Row, SparkSession
spark = # SparkSession with Iceberg catalog configured
spark.sql("DROP TABLE IF EXISTS spark_catalog.default.check_table_ntz")
spark.sql("""
CREATE TABLE spark_catalog.default.check_table_ntz (
ts timestamp_ntz,
col_a bigint
)
USING iceberg
PARTITIONED BY (days(ts))
TBLPROPERTIES ('format-version' = '2')
""")
data = [
Row(ts=datetime(2024, 1, 1, 0, 0, 0), col_a=1),
Row(ts=datetime(2024, 1, 2, 0, 0, 0), col_a=2),
]
spark.createDataFrame(data).coalesce(1).writeTo(
"spark_catalog.default.check_table_ntz"
).append()
spark.sql("""
CALL spark_catalog.system.rewrite_data_files(
table => 'spark_catalog.default.check_table_ntz',
strategy => 'sort',
sort_order => 'zorder(ts, col_a)',
options => map('rewrite-all', 'true')
)
""")Actual Behavior
IllegalArgumentException from the Z-order path: TimestampNTZType is reported as unsupported for Z-ordering.
Expected Behavior
Z-order rewrite should support timestamp_ntz columns (same ordered-bytes treatment as for TimestampType)
Problem
When a Z-order column has Spark type
TimestampNTZType,rewrite_data_fileswithstrategy => 'sort'andsort_order => 'zorder(...)'fails with:Cannot use column of type TimestampNTZType in ZOrdering, the type is unsupported
Root Cause
SparkZOrderUDF#sortedLexicographicallyhandlesTimestampType(cast to long for ordered bytes) but does not handleTimestampNTZType, so NTZ falls through to the unsupported-type branch.Steps to Reproduce
Environment:
1.10.04.0Actual Behavior
IllegalArgumentException from the Z-order path: TimestampNTZType is reported as unsupported for Z-ordering.
Expected Behavior
Z-order rewrite should support timestamp_ntz columns (same ordered-bytes treatment as for TimestampType)