Why is this change?

well the issue is, arrow's auto-detection of allocation manager type breaks when classes are shaded org.apache.iceberg.shaded.org.apache.arrow.*, because CheckAllocator.check() inspects JAR paths via ProtectionDomain.getCodeSource().getLocation() and doesn't recognize shaded package structures. Setting this property bypasses the broken-based detection, which is essential since our BigIntVector allocation for TIMESTAMP_MILLIS requires a working RootAllocator in shaded Spark runtime environments.

Could we add a brief comment explaining why we set arrow.memory.allocation.manager.type to Netty, and consider logging at debug when we default it to Netty so it’s visible at runtime?

Done.

can we also assert the value?

done.

also, in VectorizedParquetDefinitionLevelReader.java, arrow validity buffer was not being properly set for non-null values. (bit should be 1 when Value is not null). Fixed that.

Do we have a native parquet writer in the base too?
Maybe we can create a test without spark

Well, the issue is, iceberg’s native parquet writers explicitly reject TIMESTAMP_MILLIS (check here), as by design iceberg standardizes on microsecond precision.

TIMESTAMP_MILLS support exists only for reading externally-produced files (like from Spark)

To write a test without Spark will add a lot of extra code, and that too using Parquet API. IMO, it’s better to have the current end to end test with Spark.

Could we use ExampleParquetWriter or something similar?

Or maybe use AvroParquetWriter to create the Parquet file? Just an idea, maybe it is easier to write the file that way, and it supports millisec precision.

@nandorKollar well, for

(a) ExampleParquetWriter : I fear, we have to manually construct Parquet group objects and manually define the schema with TIMESTAMP_MILLIS logical type, which is much more complex than our current approach. Moreover, Iceberg never used ExampleParquetWriter, using it would require writing low-level parquet code.

(b) AvroParquetWriter : Iceberg’s AvroSchemaUtil always uses timestampMicros , never timestampMillis check here. While avro supports timestamp-millis, you cannot use it through. We have to manually create the avro schema for it, and also handle the data conversion. Again, it will bring more unnecessary code.

IMO, we should stick to our current approach.

If we plan to support reading Parquet files with TIMESTAMP_MILLIS logical type, we need to support it in all of the readers.

• Arrow
• Spark
• Flink

For this we need to find a way to test it.

@shubham19may it would be better not to rely on Spark module to cover this case with a test, but rather keep the the arrow module 'self-contained', cover it's functionality with tests there. TestVariantReaders already creates a custom Parquet writer with Avro object modell, it doesn't look too complicated. I'll try to put together an example with ExampleParquetWriter, it might be a bit more complicated. As Iceberg's Parquet writer doesn't write timestamp millis, unfortunately we need to implement our test writer for these types. Actually this is not the only case which is not covered, for example Parquet files with unsigned integer types are not covered either, probably there we can't even use the trick to test it in Spark, as unsigned types are unknown in Spark too.

Sure @pvary and @nandorKollar.

I have updated the test, moved it away from Spark to the arrow module, inside TestArrowReader.java. Please give it a review whenever you are free, and do tell me if any further changes are required.

Thanks @shubham19may!

nit: newline here is not needed.

Done.

nit: would it be possible to use InMemoryOutputFile instead of temp files? We can maybe rewrite the Arrow tests in a subsequent followup PR.