Core, Parquet, ORC: Don't write column sizes when metrics mode is None#10440
Merged
amogh-jahagirdar merged 1 commit intoJun 5, 2024
Merged
Conversation
62210e7 to
9cbead1
Compare
Contributor
Author
|
Seems like ORC and possibly Avro writers also need to be updated, the ORC tests are failing |
9cbead1 to
d55fb15
Compare
Contributor
Author
|
Avro didn't need any changes which makes sense it's row oriented anyways. Fixed ORC. |
szehon-ho
approved these changes
Jun 4, 2024
szehon-ho
left a comment
Member
There was a problem hiding this comment.
I was also thinking along these lines and so makes sense to me. Nothing seems relying on this field?
Contributor
Author
|
@szehon-ho Yeah at least from my analysis there's no expectation on this being populated in the library (which makes sense it is optional as per the spec). |
nastra
approved these changes
Jun 5, 2024
danielcweeks
approved these changes
Jun 5, 2024
Contributor
Author
|
Thanks for the reviews @szehon-ho @nastra @danielcweeks ! |
jasonf20
pushed a commit
to jasonf20/iceberg
that referenced
this pull request
Aug 4, 2024
sasankpagolu
pushed a commit
to sasankpagolu/iceberg
that referenced
this pull request
Oct 27, 2024
zachdisc
pushed a commit
to zachdisc/iceberg
that referenced
this pull request
Dec 23, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently, the Iceberg Parquet and ORC writers will write out column sizes even when metrics are disabled. This should not be the case since column sizes are optional in the spec and we should respect the property for this case.