Spark: CALL procedure for rewrite_data_files by ajantha-bhat · Pull Request #3375 · apache/iceberg

ajantha-bhat · 2021-10-26T10:47:53Z

Supported the call procedure for rewrite_data_files.
Supported input arguments: table, [strategy], [sort_order], [options], [where]
Supported output arguments: rewritten_data_files_count, added_data_files_count
Added test cases to cover the scenarios.

Fixes #3355

jackye1995 · 2021-10-26T21:55:14Z

Thanks for working on this! Almost forgot we haven't added the call procedure for this yet...

I remember Russell had laid out the arguments for the procedure in https://docs.google.com/document/d/1aXo1VzuXxSuqcTzMLSQdnivMVgtLExgDWUFMvWeXRxc, could you follow that proposal?

ajantha-bhat · 2021-10-27T03:49:48Z

@jackye1995 : Thanks for sharing this doc. I will look into it and implement as per that.

ajantha-bhat · 2021-10-27T14:03:03Z

@jackye1995 , @RussellSpitzer :
I have added some more arguments. Also updated the document.
As per me only sort related options and where arguments are pending. For where , I am not sure how to convert text to expression yet. For sort related options I am not sure whether to have as map or individual options.
If ok, I want to handle this in subsequent PR. please review the current changes.

Also I should copy the same code in spark/v3.2 folder as well ?

p.s: Also let me know if any other arguments is missed.

RussellSpitzer · 2021-10-27T14:42:56Z

@jackye1995 , @RussellSpitzer : I have added some more arguments. Also updated the document. As per me only sort related options and where arguments are pending. For where , I am not sure how to convert text to expression yet. For sort related options I am not sure whether to have as map or individual options. If ok, I want to handle this in subsequent PR. please review the current changes.

Also I should copy the same code in spark/v3.2 folder as well ?

p.s: Also let me know if any other arguments is missed.

For "where" can you check and see if you can produce expressions using the Spark session's parser? I think you may be able to convert to expressions using the parser and then translate them using our Spark3 utils ... I'm not sure though.

ajantha-bhat · 2021-11-01T05:46:26Z

@jackye1995 , @RussellSpitzer , @rdblue :
Is there any available class for converting org.apache.spark.sql.catalyst.expressions.Expression to org.apache.iceberg.expressions.Expression. I looked up based on import statement. I didn't find any.

Actually for supporting filter, I have used the spark parser to parse the string as expression. But I got spark catalyst expression. But rewrite data files action needs Iceberg expression instead of spark catalyst expression.

rdblue · 2021-11-01T23:55:55Z

There is a conversion from Spark to Iceberg expressions in SparkFilters.

ajantha-bhat · 2021-11-02T06:51:34Z

There is a conversion from Spark to Iceberg expressions in SparkFilters.

@rdblue : I think it is spark filter to Iceberg expression conversion. I needed spark expression to Iceberg expression conversion.

ajantha-bhat · 2021-11-02T06:52:24Z

@RussellSpitzer , @jackye1995 , @rdblue :
Please review and help merge this PR.

I will back sync to v3.0 folder once this PR is reviewed and merged. Thanks.

ajantha-bhat · 2021-12-01T05:12:47Z

@RussellSpitzer : So, anything else need to be done for this PR or it can be merged ?

jackye1995

overall looks good to me, just some nit

ajantha-bhat · 2021-12-03T05:55:57Z

@jackye1995 : Thanks for the review. I have pushed the nit changes also.

RussellSpitzer

Just had some minor comments which I think we need to wrap up, other than that I think this is looking pretty good. I know it's difficult for us to work this kind of functionality into Iceberg when Spark keeps a lot of these methods from us.

ajantha-bhat · 2021-12-03T17:22:27Z

@RussellSpitzer : Thanks for the review again. I have addressed the comments (also added a comment reply) and pushed the new changes.

ajantha-bhat · 2021-12-04T03:45:29Z

@RussellSpitzer : I have updated the test case with dataframe write instead of sql now. So, I think all comments are handled now. Thanks.

RussellSpitzer

Thanks so much, this is a great addition to the codebase

ajantha-bhat · 2021-12-05T17:19:20Z

@RussellSpitzer : Thanks for merging the PR. Appreciate your guidance and patient reply to my questions. It helped in finally avoiding the custom codes 👍🏻

Backport of #3375 - Support CALL procedure for rewrite_data_files

ajantha-bhat marked this pull request as draft October 26, 2021 10:48

github-actions Bot added docs spark labels Oct 26, 2021

ajantha-bhat force-pushed the call branch from 6b02bbf to 0236e0c Compare October 26, 2021 11:28

ajantha-bhat mentioned this pull request Oct 26, 2021

Support CALL procedure for rewrite_datafiles similar to rewrite_manifests #3355

Closed

ajantha-bhat force-pushed the call branch from 0236e0c to 6472d6b Compare October 27, 2021 13:57

github-actions Bot added the core label Oct 27, 2021

ajantha-bhat force-pushed the call branch from 6472d6b to 769b3ba Compare October 27, 2021 13:58

ajantha-bhat marked this pull request as ready for review October 27, 2021 14:03

ajantha-bhat force-pushed the call branch from 769b3ba to d3be42b Compare October 27, 2021 14:05

RussellSpitzer reviewed Oct 27, 2021

View reviewed changes

Comment thread site/docs/spark-procedures.md Outdated

RussellSpitzer reviewed Oct 27, 2021

View reviewed changes

Comment thread site/docs/spark-procedures.md Outdated

RussellSpitzer reviewed Oct 27, 2021

View reviewed changes

Comment thread site/docs/spark-procedures.md Outdated

ajantha-bhat force-pushed the call branch 2 times, most recently from 6dea11f to 36ed20a Compare October 27, 2021 16:08

ajantha-bhat force-pushed the call branch from 36ed20a to f69e481 Compare November 1, 2021 12:57

ajantha-bhat changed the title ~~CALL procedure for rewrite_data_files~~ Spark: CALL procedure for rewrite_data_files Nov 1, 2021

ajantha-bhat force-pushed the call branch from f69e481 to 4c0513b Compare November 2, 2021 06:49

ajantha-bhat commented Nov 2, 2021

View reviewed changes

Comment thread spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/SparkExpressions.java Outdated

jackye1995 reviewed Nov 4, 2021

View reviewed changes

Comment thread site/docs/spark-procedures.md Outdated

Comment thread ...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteDataFilesProcedure.java Outdated

Comment thread ...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteDataFilesProcedure.java Outdated

ajantha-bhat force-pushed the call branch 2 times, most recently from 2194b42 to 170fe7b Compare November 5, 2021 03:06

RussellSpitzer reviewed Nov 30, 2021

View reviewed changes

Comment thread .../v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java Outdated

RussellSpitzer reviewed Nov 30, 2021

View reviewed changes

Comment thread .../v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java Outdated

ajantha-bhat force-pushed the call branch from 0fe8d7e to 0adea41 Compare December 1, 2021 00:21

jackye1995 reviewed Dec 3, 2021

View reviewed changes

Comment thread site/docs/spark-procedures.md Outdated

jackye1995 reviewed Dec 3, 2021

View reviewed changes

Comment thread ...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteDataFilesProcedure.java Outdated

jackye1995 reviewed Dec 3, 2021

View reviewed changes

Comment thread ...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteDataFilesProcedure.java Outdated

jackye1995 reviewed Dec 3, 2021

View reviewed changes

Comment thread ...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteDataFilesProcedure.java Outdated

jackye1995 approved these changes Dec 3, 2021

View reviewed changes

ajantha-bhat force-pushed the call branch from 0adea41 to 4e1d110 Compare December 3, 2021 05:54

RussellSpitzer reviewed Dec 3, 2021

View reviewed changes

Comment thread site/docs/spark-procedures.md Outdated

RussellSpitzer reviewed Dec 3, 2021

View reviewed changes

Comment thread ...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteDataFilesProcedure.java Outdated

RussellSpitzer reviewed Dec 3, 2021

View reviewed changes

Comment thread .../v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java Outdated

RussellSpitzer reviewed Dec 3, 2021

View reviewed changes

Comment thread .../v3.2/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteDataFilesProcedure.java Outdated

RussellSpitzer approved these changes Dec 3, 2021

View reviewed changes

Spark: support CALL procedure for rewrite_data_files

a5cc356

ajantha-bhat force-pushed the call branch from 95e866d to a5cc356 Compare December 4, 2021 03:43

ajantha-bhat changed the title ~~Spark: CALL procedure for rewrite_data_files~~ Spark-3.2: CALL procedure for rewrite_data_files Dec 4, 2021

ajantha-bhat changed the title ~~Spark-3.2: CALL procedure for rewrite_data_files~~ Spark: CALL procedure for rewrite_data_files Dec 4, 2021

RussellSpitzer approved these changes Dec 4, 2021

View reviewed changes

RussellSpitzer merged commit b6554fc into apache:master Dec 4, 2021

This was referenced Dec 5, 2021

Spark-3.1: support CALL procedure for rewrite_data_files #3671

Merged

Spark-3.0: support CALL procedure for rewrite_data_files #3672

Merged

ajantha-bhat mentioned this pull request Dec 6, 2021

Spark-3.2: Move scala files of rewrite_data_files procedure to scala folder #3675

Merged

RussellSpitzer pushed a commit that referenced this pull request Dec 9, 2021

Spark-3.1: support CALL procedure for rewrite_data_files (#3671)

fc68cee

Backport of #3375 - Support CALL procedure for rewrite_data_files

RussellSpitzer pushed a commit that referenced this pull request Dec 9, 2021

Spark-3.0: support CALL procedure for rewrite_data_files (#3672)

8a944a9

Backport of #3375 - Support CALL procedure for rewrite_data_files

Uh oh!

Conversation

ajantha-bhat commented Oct 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jackye1995 commented Oct 26, 2021

Uh oh!

ajantha-bhat commented Oct 27, 2021

Uh oh!

ajantha-bhat commented Oct 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RussellSpitzer commented Oct 27, 2021

Uh oh!

ajantha-bhat commented Nov 1, 2021

Uh oh!

rdblue commented Nov 1, 2021

Uh oh!

ajantha-bhat commented Nov 2, 2021

Uh oh!

ajantha-bhat commented Nov 2, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ajantha-bhat commented Dec 1, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jackye1995 left a comment

Choose a reason for hiding this comment

Uh oh!

ajantha-bhat commented Dec 3, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RussellSpitzer left a comment

Choose a reason for hiding this comment

Uh oh!

ajantha-bhat commented Dec 3, 2021

Uh oh!

ajantha-bhat commented Dec 4, 2021

Uh oh!

RussellSpitzer left a comment

Choose a reason for hiding this comment

Uh oh!

ajantha-bhat commented Dec 5, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ajantha-bhat commented Oct 26, 2021 •

edited

Loading

ajantha-bhat commented Oct 27, 2021 •

edited

Loading