Skip to content

Spark 3.3: Add a procedure to generate table changes #6012

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aokolnychyi merged 30 commits into from
Mar 4, 2023
Merged

Spark 3.3: Add a procedure to generate table changes #6012

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
1b3ef98
Add procedure for generating table changes
flyrain Oct 4, 2022
b6dfc19
Fix the output type name
flyrain Oct 18, 2022
210a2f7
Fix the test case.
flyrain Oct 18, 2022
3fd6aa6
Use uuid as the temp column name to avoid conflicts.
flyrain Oct 19, 2022
573433c
Add test case for filtering
flyrain Oct 19, 2022
3cc8ee2
Remove the unnecessary condition
flyrain Oct 20, 2022
3b6b208
Support query by timestamps
flyrain Oct 21, 2022
46dfd98
Refactor
flyrain Oct 21, 2022
ed5dadf
Change the names to UPDATE_AFTER and UPDATE_BEFORE
flyrain Nov 3, 2022
e7de9a5
Use repartition and mapPartition
flyrain Nov 15, 2022
9ef09eb
Use repartition and mapPartition
flyrain Nov 18, 2022
aa31dff
Remove null
flyrain Nov 18, 2022
0b8ccd4
Check if two rows belong to the same partition
flyrain Nov 18, 2022
20fbbec
Fix the style issue
flyrain Nov 18, 2022
07403cb
Resolve comments
flyrain Nov 29, 2022
d138d04
Fix the style issue
flyrain Nov 30, 2022
8b16630
Create the separated class for iterator
flyrain Dec 1, 2022
fdf45bc
Allow time range query
flyrain Jan 6, 2023
2c343a3
Allow time range query
flyrain Jan 11, 2023
3a136e4
Added a new test and updated Java Doc
flyrain Jan 11, 2023
b5d137a
Rename the parameters
flyrain Jan 11, 2023
a3d4d73
Read table identifier fields
flyrain Jan 12, 2023
d25560b
Retrigger the pipeline
flyrain Jan 12, 2023
99a3728
Resolve the comments
flyrain Feb 21, 2023
41bb85b
Resolve comments
flyrain Feb 22, 2023
d9a264a
Resolve comments
flyrain Feb 24, 2023
dedf556
Resolve comments
flyrain Feb 24, 2023
1e95a66
Resolve comments
flyrain Mar 3, 2023
f659fe7
Resolve comments
flyrain Mar 3, 2023
54b3da8
Add comments
flyrain Mar 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions ...ns/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,7 @@ transformArgument
expression
: constant
| stringMap
| stringArray
;

constant
Expand All @@ -173,6 +174,10 @@ stringMap
: MAP '(' constant (',' constant)* ')'
;

stringArray

@aokolnychyi aokolnychyi Mar 2, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

: ARRAY '(' constant (',' constant)* ')'
;

booleanValue
: TRUE | FALSE
;
Expand Down Expand Up @@ -274,6 +279,7 @@ TRUE: 'TRUE';
FALSE: 'FALSE';

MAP: 'MAP';
ARRAY: 'ARRAY';

PLUS: '+';
MINUS: '-';
Expand Down
Loading