Flink: Support inspecting table by hililiwei · Pull Request #6222 · apache/iceberg

hililiwei · 2022-11-19T09:18:47Z

What is the purpose of the change

Support inspecting table in flink sql.

select * from {tableName}${metadataTableName}

The '$' syntax borrows from Flink-Table-Store, under the Flink umbrella.

List

History

To show table history:

SELECT * FROM prod.db.table$history;

Metadata Log Entries

To show table metadata log entries:

SELECT * from prod.db.table$metadata_log_entries;

Snapshots

To show the valid snapshots for a table:

SELECT * FROM prod.db.table$snapshots;

Files

To show a table's current data files:

SELECT * FROM prod.db.table$files;

Manifests

To show a table's current file manifests:

SELECT * FROM prod.db.table$manifests;

Partitions

To show a table's current partitions:

SELECT * FROM prod.db.table$partitions;

All Metadata Tables

These tables are unions of the metadata tables specific to the current snapshot, and return metadata across all snapshots.

- All Data Files

To show all of the table's data files and each file's metadata:

SELECT * FROM prod.db.table$all_data_files;

- All Manifests

To show all of the table's manifest files:

SELECT * FROM prod.db.table$all_manifests;

References

To show a table's known snapshot references:

SELECT * FROM prod.db.table$refs;

hililiwei · 2022-11-21T07:25:53Z

@stevenzwu @openinx @rdblue @Fokko could you please take a look at it when you get a chance? thx.

hililiwei · 2022-11-29T12:31:27Z

also cc @pvary, thx.

pvary · 2022-12-02T07:29:29Z

@hililiwei: Hive/Spark/Impala(?) uses SELECT * FROM prod.db.table.history like naming convention for accessing metadata tables. See: https://iceberg.apache.org/docs/latest/spark-queries/#history

Do we want to introduce a different naming convention for Flink? How widespread is the usage of the SELECT * FROM prod.db.table$history like SQL?

hililiwei · 2022-12-02T09:38:59Z

Do we want to introduce a different naming convention for Flink? How widespread is the usage of the SELECT * FROM prod.db.table$history like SQL?

@pvary, thank you for your feedback.
I personally like SELECT * FROM prod.db.table.history too, but unfortunately, flink does not support this syntax, and it always
discards .history. I tried for a long time and couldn't get it to work properly. Eventually, I had to choose the same way as flink-table-store.

pvary · 2022-12-02T17:57:14Z

Do we want to introduce a different naming convention for Flink? How widespread is the usage of the SELECT * FROM prod.db.table$history like SQL?

@pvary, thank you for your feedback. I personally like SELECT * FROM prod.db.table.history too, but unfortunately, flink does not support this syntax, and it always discards .history. I tried for a long time and couldn't get it to work properly. Eventually, I had to choose the same way as flink-table-store.

Thanks @hililiwei!

pvary · 2022-12-03T06:49:54Z

Missclicked 😢

What I was trying to comment:
Sadly I don't have enough time to properly review the PR, but I remember that we had issues with the partition evolution in the past when @szlta tested the Hive metadata tables. IIRC the issues surfaced when we changed from (a, b) partitioning to (a, c).
See the relevant PRs: https://github.com/apache/iceberg/pulls?q=is%3Apr+szlta+partition

The issues are handled on Core level, but it might worth to add some checks for edge cases too. Like:

Removed columns
Readded columns
Partition evolution
Schema evolution

stevenzwu · 2022-12-06T03:21:32Z

@hililiwei we should add comprehensive unit test for StructRowData.

I have some internal DataGenerators for unit test code with very comprehensive coverage all field types (including complex nested types). Maybe I will submit a separate PR for that, which will cover Flink RowData and Iceberg GenericRecord. You can expand it with support for Iceberg StructLike. With that, we can write unit test/assertions compare the actual and expected.

stevenzwu · 2022-12-06T03:15:36Z

+    // make a defensive copy to ensure entries do not change
+    List<Map.Entry<?, ?>> entries = ImmutableList.copyOf(map.entrySet());
+
+    ArrayData keyArray =


ok. I got it that we need run the conversion recursively (e.g. for StringData key or RowData value). I like to propose a diff structure for this class.

We can repurpose collectionToArrayData to a convertFromStructToRowData(Type type, Object value) method. If type is most primitive, it should be simple pass-thru. for other types (like timestamp, struct, map, array) there are some conversions to Flink types (like TimestampData, RowData, MapData, ArrayData).

Then for the map, we don't have to convert to key and value array. I believe the reason you did it is to reuse the collectionToArrayData.

hililiwei · 2022-12-06T07:44:40Z

@hililiwei we should add comprehensive unit test for StructRowData.

I have some internal DataGenerators for unit test code with very comprehensive coverage all field types (including complex nested types). Maybe I will submit a separate PR for that, which will cover Flink RowData and Iceberg GenericRecord. You can expand it with support for Iceberg StructLike. With that, we can write unit test/assertions compare the actual and expected.

That would be great. I'm also thinking of adding UT for it. So i'll do it based on your code.

hililiwei · 2022-12-06T16:11:07Z

UT failed because of #5376. I'm trying to fix it.

hililiwei · 2022-12-20T13:35:58Z

Added test case for StructRowData. @stevenzwu please take a look, thx.

hililiwei · 2022-12-21T06:20:38Z

like to get more feedback for adding a new public method in the api module. cc @rdblue @aokolnychyi @RussellSpitzer @szehon-ho

Here is the context why we need this. The converter in FlinkSchemaUtil creates Iceberg Schema without doc attributes. Hence @hililiwei implemented the util class and method to carry over the doc attribute from docSourceSchema to schema (converted from Flink TableSchema).
  public static Schema convert(TableSchema schema) {
  }
I suggested moving it into TypeUtil class in api module as it is very similar to the reassignIds method in the TypeUtil class.

cc @rdblue @aokolnychyi @RussellSpitzer @szehon-ho

stevenzwu · 2023-01-13T17:13:00Z

thanks @hililiwei for contributing this major feature

hililiwei · 2023-01-16T01:38:33Z

thanks @stevenzwu @pvary @szehon-ho

github-actions Bot added docs flink labels Nov 19, 2022

hililiwei commented Nov 19, 2022

View reviewed changes

Comment thread flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java Outdated

Fokko requested a review from stevenzwu November 23, 2022 09:52

stevenzwu reviewed Dec 1, 2022

View reviewed changes

Comment thread flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java Outdated

stevenzwu reviewed Dec 1, 2022

View reviewed changes

Comment thread flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/data/StructRowData.java

stevenzwu reviewed Dec 1, 2022

View reviewed changes

Comment thread flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/data/StructRowData.java

stevenzwu reviewed Dec 1, 2022

View reviewed changes

Comment thread flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/data/StructRowData.java

hililiwei force-pushed the Inspecting branch from d485cfa to 5858d17 Compare December 1, 2022 15:19

stevenzwu reviewed Dec 2, 2022

View reviewed changes

pvary closed this Dec 3, 2022

pvary reopened this Dec 3, 2022

hililiwei force-pushed the Inspecting branch 2 times, most recently from 40aeedd to ad1f420 Compare December 5, 2022 15:00

stevenzwu reviewed Dec 5, 2022

View reviewed changes

Comment thread docs/flink-getting-started.md Outdated

stevenzwu reviewed Dec 6, 2022

View reviewed changes

hililiwei force-pushed the Inspecting branch 2 times, most recently from f701467 to 6f40b74 Compare December 6, 2022 07:41

hililiwei force-pushed the Inspecting branch 3 times, most recently from d62a614 to 3e9267a Compare December 7, 2022 08:47

hililiwei force-pushed the Inspecting branch 6 times, most recently from f4a168c to 8dd0688 Compare December 20, 2022 13:31

stevenzwu reviewed Dec 20, 2022

View reviewed changes

Comment thread flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/data/StructRowData.java Outdated

stevenzwu reviewed Dec 20, 2022

View reviewed changes

Comment thread flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/data/StructRowData.java Outdated

hililiwei force-pushed the Inspecting branch 2 times, most recently from 6fe3594 to d566549 Compare December 21, 2022 06:16

hililiwei force-pushed the Inspecting branch 2 times, most recently from 7b08642 to 09dca6b Compare December 21, 2022 06:59

stevenzwu reviewed Jan 10, 2023

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/BaseFilesTable.java Outdated

stevenzwu reviewed Jan 10, 2023

View reviewed changes

Comment thread flink/v1.16/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalog.java Outdated

hililiwei added 6 commits January 13, 2023 13:10

Flink: Inspecting tables

da65860

refactor covertFromStructToRowData

41c0e28

ReadableMetrics

eac82d7

UT for StructRowData

c567985

address

661ea85

collapse case, add type check.

a157fca

hililiwei force-pushed the Inspecting branch from d64c207 to a157fca Compare January 13, 2023 05:50

stevenzwu approved these changes Jan 13, 2023

View reviewed changes

stevenzwu merged commit 6ab1986 into apache:master Jan 13, 2023

hililiwei deleted the Inspecting branch January 16, 2023 01:38

stevenzwu mentioned this pull request Jan 17, 2023

Flink: Port Support inspecting metadata table to Flink 1.14 & 1.15 #6610

Merged

xuzhiwen1255 mentioned this pull request Mar 15, 2023

Support to read iceberg table's metadata column with Flink #7110

Closed

Uh oh!

Conversation

hililiwei commented Nov 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the change

List

Uh oh!

Uh oh!

hililiwei commented Nov 21, 2022

Uh oh!

hililiwei commented Nov 29, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pvary commented Dec 2, 2022

Uh oh!

hililiwei commented Dec 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pvary commented Dec 2, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pvary commented Dec 3, 2022

Uh oh!

Uh oh!

stevenzwu commented Dec 6, 2022

Uh oh!

Uh oh!

stevenzwu Dec 6, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hililiwei commented Dec 6, 2022

Uh oh!

hililiwei commented Dec 6, 2022

Uh oh!

hililiwei commented Dec 20, 2022

Uh oh!

Uh oh!

Uh oh!

hililiwei commented Dec 21, 2022

Uh oh!

Uh oh!

Uh oh!

stevenzwu commented Jan 13, 2023

Uh oh!

hililiwei commented Jan 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hililiwei commented Nov 19, 2022 •

edited

Loading

hililiwei commented Dec 2, 2022 •

edited

Loading