Hive: HiveCatalog listTables takes minutes if there are thousands tab…#3908
Conversation
|
Can it be improved if change to |
I don't think so. Besides, in our production, we use different namespace schema for different table type. For example, tables under namespace iceberg_{biz_id} are all iceberg tables, tables under namespace hive_{biz_id} are all hive partition tables. It's wasting time for method call |
…bles in namespace
|
@rdblue @jackye1995 could you help to review this? |
|
Does it only work in Hivecatalog? If so, would it be better to name 'hive.filter-iceberg-table' or 'hive.show-iceberg-table-only'? |
Iceberg doesn't use streams for parallelism because they are quite limited. When parallelizing operations, be sure to use |
rdblue
left a comment
There was a problem hiding this comment.
I think this option is a good idea, but it should be specific to Hive and we should carefully consider naming to make it clear.
yes, only in HiveCatalog, will change the name for better understanding. |
…bles in namespace
|
Thanks, @vanliu-tx. Looking close to ready. I think you just need to rename the property. |
I was on vacation yesterday, sorry for the delay. |
|
@rdblue could you help to merge this PR? |
|
Thanks, @vanliu-tx! |
…les in namespace
A flag is added to indicate whether to filter iceberg table in HiveCatalog#listTables method. Related to #3907