Skip to content

GCP: Support multiple storage credential prefixes#12881

Merged
nastra merged 2 commits into
apache:mainfrom
nastra:gcs-multi-prefix-support
May 7, 2025
Merged

GCP: Support multiple storage credential prefixes#12881
nastra merged 2 commits into
apache:mainfrom
nastra:gcs-multi-prefix-support

Conversation

@nastra

@nastra nastra commented Apr 23, 2025

Copy link
Copy Markdown
Contributor

No description provided.

@nastra nastra force-pushed the gcs-multi-prefix-support branch from 4cc79c7 to 7da613e Compare April 23, 2025 16:36
@github-actions github-actions Bot added the core label Apr 23, 2025
@nastra nastra force-pushed the gcs-multi-prefix-support branch from 7da613e to 1fee03b Compare April 23, 2025 16:44
@github-actions github-actions Bot added the AWS label Apr 23, 2025
@nastra nastra force-pushed the gcs-multi-prefix-support branch 2 times, most recently from 48bc144 to 640ff78 Compare April 24, 2025 07:56
@github-actions github-actions Bot removed the AWS label Apr 24, 2025
@nastra nastra force-pushed the gcs-multi-prefix-support branch 2 times, most recently from 74a0ca1 to bbae5b6 Compare April 24, 2025 08:24
@github-actions github-actions Bot added the AWS label Apr 24, 2025
Comment thread gcp/src/main/java/org/apache/iceberg/gcp/gcs/PrefixedStorage.java Outdated
Comment thread gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSInputFile.java Outdated
Comment thread gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSOutputFile.java Outdated
*/
public static final int GCS_DELETE_BATCH_SIZE_DEFAULT = 50;

private final Map<String, String> allProperties;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need to keep all the properties? The goal was to no copy all properties when we should only need the FileIO specific properties (though it looks like all properties were added to the FileIO as well which is questionable)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this shouldn't be needed anymore once we remove GCPProperties from the constructors of GCSFileIO

Comment thread gcp/src/main/java/org/apache/iceberg/gcp/gcs/GCSFileIO.java Outdated
Comment thread build.gradle Outdated
@nastra nastra force-pushed the gcs-multi-prefix-support branch from bbae5b6 to 0e762f9 Compare May 5, 2025 07:14
@nastra nastra force-pushed the gcs-multi-prefix-support branch from 0e762f9 to 25e7cfa Compare May 6, 2025 06:53
@nastra nastra force-pushed the gcs-multi-prefix-support branch from 25e7cfa to 2958fe4 Compare May 6, 2025 08:15

@danielcweeks danielcweeks left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @nastra!

@nastra

nastra commented May 7, 2025

Copy link
Copy Markdown
Contributor Author

thanks for the review @danielcweeks

@nastra nastra merged commit 542a4d7 into apache:main May 7, 2025
@nastra nastra deleted the gcs-multi-prefix-support branch May 7, 2025 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants