Skip to content

AWS: remove S3URI scheme restrictions#3656

Merged
jackye1995 merged 2 commits into
apache:masterfrom
jackye1995:s3-uri
Dec 3, 2021
Merged

AWS: remove S3URI scheme restrictions#3656
jackye1995 merged 2 commits into
apache:masterfrom
jackye1995:s3-uri

Conversation

@jackye1995

Copy link
Copy Markdown
Contributor

As discussed in the mailing list: https://mail-archives.apache.org/mod_mbox/iceberg-dev/202112.mbox/%3CCAMwmD1_P2PCMpzUai5RV%2B4a9Bv62ZFqrrDdxCtLcTSvPnUxk6g%40mail.gmail.com%3E

Based on what Piotr said,

There are storages which are primarily used as "s3 compatible" and they need these settings to make them work.
We've seen these being used to access MinIO, Ceph and even S3 with some gateway

It seems to make more sense to just remove the restriction entirely instead of extending the list of allowed schemes.

@rdblue @danielcweeks @findepi @mayursrivastava

@github-actions github-actions Bot added the AWS label Dec 2, 2021
@jackye1995 jackye1995 added this to the Iceberg 0.13.0 Release milestone Dec 2, 2021
@jackye1995

Copy link
Copy Markdown
Contributor Author

looks like flaky test, restart CI

@jackye1995 jackye1995 closed this Dec 2, 2021
@jackye1995 jackye1995 reopened this Dec 2, 2021
ValidationException.check(authoritySplit.length == 2, "Invalid S3 URI: %s", location);
ValidationException.check(!authoritySplit[1].trim().isEmpty(), "Invalid S3 key: %s", location);
ValidationException.check(authoritySplit.length == 2,
"Cannot determine bucket in S3 URI path: %s", location);

@rdblue rdblue Dec 3, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bucket isn't in the URI path element, it is in the authority. Maybe just "Cannot determine bucket in S3 URI" instead? Or make it match my suggestion below with Invalid S3 URI, cannot determine bucket: %s.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I always thought authority is a part of path... updated

ValidationException.check(authoritySplit.length == 2,
"Cannot determine bucket in S3 URI path: %s", location);
ValidationException.check(!authoritySplit[1].trim().isEmpty(),
"Cannot determine object key in S3 URI path: %s", location);

@rdblue rdblue Dec 3, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about Invalid S3 URI, path is empty: %s?

@rdblue

rdblue commented Dec 3, 2021

Copy link
Copy Markdown
Contributor

Looks good overall. I'd probably fix some of the error messages that were updated to be more specific.

@jackye1995

Copy link
Copy Markdown
Contributor Author

Given Ryan approves this and all error messages are fixed, I am merging this. @danielcweeks Let me know if you have any additional concern.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants