Add service account impersonation support for BigQueryMetastoreCatalog by joyhaldar · Pull Request #14447 · apache/iceberg

joyhaldar · 2025-10-30T08:51:11Z

Description:
This PR adds service account impersonation support to BigQueryMetastoreCatalog, enabling identity separation between cluster operations and data access

Problem
BigQueryMetastoreCatalog only supports Application Default Credentials with no mechanism for service account impersonation. This prevents:

Implementing least-privilege security (cluster operations vs data access)
Running multi-tenant workloads on shared clusters
Creating proper audit trails per service account

Solution
Introduces a properties based approach for BigQuery client configuration with impersonation support using Google's ImpersonatedCredentials API.

Key changes:

Created BigQueryProperties class with metastoreOptions() for BigQuery client configuration
Added impersonation properties to GCPProperties: service account, delegates, lifetime, scopes
Updated PrefixedStorage to support impersonated credentials for GCS operations
Impersonation properties are scoped per service (gcp.bigquery.impersonate.* for BigQuery, gcs.impersonate.* for GCS)
Added deprecation annotations for constants moved to BigQueryProperties

Configuration

Minimal:

gcp.bigquery.impersonate.service-account=data-sa@project.iam.gserviceaccount.com
gcs.impersonate.service-account=data-sa@project.iam.gserviceaccount.com

Full:

# BigQuery
gcp.bigquery.impersonate.service-account=data-sa@project.iam.gserviceaccount.com
gcp.bigquery.impersonate.delegates=admin-sa@project.iam.gserviceaccount.com
gcp.bigquery.impersonate.lifetime-seconds=3600
gcp.bigquery.impersonate.scopes=bigquery

# GCS
gcs.impersonate.service-account=data-sa@project.iam.gserviceaccount.com
gcs.impersonate.delegates=admin-sa@project.iam.gserviceaccount.com
gcs.impersonate.lifetime-seconds=3600
gcs.impersonate.scopes=devstorage.read_write

Testing

Unit tests:

TestBigQueryProperties
TestPrefixedStorage

Backward Compatibility
Fully backward compatible. Catalogs without impersonation properties continue using ADC exactly as before.

Closes #14446

adding tests and formatting

Add scope expansion to support short names and http URLs

nastra · 2025-10-30T09:17:43Z

@talatuyarer can you review this one please?

joyhaldar · 2025-10-30T10:43:35Z

Hello @talatuyarer, @ebyhr, @nastra, i would really appreciate it if you could please take a look if you have some time.

yogevyuval · 2025-11-03T13:28:10Z

@joyhaldar Thanks for working at this! I think service account impersonation is relevant also outside of the BigQueryMetastoreCatalog context - basically for every workload running on GCP, so I would just make sure it's generic enough to be used regardless of the catalog type

joyhaldar · 2025-11-03T14:23:08Z

@joyhaldar Thanks for working at this! I think service account impersonation is relevant also outside of the BigQueryMetastoreCatalog context - basically for every workload running on GCP, so I would just make sure it's generic enough to be used regardless of the catalog type

Hi @yogevyuval,

Thanks for the feedback! I really appreciate it.

I wrote this to follow current patterns, for example AssumeRoleAwsClientFactory also only works with AWS catalogs if I am not wrong (please correct me if I am). I also think that users can handle impersonation at the application level for other catalog types if needed.

I personally think this would be best addressed in a follow-up PR to keep the scope focused, but I'm happy to try and expand this PR to support any catalog now if you and the other reviewers think that's a good idea.

Please let me know what you think.

Thanks,
Joy

manuzhang · 2025-11-03T14:59:26Z

              this.closeableGroup = new CloseableGroup();
              builder.setCredentials(
                  GCPAuthUtils.oauth2CredentialsFromGcpProperties(gcpProperties, closeableGroup));
+            } else if (gcpProperties.impersonateServiceAccount().isPresent()) {


Is this change tested?

Thanks for pointing this out! Added tests for the impersonation path:

impersonationPropertiesAreRead() - Verifies all impersonation properties

impersonationPropertiesWithDefaults() - Verifies defaults work

Also tested end to end with actual GCP projects to confirm credential creation works.

yogevyuval · 2025-11-03T18:45:17Z

@joyhaldar Thanks for working at this! I think service account impersonation is relevant also outside of the BigQueryMetastoreCatalog context - basically for every workload running on GCP, so I would just make sure it's generic enough to be used regardless of the catalog type

Hi @yogevyuval,

Thanks for the feedback! I really appreciate it.

I wrote this to follow current patterns, for example AssumeRoleAwsClientFactory also only works with AWS catalogs if I am not wrong (please correct me if I am). I also think that users can handle impersonation at the application level for other catalog types if needed.

I personally think this would be best addressed in a follow-up PR to keep the scope focused, but I'm happy to try and expand this PR to support any catalog now if you and the other reviewers think that's a good idea.

Please let me know what you think.

Thanks, Joy

So what I meant is a situation where a lakehouse is hosted on GCP but with a self managed catalog, such as polaris/hive metastore, but the files would still be hosted in GCS, that's where the impersonation can really be useful even when not using BigQuery

kevinjqliu · 2025-11-07T20:04:22Z

Is the service account impersonation support for the catalog, fileio, or both?

I see there's already a GoogleAuthManager class for handling auth and google credential. It uses GoogleCredentials.fromStream which already supports ImpersonatedCredentials

Could we reuse the GoogleAuthManager to abstract away the auth details?

joyhaldar · 2025-11-08T03:57:35Z

Is the service account impersonation support for the catalog, fileio, or both?

I see there's already a GoogleAuthManager class for handling auth and google credential. It uses GoogleCredentials.fromStream which already supports ImpersonatedCredentials

Could we reuse the GoogleAuthManager to abstract away the auth details?

Thank you for the comment, Kevin.

The impersonation supports both BigQuery and GCS FileIO.

Regarding GoogleAuthManager, I was under the impression that it's designed for
REST Catalog authentication, while BigQueryMetastoreCatalog uses GoogleCredentials
directly with GCP client libraries.

Please let me know if I'm misunderstanding your suggestion.

rambleraptor · 2025-11-12T21:12:05Z


+  public static final String CLIENT_FACTORY = "gcp.bigquery.client.factory";
+  private static final String GCS_IMPERSONATE_SERVICE_ACCOUNT = "gcs.impersonate.service-account";
+  private static final String GCS_PROJECT_ID = "gcs.project-id";


These 4 should have the gcp prefix instead of gcs. That matches your PR example + keeps everything under the same namespace.

Thank you so much for reviewing @rambleraptor.

I wanted to explain the flow a bit:

ImpersonatedBigQueryClientFactory defines: gcp.bigquery.impersonate.service-account (used for BigQuery client)

prepareFileIOProperties() transforms: gcp.* → gcs.* (for FileIO, based on constants defined in GCPProperties)

GCPProperties reads: gcs.impersonate.service-account

Do you think this should be changed?

I can try removing the private constants from BigQueryMetastoreCatalog and use GCPProperties.GCS_*
directly.

Please let me know what you think would be more appropriate.

Alright, this makes sense. It's fine by me.

rambleraptor · 2025-11-12T21:13:44Z

+    GoogleCredentials scopedCredentials =
+        (credentials instanceof ImpersonatedCredentials)
+            ? credentials
+            : credentials.createScoped(BigqueryScopes.all());


@talatuyarer Love your opinion on this:

I'm a little worried about defaulting this to use scopes.all() (even though that's the current functionality). Scoping is a great way to force read-only behavior at a lower-level.

I would also love Talat's opinion on this.

rambleraptor · 2025-11-12T21:15:40Z

+  private String projectId;
+  private String location;
+
+  private static final String DEFAULT_LOCATION = "us";


I'm personally not a huge fan of having a default location, but I'm happy to be overridden.

Thank you for your review comment @rambleraptor .

I preserved the default "us" from BigQueryMetastoreCatalog (DEFAULT_GCP_LOCATION) to
avoid breaking existing users. Do you think it should be removed?

Nope, that sounds great

…ttern - Removed BigQueryClientFactory, DefaultBigQueryClientFactory, ImpersonatedBigQueryClientFactory - Created BigQueryProperties with metastoreOptions() for client configuration - Added impersonation support to GCPProperties and PrefixedStorage - Single property (gcp.impersonate.service-account) enables impersonation for both BigQuery and GCS

danielcweeks

A few more comments, but this is looking pretty close.

- Add deprecation annotations for moved constants - Move credential scoping to BigQueryProperties.metastoreOptions() - Make BigQueryProperties package-private - Add configurable GCS impersonation scopes

…ables parsing to BigQueryProperties.

danielcweeks

Thanks @joyhaldar!

…atalog (apache#14447) * Add supporting impersonation in BigQueryMetastoreCatalog Co-authored-by: Joy Haldar <Joy.Haldar@target.com>

Joy Haldar and others added 5 commits October 26, 2025 13:45

initial commit for supporting impersonation in BigQueryMetastoreCatalog

4ef0a81

BigQuery: Add service account impersonation support

2b1e4e0

remove local .vscode files.

a56ba12

adding tests and formatting

3ba551a

adding tests and formatting

Add scope expansion to support short names and http URLs

d719e7a

Add scope expansion to support short names and http URLs

github-actions Bot added the GCP label Oct 30, 2025

manuzhang reviewed Nov 3, 2025

View reviewed changes

add authentication path tests for PrefixedStorage

81607aa

rambleraptor reviewed Nov 12, 2025

View reviewed changes

remove stray line to address Alex's review comment.

670cb60

rambleraptor approved these changes Nov 18, 2025

View reviewed changes

nastra requested a review from danielcweeks November 19, 2025 13:29

danielcweeks reviewed Nov 19, 2025

View reviewed changes

Comment thread bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java Outdated

danielcweeks reviewed Nov 19, 2025

View reviewed changes

Comment thread bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java Outdated

danielcweeks reviewed Nov 19, 2025

View reviewed changes

Comment thread bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryMetastoreCatalog.java Outdated

danielcweeks reviewed Nov 19, 2025

View reviewed changes

Comment thread bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryClientFactory.java Outdated

danielcweeks self-requested a review December 9, 2025 22:58

joyhaldar and others added 2 commits December 11, 2025 18:55

Merge branch 'main' into gcp-bigquery-impersonation

c2cc240

danielcweeks reviewed Dec 17, 2025

View reviewed changes

joyhaldar added 2 commits December 18, 2025 16:28

Address review feedback for BigQuery impersonation PR

0cca3ab

- Add deprecation annotations for moved constants - Move credential scoping to BigQueryProperties.metastoreOptions() - Make BigQueryProperties package-private - Add configurable GCS impersonation scopes

Fix TestBigQueryProperties

c00fc14

danielcweeks reviewed Dec 18, 2025

View reviewed changes

Scope impersonation properties to gcs and gcp.bigquery, move listAllT…

8e0000b

…ables parsing to BigQueryProperties.

danielcweeks approved these changes Dec 19, 2025

View reviewed changes

danielcweeks merged commit 554a3c1 into apache:main Dec 19, 2025
44 checks passed

Uh oh!

Conversation

joyhaldar commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nastra commented Oct 30, 2025

Uh oh!

joyhaldar commented Oct 30, 2025

Uh oh!

yogevyuval commented Nov 3, 2025

Uh oh!

joyhaldar commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yogevyuval commented Nov 3, 2025

Uh oh!

kevinjqliu commented Nov 7, 2025

Uh oh!

joyhaldar commented Nov 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joyhaldar Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joyhaldar Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danielcweeks left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danielcweeks left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

joyhaldar commented Oct 30, 2025 •

edited

Loading

joyhaldar commented Nov 3, 2025 •

edited

Loading

joyhaldar commented Nov 8, 2025 •

edited

Loading

joyhaldar Nov 13, 2025 •

edited

Loading

joyhaldar Nov 13, 2025 •

edited

Loading