Skip to content

Integration tests for Catalog Federation #2344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

poojanilangekar
Copy link
Contributor

Adds a Junit5 integration test for catalog federation.

@poojanilangekar
Copy link
Contributor Author

CC: @eric-maynard @dimas-b

Copy link
Contributor

@dimas-b dimas-b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @poojanilangekar ! The PR LGTM overall, but I have a concern with the IT test location (as commented).

public Map<String, String> getConfigOverrides() {
return Map.of(
"polaris.features.\"ENABLE_CATALOG_FEDERATION\"", "true",
"polaris.features.\"ALLOW_OVERLAPPING_CATALOG_URLS\"", "true");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to use 2 different base locations + 2 different storage configs for the two catalogs, and thus avoid setting this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to use the same base location because we are using a single Polaris instance. So unless we have two different Polaris instances (which we can't do in the current integration test setup), we need this configuration.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@poojanilangekar : Do you mean that federated catalogs check for location overlaps with internal catalogs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is correct.

Unless the ALLOW_OVERLAPPING_CATALOG_URLS flag is set, Polaris always checks if the base location of a CreateCatalog (irrespective of whether its INTERNAL vs EXTERNAL) request overlaps with an existing catalog. Therefore we need this flag to be set, at least for the JUnit tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(This has come up before, but I think the fix is that federating EXTERNAL catalogs shouldn't need or support a location)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not at the moment. Making this change would require changing the API spec and making subsequent changes in the CreateCatalog validations.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you be able to open an issue for sorting this out later?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: #2356

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is good to merge from my POV 👍

public static class CatalogFederationProfile implements QuarkusTestProfile {
@Override
public Map<String, String> getConfigOverrides() {
return Map.of(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be good to explicitly set SUPPORTED_CATALOG_CONNECTION_TYPES

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment on lines +116 to +176
newUserCredentials = managementApi.createPrincipal(PRINCIPAL_NAME);

FileStorageConfigInfo storageConfig =
FileStorageConfigInfo.builder()
.setStorageType(StorageConfigInfo.StorageTypeEnum.FILE)
.setAllowedLocations(List.of(baseLocation.toString()))
.build();

CatalogProperties catalogProperties = new CatalogProperties(baseLocation.toString());

Catalog localCatalog =
PolarisCatalog.builder()
.setType(Catalog.TypeEnum.INTERNAL)
.setName(LOCAL_CATALOG_NAME)
.setProperties(catalogProperties)
.setStorageConfigInfo(storageConfig)
.build();
managementApi.createCatalog(localCatalog);

CatalogGrant catalogGrant =
CatalogGrant.builder()
.setType(CatalogGrant.TypeEnum.CATALOG)
.setPrivilege(CatalogPrivilege.TABLE_WRITE_DATA)
.build();
managementApi.addGrant(LOCAL_CATALOG_NAME, CATALOG_ROLE_NAME, catalogGrant);
managementApi.assignPrincipalRole(PRINCIPAL_NAME, PRINCIPAL_ROLE_NAME);
CatalogRole localCatalogAdminRole =
managementApi.getCatalogRole(LOCAL_CATALOG_NAME, CATALOG_ROLE_NAME);
managementApi.grantCatalogRoleToPrincipalRole(
PRINCIPAL_ROLE_NAME, LOCAL_CATALOG_NAME, localCatalogAdminRole);

AuthenticationParameters authParams =
OAuthClientCredentialsParameters.builder()
.setAuthenticationType(AuthenticationParameters.AuthenticationTypeEnum.OAUTH)
.setTokenUri(endpoints.catalogApiEndpoint().toString() + "/v1/oauth/tokens")
.setClientId(newUserCredentials.getCredentials().getClientId())
.setClientSecret(newUserCredentials.getCredentials().getClientSecret())
.setScopes(List.of("PRINCIPAL_ROLE:ALL"))
.build();
ConnectionConfigInfo connectionConfig =
IcebergRestConnectionConfigInfo.builder()
.setConnectionType(ConnectionConfigInfo.ConnectionTypeEnum.ICEBERG_REST)
.setUri(endpoints.catalogApiEndpoint().toString())
.setRemoteCatalogName(LOCAL_CATALOG_NAME)
.setAuthenticationParameters(authParams)
.build();
ExternalCatalog externalCatalog =
ExternalCatalog.builder()
.setType(Catalog.TypeEnum.EXTERNAL)
.setName(EXTERNAL_CATALOG_NAME)
.setConnectionConfigInfo(connectionConfig)
.setProperties(catalogProperties)
.setStorageConfigInfo(storageConfig)
.build();
managementApi.createCatalog(externalCatalog);

managementApi.addGrant(EXTERNAL_CATALOG_NAME, CATALOG_ROLE_NAME, catalogGrant);
CatalogRole externalCatalogAdminRole =
managementApi.getCatalogRole(EXTERNAL_CATALOG_NAME, CATALOG_ROLE_NAME);
managementApi.grantCatalogRoleToPrincipalRole(
PRINCIPAL_ROLE_NAME, EXTERNAL_CATALOG_NAME, externalCatalogAdminRole);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only have one test right now so it's fine, but if we add more this should get refactored out

eric-maynard
eric-maynard previously approved these changes Aug 14, 2025
@github-project-automation github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Aug 14, 2025
Copy link
Contributor

@dimas-b dimas-b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but let's resolve outstanding conversations before merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants