S3 path issue - Name or service not known

[GreenBinary]

Describe the bug

I'm developing data-streaming-pipeline and have a Kafka (v3.9.0), Flink (v1.20.1), Iceberg (v1.8.1) on MinIO based data streaming pipeline all deployed on-prem on Kubernetes cluster.

For last several weeks, I've been working to establish Iceberg REST based "Apache Polaris" catalog: 1.0.x incubating.

First I deploy all my containers (including Polaris) which are running fine. Then I add a new catalog using Polaris REST API using curl command. Then, in Flink-SQL, I create a new catalog (kind of alias to the one created using curl command), database successfully. Finally I try to create an iceberg table which ends up with error.

I took care of setting required properties to ensure S3 path style access is set to true for my on-prem MinIO storage. I'll drop/attach k8s manifest files here for MinIO and Polaris YAML (you'll see I've been experimenting with a lot of env variables!).

Then I'll add the Polaris curl command to add the catalog and flink-sql commands in "To Reproduce" section, and finally put the error details/logs in the "Actual Behavior" section.

I tried a lot of things consistently for weeks with no luck at all. Please provide your feedback and let me know how to get this resolved. Thanks much.

By the way, before trying Polaris, I had been using simple HadoopCatalog and everything was working just fine.

polaris-catalog.yaml
minio-data-lake.yaml

To Reproduce

Add a Polaris catalog using REST API method via curl command as below:

curl -i -X POST -H "Authorization: Bearer $Access_Token" "http://polaris-service.my-ns.svc.cluster.local:8181/api/management/v1/catalogs" \ -H "Content-Type: application/json" \ -d '{ "catalog": { "name": "my_iceberg_polaris_catalog", "type": "INTERNAL", "properties": { "default-base-location": "s3://iceberg-bucket", "s3.endpoint": "http://minio-data-lake-service.my-ns.svc.cluster.local:9000", "s3.path-style-access": "true", "s3.access-key-id": "polaris-service-user", "s3.secret-access-key": "Passw0rd1", "s3.region": "auto" }, "storageConfigInfo": { "roleArn": "arn:aws:iam::000000000000:role/minio-polaris-role", "region": "auto", "storageType": "S3", "endpoint":"http://minio-data-lake-service.my-ns.svc.cluster.local:9000", "pathStyleAccess":"true", "allowedLocations": [ "s3://iceberg-bucket/*" ] } } }'

And below are flink-sql commands. All gets executed successfully except for the last one (CREATE TABLE one).

`CREATE CATALOG IF NOT EXISTS my_iceberg_polaris_catalog WITH (
'type'='iceberg',
'catalog-type'='rest',
'uri'='http://polaris-service.my-ns.svc.cluster.local:8181/api/catalog',
'credential'='root:s3cr3t',
'rest.credential-vending.enabled'='true',
'warehouse'='my_iceberg_polaris_catalog',
's3.region'='auto',
's3.endpoint'='http://minio-data-lake-service.my-ns.svc.cluster.local:9000',
's3.path-style-access'='true',
's3.access-key-id'='polaris-service-user',
's3.secret-access-key'='Passw0rd1',
'token-refresh-enabled'='true',
'scope'='PRINCIPAL_ROLE:ALL'
);

USE CATALOG my_iceberg_polaris_catalog;

CREATE DATABASE IF NOT EXISTS my_iceberg_db;

USE my_iceberg_db;

CREATE TABLE IF NOT EXISTS i_t_obk_cdc_topic_ciprod_CMMSDATA_MM_MAINTENANCE_ACTION_LABOR_tagged
(
...
)
PARTITIONED BY (col1)
WITH
(
'format-version'='2',
'write.format.default'='parquet',
'write.upsert.enabled'='true',
'write.merge.mode'='merge-on-read',
'write.equality-delete-field'='',
'write.distribution-mode'='hash',
'write.iceberg.hash-distribution.column'='',
'write.target-file-size-bytes'='1048576'
);`

Actual Behavior

Error below from flink-sql terminal:

**_org.apache.flink.table.api.TableException: Could not execute CreateTable in path my_iceberg_polaris_catalog.my_iceberg_db.iceberg_table_1_** at org.apache.flink.table.catalog.CatalogManager.execute(CatalogManager.java:1375) ~[flink-table-api-java-uber-1.20.1.jar:1.20.1] at org.apache.flink.table.catalog.CatalogManager.createTable(CatalogManager.java:1005) ~[flink-table-api-java-uber-1.20.1.jar:1.20.1] at org.apache.flink.table.operations.ddl.CreateTableOperation.execute(CreateTableOperation.java:86) ~[flink-table-api-java-uber-1.20.1.jar:1.20.1] at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1102) ~[flink-table-api-java-uber-1.20.1.jar:1.20.1] at org.apache.flink.table.gateway.service.operation.OperationExecutor.callOperation(OperationExecutor.java:687) ~[flink-sql-gateway-1.20.1.jar:1.20.1] at org.apache.flink.table.gateway.service.operation.OperationExecutor.executeOperation(OperationExecutor.java:522) ~[flink-sql-gateway-1.20.1.jar:1.20.1] at org.apache.flink.table.gateway.service.operation.OperationExecutor.executeStatement(OperationExecutor.java:243) ~[flink-sql-gateway-1.20.1.jar:1.20.1] at org.apache.flink.table.gateway.service.operation.OperationExecutor.executeStatement(OperationExecutor.java:199) ~[flink-sql-gateway-1.20.1.jar:1.20.1] at org.apache.flink.table.gateway.service.SqlGatewayServiceImpl.lambda$executeStatement$1(SqlGatewayServiceImpl.java:214) ~[flink-sql-gateway-1.20.1.jar:1.20.1] at org.apache.flink.table.gateway.service.operation.OperationManager.lambda$submitOperation$1(OperationManager.java:119) ~[flink-sql-gateway-1.20.1.jar:1.20.1] at org.apache.flink.table.gateway.service.operation.OperationManager$Operation.lambda$run$0(OperationManager.java:258) ~[flink-sql-gateway-1.20.1.jar:1.20.1] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?] **_Caused by: org.apache.iceberg.exceptions.ServiceFailureException: Server error: SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long._** at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:217) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:118) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:102) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:224) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:308) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.rest.BaseHTTPClient.post(BaseHTTPClient.java:88) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.rest.RESTClient.post(RESTClient.java:113) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.rest.RESTSessionCatalog$Builder.create(RESTSessionCatalog.java:848) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.CachingCatalog$CachingTableBuilder.lambda$create$0(CachingCatalog.java:264) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2406) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at java.util.concurrent.ConcurrentHashMap.compute(Unknown Source) ~[?:?] at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2404) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2387) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.CachingCatalog$CachingTableBuilder.create(CachingCatalog.java:260) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:75) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.flink.FlinkCatalog.createIcebergTable(FlinkCatalog.java:415) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.iceberg.flink.FlinkCatalog.createTable(FlinkCatalog.java:395) ~[iceberg-flink-runtime-1.20-1.8.1.jar:?] at org.apache.flink.table.catalog.CatalogManager.lambda$createTable$18(CatalogManager.java:1016) ~[flink-table-api-java-uber-1.20.1.jar:1.20.1] at org.apache.flink.table.catalog.CatalogManager.execute(CatalogManager.java:1369) ~[flink-table-api-java-uber-1.20.1.jar:1.20.1] ... 17 more 17:41:51.335 [flink-rest-server-netty-worker-thread-1] ERROR org.apache.flink.table.gateway.service.SqlGatewayServiceImpl - Failed to fetchResults. org.apache.flink.table.gateway.service.utils.SqlExecutionException: Failed to execute the operation d12e4ce3-6997-4e6c-99bb-2467b49802fd. at org.apache.flink.table.gateway.service.operation.OperationManager$Operation.processThrowable(OperationManager.java:414) ~[flink-sql-gateway-1.20.1.jar:1.20.1] at org.apache.flink.table.gateway.service.operation.OperationManager$Operation.lambda$run$0(OperationManager.java:267) ~[flink-sql-gateway-1.20.1.jar:1.20.1] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?] at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?] at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?] at java.lang.Thread.run(Unknown Source) [?:?] Caused by: org.apache.flink.table.api.TableException: Could not execute CreateTable in path my_iceberg_polaris_catalog.my_iceberg_db.iceberg_table_1`
at org.apache.flink.table.catalog.CatalogManager.execute(CatalogManager.java:1375) ~[flink-table-api-java-uber-1.20.1.jar:1.20.1]
at org.apache.flink.table.catalog.CatalogManager.createTable(CatalogManager.java:1005) ~[flink-table-api-java-uber-1.20.1.jar:1.20.1]
at org.apache.flink.table.operations.ddl.CreateTableOperation.execute(CreateTableOperation.java:86) ~[flink-table-api-java-uber-1.20.1.jar:1.20.1]
at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1102) ~[flink-table-api-java-uber-1.20.1.jar:1.20.1]
at org.apache.flink.table.gateway.service.operation.OperationExecutor.callOperation(OperationExecutor.java:687) ~[flink-sql-gateway-1.20.1.jar:1.20.1]
at org.apache.flink.table.gateway.service.operation.OperationExecutor.executeOperation(OperationExecutor.java:522) ~[flink-sql-gateway-1.20.1.jar:1.20.1]
at org.apache.flink.table.gateway.service.operation.OperationExecutor.executeStatement(OperationExecutor.java:243) ~[flink-sql-gateway-1.20.1.jar:1.20.1]
at org.apache.flink.table.gateway.service.operation.OperationExecutor.executeStatement(OperationExecutor.java:199) ~[flink-sql-gateway-1.20.1.jar:1.20.1]
at org.apache.flink.table.gateway.service.SqlGatewayServiceImpl.lambda$executeStatement$1(SqlGatewayServiceImpl.java:214) ~[flink-sql-gateway-1.20.1.jar:1.20.1]
at org.apache.flink.table.gateway.service.operation.OperationManager.lambda$submitOperation$1(OperationManager.java:119) ~[flink-sql-gateway-1.20.1.jar:1.20.1]
...

===============

And below error log from Polaris pod:

**_2025-08-19 17:41:51,220 INFO [org.apa.pol.ser.exc.IcebergExceptionMapper] [,POLARIS] [,,,] (executor-thread-1) Full RuntimeException: software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long._** at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130) at software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor.modifyException(HelpfulUnknownHostExceptionInterceptor.java:59) at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyException(ExecutionInterceptorChain.java:181) at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.runModifyException(ExceptionReportingUtils.java:54) at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.reportFailureToInterceptors(ExceptionReportingUtils.java:38) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:39) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53) at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:11227) at org.apache.iceberg.aws.s3.S3OutputStream.completeUploads(S3OutputStream.java:444) at org.apache.iceberg.aws.s3.S3OutputStream.close(S3OutputStream.java:270) at org.apache.iceberg.aws.s3.S3OutputStream.close(S3OutputStream.java:256) at java.base/sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:439) at java.base/sun.nio.cs.StreamEncoder.lmyedClose(StreamEncoder.java:237) at java.base/sun.nio.cs.StreamEncoder.close(StreamEncoder.java:222) at java.base/java.io.OutputStreamWriter.close(OutputStreamWriter.java:266) at org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:133) at org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:117) at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.writeNewMetadata(IcebergCatalog.java:1571) at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.writeNewMetadataIfRequired(IcebergCatalog.java:1560) at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.doCommit(IcebergCatalog.java:1430) at org.apache.polaris.service.catalog.iceberg.IcebergCatalog$BasePolarisTableOperations.commit(IcebergCatalog.java:1270) at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:201) at org.apache.polaris.service.catalog.iceberg.CatalogHandlerUtils.createTable(CatalogHandlerUtils.java:339) at org.apache.polaris.service.catalog.iceberg.CatalogHandlerUtils_ClientProxy.createTable(Unknown Source) at org.apache.polaris.service.catalog.iceberg.IcebergCatalogHandler.createTableDirect(IcebergCatalogHandler.java:394) at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.lambda$createTable$6(IcebergCatalogAdapter.java:360) at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.withCatalog(IcebergCatalogAdapter.java:183) at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter.createTable(IcebergCatalogAdapter.java:347) at org.apache.polaris.service.catalog.iceberg.IcebergCatalogAdapter_ClientProxy.createTable(Unknown Source) at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi.createTable(IcebergRestCatalogApi.java:193) at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.createTable$$superforward(Unknown Source) at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass$$function$$3.apply(Unknown Source) at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73) at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97) at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$8(FaultToleranceInterceptor.java:364) at io.smallrye.faulttolerance.core.Future.from(Future.java:85) at io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$9(FaultToleranceInterceptor.java:364) at io.smallrye.faulttolerance.core.FaultToleranceContext.call(FaultToleranceContext.java:20) at io.smallrye.faulttolerance.core.Invocation.apply(Invocation.java:29) at io.smallrye.faulttolerance.core.metrics.MetricsCollector.apply(MetricsCollector.java:98) at io.smallrye.faulttolerance.FaultToleranceInterceptor.syncFlow(FaultToleranceInterceptor.java:367) at io.smallrye.faulttolerance.FaultToleranceInterceptor.intercept(FaultToleranceInterceptor.java:205) at io.smallrye.faulttolerance.FaultToleranceInterceptor_Bean.intercept(Unknown Source) at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42) at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70) at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97) at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor.timedMethod(MicrometerTimedInterceptor.java:79) at io.quarkus.micrometer.runtime.MicrometerTimedInterceptor_Bean.intercept(Unknown Source) at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42) at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70) at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97) at io.quarkus.security.runtime.interceptor.SecurityHandler.handle(SecurityHandler.java:27) at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor.intercept(RolesAllowedInterceptor.java:29) at io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor_Bean.intercept(Unknown Source) at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42) at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70) at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62) at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor.intercept(StandardSecurityCheckInterceptor.java:44) at io.quarkus.resteasy.reactive.server.runtime.StandardSecurityCheckInterceptor_RolesAllowedInterceptor_Bean.intercept(Unknown Source) at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42) at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30) at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27) at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi_Subclass.createTable(Unknown Source) at org.apache.polaris.service.catalog.api.IcebergRestCatalogApi$quarkusrestinvoker$createTable_01f5a1bd6d7815fd3314a553161c943c8cd03101.invoke(Unknown Source) at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29) at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141) at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147) at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:638) at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675) at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654) at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594) at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11) at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: iceberg-bucket.minio-data-lake-service.my-ns.svc.cluster.local: Name or service not known (SDK Attempt Count: 6) at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130) at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:95) at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ... 78 more **_Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: iceberg-bucket.minio-data-lake-service.my-ns.svc.cluster.local: Name or service not known_** ...

Expected Behavior

Polaris would work well and Iceberg tables get created successfully.

Additional context

No response

System information

OS: RHEL 8

[adutra]

The problem imho is that here we are calling IcebergCatalogHandler.createTableDirect() since no Access Delegation header is present. But createTableDirect() does not generate any table config for S3 access.

You coudl maybe try to create the catalog with default table properties:

CREATE CATALOG IF NOT EXISTS my_iceberg_polaris_catalog WITH (
'type'='iceberg',
'catalog-type'='rest',
...
'table-default.s3.path-style-access'='true'
);

If that doesn't help, maybe S3 remote signing could help, but you need to wait for #2280.

[adutra]

Put differently: S3 endpoint and S3 path style access are only included in the table config IF credential subscoping is enabled for the catalog:

polaris/polaris-core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java

Lines 123 to 135 in d7d8990

	URI endpointUri = storageConfig.getEndpointUri();
	if (endpointUri != null) {
	accessConfig.put(StorageAccessProperty.AWS_ENDPOINT, endpointUri.toString());
	}
	URI internalEndpointUri = storageConfig.getInternalEndpointUri();
	if (internalEndpointUri != null) {
	accessConfig.putInternalProperty(
	StorageAccessProperty.AWS_ENDPOINT.getPropertyName(), internalEndpointUri.toString());
	}
	if (Boolean.TRUE.equals(storageConfig.getPathStyleAccess())) {
	accessConfig.put(StorageAccessProperty.AWS_PATH_STYLE_ACCESS, Boolean.TRUE.toString());
	}

When the client is using direct access to MinIO, these properties are not included.

[GreenBinary]

Thank you @adutra very much for the replies.

I trided "'table-default.s3.path-style-access'='true'", but same error.

You mentioned about credential subscoping enabled: Based on my code I provided above, do you think it's enabled? Do I need to specify/configure something explicitly to enable it?

[adutra]

You mentioned about credential subscoping enabled: Based on my code I provided above, do you think it's enabled? Do I need to specify/configure something explicitly to enable it?

It's not. You need to pass header.X-Iceberg-Access-Delegation=vended-credentials when creating the catalog.

Have you checked this getting-started example with MinIO?

https://github.com/apache/polaris/blob/main/getting-started/minio/README.md

It uses Spark but still, it will give you an idea of how to configure the catalog for MinIO.

[adutra]

The TLDR for now is: MinIO cannot work without vended credentials (credential subscoping). @dimas-b is that correct?

[dimas-b]

I'm planning to adjust the whole "subscoping" code path under #2207 and make non-credential access config available to clients in all cases.

... but it looks like this is not going to make it to the 1.1.0 release 🤷‍♂️

[dimas-b]

and yes, currently MinIO requires credential vending to be enabled.

[GreenBinary]

Thank you gentlemen, I'll try it with credential vending enabled and report you how it works.

[ricardopalazzio]

@GreenBinary
im using an 'workaround', adding full catalog dns name in kubeDNS, like network alias:

    .:53 {
        errors
        health {
            lameduck 5s
          }
        ready
        **rewrite name exact datalake.minio.svc.cluster.local minio.svc.cluster.local**
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

My Trino cfg:

datalake: |
            connector.name=iceberg
            iceberg.catalog.type=rest
            iceberg.rest-catalog.uri=http://polaris.polaris.svc.cluster.local:8181/api/catalog/
            iceberg.rest-catalog.warehouse=datalake
            iceberg.rest-catalog.vended-credentials-enabled=true
            iceberg.rest-catalog.security=OAUTH2
            iceberg.rest-catalog.oauth2.credential=root:pass
            iceberg.rest-catalog.oauth2.scope=PRINCIPAL_ROLE:ALL
            fs.native-s3.enabled=true
            s3.endpoint=http://minio.svc.cluster.local:9000
            s3.region=us-east-1

And this envs on polaris Helm values

extraEnv:
  - name: AWS_REGION
    value: "us-east-1"
  - name: AWS_ACCESS_KEY_ID
    value: "******"
  - name: AWS_SECRET_ACCESS_KEY
    value: "******"
  - name: AWS_ENDPOINT_URL_S3
    value: "http://minio.svc.cluster.local:9000"
  - name: AWS_ENDPOINT_URL_STS
    value: "http://minio.svc.cluster.local:9000"

[GreenBinary]

Still issues.
Here's what I did (I tried "header.X-Iceberg-Access-Delegation" setting):

Below is my create catalog command from flink-sql:

CREATE CATALOG IF NOT EXISTS my_iceberg_polaris_catalog WITH (
  'type'='iceberg',
  'catalog-type'='rest',
  'uri'='http://polaris-service.my-ns.svc.cluster.local:8181/api/catalog',
  'credential'='root:s3cr3t',
  'warehouse'='my_iceberg_polaris_catalog',
  'oauth2-server-uri'='http://polaris-service.my-ns.svc.cluster.local:8181/api/catalog/v1/oauth/tokens',
  's3.region'='auto',
  's3.endpoint'='http://minio-data-lake-service.my-ns.svc.cluster.local:9000',
  's3.path-style-access'='true',
  's3.access-key-id'='polaris-service-user',
  's3.secret-access-key'='Passw0rd1',
  'token-refresh-enabled'='true',
  'header.X-Iceberg-Access-Delegation'='vended-credentials',
  'scope'='PRINCIPAL_ROLE:ALL'
);

Then below is catalog I created beforehand using polaris api curl command:

curl -i -X POST -H "Authorization: Bearer $POLARIS_TOKEN" "http://polaris-service.my-ns.svc.cluster.local:8181/api/management/v1/catalogs" \
   -H "Content-Type: application/json" \
   -d '{
         "catalog": {
           "name": "my_iceberg_polaris_catalog",
           "type": "INTERNAL",
           "properties": {
             "default-base-location": "s3://iceberg-bucket",
			 "s3.endpoint": "http://minio-data-lake-service.my-ns.svc.cluster.local:9000",
			 "s3.path-style-access": "true",
			 "s3.access-key-id": "polaris-service-user",
			 "s3.secret-access-key": "Passw0rd1",
			 "s3.region": "auto"
           },
           "storageConfigInfo": {
			 "roleArn": "arn:aws:iam::000000000000:role/minio-polaris-role",
			 "region": "auto",
             "storageType": "S3",			 
			 "endpoint":"http://localhost:9000",
			 "endpointInternal":"http://minio-data-lake-service.my-ns.svc.cluster.local:9000",
             "pathStyleAccess":"true",
             "allowedLocations": [
               "s3://iceberg-bucket/*"
             ]
           }
         }
       }'

I get this error:

Caused by: org.apache.iceberg.exceptions.ForbiddenException: Forbidden: Principal 'root' with activated PrincipalRoles '[]' and activated grants via '[service_admin, catalog_admin]' is not authorized for op CREATE_TABLE_DIRECT_WITH_WRITE_DELEGATION

===

So, then I went ahead and provided the root user appropriate roles using below curl commands:

# Create a catalog admin role
curl -X PUT \
  -H "Authorization: Bearer $POLARIS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"grant":{"type":"catalog", "privilege":"CATALOG_MANAGE_CONTENT"}}' \
  "http://localhost:8181/api/management/v1/catalogs/my_iceberg_polaris_catalog/catalog-roles/catalog_admin/grants"

# Create a data engineer role
curl -X POST \
  -H "Authorization: Bearer $POLARIS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"principalRole":{"name":"data_engineer"}}' \
  "http://localhost:8181/api/management/v1/principal-roles"

# Connect the roles
curl -X PUT \
  -H "Authorization: Bearer $POLARIS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"catalogRole":{"name":"catalog_admin"}}' \
  "http://localhost:8181/api/management/v1/principal-roles/data_engineer/catalog-roles/my_iceberg_polaris_catalog"

# Give root the data engineer role
curl -X PUT \
  -H "Authorization: Bearer $POLARIS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"principalRole": {"name":"data_engineer"}}' \
  "http://localhost:8181/api/management/v1/principals/root/principal-roles"

Now I get the original error back!

Caused by: java.net.UnknownHostException: iceberg-bucket.minio-data-lake-service.ock-ns.svc.cluster.local: Name or service not known
        at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
        at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)

Please help, thanks.

[dimas-b]

@GreenBinary : If you can use the latest main (or wait for 1.1.0) I believe, it should work.

In addition to the getting stated guide that Alex linked above , I hope this (recently added) page might help too: https://polaris.apache.org/in-dev/unreleased/getting-started/minio/

If you have to use older versions of Polaris, then non-AWS S3 storage is tricky to setup, in my experience. It may be necessary to actually debug Polaris to understand what is missing or misaligned... so all in all, I'd recommend using the upcoming 1.1.0 release :)

[GreenBinary]

Hi @dimas-b

I've been using main branch, "1.0.1-incubating" tag.
Here's details of my image:
imageRegistry: hub.docker.com
imageRepository: apache/polaris
imageTag: 1.0.1-incubating

Do you think I'd use other branch/tag?

I'll check the links you provided. Thanks.

[dimas-b]

@GreenBinary : AFAIK, on recent main the s3.* settings in your catalog example should probably be inside the storageConfigInfo block ... MinIO examples follow that pattern.

1.0.1-incubating is too old for "verified" MinIO support :)