|
| 1 | +--- |
| 2 | +# |
| 3 | +# Licensed to the Apache Software Foundation (ASF) under one |
| 4 | +# or more contributor license agreements. See the NOTICE file |
| 5 | +# distributed with this work for additional information |
| 6 | +# regarding copyright ownership. The ASF licenses this file |
| 7 | +# to you under the Apache License, Version 2.0 (the |
| 8 | +# "License"); you may not use this file except in compliance |
| 9 | +# with the License. You may obtain a copy of the License at |
| 10 | +# |
| 11 | +# http://www.apache.org/licenses/LICENSE-2.0 |
| 12 | +# |
| 13 | +# Unless required by applicable law or agreed to in writing, |
| 14 | +# software distributed under the License is distributed on an |
| 15 | +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| 16 | +# KIND, either express or implied. See the License for the |
| 17 | +# specific language governing permissions and limitations |
| 18 | +# under the License. |
| 19 | +# |
| 20 | +Title: Using MinIO |
| 21 | +type: docs |
| 22 | +weight: 500 |
| 23 | +--- |
| 24 | + |
| 25 | +In this guide we walk through setting up a simple Polaris Server with local [MinIO](https://www.min.io/) storage. |
| 26 | + |
| 27 | +Similar configurations are expected to work with other S3-compatible systems that also have the |
| 28 | +[STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) API. |
| 29 | + |
| 30 | +# Setup |
| 31 | + |
| 32 | +Start MinIO. Here's a `docker` example. |
| 33 | + |
| 34 | +```shell |
| 35 | +docker run -p 9100:9000 -p 9101:9001 --name minio \ |
| 36 | + -e "MINIO_ROOT_USER=miniouser" -e "MINIO_ROOT_PASSWORD=miniopwd" \ |
| 37 | + quay.io/minio/minio:latest server /data --console-address ":9001" |
| 38 | +``` |
| 39 | + |
| 40 | +Create a bucket called "test" in MinIO UI or CLI. |
| 41 | + |
| 42 | +Edit `~/.aws/credentials` and create a `minio` profile. |
| 43 | + |
| 44 | +``` |
| 45 | +[minio] |
| 46 | +aws_access_key_id = miniouser |
| 47 | +aws_secret_access_key = miniopwd |
| 48 | +region=us-west-2 |
| 49 | +``` |
| 50 | + |
| 51 | +In the shell where the Polaris Server will be started `export AWS_PROFILE=minio`. |
| 52 | + |
| 53 | +Build and run Polaris Server. |
| 54 | + |
| 55 | +```shell |
| 56 | +./gradlew assemble |
| 57 | + |
| 58 | +env POLARIS_BOOTSTRAP_CREDENTIALS=POLARIS,root,s3cr3t \ |
| 59 | + java -jar runtime/server/build/quarkus-app/quarkus-run.jar |
| 60 | +``` |
| 61 | + |
| 62 | +Note: Apache Polaris 1.0.0-incubating does not yet have all the code required for proper operation with MinIO. |
| 63 | +The next release is expected to provide full support for S3-compatible storage with STS. |
| 64 | + |
| 65 | +Create a catalog named "polaris". |
| 66 | + |
| 67 | +```shell |
| 68 | +./polaris --client-id root --client-secret s3cr3t \ |
| 69 | + catalogs create polaris --storage-type S3 \ |
| 70 | + --default-base-location 's3://test' \ |
| 71 | + --role-arn arn:aws:iam::123456789012:role/dummy \ |
| 72 | + --region us-west-2 \ |
| 73 | + --endpoint http://127.0.0.1:9100 \ |
| 74 | + --path-style-access |
| 75 | +``` |
| 76 | + |
| 77 | +Note: the role and region parameters need to be set to avoid runtime errors in Polaris, |
| 78 | +but they will be ignored by MinIO. |
| 79 | + |
| 80 | +For the sake of simplicity, grant `CATALOG_MANAGE_CONTENT` directly to the `catalog_admin` role |
| 81 | +using `polaris` CLI. |
| 82 | + |
| 83 | +```shell |
| 84 | +./polaris --client-id root --client-secret s3cr3t \ |
| 85 | + privileges catalog grant --catalog polaris \ |
| 86 | + --catalog-role catalog_admin CATALOG_MANAGE_CONTENT |
| 87 | +``` |
| 88 | + |
| 89 | +# Connecting from Spark |
| 90 | + |
| 91 | +Start Spark. |
| 92 | + |
| 93 | +```shell |
| 94 | +bin/spark-sql \ |
| 95 | + --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.0,org.apache.iceberg:iceberg-aws-bundle:1.9.0 \ |
| 96 | + --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \ |
| 97 | + --conf spark.sql.catalog.polaris=org.apache.iceberg.spark.SparkCatalog \ |
| 98 | + --conf spark.sql.catalog.polaris.type=rest \ |
| 99 | + --conf spark.sql.catalog.polaris.uri=http://localhost:8181/api/catalog \ |
| 100 | + --conf spark.sql.catalog.polaris.token-refresh-enabled=false \ |
| 101 | + --conf spark.sql.catalog.polaris.warehouse=polaris \ |
| 102 | + --conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \ |
| 103 | + --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation=vended-credentials \ |
| 104 | + --conf spark.sql.catalog.polaris.credential=root:s3cr3t |
| 105 | +``` |
| 106 | + |
| 107 | +Create a table in Spark. |
| 108 | + |
| 109 | +```sql |
| 110 | +use polaris; |
| 111 | +create namespace ns; |
| 112 | +create table ns.t1 as select 'abc'; |
| 113 | +select * from ns.t1; |
| 114 | +``` |
0 commit comments