Skip to content

Commit f1c5ae7

Browse files
committed
feat(docs): Add Getting Stated guide for MinIO
A simple page of step-by-step instructions for setting up a local environment with Polaris, MinIO and Spark. Closes #1530
1 parent 8996132 commit f1c5ae7

File tree

1 file changed

+114
-0
lines changed
  • site/content/in-dev/unreleased/getting-started

1 file changed

+114
-0
lines changed
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
---
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
#
20+
Title: Using MinIO
21+
type: docs
22+
weight: 500
23+
---
24+
25+
In this guide we walk through setting up a simple Polaris Server with local [MinIO](https://www.min.io/) storage.
26+
27+
Similar configurations are expected to work with other S3-compatible systems that also have the
28+
[STS](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) API.
29+
30+
# Setup
31+
32+
Start MinIO. Here's a `docker` example.
33+
34+
```shell
35+
docker run -p 9100:9000 -p 9101:9001 --name minio \
36+
-e "MINIO_ROOT_USER=miniouser" -e "MINIO_ROOT_PASSWORD=miniopwd" \
37+
quay.io/minio/minio:latest server /data --console-address ":9001"
38+
```
39+
40+
Create a bucket called "test" in MinIO UI or CLI.
41+
42+
Edit `~/.aws/credentials` and create a `minio` profile.
43+
44+
```
45+
[minio]
46+
aws_access_key_id = miniouser
47+
aws_secret_access_key = miniopwd
48+
region=us-west-2
49+
```
50+
51+
In the shell where the Polaris Server will be started `export AWS_PROFILE=minio`.
52+
53+
Build and run Polaris Server.
54+
55+
```shell
56+
./gradlew assemble
57+
58+
env POLARIS_BOOTSTRAP_CREDENTIALS=POLARIS,root,s3cr3t \
59+
java -jar runtime/server/build/quarkus-app/quarkus-run.jar
60+
```
61+
62+
Note: Apache Polaris 1.0.0-incubating does not yet have all the code required for proper operation with MinIO.
63+
The next release is expected to provide full support for S3-compatible storage with STS.
64+
65+
Create a catalog named "polaris".
66+
67+
```shell
68+
./polaris --client-id root --client-secret s3cr3t \
69+
catalogs create polaris --storage-type S3 \
70+
--default-base-location 's3://test' \
71+
--role-arn arn:aws:iam::123456789012:role/dummy \
72+
--region us-west-2 \
73+
--endpoint http://127.0.0.1:9100 \
74+
--path-style-access
75+
```
76+
77+
Note: the role and region parameters need to be set to avoid runtime errors in Polaris,
78+
but they will be ignored by MinIO.
79+
80+
For the sake of simplicity, grant `CATALOG_MANAGE_CONTENT` directly to the `catalog_admin` role
81+
using `polaris` CLI.
82+
83+
```shell
84+
./polaris --client-id root --client-secret s3cr3t \
85+
privileges catalog grant --catalog polaris \
86+
--catalog-role catalog_admin CATALOG_MANAGE_CONTENT
87+
```
88+
89+
# Connecting from Spark
90+
91+
Start Spark.
92+
93+
```shell
94+
bin/spark-sql \
95+
--packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.0,org.apache.iceberg:iceberg-aws-bundle:1.9.0 \
96+
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
97+
--conf spark.sql.catalog.polaris=org.apache.iceberg.spark.SparkCatalog \
98+
--conf spark.sql.catalog.polaris.type=rest \
99+
--conf spark.sql.catalog.polaris.uri=http://localhost:8181/api/catalog \
100+
--conf spark.sql.catalog.polaris.token-refresh-enabled=false \
101+
--conf spark.sql.catalog.polaris.warehouse=polaris \
102+
--conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
103+
--conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation=vended-credentials \
104+
--conf spark.sql.catalog.polaris.credential=root:s3cr3t
105+
```
106+
107+
Create a table in Spark.
108+
109+
```sql
110+
use polaris;
111+
create namespace ns;
112+
create table ns.t1 as select 'abc';
113+
select * from ns.t1;
114+
```

0 commit comments

Comments
 (0)