Skip to content

Commit 29fe525

Browse files
dbw9580dbw9580
committed
DRILL-7745: Add storage plugin for IPFS
1 parent 0726b83 commit 29fe525

36 files changed

+3403
-15
lines changed

‎common/src/test/java/org/apache/drill/categories/IPFSStorageTest.java‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing, software
13+
* distributed under the License is distributed on an "AS IS" BASIS,
14+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
* See the License for the specific language governing permissions and
16+
* limitations under the License.
17+
*/
18+
19+
20+
package org.apache.drill.categories;
21+
22+
/**
23+
* This is a category used to mark unit tests that test the IPFS storage plugin.
24+
*/
25+
public interface IPFSStorageTest {
26+
}
27+

‎contrib/native/client/src/protobuf/UserBitShared.pb.cc‎

Lines changed: 8 additions & 7 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎contrib/native/client/src/protobuf/UserBitShared.pb.h‎

Lines changed: 3 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎contrib/pom.xml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@
5656
<module>storage-kudu</module>
5757
<module>storage-opentsdb</module>
5858
<module>storage-http</module>
59+
<module>storage-ipfs</module>
5960
<module>storage-druid</module>
6061
</modules>
6162

‎contrib/storage-ipfs/README.md‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Drill Storage Plugin for IPFS (Minerva)
2+
3+
## Contents
4+
5+
0. [Introduction](#Introduction)
6+
1. [Configuration](#Configuration)
7+
2. [Usage Notes](#Usage Notes)
8+
9+
## Introduction
10+
11+
Minerva is a storage plugin of Drill that connects IPFS's decentralized storage and Drill's flexible query engine. Any data file stored on IPFS can be easily accessed from Drill's query interface, just like a file stored on a local disk. Moreover, with Drill's capability of distributed execution, other instances who are also running Minerva can help accelerate the execution: the data stays where it is, and the queries go to the most suitable nodes which stores the data locally, and from there the operations can be performed most efficiently.
12+
13+
## Configuration
14+
15+
1. Set Drill hostname to the IP address of the node to run Drill:
16+
17+
Edit file `conf/drill-env.sh` and change the environment variable `DRILL_HOST_NAME` to the IP address of the node. Use private or global addresses, depending on whether you plan to run it in a private cluster or on the open Internet.
18+
19+
2. Configure the IPFS storage plugin:
20+
21+
The default configuration of the IPFS storage plugin is located at `src/resources/bootstrap-storage-plugins.json`:
22+
23+
```
24+
"ipfs" : {
25+
"type":"ipfs",
26+
"host": "127.0.0.1",
27+
"port": 5001,
28+
"max-nodes-per-leaf": 3,
29+
"ipfs-timeouts": {
30+
"find-provider": 4,
31+
"find-peer-info": 4,
32+
"fetch-data": 5
33+
},
34+
"ipfs-caches": {
35+
"peer": {"size": 100, "ttl": 600},
36+
"provider": {"size": 1000, "ttl": 600}
37+
},
38+
"groupscan-worker-threads": 50,
39+
"formats": null,
40+
"enabled": true
41+
}
42+
```
43+
44+
where
45+
46+
`host` and `port` are the host and API port where your IPFS daemon will be listening. Change it so that it matches the configuration of your IPFS instance.
47+
48+
`max-nodes-per-leaf` controls how many provider nodes will be considered when the query is being planned. A larger value increases the parallelization width but typically takes longer to find enough providers from DHT resolution. A smaller value does the opposite.
49+
50+
`ipfs-timeouts` set the maximum amount of time in seconds for various time-consuming operations:
51+
52+
* `find-provider` is the time allowed to do DHT queries to find providers.
53+
* `find-peer-info` is the time allowed to resolve the network addresses of the providers.
54+
* `fetch-data` is the time the actual transmission is allowed to take.
55+
56+
`ipfs-caches` control the size and TTL in seconds of cache entries of various caches used to accelerate query execution:
57+
58+
* `peer` cache caches peers addresses.
59+
* `provider` cache caches which providers provide a particular IPFS object.
60+
61+
`groupscan-worker-threads` limits the number of worker threads when the planner communicate with the IPFS daemon to resolve providers and peer info.
62+
63+
`formats` specifies the formats of the files. It is unimplemented for now and does nothing.
64+
65+
3. Configure IPFS
66+
67+
Start the IPFS daemon first.
68+
69+
Set a Drill-ready flag to the node:
70+
71+
```
72+
$ IPFS_NULL_OBJECT=$(ipfs object new)
73+
$ ipfs object patch add-link $IPFS_NULL_OBJECT "drill-ready" $IPFS_NULL_OBJECT
74+
QmeXLv7D5uV2uXHejg7St7mSXDtqwTw8LuywSBawSxy5iA
75+
$ ipfs name publish /ipfs/QmeXLv7D5uV2uXHejg7St7mSXDtqwTw8LuywSBawSxy5iA
76+
Published to <your-node-id>: /ipfs/QmeXLv7D5uV2uXHejg7St7mSXDtqwTw8LuywSBawSxy5iA
77+
```
78+
79+
This flag indicates that an IPFS node is also capable of handling Drill queries, and the planner will consider it when scheduling a query to execute distributedly. A node without this flag will be ignored.
80+
81+
Also, pin the flag so that it will stick on your node:
82+
83+
```
84+
$ ipfs pin add -r QmeXLv7D5uV2uXHejg7St7mSXDtqwTw8LuywSBawSxy5iA
85+
```
86+
87+
## Usage Notes
88+
89+
1. Compatible data formats
90+
91+
Currently only JSON files are supported by this storage plugin.
92+
93+
2. Add datasets to IPFS
94+
95+
IPFS provides the `ipfs add` command to conveniently add a file to IPFS. Unfortunately that command does not split data files into chunks on line boundaries. Use [this script](https://gist.github.com/dbw9580/250e52a54e39a34083f815dea34a89e0) to do proper chunking and add files to IPFS.
96+
97+
3. Timeout exceptions
98+
99+
IPFS operations can be time-consuming, and sometimes an operation can take forever (e.g. querying the DHT for a non-existent object). Adjust the timeout values in the config to avoid most timeout exceptions.

‎contrib/storage-ipfs/pom.xml‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
4+
Licensed to the Apache Software Foundation (ASF) under one
5+
or more contributor license agreements. See the NOTICE file
6+
distributed with this work for additional information
7+
regarding copyright ownership. The ASF licenses this file
8+
to you under the Apache License, Version 2.0 (the
9+
"License"); you may not use this file except in compliance
10+
with the License. You may obtain a copy of the License at
11+
12+
http://www.apache.org/licenses/LICENSE-2.0
13+
14+
Unless required by applicable law or agreed to in writing, software
15+
distributed under the License is distributed on an "AS IS" BASIS,
16+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
17+
See the License for the specific language governing permissions and
18+
limitations under the License.
19+
20+
-->
21+
<project xmlns="http://maven.apache.org/POM/4.0.0"
22+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
23+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
24+
<parent>
25+
<artifactId>drill-contrib-parent</artifactId>
26+
<groupId>org.apache.drill.contrib</groupId>
27+
<version>1.18.0-SNAPSHOT</version>
28+
</parent>
29+
<modelVersion>4.0.0</modelVersion>
30+
31+
<artifactId>drill-ipfs-storage</artifactId>
32+
<name>contrib/ipfs-storage-plugin</name>
33+
34+
<properties>
35+
<ipfs.TestSuite>**/IPFSTestSuit.class</ipfs.TestSuite>
36+
</properties>
37+
38+
<dependencies>
39+
<dependency>
40+
<groupId>org.apache.drill.exec</groupId>
41+
<artifactId>drill-java-exec</artifactId>
42+
<version>${project.version}</version>
43+
</dependency>
44+
45+
<dependency>
46+
<groupId>com.github.ipfs</groupId>
47+
<artifactId>java-ipfs-http-client</artifactId>
48+
<version>v1.3.3</version>
49+
</dependency>
50+
51+
<!-- Test dependencies -->
52+
<dependency>
53+
<groupId>org.apache.drill.exec</groupId>
54+
<artifactId>drill-java-exec</artifactId>
55+
<classifier>tests</classifier>
56+
<version>${project.version}</version>
57+
<scope>test</scope>
58+
</dependency>
59+
60+
<dependency>
61+
<groupId>org.apache.drill</groupId>
62+
<artifactId>drill-common</artifactId>
63+
<classifier>tests</classifier>
64+
<version>${project.version}</version>
65+
<scope>test</scope>
66+
</dependency>
67+
68+
</dependencies>
69+
70+
<build>
71+
<plugins>
72+
<plugin>
73+
<groupId>org.apache.maven.plugins</groupId>
74+
<artifactId>maven-surefire-plugin</artifactId>
75+
<configuration>
76+
<forkCount>0</forkCount>
77+
<includes>
78+
<include>${ipfs.TestSuite}</include>
79+
</includes>
80+
<excludes>
81+
<exclude>**/TestIPFSQueries.java</exclude>
82+
</excludes>
83+
</configuration>
84+
</plugin>
85+
</plugins>
86+
</build>
87+
</project>

0 commit comments

Comments
 (0)