Skip to content

Commit 767be3d

Browse files
author
rokh-conduktor
authored
Merge pull request #101 from conduktor/revert-96-use-volume-to-persist-data
Revert "Use docker volumes to persist states"
2 parents 8a92f4d + e3029c9 commit 767be3d

6 files changed

+11
-194
lines changed

‎README.md

Lines changed: 1 addition & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -62,12 +62,6 @@ docker-compose -f zk-single-kafka-single.yml up
6262
docker-compose -f zk-single-kafka-single.yml down
6363
```
6464

65-
Clean state (zookeeper & kafka) with :
66-
```
67-
docker-compose -f zk-single-kafka-single.yml down
68-
docker-compose -f zk-single-kafka-single.yml rm
69-
```
70-
7165
## Single Zookeeper / Multiple Kafka
7266

7367
If you want to have three brokers and experiment with kafka replication / fault-tolerance.
@@ -82,12 +76,6 @@ docker-compose -f zk-single-kafka-multiple.yml up
8276
docker-compose -f zk-single-kafka-multiple.yml down
8377
```
8478

85-
Clean state (zookeeper & kafka) with :
86-
```
87-
docker-compose -f zk-single-kafka-multiple.yml down
88-
docker-compose -f zk-single-kafka-multiple.yml rm
89-
```
90-
9179
## Multiple Zookeeper / Single Kafka
9280

9381
If you want to have three zookeeper nodes and experiment with zookeeper fault-tolerance.
@@ -102,11 +90,6 @@ docker-compose -f zk-multiple-kafka-single.yml up
10290
docker-compose -f zk-multiple-kafka-single.yml down
10391
```
10492

105-
Clean state (zookeeper & kafka) with :
106-
```
107-
docker-compose -f zk-multiple-kafka-single.yml down
108-
docker-compose -f zk-multiple-kafka-single.yml rm
109-
```
11093

11194
## Multiple Zookeeper / Multiple Kafka
11295

@@ -121,12 +104,6 @@ docker-compose -f zk-multiple-kafka-multiple.yml up
121104
docker-compose -f zk-multiple-kafka-multiple.yml down
122105
```
123106

124-
Clean state (zookeeper & kafka) with :
125-
```
126-
docker-compose -f zk-multiple-kafka-multiple.yml down
127-
docker-compose -f zk-multiple-kafka-multiple.yml rm
128-
```
129-
130107
## Full stack
131108

132109
Need a UI? We recommend using [Conduktor](https://conduktor.io) as your tool to bring a unified UI to all these components
@@ -146,12 +123,6 @@ Need a UI? We recommend using [Conduktor](https://conduktor.io) as your tool to
146123
docker-compose -f full-stack.yml down
147124
```
148125

149-
Clean state (zookeeper & kafka) with :
150-
```
151-
docker-compose -f full-stack.yml down
152-
docker-compose -f full-stack.yml rm
153-
```
154-
155126
# FAQ
156127

157128
## Kafka
@@ -162,7 +133,7 @@ A: Add the following line to your docker-compose environment variables: `KAFKA_L
162133

163134
**Q: How do I delete data to start fresh?**
164135

165-
A: Your data is persisted from within docker volumes , so if you want for example to reset the data in the full-stack docker compose, do a `docker-compose -f full-stack.yml rm`.
136+
A: Your data is persisted from within the docker compose folder, so if you want for example to reset the data in the full-stack docker compose, do a `docker-compose -f full-stack.yml down`.
166137

167138
**Q: Can I change the zookeeper ports?**
168139

‎zk-multiple-kafka-multiple-schema-registry.yml

Lines changed: 3 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,5 @@
1-
version: '3.8'
2-
volumes:
3-
zkmultiple-kmultiple-sr_kafka1-data:
4-
zkmultiple-kmultiple-sr_kafka2-data:
5-
zkmultiple-kmultiple-sr_kafka3-data:
6-
zkmultiple-kmultiple-sr_zoo1-data:
7-
zkmultiple-kmultiple-sr_zoo2-data:
8-
zkmultiple-kmultiple-sr_zoo3-data:
9-
zkmultiple-kmultiple-sr_zoo1-log:
10-
zkmultiple-kmultiple-sr_zoo2-log:
11-
zkmultiple-kmultiple-sr_zoo3-log:
1+
version: '2.1'
2+
123
services:
134
zoo1:
145
image: confluentinc/cp-zookeeper:6.2.1
@@ -20,13 +11,6 @@ services:
2011
ZOOKEEPER_CLIENT_PORT: 2181
2112
ZOOKEEPER_SERVER_ID: 1
2213
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
23-
volumes:
24-
- type: volume
25-
source: zkmultiple-kmultiple-sr_zoo1-data
26-
target: /data
27-
- type: volume
28-
source: zkmultiple-kmultiple-sr_zoo1-log
29-
target: /datalog
3014

3115
zoo2:
3216
image: confluentinc/cp-zookeeper:6.2.1
@@ -38,13 +22,6 @@ services:
3822
ZOOKEEPER_CLIENT_PORT: 2182
3923
ZOOKEEPER_SERVER_ID: 2
4024
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
41-
volumes:
42-
- type: volume
43-
source: zkmultiple-kmultiple-sr_zoo2-data
44-
target: /data
45-
- type: volume
46-
source: zkmultiple-kmultiple-sr_zoo2-log
47-
target: /datalog
4825

4926
zoo3:
5027
image: confluentinc/cp-zookeeper:6.2.1
@@ -56,13 +33,7 @@ services:
5633
ZOOKEEPER_CLIENT_PORT: 2183
5734
ZOOKEEPER_SERVER_ID: 3
5835
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
59-
volumes:
60-
- type: volume
61-
source: zkmultiple-kmultiple-sr_zoo3-data
62-
target: /data
63-
- type: volume
64-
source: zkmultiple-kmultiple-sr_zoo3-log
65-
target: /datalog
36+
6637

6738
kafka1:
6839
image: confluentinc/cp-kafka:6.2.1
@@ -76,10 +47,6 @@ services:
7647
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
7748
KAFKA_BROKER_ID: 1
7849
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
79-
volumes:
80-
- type: volume
81-
source: zkmultiple-kmultiple-sr_kafka1-data
82-
target: /var/lib/kafka/data
8350
depends_on:
8451
- zoo1
8552
- zoo2
@@ -97,10 +64,6 @@ services:
9764
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
9865
KAFKA_BROKER_ID: 2
9966
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
100-
volumes:
101-
- type: volume
102-
source: zkmultiple-kmultiple-sr_kafka2-data
103-
target: /var/lib/kafka/data
10467
depends_on:
10568
- zoo1
10669
- zoo2
@@ -118,10 +81,6 @@ services:
11881
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
11982
KAFKA_BROKER_ID: 3
12083
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
121-
volumes:
122-
- type: volume
123-
source: zkmultiple-kmultiple-sr_kafka3-data
124-
target: /var/lib/kafka/data
12584
depends_on:
12685
- zoo1
12786
- zoo2

‎zk-multiple-kafka-multiple.yml

Lines changed: 4 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,5 @@
1-
version: '3.8'
2-
volumes:
3-
zkmultiple-kmultiple_kafka1-data:
4-
zkmultiple-kmultiple_kafka2-data:
5-
zkmultiple-kmultiple_kafka3-data:
6-
zkmultiple-kmultiple_zoo1-data:
7-
zkmultiple-kmultiple_zoo2-data:
8-
zkmultiple-kmultiple_zoo3-data:
9-
zkmultiple-kmultiple_zoo1-log:
10-
zkmultiple-kmultiple_zoo2-log:
11-
zkmultiple-kmultiple_zoo3-log:
1+
version: '2.1'
2+
123
services:
134
zoo1:
145
image: confluentinc/cp-zookeeper:6.2.1
@@ -20,13 +11,7 @@ services:
2011
ZOOKEEPER_CLIENT_PORT: 2181
2112
ZOOKEEPER_SERVER_ID: 1
2213
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
23-
volumes:
24-
- type: volume
25-
source: zkmultiple-kmultiple_zoo1-data
26-
target: /data
27-
- type: volume
28-
source: zkmultiple-kmultiple_zoo1-log
29-
target: /datalog
14+
3015
zoo2:
3116
image: confluentinc/cp-zookeeper:6.2.1
3217
hostname: zoo2
@@ -37,13 +22,6 @@ services:
3722
ZOOKEEPER_CLIENT_PORT: 2182
3823
ZOOKEEPER_SERVER_ID: 2
3924
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
40-
volumes:
41-
- type: volume
42-
source: zkmultiple-kmultiple_zoo2-data
43-
target: /data
44-
- type: volume
45-
source: zkmultiple-kmultiple_zoo2-log
46-
target: /datalog
4725

4826
zoo3:
4927
image: confluentinc/cp-zookeeper:6.2.1
@@ -55,13 +33,7 @@ services:
5533
ZOOKEEPER_CLIENT_PORT: 2183
5634
ZOOKEEPER_SERVER_ID: 3
5735
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
58-
volumes:
59-
- type: volume
60-
source: zkmultiple-kmultiple_zoo3-data
61-
target: /data
62-
- type: volume
63-
source: zkmultiple-kmultiple_zoo3-log
64-
target: /datalog
36+
6537

6638

6739
kafka1:
@@ -76,10 +48,6 @@ services:
7648
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
7749
KAFKA_BROKER_ID: 1
7850
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
79-
volumes:
80-
- type: volume
81-
source: zkmultiple-kmultiple_kafka1-data
82-
target: /var/lib/kafka/data
8351
depends_on:
8452
- zoo1
8553
- zoo2
@@ -97,10 +65,6 @@ services:
9765
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
9866
KAFKA_BROKER_ID: 2
9967
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
100-
volumes:
101-
- type: volume
102-
source: zkmultiple-kmultiple_kafka2-data
103-
target: /var/lib/kafka/data
10468
depends_on:
10569
- zoo1
10670
- zoo2
@@ -118,10 +82,6 @@ services:
11882
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
11983
KAFKA_BROKER_ID: 3
12084
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
121-
volumes:
122-
- type: volume
123-
source: zkmultiple-kmultiple_kafka3-data
124-
target: /var/lib/kafka/data
12585
depends_on:
12686
- zoo1
12787
- zoo2

‎zk-multiple-kafka-single.yml

Lines changed: 1 addition & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,4 @@
1-
version: '3.8'
2-
volumes:
3-
zkmultiple-ksingle_kafka1-data:
4-
zkmultiple-ksingle_zoo1-data:
5-
zkmultiple-ksingle_zoo2-data:
6-
zkmultiple-ksingle_zoo3-data:
7-
zkmultiple-ksingle_zoo1-log:
8-
zkmultiple-ksingle_zoo2-log:
9-
zkmultiple-ksingle_zoo3-log:
1+
version: '2.1'
102

113
services:
124
zoo1:
@@ -19,13 +11,6 @@ services:
1911
ZOOKEEPER_CLIENT_PORT: 2181
2012
ZOOKEEPER_SERVER_ID: 1
2113
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
22-
volumes:
23-
- type: volume
24-
source: zkmultiple-ksingle_zoo1-data
25-
target: /data
26-
- type: volume
27-
source: zkmultiple-ksingle_zoo1-log
28-
target: /datalog
2914

3015
zoo2:
3116
image: confluentinc/cp-zookeeper:6.2.1
@@ -37,13 +22,6 @@ services:
3722
ZOOKEEPER_CLIENT_PORT: 2182
3823
ZOOKEEPER_SERVER_ID: 2
3924
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
40-
volumes:
41-
- type: volume
42-
source: zkmultiple-ksingle_zoo2-data
43-
target: /data
44-
- type: volume
45-
source: zkmultiple-ksingle_zoo2-log
46-
target: /datalog
4725

4826
zoo3:
4927
image: confluentinc/cp-zookeeper:6.2.1
@@ -55,13 +33,6 @@ services:
5533
ZOOKEEPER_CLIENT_PORT: 2183
5634
ZOOKEEPER_SERVER_ID: 3
5735
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
58-
volumes:
59-
- type: volume
60-
source: zkmultiple-ksingle_zoo3-data
61-
target: /data
62-
- type: volume
63-
source: zkmultiple-ksingle_zoo3-log
64-
target: /datalog
6536

6637

6738
kafka1:
@@ -82,10 +53,6 @@ services:
8253
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
8354
KAFKA_JMX_PORT: 9999
8455
KAFKA_JMX_HOSTNAME: ${DOCKER_HOST_IP:-127.0.0.1}
85-
volumes:
86-
- type: volume
87-
source: zkmultiple-ksingle_kafka1-data
88-
target: /var/lib/kafka/data
8956
depends_on:
9057
- zoo1
9158
- zoo2

‎zk-single-kafka-multiple.yml

Lines changed: 1 addition & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,4 @@
1-
version: '3.8'
2-
volumes:
3-
sksingle-kmultiple_kafka1-data:
4-
sksingle-kmultiple_kafka2-data:
5-
sksingle-kmultiple_kafka3-data:
6-
sksingle-kmultiple_zoo1-data:
7-
sksingle-kmultiple_zoo1-log:
1+
version: '2.1'
82

93
services:
104
zoo1:
@@ -17,13 +11,6 @@ services:
1711
ZOOKEEPER_CLIENT_PORT: 2181
1812
ZOOKEEPER_SERVER_ID: 1
1913
ZOOKEEPER_SERVERS: zoo1:2888:3888
20-
volumes:
21-
- type: volume
22-
source: sksingle-kmultiple_zoo1-data
23-
target: /data
24-
- type: volume
25-
source: sksingle-kmultiple_zoo1-log
26-
target: /datalog
2714

2815

2916
kafka1:
@@ -38,10 +25,6 @@ services:
3825
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
3926
KAFKA_BROKER_ID: 1
4027
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
41-
volumes:
42-
- type: volume
43-
source: sksingle-kmultiple_kafka1-data
44-
target: /var/lib/kafka/data
4528
depends_on:
4629
- zoo1
4730

@@ -57,10 +40,6 @@ services:
5740
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
5841
KAFKA_BROKER_ID: 2
5942
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
60-
volumes:
61-
- type: volume
62-
source: sksingle-kmultiple_kafka2-data
63-
target: /var/lib/kafka/data
6443
depends_on:
6544
- zoo1
6645

@@ -77,9 +56,5 @@ services:
7756
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
7857
KAFKA_BROKER_ID: 3
7958
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
80-
volumes:
81-
- type: volume
82-
source: sksingle-kmultiple_kafka3-data
83-
target: /var/lib/kafka/data
8459
depends_on:
8560
- zoo1

0 commit comments

Comments
 (0)