Skip to content
This repository was archived by the owner on Feb 6, 2024. It is now read-only.

Commit b0c8eaa

Browse files
AntonFribergAntonFriberg
authored
Allow deployment of cluster with security hardening configuration (#90)
* Allow toogle between secure ports and unsecure Change-Id: I3aae090964f16951b94e8fabb7c38ef9696fee0d * Expose openSSL config Change-Id: If67c83e1060b73e2a1feaf4c59e233d0360b7970 * Expose ability to secure internal communication Add the ability to configure the cluster so that internal communication is encrypted and password-protected. Change-Id: If40cf4fe4075f870e725c73e7af7930f3c987dc9 * Add ability to set ZooKeeper ACL config ClickHouse has the ability to set ZooKeeper access control to the nodes it is using. This provides additional layer of security since it makes sure that the zookeeper nodes used by clickhouse is password protected and cannot be modified by anyone else. Change-Id: I28cf5ea07d1264befb3628fb38a372c7ce2b8552 * Fix connection on secure tcp port If user configures secure tcp port it would be best to check connection using the secure port. Also communicate with the database on the secure connection to make sure it works. Change-Id: I8efe34b5279d63438a290e9a3919d5afbef01800 * Add security hardening documentation to README.md Add security hardening documentation. Provides the ability to enforce better security practices for network encryption and secret/credential based access for zookeeper, data replication and distributed queries. Change-Id: I5dc416ee5c48123af27cf405adf7a877aa2e6204
1 parent 35741c6 commit b0c8eaa

File tree

8 files changed

+159
-33
lines changed

8 files changed

+159
-33
lines changed

‎README.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -300,6 +300,7 @@ Including an example of how to use your role (for instance, with variables passe
300300
roles:
301301
- ansible-clickhouse
302302
```
303+
303304
To generate macros: in file host_vars\db_host_1.yml
304305
```yaml
305306
clickhouse_macros:
@@ -308,6 +309,71 @@ clickhouse_macros:
308309
replica: "db_host_1"
309310
```
310311
312+
Security harden the cluster. You can configure the cluster with extra settings
313+
which enables
314+
- HTTPS port
315+
- TLS Encrypted TCP port
316+
- HTTPS for data replication
317+
- Credentials for data replication
318+
- Secret validation for distributed queries
319+
- ZooKeeper ACL
320+
```yaml
321+
- hosts: clickhouse_cluster
322+
become: true
323+
roles:
324+
- ansible-clickhouse
325+
vars:
326+
# HTTPS instead of normal HTTP
327+
clickhouse_https_port: 8443
328+
# TLS encryption for the native TCP protocol (needs `clickhouse-client --secure`)
329+
clickhouse_tcp_secure_port: 9440
330+
# TLS encryption between nodes in cluster
331+
clickhouse_interserver_https: 9010
332+
# Credentials used to authenticate nodes during data replication
333+
clickhouse_interserver_http_credentials:
334+
user: "internal"
335+
password: "supersecretstring"
336+
# Secret used to validate nodes in cluster for distributed queries
337+
clickhouse_distributed_secret: "supersecretstring2"
338+
# Password protect zookeeper paths used by ClickHouse
339+
clickhouse_zookeeper_identity:
340+
user: "zoo_user"
341+
password: "secretzoostring"
342+
# OpenSSL settings
343+
clickhouse_ssl_server:
344+
certificate_file: "/etc/clickhouse-server/server.crt"
345+
private_key_file: "/etc/clickhouse-server/server.key"
346+
dh_params_file: "/etc/clickhouse-server/dhparam.pem"
347+
verification_mode: "none"
348+
load_default_ca_file: "true"
349+
cache_sessions: "true"
350+
disable_protocols: "sslv2,sslv3"
351+
prefer_server_ciphers: "true"
352+
clickhouse_clusters:
353+
your_cluster_name:
354+
shard_1:
355+
- host: "db_host_1"
356+
port: 9440
357+
secure: true
358+
- host: "db_host_2"
359+
port: 9440
360+
secure: true
361+
shard_2:
362+
- host: "db_host_3"
363+
port: 9440
364+
secure: true
365+
- host: "db_host_4"
366+
port: 9440
367+
secure: true
368+
clickhouse_zookeeper_nodes:
369+
- host: "zoo_host_1"
370+
port: 2181
371+
- host: "zoo_host_2"
372+
port: 2181
373+
- host: "zoo_host_3"
374+
port: 2181
375+
```
376+
311377
F: You can call separately stages(from playbook, external role etc.):
312378
313379
Tag | Action

‎defaults/main.yml‎

Lines changed: 30 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,13 +34,40 @@ clickhouse_listen_host_custom: []
3434
clickhouse_listen_host: "{{ clickhouse_listen_host_default + clickhouse_listen_host_custom }}"
3535

3636
clickhouse_http_port: 8123
37-
3837
clickhouse_tcp_port: 9000
3938

40-
clickhouse_https_port: 8443
41-
clickhouse_tcp_secure_port: 9440
39+
# clickhouse_https_port: 8443
40+
# clickhouse_tcp_secure_port: 9440
41+
42+
clickhouse_ssl_server:
43+
certificate_file: "/etc/clickhouse-server/server.crt"
44+
private_key_file: "/etc/clickhouse-server/server.key"
45+
dh_params_file: "/etc/clickhouse-server/dhparam.pem"
46+
verification_mode: "none"
47+
load_default_ca_file: "true"
48+
cache_sessions: "true"
49+
disable_protocols: "sslv2,sslv3"
50+
prefer_server_ciphers: "true"
51+
52+
clickhouse_ssl_client:
53+
load_default_ca_file: "true"
54+
cache_sessions: "true"
55+
disable_protocols: "sslv2,sslv3"
56+
prefer_server_ciphers: "true"
57+
invalid_certificate_handler_name: "RejectCertificateHandler"
4258

4359
clickhouse_interserver_http: 9009
60+
# clickhouse_interserver_https: 9010
61+
62+
# clickhouse_interserver_http_credentials:
63+
# user: "internal"
64+
# password: "secretstring"
65+
66+
# clickhouse_distributed_secret: "secretstring"
67+
68+
# clickhouse_zookeeper_identity:
69+
# user: "zoo-user"
70+
# password: "secretstring"
4471

4572
clickhouse_networks_default:
4673
- "::1"

‎tasks/configure/db.yml‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,16 @@
11
---
2+
- name: Set ClickHose Connection String
3+
set_fact: clickhouse_connection_string="clickhouse-client -h 127.0.0.1 --port {{ clickhouse_tcp_secure_port | default(clickhouse_tcp_port) }}{{' --secure' if clickhouse_tcp_secure_port is defined else '' }}"
4+
25
- name: Gather list of existing databases
3-
command: "clickhouse-client -h 127.0.0.1 --port {{ clickhouse_tcp_port }} -q 'show databases'"
6+
command: "{{ clickhouse_connection_string }} -q 'show databases'"
47
changed_when: False
58
register: existing_databases
69
tags: [config_db]
710

811
- name: Config | Delete database config
912
command: |
10-
clickhouse-client -h 127.0.0.1 --port {{ clickhouse_tcp_port }}
13+
{{ clickhouse_connection_string }}
1114
-q 'DROP DATABASE IF EXISTS `{{ item.name }}`
1215
{% if item.cluster is defined %}ON CLUSTER `{{ item.cluster }}`{% endif %}'
1316
with_items: "{{ clickhouse_dbs }}"
@@ -16,7 +19,7 @@
1619

1720
- name: Config | Create database config
1821
command: |
19-
clickhouse-client -h 127.0.0.1 --port {{ clickhouse_tcp_port }}
22+
{{ clickhouse_connection_string }}
2023
-q 'CREATE DATABASE IF NOT EXISTS `{{ item.name }}`
2124
{% if item.cluster is defined %}ON CLUSTER `{{ item.cluster }}`{% endif %}
2225
{% if item.engine is defined %}ENGINE = {{ item.engine }}{% endif %}'

‎tasks/configure/sys.yml‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -51,8 +51,6 @@
5151
mode: "ug=rw,o-rwx"
5252
become: true
5353

54-
55-
5654
- name: Config | Generate remote_servers config
5755
template:
5856
src: remote_servers.j2
@@ -84,4 +82,12 @@
8482
mode: "u=rw,og=r"
8583
notify: restart-ch
8684
become: true
87-
when: clickhouse_zookeeper_nodes is defined
85+
when: clickhouse_zookeeper_nodes is defined
86+
87+
- name: Config | Fix interserver_http_port and intersever_https_port collision
88+
lineinfile:
89+
path: "{{ clickhouse_path_configdir }}/config.xml"
90+
search_string: '<interserver_http_port>'
91+
state: absent
92+
become: true
93+
when: clickhouse_interserver_https is defined

‎tasks/main.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@
4444

4545
- name: "Wait for Clickhouse Server to Become Ready"
4646
wait_for:
47-
port: "{{ clickhouse_tcp_port }}"
47+
port: "{{ clickhouse_tcp_secure_port | default(clickhouse_tcp_port) }}"
4848
delay: "{{ clickhouse_ready_delay }}"
4949
when: not clickhouse_remove|bool
5050

‎templates/config.j2‎

Lines changed: 33 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -11,39 +11,42 @@
1111
<count>{{ clickhouse_logger.count }}</count>
1212
</logger>
1313

14+
{% if clickhouse_https_port is defined %}
15+
<https_port>{{ clickhouse_https_port }}</https_port>
16+
{% else %}
1417
<http_port>{{ clickhouse_http_port }}</http_port>
15-
<tcp_port>{{ clickhouse_tcp_port }}</tcp_port>
18+
{% endif %}
1619

17-
<!-- For HTTPS and SSL over native protocol. -->
18-
<!--
19-
<https_port>{{ clickhouse_https_port }}</https_port>
20+
{% if clickhouse_tcp_secure_port is defined %}
2021
<tcp_port_secure>{{ clickhouse_tcp_secure_port }}</tcp_port_secure>
21-
-->
22+
{% else %}
23+
<tcp_port>{{ clickhouse_tcp_port }}</tcp_port>
24+
{% endif %}
2225

2326
<!-- Used with https_port and tcp_port_secure. Full ssl options list: https://github.com/ClickHouse-Extras/poco/blob/master/NetSSL_OpenSSL/include/Poco/Net/SSLManager.h#L71 -->
2427
<openSSL>
2528
<server> <!-- Used for https server AND secure tcp port -->
2629
<!-- openssl req -subj "/CN=localhost" -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout /etc/clickhouse-server/server.key -out /etc/clickhouse-server/server.crt -->
27-
<certificateFile>/etc/clickhouse-server/server.crt</certificateFile>
28-
<privateKeyFile>/etc/clickhouse-server/server.key</privateKeyFile>
30+
<certificateFile>{{ clickhouse_ssl_server.certificate_file }}</certificateFile>
31+
<privateKeyFile>{{ clickhouse_ssl_server.private_key_file }}</privateKeyFile>
2932
<!-- openssl dhparam -out /etc/clickhouse-server/dhparam.pem 4096 -->
30-
<dhParamsFile>/etc/clickhouse-server/dhparam.pem</dhParamsFile>
31-
<verificationMode>none</verificationMode>
32-
<loadDefaultCAFile>true</loadDefaultCAFile>
33-
<cacheSessions>true</cacheSessions>
34-
<disableProtocols>sslv2,sslv3</disableProtocols>
35-
<preferServerCiphers>true</preferServerCiphers>
33+
<dhParamsFile>{{ clickhouse_ssl_server.dh_params_file }}</dhParamsFile>
34+
<verificationMode>{{ clickhouse_ssl_server.verification_mode }}</verificationMode>
35+
<loadDefaultCAFile>{{ clickhouse_ssl_server.load_default_ca_file }}</loadDefaultCAFile>
36+
<cacheSessions>{{ clickhouse_ssl_server.cache_sessions }}</cacheSessions>
37+
<disableProtocols>{{ clickhouse_ssl_server.disable_protocols }}</disableProtocols>
38+
<preferServerCiphers>{{ clickhouse_ssl_server.prefer_server_ciphers }}</preferServerCiphers>
3639
</server>
3740

3841
<client> <!-- Used for connecting to https dictionary source -->
39-
<loadDefaultCAFile>true</loadDefaultCAFile>
40-
<cacheSessions>true</cacheSessions>
41-
<disableProtocols>sslv2,sslv3</disableProtocols>
42-
<preferServerCiphers>true</preferServerCiphers>
42+
<loadDefaultCAFile>{{ clickhouse_ssl_client.load_default_ca_file }}</loadDefaultCAFile>
43+
<cacheSessions>{{ clickhouse_ssl_client.cache_sessions }}</cacheSessions>
44+
<disableProtocols>{{ clickhouse_ssl_client.disable_protocols }}</disableProtocols>
45+
<preferServerCiphers>{{ clickhouse_ssl_client.prefer_server_ciphers }}</preferServerCiphers>
4346
<!-- Use for self-signed: <verificationMode>none</verificationMode> -->
4447
<invalidCertificateHandler>
4548
<!-- Use for self-signed: <name>AcceptCertificateHandler</name> -->
46-
<name>RejectCertificateHandler</name>
49+
<name>{{ clickhouse_ssl_client.invalid_certificate_handler_name }}</name>
4750
</invalidCertificateHandler>
4851
</client>
4952
</openSSL>
@@ -54,7 +57,19 @@
5457
-->
5558

5659
<!-- Port for communication between replicas. Used for data exchange. -->
60+
{% if clickhouse_interserver_https is defined %}
61+
<interserver_https_port>{{ clickhouse_interserver_https }}</interserver_https_port>
62+
{% else %}
5763
<interserver_http_port>{{ clickhouse_interserver_http }}</interserver_http_port>
64+
{% endif %}
65+
66+
{% if clickhouse_interserver_http_credentials is defined %}
67+
<interserver_http_credentials>
68+
<user>{{ clickhouse_interserver_http_credentials.user }}</user>
69+
<password>{{ clickhouse_interserver_http_credentials.password }}</password>
70+
</interserver_http_credentials>
71+
{% endif %}
72+
5873

5974
<!-- Hostname that is used by other replicas to request this server.
6075
If not specified, than it is determined analoguous to 'hostname -f' command.

‎templates/remote_servers.j2‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,20 @@
44
<remote_servers>
55
{% for clusters_name, shards_name in clickhouse_clusters.items() | list %}
66
<{{ clusters_name }}>
7+
{% if clickhouse_distributed_secret is defined %}
8+
<secret>{{ clickhouse_distributed_secret }}</secret>
9+
{% endif %}
710
{% for shard_name, replicas in shards_name.items() %}
811
<shard>
9-
<internal_replication>true</internal_replication>
12+
<internal_replication>true</internal_replication>
1013
{% for replica in replicas %}
11-
<replica>
12-
<host>{{ replica['host'] }}</host>
13-
<port>{{ replica['port'] | default(9000) }}</port>
14-
</replica>
14+
<replica>
15+
<host>{{ replica['host'] }}</host>
16+
<port>{{ replica['port'] | default(9000) }}</port>
17+
{% if 'secure' in replica %}
18+
<secure>1</secure>
19+
{% endif %}
20+
</replica>
1521
{% endfor %}
1622
</shard>
1723
{% endfor %}

‎templates/zookeeper-servers.j2‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22
{{ ansible_managed | comment('xml') }}
33
<clickhouse>
44
<zookeeper>
5+
{% if clickhouse_zookeeper_identity is defined %}
6+
<identity>{{ clickhouse_zookeeper_identity.user }}:{{clickhouse_zookeeper_identity.password }}</identity>
7+
{% endif %}
58
{% for server in clickhouse_zookeeper_nodes %}
69
<node index="{{loop.index}}">
710
{% for key, value in server.items() %}

0 commit comments

Comments
 (0)