Cookie not writeable when readOnlyRootFilesystem: true #243
Description
Background
In order to be able to install vernemq on a production grade security enhanced Kubernetes Cluster with Pod Security Policies enabled, VerneMQ should run with the least privileges possible.
Especially when VerneMQ is public available which might be case in most cases when it comes down to IoT Devices.
Setup:
As as a security aware Developer/Operation Engineer you want a default PSP like:
restricted-psp false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs true configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
When one installs now vernemq:
helm install vernemq vernemq/vernemq
Expected Behaviour
Everything should come up and:
kubectl exec vernemq-0 /vernemq/bin/vmq-admin cluster show
should print everything is fine.
Actual Behaviour
=ERROR REPORT==== 28-Sep-2020::13:17:14.103869 ===
Failed to create cookie file '/vernemq/.erlang.cookie': erofs
=SUPERVISOR REPORT==== 28-Sep-2020::13:17:14.104081 ===
supervisor: {local,net_sup}
errorContext: start_error
reason: {"Failed to create cookie file '/vernemq/.erlang.cookie': erofs",
[{auth,init_cookie,0,[{file,"auth.erl"},{line,290}]},
{auth,init,1,[{file,"auth.erl"},{line,144}]},
{gen_server,init_it,2,[{file,"gen_server.erl"},{line,417}]},
{gen_server,init_it,6,[{file,"gen_server.erl"},{line,385}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,226}]}]}
offender: [{pid,undefined},
{id,auth},
{mfargs,{auth,start_link,[]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]
=CRASH REPORT==== 28-Sep-2020::13:17:14.104357 ===
crasher:
initial call: auth:init/1
pid: <0.82.0>
registered_name: []
exception error: "Failed to create cookie file '/vernemq/.erlang.cookie': erofs"
in function auth:init_cookie/0 (auth.erl, line 290)
in call from auth:init/1 (auth.erl, line 144)
in call from gen_server:init_it/2 (gen_server.erl, line 417)
in call from gen_server:init_it/6 (gen_server.erl, line 385)
ancestors: [net_sup,kernel_sup,<0.47.0>]
message_queue_len: 0
messages: []
links: [<0.80.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 987
stack_size: 28
reductions: 2934
neighbours:
escript: exception error: no match of right hand side value
{error,
{{shutdown,
{failed_to_start_child,auth,
{"Failed to create cookie file '/vernemq/.erlang.cookie': erofs",
[{auth,init_cookie,0,
[{file,"auth.erl"},{line,290}]},
{auth,init,1,[{file,"auth.erl"},{line,144}]},
{gen_server,init_it,2,
[{file,"gen_server.erl"},{line,417}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,385}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,226}]}]}}},
{child,undefined,net_sup_dynamic,
{erl_distribution,start_link,
[['[email protected]',
longnames],
false,net_sup_dynamic]},
permanent,1000,supervisor,
[erl_distribution]}}}
command terminated with exit code 127
Proposal
In order to fix that, data generated or changed during Runtime of a Container should not be placed into the same folder as the static compiled/backed stuff.
One possible Solution could be moving the Erlang Cookie into the /vernemq/data directory.
Another one would be moving it into an own Volume and add that at least to the Helm Chart as emptyDir.