RestateCluster: expose volumes, volumeMounts, and envFrom on spec.compute

[sevak-mnatsakanyan]

Goal

I am integrating the Kafka connector with Restate in a setup where RestateCluster is deployed via the Restate Operator on Kubernetes.

Our Kafka clusters require SASL/SSL authentication, with certificates provided via mounted file volumes.

Example configuration:

consumer = Consumer(
    {
        "bootstrap.servers": "XXXXXXXXXX",
        "security.protocol": "XXXXXXXXXX",
        "sasl.mechanisms": "XXXXXXXXXX",
        "sasl.username": "XXXXXXXXXX",
        "sasl.password": "XXXXXXXXXX",
        "group.id": "<Add-your-consumer-group-name",
        "ssl.ca.location": "/<Add-path-to-cert>/ca.crt",
        "schema.registry.url": "XXXXXXXXXX",
        "schema.registry.ssl.ca.location": "/<Add-path-to-cert>/server.pem",
        "schema.registry.ssl.certificate.location": "/<Add-path-to-cert>/client.pem",
        "schema.registry.ssl.key.location": "/<Add-path-to-cert>/key.pem"
    }
)

To support this, Restate needs access to certificate files via the filesystem, e.g.:
ssl.truststore.location=/path/to/kafka.truststore.p12

Problem

In Restate Operator, the spec.compute section of RestateCluster does not allow configuring:

• volumes
• volumeMounts
• envFrom

Currently, the only mechanism for mounting secrets as files in the CRD is: spec.security.requestSigningPrivateKey.secret[Provider].

However, this is:

• Scoped specifically to request signing
• Limited to a single key
• Not suitable for general-purpose use cases like Kafka SSL

Ask

Please expose standard Kubernetes pass-through fields on spec.compute:

• compute.volumes — []corev1.Volume
• compute.volumeMounts — []corev1.VolumeMount (merged into the Restate container)
• compute.envFrom — []corev1.EnvFromSource

[lukebond]

hi @sevak-mnatsakanyan, thanks for raising this!

just wanted to check if the recent trusted CA certs feature does what you want? see https://github.com/restatedev/restate-operator/releases/tag/v2.4.0

it doesn't sound like exactly the same thing, but wanted to double-check.

[sevak-mnatsakanyan]

Thanks a lot for this, it worked.

I also had to explicitly set the combined certificates folder so Kafka could properly bind them. Here’s the setup that worked for me:

apiVersion: restate.dev/v1
kind: RestateCluster
metadata:
  name: my-cluster
  namespace: restate
spec:
  cluster:
    autoProvision: true
  compute:
    image: xxxxx/restate:1.6.2
    env:
      - name: SERVERS
        valueFrom:
          secretKeyRef:
            name: my-secrets
            key: SERVERS
      - name: PROTOCOL
        valueFrom:
          secretKeyRef:
            name: my-secrets
            key: PROTOCOL
      - name: SASL_MECHANISM
        valueFrom:
          secretKeyRef:
            name: my-secrets
            key: SASL_MECHANISM
      - name: SASL_PASSWORD
        valueFrom:
          secretKeyRef:
            name: my-secrets
            key: SASL_PASSWORD
      # Kubernetes env composition
      - name: RESTATE_INGRESS__KAFKA_CLUSTERS
        value: >-
          [{name="kafka-cluster",
          brokers=["$(SERVERS)"],
          "security.protocol"="$(PROTOCOL)",
          "sasl.mechanism"="$(SASL_MECHANISM)",
          "sasl.username"="my-user.restate.v1",
          "sasl.password"="$(SASL_PASSWORD)",
          "ssl.ca.location"="/combined-certs/ca-certificates.crt"}]
  security:
    trustedCaCerts:
      - secretName: my-secrets
        key: truststore.pem

[lukebond]

so glad that it worked for you, and thanks for letting us know!

anything we could do to make it clearer or easier?

i wasn't sure what you meant by "I also had to explicitly set the combined certificates folder so Kafka could properly bind them"?

[sevak-mnatsakanyan]

Adding the certificate to the trusted CA certs, it appeared to me, is not enough for the Kafka library to read it.

There is a combined CA certs file generated and mounted inside the server when the restate-cluster is created, but for the Kafka library to locate and use it, the explicit configuration "ssl.ca.location"="/combined-certs/ca-certificates.crt" did the trick.

This path is extracted from the operator code and is internal to the restate operator. If you change it, my code might break.

[sevak-mnatsakanyan]

I am using a statically configured Kafka cluster on the Restate server, but with the new version, I am looking forward to using dynamic Kafka cluster registration.

While the trustedCaCerts field on the server helps, in the future this will become more of a Kafka cluster registration concern. If ssl.ca.location is really needed, then it would be cleaner to default it inside the server so that registrations don't need to pass any ca.location value.

[lukebond]

thanks for helping me understand that! i now see the problem. when i implemented the trusted certs feature i went down the route of changing the cert path because keeping it as it is meant a quite complicated solution. nevertheless, the downside of the implementation is that the rust library we use for kafka doesn't respect that envvar, forcing you to reverse-engineer the implementation and figure it out. sorry about that.

i need to switch the implementation over to one that doesn't change the location, then you wouldn't have to do this. keeping the changed-location implementation and automatically updating the kafka location would require a restate server change which i don't want.

thanks again for raising this.