OpenShift Secured Integration

This repository explores some of the integrations with credentials and certificates providers.

1. OpenShift Secured Integration
   1. 1. Introduction
   2. 2. Cert manager
      1. 2.1. Installation and configuration
      2. 2.2. Debugging cert-manager
      3. 2.3. Useful Links
   3. 3. HashiCorp Vault
      1. 3.1. Installation and access
      2. 3.2. Initializing Vault Secrets
      3. 3.3. Useful Links
   4. 4. Handling secrets on OpenShift
   5. 5. Vault Sidecar Agent Injector
      1. 6.1. Installation and configuration
      2. 6.2. Useful Links
      3. 6.3. ⚖️ Pros and Cons of Vault Sidecar Agent Injector
   6. 6. Secrets Store CSI Driver
      1. 6.1. Installation and configuration
      2. 6.2. Useful Links
      3. 6.3. ⚖️ Pros and Cons of the Secrets Store CSI Driver
   7. 7. Vault Secrets Operator (VSO)
      1. 7.1. Installation and configuration
      2. 7.2. Useful Links
      3. 7.3. ⚖️ Pros and Cons of the Vault Secrets Operator
   8. 8. External Secrets Operator (ESO)
      1. 8.1. Installation and configuration
      2. 8.2. Useful Links
      3. 8.3. ⚖️ Pros and Cons of the External Secrets Operator
   9. 9. ArgoCD Vault Plugin
      1. 9.1. Installation and configuration
      2. 9.2. Useful Links
      3. 9.3. ⚖️ Pros and Cons of the ArgoCD Vault Plugin
   10. Extra: Encrypting etcd data
       1. Testing encryption configuration

1. Introduction

This repository demonstrates how to securely integrate an OpenShift cluster with HashiCorp Vault for managing secrets and credentials, and cert-manager for automating certificate creation and renewal. You'll find deployment guides, step-by-step instructions, and example configurations to streamline your setup. Dive in to simplify and secure your cluster's secrets and certificates management!

2. Cert manager

cert-manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources. It will ensure certificates are valid and up to date periodically, and attempt to renew certificates at an appropriate time before expiry.

2.1. Installation and configuration

In order to split installation of the operator and post-configuration, I've split cert-manager resources in three different ArgoCD applications:

1. cert-manager-operator will install the operator as well as create aws credentials using the Cluster Credentials Operator to allow the operator to perform DNS requests to validate the URIs of the certificates. It also configures metrics retrieval so that you can configure its metrics (Prefixed by certmanager_).
2. cert-manager-route53 application will create the actual Certificate Issuer and certificates for the API server and the Ingress and apply them to the cluster.
3. cert-manager-self-signed is just a configuration example on how to create a self-signed certificate and use it to issue certificates for your own services. This does not have a real use-case, it is just a demonstration.

Do you want to deploy it in your cluster without ArgoCD? You can copy the following piece of code and execute it in your cluster:

# 1) Deploy the operator
oc apply -k 02-cert-manager-operator
# 2) Wait for the operator to be ready
echo -n "Waiting for cert-manager pods to be ready..."
while [[ $(oc get pods -l app.kubernetes.io/instance=cert-manager -n cert-manager -o 'jsonpath={..status.conditions[?(@.type=="Ready")].status}') != "True True True" ]]; do echo -n "." && sleep 1; done; echo -n -e "  [OK]\n"
# 3) Configure the OpenShift certificates for Ingress and API
helm template 02-cert-manager-route53 --set clusterDomain=$(oc get dns.config/cluster -o jsonpath='{.spec.baseDomain}')  | oc apply -f -
# 4) Configure custom certificates using self-signed
oc apply -k 02-cert-manager-self-signed

2.2. Debugging cert-manager

cert-manager is a simple operator, but if you are just starting to play with it, you might need some guidance to better understand what is going on. Here you can find some debugging tips:

• There should be three pods in the cert-manager namespace. If not, check the subscription status field.
• Among the pods, the cert-manager will contain the validation information, but the cert-manager-webhook will show you if cert-manager is not trusting your issuer CA.
• The Certificate resource is the main object you use to request certificates, but if you want to see the issuing status, check the certificaterequest or, more importantly, the order resource.
• If you request a cert in a secret $NAME, there is an intermediate step in which it will create an auxiliary secret named $SECRET-suffix that will then be deleted after proper issuing. Just relax and wait 😄

oc get certificate,certificaterequest,order -A

Check the certificate used in the connections:

echo Q | openssl s_client -connect $(oc get route console -n openshift-console --template="{{.spec.host}}"):443 -showcerts 2>/dev/null | openssl x509 -noout -subject -issuer -enddate

2.3. Useful Links

• Docs: cert-manager Operator for Red Hat OpenShift.
• Blog: YAUB - SSL Certificate Management for OpenShift on AWS.
• Blog: YAUB - Managing Certificates using GitOps approach.
• Blog: Automatic certificate issuing with IdM and cert-manager operator for OpenShift.
• Blog: Let's Encrypt - Challenge Types: This is a summary of how challenges work for certificates.
• Git: Cert Manager Mixin is a collection of reusable and configurable Prometheus alerts, and a Grafana dashboard to help with operating cert-manager.

3. HashiCorp Vault

HashiCorp Vault is a secrets management tool that integrates seamlessly with OpenShift (OCP) to securely store and manage sensitive information like API keys, credentials, and certificates. It enables dynamic secrets generation and automated secret renewal, reducing manual overhead and improving security. When combined with OCP, Vault ensures that applications running in your cluster can securely access secrets with fine-grained access control, enhancing the overall security posture of your workloads.

3.1. Installation and access

To install HashiCorp Vault on OpenShift, the recommended mechanism is to deploy it with the Helm Chart. For that reason, I've created the following application with the simplest configuration to deploy on OpenShift and automatically create a Route in dev mode:

cat application-03-hashicorp-vault-server.yaml | \
    CLUSTER_DOMAIN=$(oc get dns.config/cluster -o jsonpath='{.spec.baseDomain}') \
    envsubst | oc apply -f -

In order to access the deployed Vault server, just retrieve the route using the following command and access the UI using the token root:

oc get route hashicorp-vault-server -n vault --template="https://{{.spec.host}}"

3.2. Initializing Vault Secrets

As the dev Vault instance is an in-memory instance, just by deleting the vault-0 pod, you will loose all the data stored in the Vault. For that reason, I have created a script to quickly set up the Secret store with some dummy data to consume from the applications. The only thing that you need is log in to the cluster and execute the following script:

./create_vault_secrets.sh

Now, you can access the HashiCorp Vault, to the secret/ Engine and you will see the demo1, demo2, and demo3 entries.

3.3. Useful Links

• Git: GitHub - vault-helm. Official repo of the HashiCorp Vault Helm repo.
• Blog: In-Depth HashiCorp Vault setup on OpenShift using OpenShift GitOps: This is a must read. It is an updated blog on how to customize the HashiCorp Vault deployment.
• HashiCorp Tutorial: Install Vault to Red Hat OpenShift

4. Handling secrets on OpenShift

Managing secrets securely is a cornerstone of modern application security. OpenShift, while offering built-in support for Kubernetes secrets, benefits greatly from the advanced features and dynamic nature of HashiCorp Vault. This section explores multiple approaches to integrate Vault with OCP, evaluating their benefits, drawbacks, and integration with ArgoCD.

Here are some common methods for using secrets from HashiCorp Vault in OpenShift environments:

Tool	Description	Advantages	Disadvantages
HashiCorp Vault API	Direct integration with applications	Dynamic secrets, centralized management	Needs integration in application code, dependency on Vault availability.
Vault Sidecar Agent Injector	Injects secrets into pods via a sidecar	Secrets not stored in Kubernetes Secrets.	Requires sidecar for each pod. Only works for pods.
Secrets Store CSI Driver	Mounts secrets as volumes in pods	Simplifies secret consumption, works across multiple backends	Secrets not dynamically refreshed, requires additional driver. Only works for pods.
Vault Secrets Operator (VSO)	Syncs Vault secrets to Kubernetes Secrets	Kubernetes-native integration, automates secret updates	Secrets stored in Kubernetes Secrets.
External Secrets Operator (ESO)	Syncs secrets from external providers	Multi-provider support, GitOps-friendly	Secrets persisted in Kubernetes Secrets, sync delays possible.
ArgoCD Vault Plugin	Fetches secrets during manifest rendering	Tight GitOps integration, supports encrypted secrets	Adds complexity to the pipeline setup. Plugins are not in RH support.

Tip

The first method just implies using app framework libraries or pre-exec scripts to retrieve the Secrets from the HashiCorp Vault manually and add them to the application. For that reason, we are not going to explore that possibility.

Nice blogs that I've used to compile this comparison:

• Verifa - Comparing methods for accessing secrets in HashiCorp Vault from Kubernetes

5. Vault Sidecar Agent Injector

The Vault Agent Injector alters pod specifications to include Vault Agent containers that render Vault secrets to a shared memory volume using Vault Agent Templates. By rendering secrets to a shared volume, containers within the pod can consume Vault secrets without being Vault aware.

The injector is a Kubernetes Mutation Webhook Controller. The controller intercepts pod events and applies mutations to the pod if annotations exist within the request. This functionality is provided by the vault-k8s project and can be automatically installed and configured using the Vault Helm chart.

6.1. Installation and configuration

oc apply -f application-05-vault-sidecar-agent-injector.yaml

Warning

If this application is not rendered correctly in your ArgoCD, consider adding the following flag to your kustomize configuration: --enable-helm.

or deploy it locally with the following command kustomize build 05-vault-sidecar-agent-injector/ --enable-helm | oc apply -f -.

6.2. Useful Links

• Docs: HashiCorp - Vault Agent Injector.
• Tutorial: HashiCorp - Mange secrets by injecting a Vault Agent container.

6.3. ⚖️ Pros and Cons of Vault Sidecar Agent Injector

✅ Pros

• 🔧 Easy configuration based on several annotations.
• 🔑 Authentication based on ServiceAccount Tokens.

❌ Cons

• 🛠️ Requires modifying application deployment configurations.
• 🔒 Secrets can only be injected into containers, not OpenShift configuration.
• 📂 Secrets can only be injected as files, not environment variables.
• ❌ Not supported by Red Hat.
• ⚡ High resources consumption as each pod with secret needs a permanent sidecar container.

💡 Other Considerations

• 🚀 Installation does not require an operator.

6. Secrets Store CSI Driver

The Secrets Store CSI Driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container’s file system.

The following secrets store providers are available for use with the Secrets Store CSI Driver Operator:

• AWS Secrets Manager.
• AWS Systems Manager Parameter Store.
• Azure Key Vault.
• Google Secret Manager.
• HashiCorp Vault.

6.1. Installation and configuration

oc apply -f application-06-secrets-store-csi-driver.yaml

or deploy it locally with the following command kustomize build 06-secrets-store-csi-driver/ --enable-helm | oc apply -f -.

6.2. Useful Links

• Docs: OpenShift - Installing Secrets CSI.
• Docs: OpenShift - Providing sensitive data to pods by using an external secrets store.
• Docs: Kubernetes Secrets Store CSI Driver.
• Blog: OpenShift Secrets Store CSI Driver with Vault.

6.3. ⚖️ Pros and Cons of the Secrets Store CSI Driver

✅ Pros

• 🔑 Authentication based on ServiceAccount Tokens.
• 📂 Secrets can be injected as files and environment variables as well.

❌ Cons

• 🔧 More complex configuration. Requires Operator, and several k8s resources.
• 🛠️ Requires specific application deployment configuration related to the CSI Driver.
• 🔒 Secrets can only be injected into containers, not OpenShift configuration.
• 📲 The Secrets Store CSI Driver operator is Tech Preview in OpenShift 4.17.

💡 Other Considerations

• If you try to get the k8s secret, you'll see that it doesn't exist until it is requested by a pod.
• If you plan to consume your secret data as Kubernetes Secrets only, then other solutions like External Secrets Operator may be a better fit.

7. Vault Secrets Operator (VSO)

The Vault Secrets Operator allows Pods to consume Vault secrets and HCP Vault Secrets Apps natively from Kubernetes Secrets. The Operator writes the source Vault secret data directly to the destination Kubernetes Secret, ensuring that any changes made to the source are replicated to the destination over its lifetime.

Which are the main features?

• VaultStaticSecret: Synchronize a single Vault static Secret to a single Kubernetes Secret.
• VaultPKISecret: Synchronize a single Vault PKI Secret. The PKI secrets engine generates dynamic X.509 certificates. With this secrets engine, services can get certificates without going through the usual manual process
• VaultDynamicSecret: Use secrets engines like databases, aws, azure, gcp, etc. to generate a single Vault dynamic Secret and sync it to a single Kubernetes Secret. Use Vault dynamic secrets to create secrets when accessed and to invalidate the secrets after use.

Tip

Vault's database secrets engine provides a credential management solution so that the username and password can be dynamically generated upon request, and you can control the lifecycle of the credentials:

7.1. Installation and configuration

oc apply -f application-07-vault-secrets-operator.yaml

Do you want to check the Certificate content?

oc -n test-secrets get secret vso-pki -o jsonpath='{.data.certificate}' | base64 -d

Do you want to check the Certificate validity?

oc -n test-secrets exec -it vso-pki-cert-checker -- openssl x509 -in /etc/pki/certificate -noout -text | grep -E '(Subject|Issuer|Not Before|Not After ):'
        Issuer: CN=my-website.com
            Not Before: Mar 30 10:21:27 2025 GMT
            Not After : Mar 30 10:22:57 2025 GMT
        Subject: CN=www.my-website.com

Do you want to check the Dynamic Feature?

Then, you need to install the MySQL DB and you can use the following chart as example:

helm upgrade --install mysql oci://registry-1.docker.io/bitnamicharts/mysql \
   --namespace vault \
   --version 12.3.2 \
   --values 07-vault-secrets-operator/mysql-values.yaml

7.2. Useful Links

• Blog: Vault Secrets Operator for Kubernetes now GA.
• Git: learn-vault-secrets-operator
• Tutorial: Manage Kubernetes native secrets with the Vault Secrets Operator.

7.3. ⚖️ Pros and Cons of the Vault Secrets Operator

✅ Pros

• The VSO operator is a certified operator by HashiCorp.
• Integration with Vault Dynamic Secrets.
• It allows to create secrets without a pod using it, so can be used for OpenShift configuration.
• Simplified configuration compared to the previous methods.
• It supports natively Static and Dynamic Secrets.

❌ Cons

• Documentation is poor and specially when VSO is installed using OLM.
• It looks like having smaller penetration in the market compared to other alternatives.

8. External Secrets Operator (ESO)

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, CyberArk Conjur, Pulumi ESC and many more. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret.

Note

On december 2024, release 0.11.0, the ESO team stopped providing updates via OLM. The reason behind it is due to the coupling nature with our helm charts, which takes precedence over OLM as our first class release mechanism. We recommend OLM users to switch to plain helm chart installs as opposed to keep using OLM helm operator. Yo can check the official statement here and here.

8.1. Installation and configuration

This operator can be installed using the following ArgoCD application, that instantiates the official Helm chart:

oc apply -f application-08-external-secrets-operator.yaml

or deploy it locally with the following command:

helm repo add external-secrets https://charts.external-secrets.io

helm upgrade -i --create-namespace -n external-secrets external-secrets external-secrets/external-secrets  --set "installCRDs=true"

8.2. Useful Links

• External Secrets Main site.
• ESO - HashiCorp Vault Provider.
• ESO - GitHub repository.
• Blog: External Secrets with HashiCorp Vault.

8.3. ⚖️ Pros and Cons of the External Secrets Operator

✅ Pros

• It allows much more interaction types with the Secret Vault, like PushSecrets, to push secrets to Vault.
• Lot's of integrations with many Secret providers.

❌ Cons

• The provider only supports Static Secrets. For dynamic secrets you need an ESO generator.
• The OLM operator has been deprecated.

💡 Other Considerations

• Red Hat Tech Preview support is targeted for OpenShift Plus 4.19. See the Jira here.
• As of December 2024, the way to install it is by a Helm Chart supported by the ESO team.
• Only ExternalSecret, SecretStore, ClusterExternalSecret and ClusterSecretStore have v1beta1 support.

9. ArgoCD Vault Plugin

The Argo CD plugin retrieves secrets from various Secret Management tools (HashiCorp Vault, IBM Cloud Secrets Manager, AWS Secrets Manager, etc.) and inject them into Kubernetes resources.

9.1. Installation and configuration

This configuration requires deploying an instance of ArgoCD and customizing it. As I don't want to bloat the configuration with unnecessary elements, I will keep it minimal and focused on essential features. Meanwhile, we can deploy a new customized instance with the following command:

oc apply -k 09-argocd-vault-plugin-deployment

After the new instance is deployed, we can create a new ArgoCD application with a secret in that new application:

oc apply -f application-09-avp-example.yaml

9.2. Useful Links

• Git repo: argocd-vault-plugin.
• Docs: ArgoCD Vault Plugin.
• Blog: How to Use HashiCorp Vault and Argo CD for GitOps on OpenShift.
• GitHub: jtudelag - ArgoCD Vault Plugin (AVP).

9.3. ⚖️ Pros and Cons of the ArgoCD Vault Plugin

✅ Pros

• No need of extra deployments, operators, or deployments.

❌ Cons

• Changes behavior of ArgoCD App, as normal Refresh button does not recalculate secrets. Needs Hard Refresh.
• Tight couples the CD tool and the Secrets retrieval tool.
• It adds three new containers to the ArgoCD server pod.
• Not possible to evaluate the secrets without deploying using GitOps.
• Not possible to segregate authentication with different users in the same ArgoCD instance.

💡 Other Considerations

• It's not supported to use kubernetes authentication against HashiCorp Vault using ArgoCD deployed using the OLM operator, as it does not allow setting a custom ServiceAccount or mounting the SA token.

Extra: Encrypting etcd data

By default, etcd data is not encrypted in OpenShift Container Platform. You can enable etcd encryption for your cluster to provide an additional layer of data security. For example, it can help protect the loss of sensitive data if an etcd backup is exposed to the incorrect parties. When you enable etcd encryption, the following OpenShift API server and Kubernetes API server resources are encrypted:

• Secrets
• Config maps
• Routes
• OAuth access tokens
• OAuth authorize tokens

To enable etcd encryption, you would set etcdEncryption.enabled: true in your values file or pass it as a parameter when installing/upgrading the Helm chart. If you deployed the cert-manager following the steps of the previous

# 3) Configure the OpenShift certificates for Ingress and API
helm template 02-cert-manager-route53 --set clusterDomain=$(oc get dns.config/cluster -o jsonpath='{.spec.baseDomain}') --set etcdEncryption.enable="true" | oc apply -f -

Testing encryption configuration

Great! We configured that flag on the APIServer resource, but I really need to make sure that encryption is enabled on the node. That's fine! Use the following commands to check the encryption configuration for each component:

1. Review the Encrypted status condition for the OpenShift API server to verify that its resources were successfully encrypted:

oc get openshiftapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'
EncryptionCompleted
All resources encrypted: routes.route.openshift.io

2. Review the Encrypted status condition for the Kubernetes API server to verify that its resources were successfully encrypted:

$ oc get kubeapiserver -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'
EncryptionCompleted
All resources encrypted: secrets, configmaps

3. Review the Encrypted status condition for the OpenShift OAuth API server to verify that its resources were successfully encrypted:

$ oc get authentication.operator.openshift.io -o=jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}'
EncryptionCompleted
All resources encrypted: oauthaccesstokens.oauth.openshift.io, oauthauthorizetokens.oauth.openshift.io