diff --git a/k8s/k8s-workload-registrar/README.md b/k8s/k8s-workload-registrar/README.md new file mode 100644 index 0000000..6b90fcf --- /dev/null +++ b/k8s/k8s-workload-registrar/README.md @@ -0,0 +1,696 @@ +# Configure SPIRE to use the Kubernetes Workload Registrar + This tutorial builds on the [Kubernetes Quickstart Tutorial](../quickstart/) to provide an example of how to configure the SPIRE Kubernetes Workload Registrar as a container within the SPIRE Server pod. The registrar enables automatic workload registration and management in SPIRE Kubernetes implementations. The changes required to deploy the registrar and the necessary files are shown as a delta to the quickstart tutorial, so it is highly encouraged to execute, or at least read through, the Kubernetes Quickstart Tutorial first. + +This tutorial demonstrates how to use the registrar's three different modes: + + * Webhook - For historical reasons, the webhook mode is the default but reconcile and CRD modes are now preferred because webhook can create StatefulSets and pods with no entries and cause other cleanup and scalability issues. + * Reconcile - The reconcile mode uses reconciling controllers rather than webhooks. It may be slightly faster to create new entries than CRD mode and requires less configuration. + * CRD - The CRD mode provides a namespaced SpiffeID custom resource and is best for cases where you plan to manage SpiffeID custom resources directly. + +For more information, see the [Differences between modes](https://github.com/spiffe/spire/tree/master/support/k8s/k8s-workload-registrar#differences-between-modes) section of the registrar README. + +In this document you will learn how to: + * Deploy the K8s Workload Registrar as a container within the SPIRE Server Pod + * Configure the three workload registration modes + * Use the three workload registration modes + * Test successful registration entries creation + +See the SPIRE Kubernetes Workload Registrar [README](https://github.com/spiffe/spire/tree/master/support/k8s/k8s-workload-registrar) for complete configuration options. + + # Prerequisites + Before proceeding, review the following list: + * You'll need access to the Kubernetes environment configured when going through the [Kubernetes Quickstart Tutorial](../quickstart/). + * Required configuration files for this tutorial can be found in the `k8s/k8s-workload-registrar` directory in [https://github.com/spiffe/spire-tutorials](https://github.com/spiffe/spire-tutorials). If you didn't already clone the repo for the _Kubernetes Quickstart Tutorial_, please do so now. + * The steps in this document should work with Kubernetes version 1.20.2. + +We will deploy a scenario that consists of a StatefulSet containing a SPIRE Server and the Kubernetes Workload Registrar, a SPIRE Agent, and a workload, and configure the different modes to illustrate automatic registration entries creation. + +# Common configuration: socket setup + +Socket configuration is necessary in all three registrar modes. + +The SPIRE Server and the Kubernetes Workload registrar will communicate to each other using a socket mounted at the `/tmp/spire-server/private` directory, as we can see from the `volumeMounts` section of both containers. The only difference between these sections is that, for the registrar, the socket will have the `readOnly` option set to `true`, while for the SPIRE Server container it will have its value set to `false`. For example, here is the registrar container's `volumeMounts` section from `spire-server.yaml`: +``` +volumeMounts: +- name: spire-server-socket + mountPath: /tmp/spire-server/private + readOnly: true +``` + +Continue with the registrar mode that you want to try out: +* [Webhook](#configure-webhook-mode) +* [Reconcile](#configure-reconcile-mode) +* [CRD](#configure-crd-mode) + +# Configure Webhook mode + +This section describes the older, default webhook mode of the Kubernetes Workload Registrar. We will review the important files needed to configure it. + +This mode makes use of the `ValidatingWebhookConfiguration` feature from Kubernetes, which is called by the Kubernetes API server every time a new pod is created or deleted in the cluster, as we can see from the rules of the resource below: + +``` +ApiVersion: admissionregistration.k8s.io/v1beta1 +kind: ValidatingWebhookConfiguration +metadata: + name: k8s-workload-registrar-webhook +webhooks: + - name: k8s-workload-registrar.spire.svc + clientConfig: + service: + name: k8s-workload-registrar + namespace: spire + path: "/validate" + caBundle: ... + admissionReviewVersions: + - v1beta1 + rules: + - apiGroups: [""] + apiVersions: ["v1"] + operations: ["CREATE", "DELETE"] + resources: ["pods"] + scope: "Namespaced" +``` + +The webhook authenticates the API server, and for this reason we provide a CA bundle with the `caBundle` option, as we can see in the stanza above (value omitted for brevity). This authentication must be done to ensure that it is the API server that is contacting the webhook, because this situation will lead to registration entry creation or deletion on the SPIRE Server, something that is a key point in the SPIRE infrastructure and should be strongly secured. + +Also, a secret is volume mounted in the `/run/spire/k8s-workload-registrar/secret` directory inside the SPIRE Server container. This secret contains the K8s Workload Registrar server key. We can see this in the `volumeMounts` section of the SPIRE Server statefulset configuration file: + +``` +- name: k8s-workload-registrar-secret + mountPath: /run/spire/k8s-workload-registrar/secret + readOnly: true +``` + +The secret itself is named `k8s-workload-registrar-secret` and is shown below: + +``` +apiVersion: v1 +kind: Secret +metadata: + name: k8s-workload-registrar-secret + namespace: spire +type: Opaque +data: + server-key.pem: ... +``` + +Again, the value of the key is omitted. + +Another setting that is relevant in this mode is the registrar certificate's `ConfigMap`, that contains the K8s Workload Registrar server certificate and CA bundle used to verify the client certificate presented by the API server. This is mounted in the `/run/spire/k8s-workload-registrar/certs` directory. This is defined in the `volumeMounts` section of the SPIRE Server statefulset configuration file, which is shown below: + +``` +- name: k8s-workload-registrar-certs + mountPath: /run/spire/k8s-workload-registrar/certs + readOnly: true +``` + +These certificates are stored in a `ConfigMap`: + +``` +apiVersion: v1 +kind: ConfigMap +metadata: + name: k8s-workload-registrar-certs + namespace: spire +data: + server-cert.pem: | + -----BEGIN CERTIFICATE----- + + ... + + -----END CERTIFICATE----- + + cacert.pem: | + -----BEGIN CERTIFICATE----- + + ... + + -----END CERTIFICATE----- +``` + +With all of this set, we can look at at the registrar's container configuration: + +``` +apiVersion: v1 +kind: ConfigMap +metadata: + name: k8s-workload-registrar + namespace: spire +data: + k8s-workload-registrar.conf: | + trust_domain = "example.org" + server_socket_path = "/tmp/spire-server/private/api.sock" + cluster = "demo-cluster" + mode = "webhook" + cert_path = "/run/spire/k8s-workload-registrar/certs/server-cert.pem" + key_path = "/run/spire/k8s-workload-registrar/secret/server-key.pem" + cacert_path = "/run/spire/k8s-workload-registrar/certs/cacert.pem" +``` + +As we can see, the `key_path` points to where the secret containing the server key is mounted, which was shown earlier. The `cert_path` and `cacert_path` entries point to the directory where the `ConfigMap` with the PEM-encoded certificates for the server and for the CA are mounted. When the webhook is triggered, the registrar acts as the server and validates the identity of the client, which is the Kubernetes API server in this case. We can disable this authentication by setting the ```insecure_skip_client_verification``` option to `true` (though it is not recommended). + +For authentication, a `KubeConfig` file with the client certificate and the key the API server should use to authenticate with the registrar is mounted inside the filesystem of the Kubernetes node. This file is shown below: + +``` +apiVersion: v1 +kind: Config +users: +- name: k8s-workload-registrar.spire.svc + user: + client-certificate-data: ... + client-key-data: ... +``` + +An `AdmissionConfiguration` is mounted inside the node too, and it describes where the API server can locate the file containing the `KubeConfig` entry used in the authentication process. + +``` +apiVersion: apiserver.k8s.io/v1alpha1 +kind: AdmissionConfiguration +plugins: +- name: ValidatingAdmissionWebhook + configuration: + apiVersion: apiserver.config.k8s.io/v1alpha1 + kind: WebhookAdmission + kubeConfigFile: /var/lib/minikube/certs/admctrl/kubeconfig.yaml +``` + +To mount the two files into the node, and as we are using the docker driver to start minikube, we will use the `docker cp` directive. Once the files are placed into the node's filesystem, we use the `apiserver.admission-control-config-file` extra flag to specify the location of the admission control configuration file, which will be put in `/var/lib/minikube/certs/admctrl/admission-control.yaml`. + +## Run the registrar in webhook mode + +We have looked at the key points of the webhook mode's configuration, so let's apply the necessary files to enable our scenario with a SPIRE Server with the registrar container in it, an Agent, and a workload by issuing the following command in the `mode-webhook` directory: + +```console +$ bash scripts/deploy-scenario.sh +``` + +This is all we need to have the registration entries created on the server. We will run the server command to see the registration entries created, by executing the command below: + +```console +$ kubectl exec statefulset/spire-server -n spire -c spire-server -- bin/spire-server entry show -registrationUDSPath /tmp/spire-server/private/api.sock +``` + +You should see the following three registration entries, corresponding to the node, the agent, and the workload (the order of the results may differ in your output). + +```console +Found 3 entries +Entry ID : ... +SPIFFE ID : spiffe://example.org/k8s-workload-registrar/demo-cluster/node +Parent ID : spiffe://example.org/spire/server +Revision : 0 +TTL : default +Selector : k8s_psat:cluster:demo-cluster + +Entry ID : ... +SPIFFE ID : spiffe://example.org/ns/spire/sa/spire-agent +Parent ID : spiffe://example.org/k8s-workload-registrar/demo-cluster/node +Revision : 0 +TTL : default +Selector : k8s:ns:spire +Selector : k8s:pod-name:spire-agent-wtx7b + +Entry ID : ... +SPIFFE ID : spiffe://example.org/ns/spire/sa/default +Parent ID : spiffe://example.org/k8s-workload-registrar/demo-cluster/node +Revision : 0 +TTL : default +Selector : k8s:ns:spire +Selector : k8s:pod-name:example-workload-6877cd47d5-2fmpq +``` + +We omitted the entry IDs, as those may change with every run. Let's see how the other fields are built: + +The cluster name *demo-cluster* is used in the Parent ID field for the entries that correspond to the agent and the workload (second and third, respectively), but there is no reference to the node that these pods belong to, this is, the registration entries are mapped to a single node entry inside the cluster. This represents a drawback for this mode, as all the nodes in the cluster have permission to get identities for all the workloads that belong to the Kubernetes cluster, which increases the blast radius in case of a node being compromised, among other disadvantages. + +Taking a look at the assigned SPIFFE IDs for the agent and the workload, we can see that they have the following form: +*spiffe://\/ns/\/sa/\*. +From this, we can conclude that we are using the registrar configured with the Service Account Based workload registration (which is the default behaviour). For instance, as the workload uses the *default* service account, into the *spire* namespace, its SPIFFE ID is: *spiffe://example.org/ns/spire/sa/default* + +Another thing that is worthwhile to examine is the registrar log to find out if the entries were created by this container. Run the following command to display lines in the log that match *Created pod entry*. + +```console +$ kubectl logs statefulset/spire-server -n spire -c k8s-workload-registrar | grep "Created pod entry" +``` + +The output of this command includes three lines, one for every entry created. We can conclude that the three entries that were present on the SPIRE Server were created by the registrar. They correspond to the node, agent, and workload, in that specific order. + +## Pod deletion + +Let's see how the registrar handles a pod deletion, and the impact it has on the registration entries. Run the following command to delete the workload deployment: + +```console +$ kubectl delete deployment/example-workload -n spire +``` + +Again, check for the registration entries with the command below: + +```console +$ kubectl exec statefulset/spire-server -n spire -c spire-server -- bin/spire-server entry show -registrationUDSPath /tmp/spire-server/private/api.sock +``` + +The output of the command will not include the registration entry that corresponds to the workload, because the pod was deleted, and should be similar to: + +```console +Found 2 entries +Entry ID : ... +SPIFFE ID : spiffe://example.org/k8s-workload-registrar/demo-cluster/node +Parent ID : spiffe://example.org/spire/server +Revision : 0 +TTL : default +Selector : k8s_psat:cluster:demo-cluster + +Entry ID : ... +SPIFFE ID : spiffe://example.org/ns/spire/sa/spire-agent +Parent ID : spiffe://example.org/k8s-workload-registrar/demo-cluster/node +Revision : 0 +TTL : default +Selector : k8s:ns:spire +Selector : k8s:pod-name:spire-agent-wtx7b +``` + +We will check the registrar logs to find out if it deleted the entry, looking for the "Deleting pod entries" keyword, with the command shown below: + +```console +$ kubectl logs statefulset/spire-server -n spire -c k8s-workload-registrar | grep "Deleting pod entries" +``` + +The registrar successfully deleted the corresponding entry for the *example-workload* pod. + +## Teardown + +To delete the resources used for this mode, we'll issue the `delete-scenario.sh` script: + +```console +$ bash scripts/delete-scenario.sh +``` + +# Configure reconcile mode + +This mode, as opposed to webhook mode, does not use a validating webhook but two reconciling controllers instead: one for the nodes and one for the pods. For this reason, it isn't necessary to configure Kubernetes API server authentication with secrets and the `KubeConfig` entry, making the configuration much simpler. + +The registrar's container configuration is: + +``` +apiVersion: v1 +kind: ConfigMap +metadata: + name: k8s-workload-registrar + namespace: spire +data: + k8s-workload-registrar.conf: | + trust_domain = "example.org" + server_socket_path = "/tmp/spire-server/private/api.sock" + cluster = "demo-cluster" + mode = "reconcile" + pod_label = "spire-workload" + metrics_addr = "0" +``` + +We are explicitly indicating that *reconcile* mode is used. For the sake of the tutorial, we will be using Label Based workload registration for this mode (as we can see from the `pod_label` configurable), though every workload registration mode can be used with every registrar mode. This is all the configuration that is needed to have the containers working properly. + +## Run the registrar in reconcile mode + +We will deploy the same scenario as the previous mode, with the difference in the agent and workload pods: they will be labeled with the *spire-workload* label that corresponds to the value indicated in the `pod_label` option of the `ConfigMap` shown above. Ensure that your working directory is `mode-reconcile` and run the following command to start the scenario: + +```console +$ bash scripts/deploy-scenario.sh +``` + +With the reconcile scenario running, we will check the registration entries and some special considerations for this mode. Let's issue the command below to show the existing registration entries. + +```console +$ kubectl exec statefulset/spire-server -n spire -c spire-server -- bin/spire-server entry show -registrationUDSPath /tmp/spire-server/private/api.sock +``` + +Your output should similar to the following, and shows the entries for the node, the agent and the workload: + +```console +Found 3 entries +Entry ID : ... +SPIFFE ID : spiffe://example.org/spire-k8s-registrar/demo-cluster/node/minikube +Parent ID : spiffe://example.org/spire/server +Revision : 0 +TTL : default +Selector : k8s_psat:agent_node_name:minikube +Selector : k8s_psat:cluster:demo-cluster + +Entry ID : ... +SPIFFE ID : spiffe://example.org/agent +Parent ID : spiffe://example.org/spire-k8s-registrar/demo-cluster/node/minikube +Revision : 0 +TTL : default +Selector : k8s:ns:spire +Selector : k8s:pod-name:spire-agent-c5c5f + +Entry ID : ... +SPIFFE ID : spiffe://example.org/example-workload +Parent ID : spiffe://example.org/spire-k8s-registrar/demo-cluster/node/minikube +Revision : 0 +TTL : default +Selector : k8s:ns:spire +Selector : k8s:pod-name:example-workload-b98cc787d-kzxz6 +``` + +If we compare these entries to those created using webhook mode, the difference is that the Parent ID of the agent and workload registration entries (second and third, respectively) contains a reference to the node where the pods are scheduled on, in this case, using its name `minikube`. We mentioned that this doesn't happen using the webhook mode, and this was one of the principal drawbacks of that mode. Also, the pod name and namespace are used in the selectors. For the node registration entry (the one that has the SPIRE Server SPIFFE ID as the Parent ID), the node name is used in the selectors, along with the cluster name. + +As we are using Label workload registration mode, the SPIFFE IDs for the agent and the workload (which are labeled as we mentioned before) have the form: *spiffe://\/\*. For example, as the agent has the label value equal to `agent`, it has the following SPIFFE ID: *spiffe://example.org/agent*. + +Let's check if the registrar indeed created the registration entries by checking its logs, and looking for the *Created new spire entry* keyword. Run the command that is shown below: + +```console +$ kubectl logs statefulset/spire-server -n spire -c k8s-workload-registrar | grep "controllers.*Created new spire entry" +``` + +We mentioned before that there were two reconciling controllers, and from the output of the command above, we can see that the node controller created the entry for the single node in the cluster, and that the pod controller created the entries for the two labeled pods: agent and workload. + +## Pod deletion + +The Kubernetes Workload Registrar automatically handles the creation and deletion of registration entries. We just saw how the entries are created, and now we will test deletion. Let's delete the workload deployment: + +```console +$ kubectl delete deployment/example-workload -n spire +``` + +We will check if its corresponding entry is deleted too. Run the following command to see the registration entries on the SPIRE Server: + +```console +$ kubectl exec statefulset/spire-server -n spire -c spire-server -- bin/spire-server entry show -registrationUDSPath /tmp/spire-server/private/api.sock +``` + +The output will only show two registration entries, because the workload entry was deleted by the registrar: + +```console +Found 2 entries +Entry ID : ... +SPIFFE ID : spiffe://example.org/agent +Parent ID : spiffe://example.org/spire-k8s-registrar/demo-cluster/node/minikube +Revision : 0 +TTL : default +Selector : k8s:ns:spire +Selector : k8s:pod-name:spire-agent-c5c5f + +Entry ID : ... +SPIFFE ID : spiffe://example.org/spire-k8s-registrar/demo-cluster/node/minikube +Parent ID : spiffe://example.org/spire/server +Revision : 0 +TTL : default +Selector : k8s_psat:agent_node_name:minikube +Selector : k8s_psat:cluster:demo-cluster +``` + +If we look for the *Deleted entry* keyword on the registrar logs, we will find out that the registrar deleted the entry. Issue the following command: + +```console +$ kubectl logs statefulset/spire-server -n spire -c k8s-workload-registrar | grep "controllers.*Deleted entry" +``` + +The pod controller successfully deleted the entry. + +## Non-labeled pods + +As we are using Label Based workload registration, only pods that have the *spire-workload* label will have their registration entries automatically created. Let's deploy a pod that has no label by executing the command below from the `mode-reconcile` directory: + +```console +$ kubectl apply -f k8s/not-labeled-workload.yaml +``` + +Let's see the existing registration entries with the command: + +```console +$ kubectl exec statefulset/spire-server -n spire -c spire-server -- bin/spire-server entry show -registrationUDSPath /tmp/spire-server/private/api.sock +``` + +The output should be the same as the output that we obtained in the *Pod deletion* section. This implies that the registrar only creates entries for pods that are using the matching label. + +## Teardown + +To delete the resources used for this mode, issue the `delete-scenario.sh` script: + +```console +$ bash scripts/delete-scenario.sh +``` + +# Configure CRD mode + +This mode takes advantage of the `CustomResourceDefinition` feature from Kubernetes, which allows SPIRE to integrate with this tool and its control plane. A SPIFFE ID is defined as a custom resource, with a structure that matches the form of a registration entry. Below is a simplified example of the definition of a SPIFFE ID CRD. + +``` +apiVersion: spiffeid.spiffe.io/v1beta1 +kind: SpiffeID +metadata: + name: my-test-spiffeid + namespace: default +spec: + parentId: spiffe://example.org/spire/server + selector: + namespace: default + podName: my-test-pod + spiffeId: spiffe://example.org/test +``` + +The main goal of the custom resource is to track the intent of what and how the registration entries should look on the SPIRE Server by keeping these resources in sync with any modification made to the registration entries. This means that every SPIFFE ID CRD will have a matching registration entry whose existence will be closely linked. Every modification done to a registration entry will have an impact on its corresponding SPIFFE ID CRD, and vice versa. + +The `ConfigMap` for the registrar below shows that we will be using the *crd* mode, and that Annotation Based workload registration is used along with it. The annotation that the registrar will look for is *spiffe.io/spiffe-id*. + +``` +apiVersion: v1 +kind: ConfigMap +metadata: + name: k8s-workload-registrar + namespace: spire +data: + k8s-workload-registrar.conf: | + trust_domain = "example.org" + server_socket_path = "/tmp/spire-server/private/api.sock" + cluster = "demo-cluster" + mode = "crd" + pod_annotation = "spiffe.io/spiffe-id" + metrics_bind_addr = "0" +``` + +## Run the registrar in CRD mode + +Let's deploy the necessary files, including the base scenario plus the SPIFFE ID CRD definition, and examine the automatically created registration entries. Ensure that your working directory is `mode-crd`, and run: + +```console +$ bash scripts/deploy-scenario.sh +``` + +Run the entry show command by executing: + +```console +$ kubectl exec statefulset/spire-server -n spire -c spire-server -- bin/spire-server entry show -registrationUDSPath /tmp/spire-server/private/api.sock +``` + +The output should show the following registration entries: + +```console +Found 3 entries +Entry ID : ... +SPIFFE ID : spiffe://example.org/k8s-workload-registrar/demo-cluster/node/minikube +Parent ID : spiffe://example.org/spire/server +Revision : 1 +TTL : default +Selector : k8s_psat:agent_node_uid:08990bfd-3551-4761-8a1b-2e652984ffdd +Selector : k8s_psat:cluster:demo-cluster + +Entry ID : ... +SPIFFE ID : spiffe://example.org/testing/agent +Parent ID : spiffe://example.org/k8s-workload-registrar/demo-cluster/node/minikube +Revision : 1 +TTL : default +Selector : k8s:node-name:minikube +Selector : k8s:ns:spire +Selector : k8s:pod-uid:538886bb-48e1-4795-b386-10e97f50e34f +DNS name : spire-agent-jzc8w + +Entry ID : ... +SPIFFE ID : spiffe://example.org/testing/example-workload +Parent ID : spiffe://example.org/k8s-workload-registrar/demo-cluster/node/minikube +Revision : 1 +TTL : default +Selector : k8s:node-name:minikube +Selector : k8s:ns:spire +Selector : k8s:pod-uid:78ed3fc5-4cff-476a-90f5-37d3abd47823 +DNS name : example-workload-6877cd47d5-l4hv5 +``` + +Three entries were created corresponding to the node, agent, and workload. For the node entry (the one that has the SPIRE Server SPIFFE ID as Parent ID), we see a difference in the selectors compared to reconcile mode: instead of using the node name, CRD mode stores the UID of the node where the agent is running on, and as the node name is used in the SPIFFE ID, we can take this as a mapping from node UID to node name. + +Something similar happens with the pod entries, but this time the pod UID where the workload is running is stored in the selectors instead of the node UID. + +If we now focus our attention on the SPIFFE IDs assigned to the workloads, we see that it takes the form of *spiffe://\/\*. By using Annotation Based workload registration, it is possible to freely set the SPIFFE ID path. In this case, for the workload, we set the annotation value to *example-workload*. + +Obtain the registrar logs by issuing: + +```console +$ kubectl logs statefulset/spire-server -n spire -c k8s-workload-registrar | grep "Created entry" +``` + +This will show that the registrar created the three entries in the SPIRE Server. + +In addition to the SPIRE entries, the registrar in this mode is configured to create the corresponding custom resources. Let's check for this using a Kubernetes native command such as: + +```console +$ kubectl get spiffeids -n spire +``` + +This command will show the custom resources for each one of the pods: + +```console +NAME AGE +minikube 24m +example-workload-5bffcd75d-stl5w 24m +spire-agent-r86rz 24m +``` + +## Pod deletion + +As in the previous modes, if we delete the workload deployment, we will see that its corresponding registration entry will be deleted too. Let's check it by running the command to delete the workload pod: + +```console +$ kubectl delete deployment/example-workload -n spire +``` + +And now, check the registration entries in the SPIRE Server by executing: + +```console +$ kubectl exec statefulset/spire-server -n spire -c spire-server -- bin/spire-server entry show -registrationUDSPath /tmp/spire-server/private/api.sock +``` + +The output should look like: + +```console +Found 2 entries +Entry ID : ... +SPIFFE ID : spiffe://example.org/k8s-workload-registrar/demo-cluster/node/minikube +Parent ID : spiffe://example.org/spire/server +Revision : 1 +TTL : default +Selector : k8s_psat:agent_node_uid:08990bfd-3551-4761-8a1b-2e652984ffdd +Selector : k8s_psat:cluster:demo-cluster + +Entry ID : ... +SPIFFE ID : spiffe://example.org/testing/agent +Parent ID : spiffe://example.org/k8s-workload-registrar/demo-cluster/node/minikube +Revision : 1 +TTL : default +Selector : k8s:node-name:minikube +Selector : k8s:ns:spire +Selector : k8s:pod-uid:538886bb-48e1-4795-b386-10e97f50e34f +DNS name : spire-agent-jzc8w +``` + +The only entries that should exist now are the ones that match the node and the SPIRE Agent, because the workload one was deleted by the registrar, something that we can check if we examine the registrar logs, but this time looking for the keyword "Deleted entry": + +```console +$ kubectl logs statefulset/spire-server -n spire -c k8s-workload-registrar | grep -A 1 "Deleted entry" +``` + +As the registrar handles the custom resources automatically, it also deleted the corresponding SPIFFE ID CRD, something that we can also check by querying the Kubernetes control plane (`kubectl get spiffeids -n spire`), command which should display the following: + +```console +NAME AGE +minikube 41m +spire-agent-r86rz 40m +``` + +## Non-annotated pods + +Let's check if a pod that has no annotations is detected by the registrar. Deploy a new workload without annotations using the following command: + +```console +$ kubectl apply -f k8s/not-annotated-workload.yaml +``` + +As in the previous section, let's see the registration entries that are present in the SPIRE Server: + +```console +$ kubectl exec statefulset/spire-server -n spire -c spire-server -- bin/spire-server entry show -registrationUDSPath /tmp/spire-server/private/api.sock +``` + +The result of the command should be equal to the one shown in *Pod deletion* section, because no new entry has been created, as expected. + +## SPIFFE ID CRD creation + +One of the benefits of using the CRD mode is that we can manipulate the SPIFFE IDs as if they were resources inside Kubernetes environment, in other words using the `kubectl` command. + +Let's create a new SPIFFE ID CRD by using: + +```console +$ kubectl apply -f k8s/test_spiffeid.yaml +``` + +We will check if it was created, consulting the custom resources with `kubectl get spiffeids -n spire`, the output of which should show the following: + +```console +NAME AGE +example-cluster-control-plane 45m +my-test-spiffeid 19s +spire-agent-r86rz 45m +``` + +The resource was succesfully created, but had it any impact on the SPIRE Server? Let's execute the command below to see the registration entries: + +```console +$ kubectl exec statefulset/spire-server -n spire -c spire-server -- bin/spire-server entry show -registrationUDSPath /tmp/spire-server/private/api.sock +``` + +You'll get an output similar to this: + +```console +Found 3 entries +Entry ID : ... +SPIFFE ID : spiffe://example.org/k8s-workload-registrar/demo-cluster/node/minikube +Parent ID : spiffe://example.org/spire/server +Revision : 1 +TTL : default +Selector : k8s_psat:agent_node_uid:08990bfd-3551-4761-8a1b-2e652984ffdd +Selector : k8s_psat:cluster:demo-cluster + +Entry ID : ... +SPIFFE ID : spiffe://example.org/test +Parent ID : spiffe://example.org/spire/server +Revision : 1 +TTL : default +Selector : k8s:ns:spire +Selector : k8s:pod-name:my-test-pod + +Entry ID : ... +SPIFFE ID : spiffe://example.org/testing/agent +Parent ID : spiffe://example.org/k8s-workload-registrar/demo-cluster/node/minikube +Revision : 1 +TTL : default +Selector : k8s:node-name:minikube +Selector : k8s:ns:spire +Selector : k8s:pod-uid:538886bb-48e1-4795-b386-10e97f50e34f +DNS name : spire-agent-jzc8w +``` + +As we can see, SPIFFE ID CRD creation triggers registration entry creation on the SPIRE Server, too. + +## SPIFFE ID CRD deletion + +The lifecycle of a SPIFFE ID CRD can be managed by Kubernetes, and has a direct impact on the corresponding registration entry stored in the SPIRE Server. We already saw how SPIFFE ID CRD creation activates registration entry creation. We will prove that the same applies for a CRD deletion. + +Let's delete the previously created SPIFFE ID CRD, and later check for the registration entries on the server. Run the following command to delete the CRD: + +```console +$ kubectl delete spiffeid/my-test-spiffeid -n spire +``` + +Now, we will check the registration entries: + +```console +$ kubectl exec statefulset/spire-server -n spire -c spire-server -- bin/spire-server entry show -registrationUDSPath /tmp/spire-server/private/api.sock +``` + +The output from this command should include only the entries for the node and the agent, because the recently created SPIFFE ID CRD was deleted, along with the entry. + +## Teardown + +To delete the resources used for this mode, we will run the `delete-scenario.sh` script: + +```console +$ bash scripts/delete-scenario.sh +``` diff --git a/k8s/k8s-workload-registrar/mode-crd/k8s/namespace.yaml b/k8s/k8s-workload-registrar/mode-crd/k8s/namespace.yaml new file mode 100644 index 0000000..c6ba349 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-crd/k8s/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: spire diff --git a/k8s/k8s-workload-registrar/mode-crd/k8s/not-annotated-workload.yaml b/k8s/k8s-workload-registrar/mode-crd/k8s/not-annotated-workload.yaml new file mode 100644 index 0000000..9aad1c7 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-crd/k8s/not-annotated-workload.yaml @@ -0,0 +1,35 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: example-workload + namespace: spire + labels: + app: example-workload +spec: + selector: + matchLabels: + app: example-workload + template: + metadata: + namespace: spire + labels: + app: example-workload + spec: + hostPID: true + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: example-workload + image: gcr.io/spiffe-io/spire-agent:0.12.0 + imagePullPolicy: IfNotPresent + command: ["/usr/bin/dumb-init", "/opt/spire/bin/spire-agent", "api", "watch"] + args: ["-socketPath", "/tmp/spire-agent/public/api.sock"] + volumeMounts: + - name: spire-agent-socket + mountPath: /tmp/spire-agent/public + readOnly: true + volumes: + - name: spire-agent-socket + hostPath: + path: /run/spire/agent-sockets + type: Directory diff --git a/k8s/k8s-workload-registrar/mode-crd/k8s/spiffeid.spiffe.io_spiffeids.yaml b/k8s/k8s-workload-registrar/mode-crd/k8s/spiffeid.spiffe.io_spiffeids.yaml new file mode 100644 index 0000000..302c2b7 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-crd/k8s/spiffeid.spiffe.io_spiffeids.yaml @@ -0,0 +1,106 @@ + +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.2.4 + creationTimestamp: null + name: spiffeids.spiffeid.spiffe.io +spec: + group: spiffeid.spiffe.io + names: + kind: SpiffeID + listKind: SpiffeIDList + plural: spiffeids + singular: spiffeid + scope: Namespaced + subresources: + status: {} + validation: + openAPIV3Schema: + description: SpiffeID is the Schema for the spiffeid API + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: SpiffeIDSpec defines the desired state of SpiffeID + properties: + dnsNames: + items: + type: string + type: array + parentId: + type: string + selector: + properties: + arbitrary: + description: Arbitrary selectors + items: + type: string + type: array + containerImage: + description: Container image to match for this spiffe ID + type: string + containerName: + description: Container name to match for this spiffe ID + type: string + namespace: + description: Namespace to match for this spiffe ID + type: string + nodeName: + description: Node name to match for this spiffe ID + type: string + podLabel: + additionalProperties: + type: string + description: Pod label name/value to match for this spiffe ID + type: object + podName: + description: Pod name to match for this spiffe ID + type: string + podUid: + description: Pod UID to match for this spiffe ID + type: string + serviceAccount: + description: ServiceAccount to match for this spiffe ID + type: string + type: object + spiffeId: + type: string + required: + - parentId + - selector + - spiffeId + type: object + status: + description: SpiffeIDStatus defines the observed state of SpiffeID + properties: + entryId: + description: 'INSERT ADDITIONAL STATUS FIELD - define observed state + of cluster Important: Run "make" to regenerate code after modifying + this file' + type: string + type: object + type: object + version: v1beta1 + versions: + - name: v1beta1 + served: true + storage: true +status: + acceptedNames: + kind: "" + plural: "" + conditions: [] + storedVersions: [] diff --git a/k8s/k8s-workload-registrar/mode-crd/k8s/spire-agent.yaml b/k8s/k8s-workload-registrar/mode-crd/k8s/spire-agent.yaml new file mode 100644 index 0000000..52ba97e --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-crd/k8s/spire-agent.yaml @@ -0,0 +1,163 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: spire-agent + namespace: spire + +--- + +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spire-agent-cluster-role +rules: +- apiGroups: [""] + resources: ["pods","nodes","nodes/proxy"] + verbs: ["get"] + +--- + +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spire-agent-cluster-role-binding +subjects: +- kind: ServiceAccount + name: spire-agent + namespace: spire +roleRef: + kind: ClusterRole + name: spire-agent-cluster-role + apiGroup: rbac.authorization.k8s.io + +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: spire-agent + namespace: spire +data: + agent.conf: | + agent { + data_dir = "/run/spire" + log_level = "DEBUG" + server_address = "spire-server" + server_port = "8081" + trust_bundle_path = "/run/spire/bundle/bundle.crt" + trust_domain = "example.org" + socket_path = "/tmp/spire-agent/public/api.sock" + } + + plugins { + NodeAttestor "k8s_psat" { + plugin_data { + cluster = "demo-cluster" + } + } + + KeyManager "memory" { + plugin_data { + } + } + + WorkloadAttestor "k8s" { + plugin_data { + # Defaults to the secure kubelet port by default. + # Minikube does not have a cert in the cluster CA bundle that + # can authenticate the kubelet cert, so skip validation. + skip_kubelet_verification = true + } + } + } + + health_checks { + listener_enabled = true + bind_address = "0.0.0.0" + bind_port = "8080" + live_path = "/live" + ready_path = "/ready" + } + +--- + +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: spire-agent + namespace: spire + labels: + app: spire-agent +spec: + selector: + matchLabels: + app: spire-agent + updateStrategy: + type: RollingUpdate + template: + metadata: + namespace: spire + labels: + app: spire-agent + spire-workload: agent + annotations: + spiffe.io/spiffe-id: "testing/agent" + spec: + hostPID: true + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + serviceAccountName: spire-agent + initContainers: + - name: init + # This is a small image with wait-for-it, choose whatever image + # you prefer that waits for a service to be up. This image is built + # from https://github.com/lqhl/wait-for-it + image: gcr.io/spiffe-io/wait-for-it + args: ["-t", "30", "spire-server:8081"] + containers: + - name: spire-agent + image: gcr.io/spiffe-io/spire-agent:0.12.0 + imagePullPolicy: IfNotPresent + args: ["-config", "/run/spire/config/agent.conf"] + volumeMounts: + - name: spire-config + mountPath: /run/spire/config + readOnly: true + - name: spire-bundle + mountPath: /run/spire/bundle + readOnly: true + - name: spire-agent-socket + mountPath: /tmp/spire-agent/public + readOnly: false + - name: spire-token + mountPath: /var/run/secrets/tokens + livenessProbe: + httpGet: + path: /live + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /ready + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 10 + volumes: + - name: spire-config + configMap: + name: spire-agent + - name: spire-bundle + configMap: + name: spire-bundle + - name: spire-agent-socket + hostPath: + path: /run/spire/agent-sockets + type: DirectoryOrCreate + - name: spire-token + projected: + sources: + - serviceAccountToken: + path: spire-agent + expirationSeconds: 7200 + audience: spire-server diff --git a/k8s/k8s-workload-registrar/mode-crd/k8s/spire-server.yaml b/k8s/k8s-workload-registrar/mode-crd/k8s/spire-server.yaml new file mode 100644 index 0000000..d925e7e --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-crd/k8s/spire-server.yaml @@ -0,0 +1,258 @@ +# ServiceAccount used by the SPIRE server. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: spire-server + namespace: spire + +--- + +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: k8s-workload-registrar-role +rules: +- apiGroups: ["authentication.k8s.io"] + resources: ["tokenreviews"] + verbs: ["get", "create"] +- apiGroups: [""] + resources: ["endpoints", "nodes", "pods"] + verbs: ["get", "list", "watch"] +- apiGroups: ["spiffeid.spiffe.io"] + resources: ["spiffeids"] + verbs: ["create", "delete", "get", "list", "patch", "update", "watch"] +- apiGroups: ["spiffeid.spiffe.io"] + resources: ["spiffeids/status"] + verbs: ["get", "patch", "update"] +- apiGroups: ["admissionregistration.k8s.io"] + resources: ["validatingwebhookconfigurations"] + verbs: ["get", "list", "update", "watch"] + +--- + +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: k8s-workload-registrar-role-binding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: k8s-workload-registrar-role +subjects: +- kind: ServiceAccount + name: spire-server + namespace: spire + +--- + +kind: Role +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + namespace: spire + name: spire-server-role +rules: +- apiGroups: [""] + resources: ["pods"] + verbs: ["get"] +- apiGroups: [""] + resources: ["configmaps"] + resourceNames: ["spire-bundle"] + verbs: ["get", "patch"] + +--- + +kind: RoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spire-server-role-binding + namespace: spire +subjects: +- kind: ServiceAccount + name: spire-server + namespace: spire +roleRef: + kind: Role + name: spire-server-role + apiGroup: rbac.authorization.k8s.io + +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: spire-bundle + namespace: spire + +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: spire-server + namespace: spire +data: + server.conf: | + server { + bind_address = "0.0.0.0" + bind_port = "8081" + trust_domain = "example.org" + data_dir = "/run/spire/data" + log_level = "DEBUG" + default_svid_ttl = "1h" + ca_ttl = "12h" + registration_uds_path = "/tmp/spire-server/private/api.sock" + ca_subject { + country = ["US"] + organization = ["SPIFFE"] + common_name = "" + } + } + + plugins { + DataStore "sql" { + plugin_data { + database_type = "sqlite3" + connection_string = "/run/spire/data/datastore.sqlite3" + } + } + + NodeAttestor "k8s_psat" { + plugin_data { + clusters = { + "demo-cluster" = { + service_account_whitelist = ["spire:spire-agent"] + } + } + } + } + + KeyManager "disk" { + plugin_data { + keys_path = "/run/spire/data/keys.json" + } + } + + Notifier "k8sbundle" { + plugin_data { + # This plugin updates the bundle.crt value in the spire:spire-bundle + # ConfigMap by default, so no additional configuration is necessary. + } + } + } + + health_checks { + listener_enabled = true + bind_address = "0.0.0.0" + bind_port = "8080" + live_path = "/live" + ready_path = "/ready" + } + +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: k8s-workload-registrar + namespace: spire +data: + k8s-workload-registrar.conf: | + trust_domain = "example.org" + server_socket_path = "/tmp/spire-server/private/api.sock" + cluster = "demo-cluster" + mode = "crd" + pod_annotation = "spiffe.io/spiffe-id" + metrics_bind_addr = "0" + +--- + +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: spire-server + namespace: spire + labels: + app: spire-server +spec: + replicas: 1 + selector: + matchLabels: + app: spire-server + serviceName: spire-server + template: + metadata: + namespace: spire + labels: + app: spire-server + spec: + serviceAccountName: spire-server + shareProcessNamespace: true + containers: + - name: spire-server + image: gcr.io/spiffe-io/spire-server:0.12.0 + args: + - -config + - /run/spire/config/server.conf + livenessProbe: + httpGet: + path: /live + port: 8080 + initialDelaySeconds: 5 + periodSeconds: 5 + readinessProbe: + httpGet: + path: /ready + port: 8080 + initialDelaySeconds: 5 + periodSeconds: 5 + ports: + - containerPort: 8081 + volumeMounts: + - name: spire-config + mountPath: /run/spire/config + readOnly: true + - name: spire-registration-socket + mountPath: /tmp/spire-server/private + readOnly: false + - name: k8s-workload-registrar + image: gcr.io/spiffe-io/k8s-workload-registrar:0.12.0 + args: ["-config", "/run/spire/k8s-workload-registrar/conf/k8s-workload-registrar.conf"] + ports: + - containerPort: 9443 + name: webhook + protocol: TCP + volumeMounts: + - name: spire-registration-socket + mountPath: /tmp/spire-server/private + readOnly: true + - name: k8s-workload-registrar + mountPath: /run/spire/k8s-workload-registrar/conf + readOnly: true + volumes: + - name: k8s-workload-registrar + configMap: + name: k8s-workload-registrar + - name: spire-registration-socket + hostPath: + path: /run/spire/server-sockets + type: DirectoryOrCreate + - name: spire-config + configMap: + name: spire-server + +--- + +apiVersion: v1 +kind: Service +metadata: + name: spire-server + namespace: spire +spec: + type: NodePort + ports: + - name: grpc + port: 8081 + targetPort: 8081 + protocol: TCP + selector: + app: spire-server diff --git a/k8s/k8s-workload-registrar/mode-crd/k8s/test_spiffeid.yaml b/k8s/k8s-workload-registrar/mode-crd/k8s/test_spiffeid.yaml new file mode 100644 index 0000000..2a5ba52 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-crd/k8s/test_spiffeid.yaml @@ -0,0 +1,11 @@ +apiVersion: spiffeid.spiffe.io/v1beta1 +kind: SpiffeID +metadata: + name: my-test-spiffeid + namespace: spire +spec: + parentId: spiffe://example.org/spire/server + selector: + namespace: spire + podName: my-test-pod + spiffeId: spiffe://example.org/test diff --git a/k8s/k8s-workload-registrar/mode-crd/k8s/workload.yaml b/k8s/k8s-workload-registrar/mode-crd/k8s/workload.yaml new file mode 100644 index 0000000..a004396 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-crd/k8s/workload.yaml @@ -0,0 +1,37 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: example-workload + namespace: spire + labels: + app: example-workload +spec: + selector: + matchLabels: + app: example-workload + template: + metadata: + namespace: spire + labels: + app: example-workload + annotations: + spiffe.io/spiffe-id: "testing/example-workload" + spec: + hostPID: true + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: example-workload + image: gcr.io/spiffe-io/spire-agent:0.12.0 + imagePullPolicy: IfNotPresent + command: ["/usr/bin/dumb-init", "/opt/spire/bin/spire-agent", "api", "watch"] + args: ["-socketPath", "/tmp/spire-agent/public/api.sock"] + volumeMounts: + - name: spire-agent-socket + mountPath: /tmp/spire-agent/public + readOnly: true + volumes: + - name: spire-agent-socket + hostPath: + path: /run/spire/agent-sockets + type: Directory diff --git a/k8s/k8s-workload-registrar/mode-crd/scripts/delete-scenario.sh b/k8s/k8s-workload-registrar/mode-crd/scripts/delete-scenario.sh new file mode 100755 index 0000000..b7514bd --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-crd/scripts/delete-scenario.sh @@ -0,0 +1,22 @@ +#!/bin/bash + +PARENT_DIR="$(dirname "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )")" + +kubectl delete -f "${PARENT_DIR}"/k8s/workload.yaml --ignore-not-found + +kubectl delete -f "${PARENT_DIR}"/k8s/spire-agent.yaml --ignore-not-found + +kubectl delete -f "${PARENT_DIR}"/k8s/spire-server.yaml --ignore-not-found + +SPIFFE_ID_CRDS=$(kubectl get spiffeids --no-headers -o custom-columns=":metadata.name" -n spire) +for SPIFFE_ID_CRD in $SPIFFE_ID_CRDS +do + kubectl patch spiffeid.spiffeid.spiffe.io/"${SPIFFE_ID_CRD}" --type=merge -p '{"metadata":{"finalizers":[]}}' -n spire + kubectl delete spiffeid "${SPIFFE_ID_CRD}" -n spire --ignore-not-found +done + +kubectl patch customresourcedefinition.apiextensions.k8s.io/spiffeids.spiffeid.spiffe.io --type=merge -p '{"metadata":{"finalizers":[]}}' +kubectl delete -f "${PARENT_DIR}"/k8s/spiffeid.spiffe.io_spiffeids.yaml --ignore-not-found + + +kubectl delete -f "${PARENT_DIR}"/k8s/namespace.yaml --ignore-not-found diff --git a/k8s/k8s-workload-registrar/mode-crd/scripts/deploy-scenario.sh b/k8s/k8s-workload-registrar/mode-crd/scripts/deploy-scenario.sh new file mode 100755 index 0000000..33bae24 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-crd/scripts/deploy-scenario.sh @@ -0,0 +1,13 @@ +#!/bin/bash + +PARENT_DIR="$(dirname "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )")" + +kubectl apply -f "${PARENT_DIR}"/k8s/namespace.yaml +kubectl apply -f "${PARENT_DIR}"/k8s/spiffeid.spiffe.io_spiffeids.yaml +kubectl apply -f "${PARENT_DIR}"/k8s/spire-server.yaml +kubectl rollout status statefulset/spire-server -n spire + +kubectl apply -f "${PARENT_DIR}"/k8s/spire-agent.yaml +kubectl rollout status daemonset/spire-agent -n spire + +kubectl apply -f "${PARENT_DIR}"/k8s/workload.yaml diff --git a/k8s/k8s-workload-registrar/mode-crd/test.sh b/k8s/k8s-workload-registrar/mode-crd/test.sh new file mode 100755 index 0000000..535c9c6 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-crd/test.sh @@ -0,0 +1,57 @@ +#!/bin/bash + +DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + +bold=$(tput bold) || true +norm=$(tput sgr0) || true +red=$(tput setaf 1) || true +green=$(tput setaf 2) || true + +echo $DIR + +set_env() { + echo "${bold}Setting up CRD mode environment...${norm}" + "${DIR}"/scripts/deploy-scenario.sh > /dev/null +} + +cleanup() { + echo "${bold}Cleaning up...${norm}" + "${DIR}"/scripts/delete-scenario.sh > /dev/null +} + +trap cleanup EXIT + +cleanup +set_env + +NODE_SPIFFE_ID="spiffe://example.org/k8s-workload-registrar/demo-cluster/node/" +AGENT_SPIFFE_ID="spiffe://example.org/testing/agent" +WORKLOAD_SPIFFE_ID="spiffe://example.org/testing/example-workload" + +MAX_FETCH_CHECKS=60 +FETCH_CHECK_INTERVAL=5 + +for ((i=0;i<"$MAX_FETCH_CHECKS";i++)); do + if [[ -n $(kubectl exec -t statefulset/spire-server -n spire -c spire-server -- \ + /opt/spire/bin/spire-server entry show -registrationUDSPath /tmp/spire-server/private/api.sock \ + | grep "$NODE_SPIFFE_ID") ]] && + [[ -n $(kubectl exec -t daemonset/spire-agent -n spire -c spire-agent -- \ + /opt/spire/bin/spire-agent api fetch -socketPath /tmp/spire-agent/public/api.sock \ + | grep "$AGENT_SPIFFE_ID") ]] && + [[ -n $(kubectl exec -t deployment/example-workload -n spire -- \ + /opt/spire/bin/spire-agent api fetch -socketPath /tmp/spire-agent/public/api.sock \ + | grep "$WORKLOAD_SPIFFE_ID") ]]; then + DONE=1 + break + fi + sleep "$FETCH_CHECK_INTERVAL" +done + +if [ "${DONE}" -eq 1 ]; then + exit 0 +else + echo "${red}CRD mode test failed.${norm}" + exit 1 +fi + + exit 0 diff --git a/k8s/k8s-workload-registrar/mode-reconcile/k8s/namespace.yaml b/k8s/k8s-workload-registrar/mode-reconcile/k8s/namespace.yaml new file mode 100644 index 0000000..c6ba349 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-reconcile/k8s/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: spire diff --git a/k8s/k8s-workload-registrar/mode-reconcile/k8s/not-labeled-workload.yaml b/k8s/k8s-workload-registrar/mode-reconcile/k8s/not-labeled-workload.yaml new file mode 100644 index 0000000..007c15b --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-reconcile/k8s/not-labeled-workload.yaml @@ -0,0 +1,35 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: example-workload + namespace: spire + labels: + app: example-workload +spec: + selector: + matchLabels: + app: example-workload + template: + metadata: + namespace: spire + labels: + app: example-workload + spec: + hostPID: true + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: example-workload + image: gcr.io/spiffe-io/spire-agent:0.12.0 + imagePullPolicy: IfNotPresent + command: ["/usr/bin/dumb-init", "/opt/spire/bin/spire-agent", "api", "watch"] + args: ["-socketPath", "/tmp/spire-agent/public/api.sock"] + volumeMounts: + - name: spire-agent-socket + mountPath: /tmp/spire-agent/public + readOnly: true + volumes: + - name: spire-agent-socket + hostPath: + path: /run/spire/agent-sockets + type: Directory diff --git a/k8s/k8s-workload-registrar/mode-reconcile/k8s/spire-agent.yaml b/k8s/k8s-workload-registrar/mode-reconcile/k8s/spire-agent.yaml new file mode 100644 index 0000000..7a07ac4 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-reconcile/k8s/spire-agent.yaml @@ -0,0 +1,162 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: spire-agent + namespace: spire + +--- + +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spire-agent-cluster-role +rules: +- apiGroups: [""] + resources: ["pods","nodes","nodes/proxy"] + verbs: ["get"] + +--- + +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spire-agent-cluster-role-binding +subjects: +- kind: ServiceAccount + name: spire-agent + namespace: spire +roleRef: + kind: ClusterRole + name: spire-agent-cluster-role + apiGroup: rbac.authorization.k8s.io + + +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: spire-agent + namespace: spire +data: + agent.conf: | + agent { + data_dir = "/run/spire" + log_level = "DEBUG" + server_address = "spire-server" + server_port = "8081" + trust_bundle_path = "/run/spire/bundle/bundle.crt" + trust_domain = "example.org" + socket_path = "/tmp/spire-agent/public/api.sock" + } + + plugins { + NodeAttestor "k8s_psat" { + plugin_data { + cluster = "demo-cluster" + } + } + + KeyManager "memory" { + plugin_data { + } + } + + WorkloadAttestor "k8s" { + plugin_data { + # Defaults to the secure kubelet port by default. + # Minikube does not have a cert in the cluster CA bundle that + # can authenticate the kubelet cert, so skip validation. + skip_kubelet_verification = true + } + } + } + + health_checks { + listener_enabled = true + bind_address = "0.0.0.0" + bind_port = "8080" + live_path = "/live" + ready_path = "/ready" + } + +--- + +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: spire-agent + namespace: spire + labels: + app: spire-agent +spec: + selector: + matchLabels: + app: spire-agent + updateStrategy: + type: RollingUpdate + template: + metadata: + namespace: spire + labels: + app: spire-agent + spire-workload: agent + spec: + hostPID: true + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + serviceAccountName: spire-agent + initContainers: + - name: init + # This is a small image with wait-for-it, choose whatever image + # you prefer that waits for a service to be up. This image is built + # from https://github.com/lqhl/wait-for-it + image: gcr.io/spiffe-io/wait-for-it + args: ["-t", "30", "spire-server:8081"] + containers: + - name: spire-agent + image: gcr.io/spiffe-io/spire-agent:0.12.0 + imagePullPolicy: IfNotPresent + args: ["-config", "/run/spire/config/agent.conf"] + volumeMounts: + - name: spire-config + mountPath: /run/spire/config + readOnly: true + - name: spire-bundle + mountPath: /run/spire/bundle + readOnly: true + - name: spire-agent-socket + mountPath: /tmp/spire-agent/public + readOnly: false + - name: spire-token + mountPath: /var/run/secrets/tokens + livenessProbe: + httpGet: + path: /live + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /ready + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 10 + volumes: + - name: spire-config + configMap: + name: spire-agent + - name: spire-bundle + configMap: + name: spire-bundle + - name: spire-agent-socket + hostPath: + path: /run/spire/agent-sockets + type: DirectoryOrCreate + - name: spire-token + projected: + sources: + - serviceAccountToken: + path: spire-agent + expirationSeconds: 7200 + audience: spire-server diff --git a/k8s/k8s-workload-registrar/mode-reconcile/k8s/spire-server.yaml b/k8s/k8s-workload-registrar/mode-reconcile/k8s/spire-server.yaml new file mode 100644 index 0000000..5f25135 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-reconcile/k8s/spire-server.yaml @@ -0,0 +1,259 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: spire-server + namespace: spire + +--- + +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spire-server-cluster-role +rules: +- apiGroups: [""] + resources: ["pods", "nodes"] + verbs: ["get", "list", "watch"] +- apiGroups: ["authentication.k8s.io"] + resources: ["tokenreviews"] + verbs: ["get", "create"] + +--- + +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spire-server-cluster-role-binding + namespace: spire +subjects: +- kind: ServiceAccount + name: spire-server + namespace: spire +roleRef: + kind: ClusterRole + name: spire-server-cluster-role + apiGroup: rbac.authorization.k8s.io + +--- + +kind: Role +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + namespace: spire + name: spire-server-role +rules: +- apiGroups: [""] + resources: ["pods"] + verbs: ["get"] +- apiGroups: [""] + resources: ["configmaps"] + resourceNames: ["spire-bundle"] + verbs: ["get", "patch"] +- apiGroups: [""] + resources: ["configmaps"] + verbs: ["create"] +- apiGroups: [""] + resources: ["configmaps"] + resourceNames: ["spire-k8s-registrar-leader-election"] + verbs: ["update", "get"] +- apiGroups: [""] + resources: ["events"] + verbs: ["create"] + +--- + +kind: RoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spire-server-role-binding + namespace: spire +subjects: +- kind: ServiceAccount + name: spire-server + namespace: spire +roleRef: + kind: Role + name: spire-server-role + apiGroup: rbac.authorization.k8s.io + +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: spire-bundle + namespace: spire + +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: spire-server + namespace: spire +data: + server.conf: | + server { + bind_address = "0.0.0.0" + bind_port = "8081" + trust_domain = "example.org" + data_dir = "/run/spire/data" + log_level = "DEBUG" + default_svid_ttl = "1h" + ca_ttl = "12h" + registration_uds_path = "/tmp/spire-server/private/api.sock" + ca_subject { + country = ["US"] + organization = ["SPIFFE"] + common_name = "" + } + } + + plugins { + DataStore "sql" { + plugin_data { + database_type = "sqlite3" + connection_string = "/run/spire/data/datastore.sqlite3" + } + } + + NodeAttestor "k8s_psat" { + plugin_data { + clusters = { + "demo-cluster" = { + service_account_whitelist = ["spire:spire-agent"] + } + } + } + } + + KeyManager "disk" { + plugin_data { + keys_path = "/run/spire/data/keys.json" + } + } + + Notifier "k8sbundle" { + plugin_data { + # This plugin updates the bundle.crt value in the spire:spire-bundle + # ConfigMap by default, so no additional configuration is necessary. + } + } + } + + health_checks { + listener_enabled = true + bind_address = "0.0.0.0" + bind_port = "8080" + live_path = "/live" + ready_path = "/ready" + } + +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: k8s-workload-registrar + namespace: spire +data: + k8s-workload-registrar.conf: | + trust_domain = "example.org" + server_socket_path = "/tmp/spire-server/private/api.sock" + cluster = "demo-cluster" + mode = "reconcile" + pod_label = "spire-workload" + metrics_addr = "0" + controller_name = "k8s-workload-registrar" + +--- + +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: spire-server + namespace: spire + labels: + app: spire-server +spec: + replicas: 1 + selector: + matchLabels: + app: spire-server + serviceName: spire-server + template: + metadata: + namespace: spire + labels: + app: spire-server + spec: + serviceAccountName: spire-server + shareProcessNamespace: true + containers: + - name: spire-server + image: gcr.io/spiffe-io/spire-server:0.12.0 + imagePullPolicy: IfNotPresent + args: ["-config", "/run/spire/config/server.conf"] + ports: + - containerPort: 8081 + volumeMounts: + - name: spire-config + mountPath: /run/spire/config + readOnly: true + - name: spire-server-socket + mountPath: /tmp/spire-server/private + readOnly: false + livenessProbe: + httpGet: + path: /live + port: 8080 + initialDelaySeconds: 5 + periodSeconds: 5 + readinessProbe: + httpGet: + path: /ready + port: 8080 + initialDelaySeconds: 5 + periodSeconds: 5 + - name: k8s-workload-registrar + image: gcr.io/spiffe-io/k8s-workload-registrar:0.12.0 + imagePullPolicy: IfNotPresent + args: ["-config", "/run/spire/k8s-workload-registrar/conf/k8s-workload-registrar.conf"] + ports: + - containerPort: 8443 + name: registrar-port + volumeMounts: + - name: spire-server-socket + mountPath: /tmp/spire-server/private + readOnly: true + - name: k8s-workload-registrar + mountPath: /run/spire/k8s-workload-registrar/conf + readOnly: true + volumes: + - name: spire-config + configMap: + name: spire-server + - name: spire-server-socket + hostPath: + path: /run/spire/server-sockets + type: DirectoryOrCreate + - name: k8s-workload-registrar + configMap: + name: k8s-workload-registrar + +--- + +apiVersion: v1 +kind: Service +metadata: + name: spire-server + namespace: spire +spec: + type: NodePort + ports: + - name: grpc + port: 8081 + targetPort: 8081 + protocol: TCP + selector: + app: spire-server diff --git a/k8s/k8s-workload-registrar/mode-reconcile/k8s/workload.yaml b/k8s/k8s-workload-registrar/mode-reconcile/k8s/workload.yaml new file mode 100644 index 0000000..df35b43 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-reconcile/k8s/workload.yaml @@ -0,0 +1,36 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: example-workload + namespace: spire + labels: + app: example-workload +spec: + selector: + matchLabels: + app: example-workload + template: + metadata: + namespace: spire + labels: + app: example-workload + spire-workload: example-workload + spec: + hostPID: true + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: example-workload + image: gcr.io/spiffe-io/spire-agent:0.12.0 + imagePullPolicy: IfNotPresent + command: ["/usr/bin/dumb-init", "/opt/spire/bin/spire-agent", "api", "watch"] + args: ["-socketPath", "/tmp/spire-agent/public/api.sock"] + volumeMounts: + - name: spire-agent-socket + mountPath: /tmp/spire-agent/public + readOnly: true + volumes: + - name: spire-agent-socket + hostPath: + path: /run/spire/agent-sockets + type: Directory diff --git a/k8s/k8s-workload-registrar/mode-reconcile/scripts/delete-scenario.sh b/k8s/k8s-workload-registrar/mode-reconcile/scripts/delete-scenario.sh new file mode 100755 index 0000000..7e1e089 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-reconcile/scripts/delete-scenario.sh @@ -0,0 +1,10 @@ +#!/bin/bash + +PARENT_DIR="$(dirname "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )")" + +kubectl delete -f "${PARENT_DIR}"/k8s/workload.yaml --ignore-not-found + +kubectl delete -f "${PARENT_DIR}"/k8s/spire-agent.yaml --ignore-not-found + +kubectl delete -f "${PARENT_DIR}"/k8s/spire-server.yaml --ignore-not-found +kubectl delete -f "${PARENT_DIR}"/k8s/namespace.yaml --ignore-not-found diff --git a/k8s/k8s-workload-registrar/mode-reconcile/scripts/deploy-scenario.sh b/k8s/k8s-workload-registrar/mode-reconcile/scripts/deploy-scenario.sh new file mode 100755 index 0000000..bbabc88 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-reconcile/scripts/deploy-scenario.sh @@ -0,0 +1,13 @@ +#!/bin/bash + +PARENT_DIR="$(dirname "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )")" + +kubectl apply -f "${PARENT_DIR}"/k8s/namespace.yaml +kubectl apply -f "${PARENT_DIR}"/k8s/spire-server.yaml +kubectl rollout status statefulset/spire-server -n spire + +kubectl apply -f "${PARENT_DIR}"/k8s/spire-agent.yaml +kubectl rollout status daemonset/spire-agent -n spire + +kubectl apply -f "${PARENT_DIR}"/k8s/workload.yaml +kubectl rollout status deployment/example-workload -n spire diff --git a/k8s/k8s-workload-registrar/mode-reconcile/test.sh b/k8s/k8s-workload-registrar/mode-reconcile/test.sh new file mode 100755 index 0000000..3706d18 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-reconcile/test.sh @@ -0,0 +1,53 @@ +#!/bin/bash + +DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + +bold=$(tput bold) || true +norm=$(tput sgr0) || true +red=$(tput setaf 1) || true +green=$(tput setaf 2) || true + +set_env() { + echo "${bold}Setting up reconcile environment...${norm}" + "${DIR}"/scripts/deploy-scenario.sh > /dev/null +} + +cleanup() { + echo "${bold}Cleaning up...${norm}" + "${DIR}"/scripts/delete-scenario.sh > /dev/null +} + +trap cleanup EXIT + +cleanup +set_env + +NODE_SPIFFE_ID="spiffe://example.org/k8s-workload-registrar/demo-cluster/node/" +AGENT_SPIFFE_ID="spiffe://example.org/agent" +WORKLOAD_SPIFFE_ID="spiffe://example.org/example-workload" + +MAX_FETCH_CHECKS=60 +FETCH_CHECK_INTERVAL=5 + +for ((i=0;i<"$MAX_FETCH_CHECKS";i++)); do + if [[ -n $(kubectl exec -t statefulset/spire-server -n spire -c spire-server -- \ + /opt/spire/bin/spire-server entry show -registrationUDSPath /tmp/spire-server/private/api.sock \ + | grep "$NODE_SPIFFE_ID") ]] && + [[ -n $(kubectl exec -t daemonset/spire-agent -n spire -c spire-agent -- \ + /opt/spire/bin/spire-agent api fetch -socketPath /tmp/spire-agent/public/api.sock \ + | grep "$AGENT_SPIFFE_ID") ]] && + [[ -n $(kubectl exec -t deployment/example-workload -n spire -- \ + /opt/spire/bin/spire-agent api fetch -socketPath /tmp/spire-agent/public/api.sock \ + | grep "$WORKLOAD_SPIFFE_ID") ]]; then + DONE=1 + break + fi + sleep "$FETCH_CHECK_INTERVAL" +done + +if [ "${DONE}" -eq 1 ]; then + exit 0 +else + echo "${red}Reconcile mode test failed.${norm}" + exit 1 +fi diff --git a/k8s/k8s-workload-registrar/mode-webhook/k8s/admctrl/admission-control.yaml b/k8s/k8s-workload-registrar/mode-webhook/k8s/admctrl/admission-control.yaml new file mode 100644 index 0000000..7bb684b --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-webhook/k8s/admctrl/admission-control.yaml @@ -0,0 +1,8 @@ +apiVersion: apiserver.k8s.io/v1alpha1 +kind: AdmissionConfiguration +plugins: +- name: ValidatingAdmissionWebhook + configuration: + apiVersion: apiserver.config.k8s.io/v1alpha1 + kind: WebhookAdmission + kubeConfigFile: /var/lib/minikube/certs/admctrl/kubeconfig.yaml diff --git a/k8s/k8s-workload-registrar/mode-webhook/k8s/admctrl/kubeconfig.yaml b/k8s/k8s-workload-registrar/mode-webhook/k8s/admctrl/kubeconfig.yaml new file mode 100644 index 0000000..f52c150 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-webhook/k8s/admctrl/kubeconfig.yaml @@ -0,0 +1,7 @@ +apiVersion: v1 +kind: Config +users: +- name: k8s-workload-registrar.spire.svc + user: + client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJ1VENDQVYrZ0F3SUJBZ0lJVVNIdmpGQTFxRHd3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF4TVoKU3poVElGZFBVa3RNVDBGRUlGSkZSMGxUVkZKQlVpQkRRVEFnRncweE9UQTFNVE14T1RFME1qTmFHQTg1T1RrNQpNVEl6TVRJek5UazFPVm93S0RFbU1DUUdBMVVFQXhNZFN6aFRJRmRQVWt0TVQwRkVJRkpGUjBsVFZGSkJVaUJEClRFbEZUbFF3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVM3SDIrMjJOcEFhTmVRdXQvZEYwdUYKMXk0VDVKTVdBczJOYm9NOXhZdlFKb1FXTVVNNERobWZQT1hVaE5STXdkb1JzTmhSdXZsYkROY2FEU29tNE1DYQpvM1V3Y3pBT0JnTlZIUThCQWY4RUJBTUNBNmd3RXdZRFZSMGxCQXd3Q2dZSUt3WUJCUVVIQXdJd0RBWURWUjBUCkFRSC9CQUl3QURBZEJnTlZIUTRFRmdRVW9EYlBiOUpWNXhqZlZVMnBhSzd2UUNsZ2d3SXdId1lEVlIwakJCZ3cKRm9BVW02eFNULzJCUzRYdmhVcXVzaDJCTEwwdlJNSXdDZ1lJS29aSXpqMEVBd0lEU0FBd1JRSWdHNzRQeWkyZQpONlBEcVRGRnY1UDFjNFhjVVdERzMwdzJIZEU4Wm8rMStVWUNJUURUL2xMa2dUUjUzV01INVRqWkllblhmYzFjCmxkMGlqSmpvRFJIR3lIRjJxdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K + client-key-data: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JR0hBZ0VBTUJNR0J5cUdTTTQ5QWdFR0NDcUdTTTQ5QXdFSEJHMHdhd0lCQVFRZ1BhSWtTTVowUmduQllWYncKMDIrdlN5UUpDM2RtZ0VDNFBLN2svTnk4Qnh1aFJBTkNBQVM3SDIrMjJOcEFhTmVRdXQvZEYwdUYxeTRUNUpNVwpBczJOYm9NOXhZdlFKb1FXTVVNNERobWZQT1hVaE5STXdkb1JzTmhSdXZsYkROY2FEU29tNE1DYQotLS0tLUVORCBQUklWQVRFIEtFWS0tLS0tCg== diff --git a/k8s/k8s-workload-registrar/mode-webhook/k8s/k8s-workload-registrar-secret.yaml b/k8s/k8s-workload-registrar/mode-webhook/k8s/k8s-workload-registrar-secret.yaml new file mode 100644 index 0000000..8e79c6f --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-webhook/k8s/k8s-workload-registrar-secret.yaml @@ -0,0 +1,8 @@ +apiVersion: v1 +kind: Secret +metadata: + name: k8s-workload-registrar-secret + namespace: spire +type: Opaque +data: + server-key.pem: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JR0hBZ0VBTUJNR0J5cUdTTTQ5QWdFR0NDcUdTTTQ5QXdFSEJHMHdhd0lCQVFRZ3RqS0h2ckVjVWJDdWtlUG8KaXJSMDRqSnZyWW1ONlF3cHlQSlFFTWtsZ3MraFJBTkNBQVJVdzRwSG1XQ3pyZmprWHNlbjkrbVNQemlmV1Y0MwpzNlNaMUorK3h2RFhNMmpPaE04NlZwL1JkQzBtMkZOajNXWWc2c3VSbEV6dmYvRncyQ3N1WmJtbwotLS0tLUVORCBQUklWQVRFIEtFWS0tLS0tCg== diff --git a/k8s/k8s-workload-registrar/mode-webhook/k8s/namespace.yaml b/k8s/k8s-workload-registrar/mode-webhook/k8s/namespace.yaml new file mode 100644 index 0000000..c6ba349 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-webhook/k8s/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: spire diff --git a/k8s/k8s-workload-registrar/mode-webhook/k8s/spire-agent.yaml b/k8s/k8s-workload-registrar/mode-webhook/k8s/spire-agent.yaml new file mode 100644 index 0000000..52ba97e --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-webhook/k8s/spire-agent.yaml @@ -0,0 +1,163 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: spire-agent + namespace: spire + +--- + +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spire-agent-cluster-role +rules: +- apiGroups: [""] + resources: ["pods","nodes","nodes/proxy"] + verbs: ["get"] + +--- + +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spire-agent-cluster-role-binding +subjects: +- kind: ServiceAccount + name: spire-agent + namespace: spire +roleRef: + kind: ClusterRole + name: spire-agent-cluster-role + apiGroup: rbac.authorization.k8s.io + +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: spire-agent + namespace: spire +data: + agent.conf: | + agent { + data_dir = "/run/spire" + log_level = "DEBUG" + server_address = "spire-server" + server_port = "8081" + trust_bundle_path = "/run/spire/bundle/bundle.crt" + trust_domain = "example.org" + socket_path = "/tmp/spire-agent/public/api.sock" + } + + plugins { + NodeAttestor "k8s_psat" { + plugin_data { + cluster = "demo-cluster" + } + } + + KeyManager "memory" { + plugin_data { + } + } + + WorkloadAttestor "k8s" { + plugin_data { + # Defaults to the secure kubelet port by default. + # Minikube does not have a cert in the cluster CA bundle that + # can authenticate the kubelet cert, so skip validation. + skip_kubelet_verification = true + } + } + } + + health_checks { + listener_enabled = true + bind_address = "0.0.0.0" + bind_port = "8080" + live_path = "/live" + ready_path = "/ready" + } + +--- + +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: spire-agent + namespace: spire + labels: + app: spire-agent +spec: + selector: + matchLabels: + app: spire-agent + updateStrategy: + type: RollingUpdate + template: + metadata: + namespace: spire + labels: + app: spire-agent + spire-workload: agent + annotations: + spiffe.io/spiffe-id: "testing/agent" + spec: + hostPID: true + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + serviceAccountName: spire-agent + initContainers: + - name: init + # This is a small image with wait-for-it, choose whatever image + # you prefer that waits for a service to be up. This image is built + # from https://github.com/lqhl/wait-for-it + image: gcr.io/spiffe-io/wait-for-it + args: ["-t", "30", "spire-server:8081"] + containers: + - name: spire-agent + image: gcr.io/spiffe-io/spire-agent:0.12.0 + imagePullPolicy: IfNotPresent + args: ["-config", "/run/spire/config/agent.conf"] + volumeMounts: + - name: spire-config + mountPath: /run/spire/config + readOnly: true + - name: spire-bundle + mountPath: /run/spire/bundle + readOnly: true + - name: spire-agent-socket + mountPath: /tmp/spire-agent/public + readOnly: false + - name: spire-token + mountPath: /var/run/secrets/tokens + livenessProbe: + httpGet: + path: /live + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /ready + port: 8080 + initialDelaySeconds: 10 + periodSeconds: 10 + volumes: + - name: spire-config + configMap: + name: spire-agent + - name: spire-bundle + configMap: + name: spire-bundle + - name: spire-agent-socket + hostPath: + path: /run/spire/agent-sockets + type: DirectoryOrCreate + - name: spire-token + projected: + sources: + - serviceAccountToken: + path: spire-agent + expirationSeconds: 7200 + audience: spire-server diff --git a/k8s/k8s-workload-registrar/mode-webhook/k8s/spire-server.yaml b/k8s/k8s-workload-registrar/mode-webhook/k8s/spire-server.yaml new file mode 100644 index 0000000..8639742 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-webhook/k8s/spire-server.yaml @@ -0,0 +1,313 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: spire-server + namespace: spire + +--- + +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spire-server-cluster-role +rules: +- apiGroups: [""] + resources: ["nodes"] + verbs: ["get"] + # allow TokenReview requests (to verify service account tokens for PSAT + # attestation) +- apiGroups: ["authentication.k8s.io"] + resources: ["tokenreviews"] + verbs: ["get", "create"] + +--- + +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spire-server-cluster-role-binding + namespace: spire +subjects: +- kind: ServiceAccount + name: spire-server + namespace: spire +roleRef: + kind: ClusterRole + name: spire-server-cluster-role + apiGroup: rbac.authorization.k8s.io + +--- + +kind: Role +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + namespace: spire + name: spire-server-role +rules: +- apiGroups: [""] + resources: ["pods"] + verbs: ["get"] +- apiGroups: [""] + resources: ["configmaps"] + resourceNames: ["spire-bundle"] + verbs: ["get", "patch"] + +--- + +kind: RoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: spire-server-role-binding + namespace: spire +subjects: +- kind: ServiceAccount + name: spire-server + namespace: spire +roleRef: + kind: Role + name: spire-server-role + apiGroup: rbac.authorization.k8s.io + +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: spire-bundle + namespace: spire + +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: spire-server + namespace: spire +data: + server.conf: | + server { + bind_address = "0.0.0.0" + bind_port = "8081" + trust_domain = "example.org" + data_dir = "/run/spire/data" + log_level = "DEBUG" + default_svid_ttl = "1h" + ca_ttl = "12h" + registration_uds_path = "/tmp/spire-server/private/api.sock" + ca_subject { + country = ["US"] + organization = ["SPIFFE"] + common_name = "" + } + } + + plugins { + DataStore "sql" { + plugin_data { + database_type = "sqlite3" + connection_string = "/run/spire/data/datastore.sqlite3" + } + } + + NodeAttestor "k8s_psat" { + plugin_data { + clusters = { + "demo-cluster" = { + service_account_whitelist = ["spire:spire-agent"] + } + } + } + } + + KeyManager "disk" { + plugin_data { + keys_path = "/run/spire/data/keys.json" + } + } + + Notifier "k8sbundle" { + plugin_data { + # This plugin updates the bundle.crt value in the spire:spire-bundle + # ConfigMap by default, so no additional configuration is necessary. + } + } + } + + health_checks { + listener_enabled = true + bind_address = "0.0.0.0" + bind_port = "8080" + live_path = "/live" + ready_path = "/ready" + } + +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: k8s-workload-registrar + namespace: spire +data: + k8s-workload-registrar.conf: | + trust_domain = "example.org" + server_socket_path = "/tmp/spire-server/private/api.sock" + cluster = "demo-cluster" + mode = "webhook" + cert_path = "/run/spire/k8s-workload-registrar/certs/server-cert.pem" + key_path = "/run/spire/k8s-workload-registrar/secret/server-key.pem" + cacert_path = "/run/spire/k8s-workload-registrar/certs/cacert.pem" + +--- + +apiVersion: v1 +kind: ConfigMap +metadata: + name: k8s-workload-registrar-certs + namespace: spire +data: + server-cert.pem: | + -----BEGIN CERTIFICATE----- + MIIB5zCCAY6gAwIBAgIIQhiO2hfTsKQwCgYIKoZIzj0EAwIwJDEiMCAGA1UEAxMZ + SzhTIFdPUktMT0FEIFJFR0lTVFJBUiBDQTAgFw0xOTA1MTMxOTE0MjNaGA85OTk5 + MTIzMTIzNTk1OVowKDEmMCQGA1UEAxMdSzhTIFdPUktMT0FEIFJFR0lTVFJBUiBT + RVJWRVIwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAARUw4pHmWCzrfjkXsen9+mS + PzifWV43s6SZ1J++xvDXM2jOhM86Vp/RdC0m2FNj3WYg6suRlEzvf/Fw2CsuZbmo + o4GjMIGgMA4GA1UdDwEB/wQEAwIDqDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNV + HRMBAf8EAjAAMB0GA1UdDgQWBBS+rw+LUFZAT45Ia8SnrfdWOBtAAzAfBgNVHSME + GDAWgBSbrFJP/YFLhe+FSq6yHYEsvS9EwjArBgNVHREEJDAigiBrOHMtd29ya2xv + YWQtcmVnaXN0cmFyLnNwaXJlLnN2YzAKBggqhkjOPQQDAgNHADBEAiBSaDzjPws6 + Kt68mcJGAYBuWasdgdXJXeySzcnfieXe5AIgXwwaeq+deuF4+ckEY6WIzNWoIPOd + SDoLJWybQN17R0M= + -----END CERTIFICATE----- + + cacert.pem: | + -----BEGIN CERTIFICATE----- + MIIBgTCCASigAwIBAgIIVLxbHbQsZQMwCgYIKoZIzj0EAwIwJDEiMCAGA1UEAxMZ + SzhTIFdPUktMT0FEIFJFR0lTVFJBUiBDQTAgFw0xOTA1MTMxOTE0MjNaGA85OTk5 + MTIzMTIzNTk1OVowJDEiMCAGA1UEAxMZSzhTIFdPUktMT0FEIFJFR0lTVFJBUiBD + QTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABJNq7IL77XWiWbohBOsmrCKMj+g3 + z/+U0c5HmXRj7lbSpjofS0Y1RkTHMEJSvAoMHzssCe5/MDMHX5Xnn4r/LSGjQjBA + MA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSbrFJP + /YFLhe+FSq6yHYEsvS9EwjAKBggqhkjOPQQDAgNHADBEAiBaun9z1WGCSkjx4P+x + mhZkiu1HsOifT9SGQx3in48OSgIgJm02lvnuuKcO/YT2CGHqZ7QjGAnJQY6uLgEQ + 7CXLvcI= + -----END CERTIFICATE----- + +--- + +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: spire-server + namespace: spire + labels: + app: spire-server +spec: + replicas: 1 + selector: + matchLabels: + app: spire-server + serviceName: spire-server + template: + metadata: + namespace: spire + labels: + app: spire-server + spec: + serviceAccountName: spire-server + shareProcessNamespace: true + containers: + - name: spire-server + image: gcr.io/spiffe-io/spire-server:0.12.0 + imagePullPolicy: IfNotPresent + args: ["-config", "/run/spire/config/server.conf"] + ports: + - containerPort: 8081 + volumeMounts: + - name: spire-config + mountPath: /run/spire/config + readOnly: true + - name: spire-server-socket + mountPath: /tmp/spire-server/private + readOnly: false + livenessProbe: + httpGet: + path: /live + port: 8080 + initialDelaySeconds: 5 + periodSeconds: 5 + readinessProbe: + httpGet: + path: /ready + port: 8080 + initialDelaySeconds: 5 + periodSeconds: 5 + - name: k8s-workload-registrar + image: gcr.io/spiffe-io/k8s-workload-registrar:0.12.0 + imagePullPolicy: IfNotPresent + args: ["-config", "/run/spire/k8s-workload-registrar/conf/k8s-workload-registrar.conf"] + ports: + - containerPort: 8443 + name: registrar-port + volumeMounts: + - name: spire-server-socket + mountPath: /tmp/spire-server/private + readOnly: true + - name: k8s-workload-registrar + mountPath: /run/spire/k8s-workload-registrar/conf + readOnly: true + - name: k8s-workload-registrar-certs + mountPath: /run/spire/k8s-workload-registrar/certs + readOnly: true + - name: k8s-workload-registrar-secret + mountPath: /run/spire/k8s-workload-registrar/secret + readOnly: true + volumes: + - name: spire-config + configMap: + name: spire-server + - name: spire-server-socket + hostPath: + path: /run/spire/server-sockets + type: DirectoryOrCreate + - name: k8s-workload-registrar + configMap: + name: k8s-workload-registrar + - name: k8s-workload-registrar-certs + configMap: + name: k8s-workload-registrar-certs + - name: k8s-workload-registrar-secret + secret: + secretName: k8s-workload-registrar-secret + +--- + +apiVersion: v1 +kind: Service +metadata: + name: spire-server + namespace: spire +spec: + type: NodePort + ports: + - name: grpc + port: 8081 + targetPort: 8081 + protocol: TCP + selector: + app: spire-server + +--- + +apiVersion: v1 +kind: Service +metadata: + name: k8s-workload-registrar + namespace: spire +spec: + selector: + app: spire-server + ports: + - port: 443 + targetPort: registrar-port diff --git a/k8s/k8s-workload-registrar/mode-webhook/k8s/validation-webhook.yaml b/k8s/k8s-workload-registrar/mode-webhook/k8s/validation-webhook.yaml new file mode 100644 index 0000000..5509aba --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-webhook/k8s/validation-webhook.yaml @@ -0,0 +1,21 @@ +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingWebhookConfiguration +metadata: + name: k8s-workload-registrar-webhook +webhooks: + - name: k8s-workload-registrar.spire.svc + clientConfig: + service: + name: k8s-workload-registrar + namespace: spire + path: "/validate" + caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJnVENDQVNpZ0F3SUJBZ0lJVkx4YkhiUXNaUU13Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF4TVoKU3poVElGZFBVa3RNVDBGRUlGSkZSMGxUVkZKQlVpQkRRVEFnRncweE9UQTFNVE14T1RFME1qTmFHQTg1T1RrNQpNVEl6TVRJek5UazFPVm93SkRFaU1DQUdBMVVFQXhNWlN6aFRJRmRQVWt0TVQwRkVJRkpGUjBsVFZGSkJVaUJEClFUQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJKTnE3SUw3N1hXaVdib2hCT3NtckNLTWorZzMKei8rVTBjNUhtWFJqN2xiU3Bqb2ZTMFkxUmtUSE1FSlN2QW9NSHpzc0NlNS9NRE1IWDVYbm40ci9MU0dqUWpCQQpNQTRHQTFVZER3RUIvd1FFQXdJQmhqQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCU2JyRkpQCi9ZRkxoZStGU3E2eUhZRXN2UzlFd2pBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlCYXVuOXoxV0dDU2tqeDRQK3gKbWhaa2l1MUhzT2lmVDlTR1F4M2luNDhPU2dJZ0ptMDJsdm51dUtjTy9ZVDJDR0hxWjdRakdBbkpRWTZ1TGdFUQo3Q1hMdmNJPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== + admissionReviewVersions: + - v1beta1 + rules: + - apiGroups: [""] + apiVersions: ["v1"] + operations: ["CREATE", "DELETE"] + resources: ["pods"] + scope: "Namespaced" + sideEffects: None diff --git a/k8s/k8s-workload-registrar/mode-webhook/k8s/workload.yaml b/k8s/k8s-workload-registrar/mode-webhook/k8s/workload.yaml new file mode 100644 index 0000000..9aad1c7 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-webhook/k8s/workload.yaml @@ -0,0 +1,35 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: example-workload + namespace: spire + labels: + app: example-workload +spec: + selector: + matchLabels: + app: example-workload + template: + metadata: + namespace: spire + labels: + app: example-workload + spec: + hostPID: true + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: example-workload + image: gcr.io/spiffe-io/spire-agent:0.12.0 + imagePullPolicy: IfNotPresent + command: ["/usr/bin/dumb-init", "/opt/spire/bin/spire-agent", "api", "watch"] + args: ["-socketPath", "/tmp/spire-agent/public/api.sock"] + volumeMounts: + - name: spire-agent-socket + mountPath: /tmp/spire-agent/public + readOnly: true + volumes: + - name: spire-agent-socket + hostPath: + path: /run/spire/agent-sockets + type: Directory diff --git a/k8s/k8s-workload-registrar/mode-webhook/scripts/delete-scenario.sh b/k8s/k8s-workload-registrar/mode-webhook/scripts/delete-scenario.sh new file mode 100755 index 0000000..c10e22e --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-webhook/scripts/delete-scenario.sh @@ -0,0 +1,11 @@ +#!/bin/bash +PARENT_DIR="$(dirname "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )")" + +kubectl delete -f "${PARENT_DIR}"/k8s/workload.yaml --ignore-not-found + +kubectl delete -f "${PARENT_DIR}"/k8s/spire-agent.yaml --ignore-not-found +kubectl delete -f "${PARENT_DIR}"/k8s/validation-webhook.yaml --ignore-not-found + +kubectl delete -f "${PARENT_DIR}"/k8s/spire-server.yaml --ignore-not-found +kubectl delete -f "${PARENT_DIR}"/k8s/k8s-workload-registrar-secret.yaml --ignore-not-found +kubectl delete -f "${PARENT_DIR}"/k8s/namespace.yaml --ignore-not-found diff --git a/k8s/k8s-workload-registrar/mode-webhook/scripts/deploy-scenario.sh b/k8s/k8s-workload-registrar/mode-webhook/scripts/deploy-scenario.sh new file mode 100755 index 0000000..e54a646 --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-webhook/scripts/deploy-scenario.sh @@ -0,0 +1,31 @@ +#!/bin/bash +DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" +PARENT_DIR="$(dirname "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )")" + +if [ -n "${TRAVIS}" ]; then + minikube stop + sudo cp -R "${PARENT_DIR}"/k8s/admctrl /var/lib/minikube/certs/ + minikube start --driver=none --bootstrapper=kubeadm --extra-config=apiserver.admission-control-config-file=/var/lib/minikube/certs/admctrl/admission-control.yaml +else + docker cp "${PARENT_DIR}"/k8s/admctrl minikube:/var/lib/minikube/certs/ + minikube stop + minikube start \ + --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/sa.key \ + --extra-config=apiserver.service-account-key-file=/var/lib/minikube/certs/sa.pub \ + --extra-config=apiserver.service-account-issuer=api \ + --extra-config=apiserver.service-account-api-audiences=api,spire-server \ + --extra-config=apiserver.authorization-mode=Node,RBAC \ + --extra-config=apiserver.admission-control-config-file=/var/lib/minikube/certs/admctrl/admission-control.yaml +fi + +kubectl apply -f "${PARENT_DIR}"/k8s/namespace.yaml +kubectl apply -f "${PARENT_DIR}"/k8s/k8s-workload-registrar-secret.yaml +kubectl apply -f "${PARENT_DIR}"/k8s/spire-server.yaml +kubectl rollout status statefulset/spire-server -n spire + +kubectl apply -f "${PARENT_DIR}"/k8s/validation-webhook.yaml +kubectl apply -f "${PARENT_DIR}"/k8s/spire-agent.yaml +kubectl rollout status daemonset/spire-agent -n spire + +kubectl apply -f "${PARENT_DIR}"/k8s/workload.yaml +kubectl rollout status deployment/example-workload -n spire diff --git a/k8s/k8s-workload-registrar/mode-webhook/test.sh b/k8s/k8s-workload-registrar/mode-webhook/test.sh new file mode 100755 index 0000000..7751c6d --- /dev/null +++ b/k8s/k8s-workload-registrar/mode-webhook/test.sh @@ -0,0 +1,53 @@ +#!/bin/bash + +DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + +bold=$(tput bold) || true +norm=$(tput sgr0) || true +red=$(tput setaf 1) || true +green=$(tput setaf 2) || true + +set_env() { + echo "${bold}Setting up webhook environment...${norm}" + "${DIR}"/scripts/deploy-scenario.sh > /dev/null +} + +cleanup() { + echo "${bold}Cleaning up...${norm}" + "${DIR}"/scripts/delete-scenario.sh > /dev/null +} + +trap cleanup EXIT + +cleanup +set_env + +NODE_SPIFFE_ID="spiffe://example.org/k8s-workload-registrar/demo-cluster/node" +AGENT_SPIFFE_ID="spiffe://example.org/ns/spire/sa/spire-agent" +WORKLOAD_SPIFFE_ID="spiffe://example.org/ns/spire/sa/default" + +MAX_FETCH_CHECKS=60 +FETCH_CHECK_INTERVAL=5 + +for ((i=0;i<"$MAX_FETCH_CHECKS";i++)); do + if [[ -n $(kubectl exec -t statefulset/spire-server -n spire -c spire-server -- \ + /opt/spire/bin/spire-server entry show -registrationUDSPath /tmp/spire-server/private/api.sock \ + | grep "$NODE_SPIFFE_ID") ]] && + [[ -n $(kubectl exec -t daemonset/spire-agent -n spire -c spire-agent -- \ + /opt/spire/bin/spire-agent api fetch -socketPath /tmp/spire-agent/public/api.sock \ + | grep "$AGENT_SPIFFE_ID") ]] && + [[ -n $(kubectl exec -t deployment/example-workload -n spire -- \ + /opt/spire/bin/spire-agent api fetch -socketPath /tmp/spire-agent/public/api.sock \ + | grep "$WORKLOAD_SPIFFE_ID") ]]; then + DONE=1 + break + fi + sleep "$FETCH_CHECK_INTERVAL" +done + +if [ "${DONE}" -eq 1 ]; then + exit 0 +else + echo "${red}Webhook mode test failed.${norm}" + exit 1 +fi diff --git a/k8s/k8s-workload-registrar/test.sh b/k8s/k8s-workload-registrar/test.sh new file mode 100755 index 0000000..1f052af --- /dev/null +++ b/k8s/k8s-workload-registrar/test.sh @@ -0,0 +1,30 @@ +#!/bin/bash + +DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + +bold=$(tput bold) || true +norm=$(tput sgr0) || true +red=$(tput setaf 1) || true +green=$(tput setaf 2) || true + +fail() { + echo "${red}$*${norm}." + exit 1 +} + +for testdir in "${DIR}"/*; do + if [[ -x "${testdir}/test.sh" ]]; then + testname=$(basename "$testdir") + echo "${bold}Running \"$testname\" test...${norm}" + if ${testdir}/test.sh; then + echo "${green}\"$testname\" test succeeded${norm}" + else + echo "${red}\"$testname\" test failed${norm}" + FAILED=true + fi + fi +done + +if [ -n "${FAILED}" ]; then + fail "There were test failures" +fi