error creating training portal - secret already exists

[djschny]

Describe the bug

1. Delete training portal CRD
2. Create training portal CRD with same name

Observe training portal does not start and there are errors in the session manager logs.

Additional information

Session Mananager logs:

INFO:educates.trainingportal:Training portal creation request for tanzu-snowflake being processed.
ERROR:educates.trainingportal:Unexpected error creating training portal tanzu-snowflake.
Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.13/site-packages/pykube/http.py", line 437, in raise_for_status
    resp.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/app-root/venv/lib64/python3.13/site-packages/requests/models.py", line 1026, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: https://10.2.0.1:443/apis/secrets.educates.dev/v1beta1/secretcopiers

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/app-root/src/handlers/trainingportal.py", line 1043, in training_portal_create
    SecretCopier(api, ingress_secret_copier_body).create()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/app-root/venv/lib64/python3.13/site-packages/pykube/objects.py", line 131, in create
    self.api.raise_for_status(r)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "/opt/app-root/venv/lib64/python3.13/site-packages/pykube/http.py", line 444, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
pykube.exceptions.HTTPError: object is being deleted: secretcopiers.secrets.educates.dev "educates-ingress-secret-tanzu-snowflake-ui" already exists

[jorgemoralespou]

Hi @djschny
I'm assuming this is still for educates 3.3.3?
Also, what is the time between deletion of the trainingportal CR and the creation of the same trainingportal CR?
Is there something holding to that secret so that it can not be deleted? It seems as when the trainingportal is deleted that specific secret (which I assume is created by a secretcopier in a workshop.session.objects) is not deleted probably because something in kubernetes is preventing the deletion. Can you verify this?

[djschny]

I'm assuming this is still for educates 3.3.3?

That is correct.

I will look to see if I can find the information you're requesting.

[djschny]

what is the time between deletion of the trainingportal CR and the creation of the same trainingportal CR?

According to the logs it looks like approximately 5 seconds?

Is there something holding to that secret so that it can not be deleted?

Not that I'm aware of, all sessions were deleted and nothing else holding onto it from what I can tell.

See logs below for more details:

2026-08-17T12:04:48.512777456Z INFO:educates.trainingportal:Training portal tanzu-snowflake has been deleted.
2026-08-17T12:04:48.566646577Z INFO:educates.workshopenvironment:Workshop environment tanzu-snowflake-w01 has been deleted.
2026-08-17T12:04:48.567291642Z INFO:educates.workshopenvironment:Workshop environment tanzu-snowflake-w02 has been deleted.
2026-08-17T12:04:48.607636295Z INFO:educates.workshopsession:Workshop session tanzu-snowflake-w01-s005 has been deleted.
2026-08-17T12:04:48.607852252Z INFO:educates.workshopsession:Workshop session tanzu-snowflake-w01-s004 has been deleted.
2026-08-17T12:04:48.652578462Z INFO:educates.workshopallocation:Workshop allocation request tanzu-snowflake-w01-s004 has been deleted.
2026-08-17T12:04:49.921153597Z INFO:educates.workshopsession:Workshop session tanzu-snowflake-w01-s006 has been deleted.
2026-08-17T12:04:53.876989360Z INFO:educates.trainingportal:Training portal pod training-portal-69d859599c-7j5vb in namespace tanzu-snowflake-ui has been modified with current status of Running.
2026-08-17T12:04:55.275922906Z INFO:educates.workshopsession:Workshop session pod tanzu-snowflake-w01-s005-5b89449f48-m5gds in namespace tanzu-snowflake-w01 has been modified with current status of Pending.
2026-08-17T12:04:55.286205215Z INFO:educates.workshopsession:Workshop session pod tanzu-snowflake-w01-s004-69ffff7f6-kgzz8 in namespace tanzu-snowflake-w01 has been modified with current status of Pending.
2026-08-17T12:04:55.395074551Z INFO:educates.workshopsession:Workshop session pod tanzu-snowflake-w01-s005-5b89449f48-m5gds in namespace tanzu-snowflake-w01 has been modified with current status of Pending.
2026-08-17T12:04:55.440864679Z INFO:educates.workshopsession:Workshop session pod tanzu-snowflake-w01-s004-69ffff7f6-kgzz8 in namespace tanzu-snowflake-w01 has been modified with current status of Pending.
2026-08-17T12:04:55.686343020Z INFO:educates.workshopsession:Workshop session pod tanzu-snowflake-w01-s004-69ffff7f6-kgzz8 in namespace tanzu-snowflake-w01 has been modified with current status of Pending.
2026-08-17T12:04:55.695868849Z INFO:educates.workshopsession:Workshop session pod tanzu-snowflake-w01-s005-5b89449f48-m5gds in namespace tanzu-snowflake-w01 has been modified with current status of Pending.
2026-08-17T12:04:55.868569812Z INFO:educates.workshopsession:Workshop session pod tanzu-snowflake-w01-s005-5b89449f48-m5gds in namespace tanzu-snowflake-w01 has been modified with current status of Pending.
2026-08-17T12:04:55.914481146Z INFO:educates.workshopsession:Workshop session pod tanzu-snowflake-w01-s004-69ffff7f6-kgzz8 in namespace tanzu-snowflake-w01 has been modified with current status of Pending.
2026-08-17T12:04:56.485790117Z INFO:educates.workshopsession:Workshop session pod tanzu-snowflake-w01-s005-5b89449f48-m5gds in namespace tanzu-snowflake-w01 has been deleted with current status of Failed.
2026-08-17T12:04:56.606039535Z INFO:educates.workshopsession:Workshop session pod tanzu-snowflake-w01-s004-69ffff7f6-kgzz8 in namespace tanzu-snowflake-w01 has been deleted with current status of Failed.
2026-08-17T12:04:57.290701267Z INFO:educates.trainingportal:Training portal pod training-portal-69d859599c-7j5vb in namespace tanzu-snowflake-ui has been modified with current status of Running.
2026-08-17T12:04:57.414142938Z INFO:educates.trainingportal:Training portal pod training-portal-69d859599c-7j5vb in namespace tanzu-snowflake-ui has been modified with current status of Succeeded.
2026-08-17T12:04:58.097132172Z INFO:educates.trainingportal:Training portal pod training-portal-69d859599c-7j5vb in namespace tanzu-snowflake-ui has been deleted with current status of Succeeded.
2026-08-17T12:05:26.673082576Z INFO:educates.trainingportal:Training portal creation request for tanzu-snowflake being processed.
2026-08-17T12:05:26.801849659Z ERROR:educates.trainingportal:Unexpected error creating training portal tanzu-snowflake.
2026-08-17T12:05:26.801928541Z Traceback (most recent call last):
2026-08-17T12:05:26.801937102Z   File "/opt/app-root/venv/lib64/python3.13/site-packages/pykube/http.py", line 437, in raise_for_status
2026-08-17T12:05:26.801942765Z     resp.raise_for_status()
2026-08-17T12:05:26.801947597Z     ~~~~~~~~~~~~~~~~~~~~~^^
2026-08-17T12:05:26.801952229Z   File "/opt/app-root/venv/lib64/python3.13/site-packages/requests/models.py", line 1026, in raise_for_status
2026-08-17T12:05:26.801958508Z     raise HTTPError(http_error_msg, response=self)
2026-08-17T12:05:26.801964076Z requests.exceptions.HTTPError: 409 Client Error: Conflict for url: https://10.2.0.1:443/apis/secrets.educates.dev/v1beta1/secretcopiers
2026-08-17T12:05:26.801968427Z 
2026-08-17T12:05:26.801973576Z During handling of the above exception, another exception occurred:
2026-08-17T12:05:26.801977740Z 
2026-08-17T12:05:26.801982708Z Traceback (most recent call last):
2026-08-17T12:05:26.801987832Z   File "/opt/app-root/src/handlers/trainingportal.py", line 1043, in training_portal_create
2026-08-17T12:05:26.801992718Z     SecretCopier(api, ingress_secret_copier_body).create()
2026-08-17T12:05:26.801997612Z     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
2026-08-17T12:05:26.802030523Z   File "/opt/app-root/venv/lib64/python3.13/site-packages/pykube/objects.py", line 131, in create
2026-08-17T12:05:26.802035155Z     self.api.raise_for_status(r)
2026-08-17T12:05:26.802039631Z     ~~~~~~~~~~~~~~~~~~~~~~~~~^^^
2026-08-17T12:05:26.802044024Z   File "/opt/app-root/venv/lib64/python3.13/site-packages/pykube/http.py", line 444, in raise_for_status
2026-08-17T12:05:26.802048868Z     raise HTTPError(resp.status_code, payload["message"])
2026-08-17T12:05:26.802055106Z pykube.exceptions.HTTPError: object is being deleted: secretcopiers.secrets.educates.dev "educates-ingress-secret-tanzu-snowflake-ui" already exists
2026-08-17T12:05:26.802743582Z ERROR:kopf.objects:Handler 'training_portal_create' failed temporarily: Unexpected error creating training portal tanzu-snowflake.
2026-08-17T12:05:56.966180394Z INFO:educates.trainingportal:Training portal creation request for tanzu-snowflake being retried, retries 1.
2026-08-17T12:05:57.012736393Z ERROR:kopf.objects:Handler 'training_portal_create' failed temporarily: Deleting tanzu-snowflake-ui and retrying.
2026-08-17T12:06:27.176741497Z INFO:educates.trainingportal:Training portal creation request for tanzu-snowflake being retried, retries 2.

[GrahamDumpleton]

I'll explain properly later the reasons, but the short of it is that if using kubectl to delete the training portal, you should use the --cascade='foreground' option if using scripting to immediately create straight away after.

[djschny]

I didn't see that mentioned in the docs, my apologies if I missed it.

[GrahamDumpleton]

There did use to be something about it in the docs, but it seems it vanished when the getting started guide was changed to demonstrate the CLI and not direct use of TrainingPortal. :-(

[GrahamDumpleton]

Here is the explanation of what is occurring.

Unlike a lot of Kubernetes operators, the Educates session-manager operator does not use a reconciliation model whereby it is always trying to ensure what is in the cluster is what it thinks should be there. Also, on deletion, it does not react to the deletion of the holding custom resources, eg., TrainingPortal, and perform the deletion of any resources created because of the custom resource. Instead, all resources created because of TrainingPortal have their owner references set to be the TrainingPortal. That way when the TrainingPortal resource is deleted, all child resources will be automatically cleaned up.

The consequence of this is that if you run plain kubectl delete on TrainingPortal, deletion of everything effectively happens in parallel and so technically the TrainingPortal resource may not be the last resource to be deleted. As such, it may be possible to create a new TrainingPortal of the same name while there are still resources from prior instance still not cleaned up. For TrainingPortal, since you use the same name, the child resources will also have the same name.

Usually this is not a problem because most things are in the namespace which holds the training portal instance. The session-manager operator is written to detect if the prior namespace has not been cleaned up and will backoff and retry later. In other words, the existence of the prior namespace gates it doing anything for the new instance.

Beyond the namespace though there is also a SecretCopier resource (can't remember if any others) connected to the TrainingPortal.

Usually it would take a lot longer to clean up the prior namespace for the training portal instance than the SecretCopier, so gating on namespace means you are all good. Seems though that in your case the SecretCopier took the longest to be deleted and so the operator made it past the namespace gate and then failed when creating the SecretCopier since prior one was still being deleted.

Since SecretCopier is created by the operator after the namespace gate, one solution would be to use the generateName mechanism on the SecretCopier created so that it gets the typical 5 character random sequence at the end. That way it would not clash with the old SecretCopier. If there are other cluster scoped resources for a TrainingPortal which are created, they could perhaps be treated the same way.

As a I mentioned, the operational change you can make if using kubectl delete, is to tell it to wait until all resources are deleted, which is done with --cascade='foreground'.

[djschny]

FYI - I tried with the --cascade='foreground' option and unfortunately the secretcopier still exists after returning.

[djschny]

FYI - ultimately it required the session-manager to be restarted for things to start working.

[GrahamDumpleton]

You definitely aren't creating the SecretCopier called educates-ingress-secret-tanzu-snowflake-ui yourself using automation? In typical config you would tell Educates the location of your secret and it would create the SecretCopier. In some situations you may not want to rely on that mechanism and instead create the SecretCopier yourself. If you had the latter, but then decided to rely on the more typical setup, then maybe you have a conflict over who owns it.

What do the owner references in the SecretCopier have? If managed by Educates, I would expect the owner references to be back to TrainingPortal. If they are missing, would imply manually managed. If the owner reference is there, I don't understand how foreground deletion wouldn't just block on deleting TrainingPortal until the SecretCopier had been deleted, which logically should then prevent you from creating a new TrainingPortal. Either way, restarting session-manager shouldn't make a difference, since SecretCopier is managed by secrets-manager. Even restart secrets-manager shouldn't do anything unless there was an issue with cleanup of SecretCopier as for reasons can't remember exactly it may be attaching a finalizer to the SecretCopier and thus the on delete handler in the operator must run to properly release the SecretCopier.

[djschny]

You definitely aren't creating the SecretCopier called educates-ingress-secret-tanzu-snowflake-ui yourself using automation?

Negative.

What do the owner references in the SecretCopier have?

  Owner References:
    API Version:           v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Namespace
    Name:                  tanzu-snowflake-ui
    UID:                   ca9da063-4698-4a46-9730-0738c196b73b

[GrahamDumpleton]

Ahhh, forgot the relationship is TrainingPortal->namespace->SecretCopier. Still should mean that foreground deletion would block TrainingPortal. Only possibility is that forgetting block owner deletion flag as true on something.