Skip to content

fix: infinite status reconciliation loop in notebook controller - #1343

Open
juliusvonkohout wants to merge 2 commits into
notebooks-v1from
notebook-controller-scalability-fix
Open

fix: infinite status reconciliation loop in notebook controller#1343
juliusvonkohout wants to merge 2 commits into
notebooks-v1from
notebook-controller-scalability-fix

Conversation

@juliusvonkohout

@juliusvonkohout juliusvonkohout commented Aug 21, 2026

Copy link
Copy Markdown
Member

Alternative to #1341

The reconcilliation loop creates a lot of logging costs and api server load if you have for example 250 notebooks on a cluster.

│ 1.7873218948448386e+09    INFO    controllers.Notebook    Reconciliation loop started    {"notebook": "my-namespace/my-notebook"}                    │
│ 1.787321894844961e+09    INFO    controllers.Notebook    Updating StatefulSet    {"notebook": "my-namespace/my-notebook", "namespace": "xdiv-finance │
│ -ior-prediction-dev", "name": "daniel"}                                                                                                                            │
│ 1.7873218949141026e+09    INFO    controllers.Notebook    Initializing Notebook CR Status    {"notebook": "my-namespace/my-notebook"}                │
│ 1.7873218949141288e+09    INFO    controllers.Notebook    Calculating Notebook's  containerState    {"notebook": "my-namespace/my-notebook"}         │
│ 1.7873218949141347e+09    INFO    controllers.Notebook    Updating Notebook CR state:     {"notebook": "my-namespace/my-notebook", "state": {"runnin │
│ g":{"startedAt":"2026-08-18T13:11:00Z"}}}                                                                                                                          │
│ 1.7873218949141548e+09    INFO    controllers.Notebook    Calculating Notebook's Conditions    {"notebook": "my-namespace/my-notebook"}              │
│ 1.787321894914165e+09    INFO    controllers.Notebook    Updating Notebook CR Status    {"notebook": "my-namespace/my-notebook", "status": {"conditi │
│ ons":[{"type":"PodReadyToStartContainers","status":"True","lastProbeTime":"2026-08-21T14:18:14Z","lastTransitionTime":"2026-08-18T13:10:58Z"},{"type":"Initialized │
│ ","status":"True","lastProbeTime":"2026-08-21T14:18:14Z","lastTransitionTime":"2026-08-18T13:11:00Z"},{"type":"Ready","status":"True","lastProbeTime":"2026-08-21T │
│ 14:18:14Z","lastTransitionTime":"2026-08-18T13:11:07Z"},{"type":"ContainersReady","status":"True","lastProbeTime":"2026-08-21T14:18:14Z","lastTransitionTime":"202 │
│ 6-08-18T13:11:07Z"},{"type":"PodScheduled","status":"True","lastProbeTime":"2026-08-21T14:18:14Z","lastTransitionTime":"2026-08-18T13:10:37Z"}],"readyReplicas":1, │
│ "containerState":{"running":{"startedAt":"2026-08-18T13:11:00Z"}}}}   

Imagine this continously at full speed without interruption increasing the revisions.

Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com>
@github-project-automation github-project-automation Bot moved this to Needs Triage in Kubeflow Notebooks Aug 21, 2026
@google-oss-prow google-oss-prow Bot added the area/controller area - related to controller components label Aug 21, 2026
@google-oss-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign thesuperzapper for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow Bot added area/v1 area - version - kubeflow notebooks v1 size/L labels Aug 21, 2026
Signed-off-by: juliusvonkohout <45896133+juliusvonkohout@users.noreply.github.com>
@juliusvonkohout

Copy link
Copy Markdown
Member Author

docker.io/jvonkoho/notebook-controller:pr-1343 if you want to test it. It works well in practice.

@christian-heusel christian-heusel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request is currently lacking a few quite basic things, please add them and we can proceed with the review itself:

  1. Motivation: What is the bug that is being fixed here?
  2. Reproducer: How can the bug be reproduced by a developer?
  3. Implementation: How is the submitted changeset fixing the issue?

Also I suspect you used AI in the creation of this change, please add appropriate trailers in order comply with the Kubeflow AI policy 😊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/controller area - related to controller components area/v1 area - version - kubeflow notebooks v1 size/L

Projects

Status: Needs Triage

Development

Successfully merging this pull request may close these issues.

2 participants