feat: improve deployment creation performance by using informer cache… #7128
Conversation
… for status updates
karmada-bot
added
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Summary of ChangesHello @ljluestc, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a significant performance optimization for resource status updates, particularly impacting deployment creation. By leveraging an informer cache for initial resource fetches, it aims to reduce the load on the API server and decrease latency. A robust fallback mechanism ensures data consistency by retrying updates directly from the API server in cases of cache misses or conflicts. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
karmada-bot
added
the
size/M
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a valuable performance optimization by leveraging an informer cache to fetch resources before updating their status, which should effectively reduce the load on the API server. The overall implementation is sound, with a fallback mechanism to ensure correctness. However, I've identified a few areas for improvement concerning code maintainability and minor correctness issues. My review includes suggestions to refactor for better readability and to fix an issue with event recording. Addressing these points will enhance the robustness and long-term maintainability of the code.
| newObj, err := interpreter.AggregateStatus(resource, bindingStatus.AggregatedStatus) | ||
| if err != nil { | ||
| klog.Errorf("Failed to aggregate status for resource template(%s/%s/%s), Error: %v", gvr, resource.GetNamespace(), resource.GetName(), err) | ||
| return err | ||
| } | ||
|
|
||
| oldStatus, _, _ := unstructured.NestedFieldNoCopy(resource.Object, "status") | ||
| newStatus, _, _ := unstructured.NestedFieldNoCopy(newObj.Object, "status") | ||
| if reflect.DeepEqual(oldStatus, newStatus) { | ||
| klog.V(3).Infof("Ignore update resource(%s/%s/%s) status as up to date.", gvr, resource.GetNamespace(), resource.GetName()) | ||
| return nil | ||
| } | ||
|
|
||
| _, err = dynamicClient.Resource(gvr).Namespace(resource.GetNamespace()).UpdateStatus(ctx, newObj, metav1.UpdateOptions{}) | ||
| if err == nil { | ||
| eventRecorder.Event(resource, corev1.EventTypeNormal, events.EventReasonAggregateStatusSucceed, "Update Resource with AggregatedStatus successfully.") | ||
| klog.V(3).Infof("Update resource(%s/%s/%s) status successfully.", gvr, resource.GetNamespace(), resource.GetName()) | ||
| return nil | ||
| } | ||
|
|
||
| if !apierrors.IsConflict(err) { | ||
| klog.Errorf("Failed to update resource(%s/%s/%s), Error: %v", gvr, resource.GetNamespace(), resource.GetName(), err) | ||
| return err | ||
| } | ||
| // If conflict happens, we will retry to fetch the latest resource from apiserver and update again. | ||
| klog.V(3).Infof("Conflict updating resource(%s/%s/%s) from cache, falling back to API server.", gvr, resource.GetNamespace(), resource.GetName()) | ||
| } |
There was a problem hiding this comment.
There is significant code duplication between the new optimistic update path (from cache, lines 191-209) and the fallback path within retry.RetryOnConflict (lines 236-256). The logic for aggregating status, comparing statuses, and updating the resource is repeated.
To improve maintainability and reduce redundancy, consider refactoring the common logic into a helper function. This would make the code cleaner and less error-prone for future modifications.
| _, err = dynamicClient.Resource(gvr).Namespace(resource.GetNamespace()).UpdateStatus(ctx, newObj, metav1.UpdateOptions{}) | ||
| if err == nil { | ||
| eventRecorder.Event(resource, corev1.EventTypeNormal, events.EventReasonAggregateStatusSucceed, "Update Resource with AggregatedStatus successfully.") | ||
| klog.V(3).Infof("Update resource(%s/%s/%s) status successfully.", gvr, resource.GetNamespace(), resource.GetName()) | ||
| return nil | ||
| } |
There was a problem hiding this comment.
The event is being recorded on the resource object, which is the state of the object before the status update. The UpdateStatus function returns the updated object, and it's a best practice to use this returned object for event recording to ensure the event is associated with the most recent version of the object.
This issue also exists in the retry.RetryOnConflict block on line 254. Please apply a similar fix there.
| _, err = dynamicClient.Resource(gvr).Namespace(resource.GetNamespace()).UpdateStatus(ctx, newObj, metav1.UpdateOptions{}) | |
| if err == nil { | |
| eventRecorder.Event(resource, corev1.EventTypeNormal, events.EventReasonAggregateStatusSucceed, "Update Resource with AggregatedStatus successfully.") | |
| klog.V(3).Infof("Update resource(%s/%s/%s) status successfully.", gvr, resource.GetNamespace(), resource.GetName()) | |
| return nil | |
| } | |
| updatedObj, err := dynamicClient.Resource(gvr).Namespace(resource.GetNamespace()).UpdateStatus(ctx, newObj, metav1.UpdateOptions{}) | |
| if err == nil { | |
| eventRecorder.Event(updatedObj, corev1.EventTypeNormal, events.EventReasonAggregateStatusSucceed, "Update Resource with AggregatedStatus successfully.") | |
| klog.V(3).Infof("Update resource(%s/%s/%s) status successfully.", gvr, resource.GetNamespace(), resource.GetName()) | |
| return nil | |
| } |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #7128 +/- ##
==========================================
- Coverage 46.54% 46.52% -0.02%
==========================================
Files 700 700
Lines 48134 48163 +29
==========================================
+ Hits 22403 22410 +7
- Misses 24045 24065 +20
- Partials 1686 1688 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
3 participants
What type of PR is this?
/kind feature
What this PR does / why we need it:
Performance improvement for create deployment.
Currently,
updateResourceStatusinpkg/controllers/status/common.goperforms a direct APIGetandUpdateon the resource status for every Work status update. This bypasses the controller-runtime cache, leading to higher latency and increased load on the API server, especially when creating a large number of deployments concurrently.This PR optimizes
updateResourceStatusto attempt fetching the resource from the informer cache first.InformerManager.This "optimistic cache fetch" strategy significantly reduces the number of read operations against the API server during normal operation, improving performance for high-concurrency scenarios.
Which issue(s) this PR fixes:
Fixes #4225
Special notes for your reviewer:
Verified with
go test ./pkg/controllers/status/.... The retry logic ensures data consistency even if the cache is stale.Does this PR introduce a user-facing change?: