Skip to content

[WIP] Create Private Package for Scale Controller#3185

Draft
nytian wants to merge 16 commits intonytian/v3.1.0from
nytian/sc-hotfix-v3.1.0
Draft

[WIP] Create Private Package for Scale Controller#3185
nytian wants to merge 16 commits intonytian/v3.1.0from
nytian/sc-hotfix-v3.1.0

Conversation

@nytian
Copy link
Collaborator

@nytian nytian commented Aug 19, 2025

Issue describing the changes in this PR

Don't merge this PR
This branch is to create private package for Scale Controller and this PR is mainly for test running.

Pull request checklist

  • My changes do not require documentation changes
    • Otherwise: Documentation PR is ready to merge and referenced in pending_docs.md
  • My changes should not be added to the release notes for the next release
    • Otherwise: I've added my notes to release_notes.md
  • My changes do not need to be backported to a previous version
    • Otherwise: Backport tracked by issue/PR #issue_or_pr
  • I have added all required tests (Unit tests, E2E tests)
  • My changes do not require any extra work to be leveraged by OutOfProc SDKs
    • Otherwise: That work is being tracked here: #issue_or_pr_in_each_sdk
  • My changes do not change the version of the WebJobs.Extensions.DurableTask package
    • Otherwise: major or minor version updates are reflected in /src/Worker.Extensions.DurableTask/AssemblyInfo.cs
  • My changes do not add EventIds to our EventSource logs
    • Otherwise: Ensure the EventIds are within the supported range in our existing Windows infrastructure. You may validate this with a deployed app's telemetry. You may also extend the range by completing a PR such as this one.
  • My changes should be added to v2.x branch.
    • Otherwise: This change applies exclusively to WebJobs.Extensions.DurableTask v3.x. It will be retained only in the dev and main branches and will not be merged into the v2.x branch.

nytian and others added 16 commits August 19, 2025 09:55
…TaskTargetScaler

When DurableTaskMetricsProvider.GetMetricsAsync() fails to obtain a heartbeat
(e.g., PulseAsync throws or hasn't run yet), the ControlQueueLengths property
on DurableTaskTriggerMetrics remains null. DurableTaskTargetScaler then passes
this null value to JsonConvert.DeserializeObject, which throws:
  System.ArgumentNullException: Value cannot be null. (Parameter 'value')

This generates ~640 million error events per day in ScaleControllerEvents across
12,000+ function apps.

Fix: Add a null/empty check before deserializing ControlQueueLengths. When null
or empty, treat control queue metrics as zero (no orchestrator messages), which
is the correct behavior since no heartbeat data is available.

Added test cases for null and empty ControlQueueLengths scenarios.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants