[Storage] [Webjobs] Implementing lightweight detection for new write operations

[amnguye]

From #52977 @alrod :

"This PR introduces a streamlined approach to zero-to-one scaling logic within the Azure SDK. The current implementation reads and parses all classic monitoring logs in the $log container to extract modified blob paths. However, for scale-out from zero to one, this level of detail is unnecessary.
The updated logic simplifies the process by checking only for the presence of write operations within the last two hours—without downloading or parsing the full log content. This optimization reduces memory consumption and improves performance, especially in environments with high write volumes."

[Copilot]

Pull Request Overview

This PR implements a lightweight detection mechanism for new write operations in Azure Storage blob monitoring, optimizing the zero-to-one scaling logic. Instead of reading and parsing complete log files, it now checks only for the presence of write operations in metadata within a specified time window.

Key changes:

• Added a new HasBlobWritesAsync method that checks for write operations without downloading log content
• Simplified the zero-to-one scaling monitor to use the lightweight detection
• Updated logging messages and removed debugging-related code

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
BlobLogListener.cs	Added HasBlobWritesAsync method for lightweight write detection and made constructor public
BlobScalerMonitorProvider.cs	Replaced detailed write enumeration with simple boolean check and updated logging
BlobLogListenerTests.cs	Added comprehensive tests for the new HasBlobWritesAsync method with various scenarios

[Copilot]

The null-forgiving operator (!) is used without null checks. If these properties don't exist, this will throw at runtime. Add proper null validation or use more defensive reflection patterns.

Suggested change

	private static readonly System.Reflection.PropertyInfo NameProp = BlobItemType.GetProperty(nameof(BlobItem.Name))!;
	private static readonly System.Reflection.PropertyInfo MetadataProp = BlobItemType.GetProperty(nameof(BlobItem.Metadata))!;
	private static readonly System.Reflection.PropertyInfo NameProp = BlobItemType.GetProperty(nameof(BlobItem.Name));
	private static readonly System.Reflection.PropertyInfo MetadataProp = BlobItemType.GetProperty(nameof(BlobItem.Metadata));
	static BlobItemFactory()
	{
	if (NameProp == null)
	{
	throw new InvalidOperationException($"Property '{nameof(BlobItem.Name)}' not found on type '{BlobItemType.FullName}'.");
	}
	if (MetadataProp == null)
	{
	throw new InvalidOperationException($"Property '{nameof(BlobItem.Metadata)}' not found on type '{BlobItemType.FullName}'.");
	}
	}

[mathewc]

Looks like this isn't really used and can be removed

[mathewc]

How did you choose this 200 - is it a default we use elsewhere?

[alrod]

I don’t believe paging is necessary here. When I tested uploading 100K blobs, it only generated three logs. I’m removing the paging logic. The blob log itself is quite large—tens of megabytes.

[mathewc]

nit: could make this a little more succinct:

if (blob.Metadata?.TryGetValue(LogType, out string logType) &&
    logType?.Contains("write", StringComparison.OrdinalIgnoreCase))
{
    return true;
}

[alrod]

This does not compile. Copilot:

The shown replacement is problematic for multiple reasons:

1. Null-conditional invocation result type: blob.Metadata?.TryGetValue(...) returns bool? (nullable bool). You can’t use && with a bool? left operand. Won’t compile.
2. API availability: string.Contains(string, StringComparison) is NOT available in .NET Standard 2.0. This project multi-targets .NET 8 and .NET Standard 2.0, so that overload breaks the netstandard2.0 build. The original IndexOf(..., StringComparison.OrdinalIgnoreCase) works on both.
3. Semantics / safety: The original guarded against null or empty logType explicitly. Your version relies on logType?.Contains, which is fine, but coupled with point (1) & (2) it fails anyway.
4. Consistency: Elsewhere in this file (GetLogsWithPrefixAsync) you have logType.Contains("write") without a StringComparison, which is culture-sensitive. The original pattern (IndexOf + StringComparison.OrdinalIgnoreCase) is the safer, consistent approach.
   Recommended fixed version (compiles for both targets and remains clear):

if (blob.Metadata is not null &&
    blob.Metadata.TryGetValue(LogType, out var logType) &&
    !string.IsNullOrEmpty(logType) &&
    logType.IndexOf("write", StringComparison.OrdinalIgnoreCase) >= 0)
{
    return true;
}

[mathewc]

Not sure if @amnguye knows a better way to mock this stuff?

[alrod]

BlobItem does not have public ctor. It seems there is no other way.

[amnguye]

If you use the BlobsModelFactory we have some constructors of the BlobItem you can mock.

For example:

BlobItem blobItem = BlobsModelFactory.BlobItem(
    name: result.Item1,
    properties: BlobsModelFactory.BlobItemProperties(
        accessTierInferred: true,
        contentLength: 1024,
        blobType: BlobType.Block,
        eTag: new ETag("etag")));
blobHierarchyItems.Add(new BlobHierarchyItem(default, blobItem));

[mathewc]

Thanks - yes that's what I recalled in the past for other SDK types - there was a model factory pattern that could be used

[alrod]

Closed in favor: #53263