Skip to content

fix: Improve usage reporting #1108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jsonbailey wants to merge 6 commits into
base: main
Choose a base branch
from
Open

fix: Improve usage reporting #1108

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
621be12
[aic-1636] fix: Improve usage reporting
jsonbailey Feb 18, 2026
6b5da53
fix lint issue
jsonbailey Feb 19, 2026
21f6e76
fix lint on test
jsonbailey Feb 19, 2026
ef82f52
Merge branch 'main' into jb/aic-1636/improve-reporting
jsonbailey Feb 19, 2026
903f126
update to follow spec change
jsonbailey Feb 23, 2026
a765834
Merge branch 'main' into jb/aic-1636/improve-reporting
jsonbailey Feb 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 21 additions & 6 deletions packages/sdk/server-ai/__tests__/LDAIClientImpl.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { Judge } from '../src/api/judge/Judge';
import { AIProviderFactory } from '../src/api/providers/AIProviderFactory';
import { LDAIClientImpl } from '../src/LDAIClientImpl';
import { LDClientMin } from '../src/LDClientMin';
import { aiSdkLanguage, aiSdkName, aiSdkVersion } from '../src/sdkInfo';

// Mock Judge and AIProviderFactory
jest.mock('../src/api/judge/Judge');
Expand All @@ -26,6 +27,20 @@ beforeEach(() => {

const testContext: LDContext = { kind: 'user', key: 'test-user' };

describe('init tracking', () => {
it('tracks init in constructor with SDK name, version, and language in data', () => {
const client = new LDAIClientImpl(mockLdClient);
expect(client).toBeDefined();
expect(mockLdClient.track).toHaveBeenNthCalledWith(
1,
'$ld:ai:sdk:info',
{ kind: 'ld_ai', key: 'ld-internal-tracking', anonymous: true },
{ aiSdkName, aiSdkVersion, aiSdkLanguage },
1,
);
});
});

describe('config evaluation', () => {
it('evaluates completion config successfully with variable interpolation', async () => {
const client = new LDAIClientImpl(mockLdClient);
Expand Down Expand Up @@ -403,7 +418,7 @@ describe('completionConfig method', () => {
const result = await client.completionConfig(key, testContext, defaultValue, variables);

expect(mockLdClient.track).toHaveBeenCalledWith(
'$ld:ai:config:function:single',
'$ld:ai:usage:completion-config',
testContext,
key,
1,
Expand Down Expand Up @@ -444,7 +459,7 @@ describe('agentConfig method', () => {
const result = await client.agentConfig(key, testContext, defaultValue, variables);

expect(mockLdClient.track).toHaveBeenCalledWith(
'$ld:ai:agent:function:single',
'$ld:ai:usage:agent-config',
testContext,
key,
1,
Expand Down Expand Up @@ -529,7 +544,7 @@ describe('agents method', () => {
});

expect(mockLdClient.track).toHaveBeenCalledWith(
'$ld:ai:agent:function:multiple',
'$ld:ai:usage:agent-configs',
testContext,
agentConfigs.length,
agentConfigs.length,
Expand All @@ -544,7 +559,7 @@ describe('agents method', () => {
expect(result).toEqual({});

expect(mockLdClient.track).toHaveBeenCalledWith(
'$ld:ai:agent:function:multiple',
'$ld:ai:usage:agent-configs',
testContext,
0,
0,
Expand Down Expand Up @@ -577,7 +592,7 @@ describe('judgeConfig method', () => {
const result = await client.judgeConfig(key, testContext, defaultValue, variables);

expect(mockLdClient.track).toHaveBeenCalledWith(
'$ld:ai:judge:function:single',
'$ld:ai:usage:judge-config',
testContext,
key,
1,
Expand Down Expand Up @@ -633,7 +648,7 @@ describe('createJudge method', () => {
const result = await client.createJudge(key, testContext, defaultValue);

expect(mockLdClient.track).toHaveBeenCalledWith(
'$ld:ai:judge:function:createJudge',
'$ld:ai:usage:create-judge',
testContext,
key,
1,
Expand Down
3 changes: 0 additions & 3 deletions packages/sdk/server-ai/__tests__/LDAIConfigTrackerImpl.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ import { LDContext } from '@launchdarkly/js-server-sdk-common';
import { LDFeedbackKind } from '../src/api/metrics';
import { LDAIConfigTrackerImpl } from '../src/LDAIConfigTrackerImpl';
import { LDClientMin } from '../src/LDClientMin';
import { aiSdkName, aiSdkVersion } from '../src/sdkInfo';

const mockTrack = jest.fn();
const mockVariation = jest.fn();
Expand All @@ -25,8 +24,6 @@ const getExpectedTrackData = () => ({
version,
modelName,
providerName,
aiSdkName,
aiSdkVersion,
});

beforeEach(() => {
Expand Down
47 changes: 35 additions & 12 deletions packages/sdk/server-ai/src/LDAIClientImpl.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,22 +23,40 @@ import { LDAIClient } from './api/LDAIClient';
import { AIProviderFactory, SupportedAIProvider } from './api/providers';
import { LDAIConfigTrackerImpl } from './LDAIConfigTrackerImpl';
import { LDClientMin } from './LDClientMin';
import { aiSdkLanguage, aiSdkName, aiSdkVersion } from './sdkInfo';

/**
* Tracking event keys for AI SDK usage metrics.
*/
const TRACK_CONFIG_SINGLE = '$ld:ai:config:function:single';
const TRACK_CONFIG_CREATE_CHAT = '$ld:ai:config:function:createChat';
const TRACK_JUDGE_SINGLE = '$ld:ai:judge:function:single';
const TRACK_JUDGE_CREATE = '$ld:ai:judge:function:createJudge';
const TRACK_AGENT_SINGLE = '$ld:ai:agent:function:single';
const TRACK_AGENT_MULTIPLE = '$ld:ai:agent:function:multiple';
const TRACK_SDK_INFO = '$ld:ai:sdk:info';
const TRACK_USAGE_COMPLETION_CONFIG = '$ld:ai:usage:completion-config';
const TRACK_USAGE_CREATE_CHAT = '$ld:ai:usage:create-chat';
const TRACK_USAGE_JUDGE_CONFIG = '$ld:ai:usage:judge-config';
const TRACK_USAGE_CREATE_JUDGE = '$ld:ai:usage:create-judge';
const TRACK_USAGE_AGENT_CONFIG = '$ld:ai:usage:agent-config';
const TRACK_USAGE_AGENT_CONFIGS = '$ld:ai:usage:agent-configs';

const INIT_TRACK_CONTEXT: LDContext = {
kind: 'ld_ai',
key: 'ld-internal-tracking',
anonymous: true,
};

export class LDAIClientImpl implements LDAIClient {
private _logger?: LDLogger;

constructor(private _ldClient: LDClientMin) {
this._logger = _ldClient.logger;
this._ldClient.track(
TRACK_SDK_INFO,
INIT_TRACK_CONTEXT,
{
aiSdkName,
aiSdkVersion,
aiSdkLanguage,
},
1,
);
Copy link

@cursor cursor bot Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Constructor tracking can spam events

Medium Severity

The LDAIClientImpl constructor now unconditionally calls track for $ld:ai:sdk:info. If client instances are created per request/job (common in server code), this can generate high-volume duplicate events and unexpected side effects during object creation, increasing telemetry cost and potentially impacting latency.

Fix in Cursor Fix in Web

}

private _interpolateTemplate(template: string, variables: Record<string, unknown>): string {
Expand Down Expand Up @@ -146,7 +164,7 @@ export class LDAIClientImpl implements LDAIClient {
defaultValue: LDAICompletionConfigDefault,
variables?: Record<string, unknown>,
): Promise<LDAICompletionConfig> {
this._ldClient.track(TRACK_CONFIG_SINGLE, context, key, 1);
this._ldClient.track(TRACK_USAGE_COMPLETION_CONFIG, context, key, 1);

const config = await this._evaluate(key, context, defaultValue, 'completion', variables);
return config as LDAICompletionConfig;
Expand All @@ -170,7 +188,7 @@ export class LDAIClientImpl implements LDAIClient {
defaultValue: LDAIJudgeConfigDefault,
variables?: Record<string, unknown>,
): Promise<LDAIJudgeConfig> {
this._ldClient.track(TRACK_JUDGE_SINGLE, context, key, 1);
this._ldClient.track(TRACK_USAGE_JUDGE_CONFIG, context, key, 1);

const config = await this._evaluate(key, context, defaultValue, 'judge', variables);
return config as LDAIJudgeConfig;
Expand All @@ -182,7 +200,7 @@ export class LDAIClientImpl implements LDAIClient {
defaultValue: LDAIAgentConfigDefault,
variables?: Record<string, unknown>,
): Promise<LDAIAgentConfig> {
this._ldClient.track(TRACK_AGENT_SINGLE, context, key, 1);
this._ldClient.track(TRACK_USAGE_AGENT_CONFIG, context, key, 1);

const config = await this._evaluate(key, context, defaultValue, 'agent', variables);
return config as LDAIAgentConfig;
Expand All @@ -204,7 +222,12 @@ export class LDAIClientImpl implements LDAIClient {
agentConfigs: T,
context: LDContext,
): Promise<Record<T[number]['key'], LDAIAgentConfig>> {
this._ldClient.track(TRACK_AGENT_MULTIPLE, context, agentConfigs.length, agentConfigs.length);
this._ldClient.track(
TRACK_USAGE_AGENT_CONFIGS,
context,
agentConfigs.length,
agentConfigs.length,
);

const agents = {} as Record<T[number]['key'], LDAIAgentConfig>;

Expand Down Expand Up @@ -241,7 +264,7 @@ export class LDAIClientImpl implements LDAIClient {
variables?: Record<string, unknown>,
defaultAiProvider?: SupportedAIProvider,
): Promise<TrackedChat | undefined> {
this._ldClient.track(TRACK_CONFIG_CREATE_CHAT, context, key, 1);
this._ldClient.track(TRACK_USAGE_CREATE_CHAT, context, key, 1);

const config = await this.completionConfig(key, context, defaultValue, variables);

Expand Down Expand Up @@ -272,7 +295,7 @@ export class LDAIClientImpl implements LDAIClient {
variables?: Record<string, unknown>,
defaultAiProvider?: SupportedAIProvider,
): Promise<Judge | undefined> {
this._ldClient.track(TRACK_JUDGE_CREATE, context, key, 1);
Copy link

@cursor cursor bot Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate usage events for chat creation

Medium Severity

createChat now emits a create-chat usage event and then calls completionConfig, which also emits a completion-config usage event for the same operation. This can double-count usage when consumers use createChat as their primary entrypoint, skewing usage reporting compared to direct completionConfig calls.

Additional Locations (1)

Fix in Cursor Fix in Web

this._ldClient.track(TRACK_USAGE_CREATE_JUDGE, context, key, 1);
Copy link

@cursor cursor bot Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate usage events for judge creation

Medium Severity

createJudge now emits a create-judge usage event and then calls judgeConfig, which also emits a judge-config usage event. This changes usage semantics and can double-count “judge config usage” for callers that primarily use createJudge.

Additional Locations (1)

Fix in Cursor Fix in Web


try {
if (variables?.message_history !== undefined) {
Expand Down
5 changes: 0 additions & 5 deletions packages/sdk/server-ai/src/LDAIConfigTrackerImpl.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ import {
LDTokenUsage,
} from './api/metrics';
import { LDClientMin } from './LDClientMin';
import { aiSdkName, aiSdkVersion } from './sdkInfo';

export class LDAIConfigTrackerImpl implements LDAIConfigTracker {
private _trackedMetrics: LDAIMetricSummary = {};
Expand All @@ -33,17 +32,13 @@ export class LDAIConfigTrackerImpl implements LDAIConfigTracker {
version: number;
modelName: string;
providerName: string;
aiSdkName: string;
aiSdkVersion: string;
Copy link

@cursor cursor bot Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SDK info removed from metrics payloads

Medium Severity

getTrackData no longer includes aiSdkName/aiSdkVersion, and the LDAIConfigTracker type was updated accordingly. Any analytics pipeline or downstream consumer that relied on these fields in token/duration/success events will silently lose SDK attribution, even though only a separate $ld:ai:sdk:info event is now emitted.

Additional Locations (1)

Fix in Cursor Fix in Web

Copy link
Contributor Author

@jsonbailey jsonbailey Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional and we are making the change while still in pre-release. This is not something downstream consumers should be relying on which is part of the reason for this change.

} {
return {
variationKey: this._variationKey,
configKey: this._configKey,
version: this._version,
modelName: this._modelName,
providerName: this._providerName,
aiSdkName,
aiSdkVersion,
};
}

Expand Down
2 changes: 0 additions & 2 deletions packages/sdk/server-ai/src/api/config/LDAIConfigTracker.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,6 @@ export interface LDAIConfigTracker {
version: number;
modelName: string;
providerName: string;
aiSdkName: string;
aiSdkVersion: string;
};
/**
* Track the duration of generation.
Expand Down
1 change: 1 addition & 0 deletions packages/sdk/server-ai/src/sdkInfo.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
export const aiSdkName = '@launchdarkly/server-sdk-ai';
export const aiSdkVersion = '0.16.2'; // x-release-please-version
export const aiSdkLanguage = 'javascript';