feat(go/ai): add streaming support for GenerateData with partial JSON parsing

[nozomi-koborinai]

Description

This PR adds streaming support for GenerateData in Genkit Go, enabling partial JSON parsing during streaming responses.

Background

When using GenerateData with streaming callbacks, the following error occurs:

Bad stream: SyntaxError: Unexpected end of JSON input

This happens because GenerateData doesn't support parsing partial JSON during streaming, which is a common use case when generating structured data (like menus, configurations, etc.) with streaming enabled for better user experience.

Solution

Following the JavaScript implementation approach, this PR implements:

1. Partial JSON parsing capability - Added functions to parse incomplete JSON structures during streaming, based on js/ai/src/extract.ts
2. Formatter streaming support - Added ParseChunk method to all formatters to handle streaming data
3. Metadata field for chunks - Added to ModelResponseChunk to store parsed partial data, matching js/ai/src/generate/chunk.ts

Example

This allows for an implementation as per the document: https://genkit.dev/go/docs/flows/#streaming-flows

type Menu struct {
    Theme  string     `json:"theme"`
    Items  []MenuItem `json:"items"`
}

type MenuItem struct {
    Name        string `json:"name"`
    Description string `json:"description"`
}

menuSuggestionFlow := genkit.DefineStreamingFlow(g, "menuSuggestionFlow",
    func(ctx context.Context, theme string, callback core.StreamCallback[string]) (Menu, error) {
        item, _, err := genkit.GenerateData[MenuItem](ctx, g,
            ai.WithPrompt("Invent a menu item for a %s themed restaurant.", theme),
            ai.WithStreaming(func(ctx context.Context, chunk *ai.ModelResponseChunk) error {
                // Here, you could process the chunk in some way before sending it to
                // the output stream using StreamCallback. In this example, we output
                // the text of the chunk, unmodified.
                return callback(ctx, chunk.Text())
            }),
        )
        if err != nil {
            return nil, err
        }

        return Menu{
            Theme: theme,
            Items: []MenuItem{item},
        }, nil
    })

This brings feature parity with the JavaScript SDK and resolves the TODO comment in the code.

Test Results

All tests pass successfully:

$ go test ./ai/... -v
=== RUN   TestGenerateAction
...
PASS
ok      github.com/firebase/genkit/go/ai        0.357s

$ go test ./internal/base/... -v
=== RUN   TestExtractJSONFromMarkdown
...
=== RUN   TestParsePartialJSON
...
=== RUN   TestExtractJSON
...
PASS
ok      github.com/firebase/genkit/go/internal/base     0.220s

$ go test ./... -short
ok      github.com/firebase/genkit/go/ai        0.304s
ok      github.com/firebase/genkit/go/core      0.644s
ok      github.com/firebase/genkit/go/genkit    0.966s
ok      github.com/firebase/genkit/go/internal/base     1.396s
... (all packages pass)

Checklist (if applicable):

• PR title is following https://www.conventionalcommits.org/en/v1.0.0/
• Tested (manually, unit tested, etc.)
• Docs updated (updated docs or a docs bug required)