Allow custom type to be streamed and use native response

[TomeHirata]

This PR increases the extensibility of dspy.Type and allows users to define a custom type that can be streamed or use native LM response. This is useful when users want to use provider specific response field such as tool call or citation.

Following methods are introduced into dspy.Type

• is_streamable: Whether the custom type is streamable
• parse_stream_chunk: Parse a stream chunk into the custom type
• use_native_response: Whether to use the native response of the LM such as function calling

[okhat]

No concerns on a quick scan. If @chenmoneygithub approves, feel free to merge.

[chenmoneygithub]

Very clean and cool! Left some comments for discussion.

[TomeHirata]

@chenmoneygithub I've moved the native response type config to adapter, can you take a look again?

[chenmoneygithub]

Some minor comments, otherwise LGTM!

[chenmoneygithub]

when will this throw an exception?

[TomeHirata]

It does not fail in normal cases, but since predict or signature_field_name is not guaranteed to be non-null, it's better to have a safeguard

[chenmoneygithub]

ic, actually I think predict and signature_field_name must be configured for stream listener, otherwise it is not doing anything.

[TomeHirata]

I know, but typing-wise they are nullable (

dspy/dspy/streaming/streaming_listener.py

Lines 24 to 29 in 28a7f78

	def __init__(
	self,
	signature_field_name: str,
	predict: Any = None,
	predict_name: str | None = None,
	allow_reuse: bool = False,

), so I still want to keep the check. We can do if chain, as well.

[chenmoneygithub]

LGTM after #8778 (comment) is resolved.