get_job_logs silently returns empty iterator when no pod is found

[kevo-1]

When no pod is found for the requested step, the generator silently returns. The caller gets an empty iterator with no way to distinguish "job produced no logs" from "pod was never found."

if pod_name is None:
    return

File affected	Line
kubernetes/backend.py	line: 443
optimizer/backends/kubernetes/backend.py	line: 245

Same behavior appears in both files above

Was this behavior intended or should it raise an error?

[ahmedtabdelhamed]

Hi @kevo-1, thanks for catching this. I would like to work on this. My plan is to update both kubernetes/backend.py and optimizer/backends/kubernetes/backend.py to raise a RuntimeError (or a specific SDK exception if the maintainers prefer) when pod_name is None, and then update the corresponding unit tests to expect this exception.

[kevo-1]

@ahmedtabdelhamed, thanks for your enthusiasm! This is more of a design question for the maintainers to decide on, it would be worth opening a discussion or asking them directly before proceeding, so we can align on the preferred approach.

[ahmedtabdelhamed]

@kevo-1, thanks for the heads-up. You are completely right.

[sjiang83]

Quick thought on get_job_logs(): currently it silently returns an empty iterator when no pod is found.

This creates some ambiguity for callers, especially in the context of MCP. It's hard to tell if it's a case of "no logs yet" or if the pod actually doesn't exist permissions.

Would maintainers be open to distinguishing these cases? I'd be happy to put together a quick PR for this — just looking for some feedback on whether you'd prefer raising a specific exception or changing it to a structured result.

[CREVIOS]

I ran into this while looking at the SDK internals for the MCP server project (#238). If the MCP get_training_logs tool wraps this method, an agent would receive an empty iterator and have no way to distinguish between "job hasn't produced logs yet" vs "pod was never scheduled" vs "wrong job name". That's a real problem for agentic workflows where the LLM needs to decide what to do next based on the response.

I think a simple approach would be:

• When pod_name is None, raise a specific exception (something like PodNotFoundError) instead of silently returning
• Callers that want the current silent behavior can catch it explicitly

This keeps backward compat straightforward — existing code that doesn't catch the exception will get a clear error instead of silently empty results. The same fix applies to both kubernetes/backend.py:443 and optimizer/backends/kubernetes/backend.py:245.

I'd be happy to send a PR for this if maintainers agree on the approach. Wanted to check first since @kevo-1 mentioned it's a design decision.

[sjiang83]

I ran into this while looking at the SDK internals for the MCP server project (#238). If the MCP get_training_logs tool wraps this method, an agent would receive an empty iterator and have no way to distinguish between "job hasn't produced logs yet" vs "pod was never scheduled" vs "wrong job name". That's a real problem for agentic workflows where the LLM needs to decide what to do next based on the response.

I think a simple approach would be:

• When pod_name is None, raise a specific exception (something like PodNotFoundError) instead of silently returning
• Callers that want the current silent behavior can catch it explicitly

This keeps backward compat straightforward — existing code that doesn't catch the exception will get a clear error instead of silently empty results. The same fix applies to both kubernetes/backend.py:443 and optimizer/backends/kubernetes/backend.py:245.

I'd be happy to send a PR for this if maintainers agree on the approach. Wanted to check first since @kevo-1 mentioned it's a design decision.

Thanks, +1. This is exactly the ambiguity I'm trying to avoid for the LLM agents and MCP integrations.

I can easily open a tightly scoped PR to fix this. Instead of returning an empty iterator, I can make the missing pod scenario explicit. I'll also add tests and ensure both the trainer and optimizer K8s backends are updated to match.

Happy to follow whatever the maintainers prefer for the return format. That said, raising a specific PodNotFound exception feels like the cleanest and most debuggable approach to me.

[sjiang83]

I ran into this while looking at the SDK internals for the MCP server project (#238). If the MCP get_training_logs tool wraps this method, an agent would receive an empty iterator and have no way to distinguish between "job hasn't produced logs yet" vs "pod was never scheduled" vs "wrong job name". That's a real problem for agentic workflows where the LLM needs to decide what to do next based on the response.
I think a simple approach would be:

• When pod_name is None, raise a specific exception (something like PodNotFoundError) instead of silently returning
• Callers that want the current silent behavior can catch it explicitly

This keeps backward compat straightforward — existing code that doesn't catch the exception will get a clear error instead of silently empty results. The same fix applies to both kubernetes/backend.py:443 and optimizer/backends/kubernetes/backend.py:245.
I'd be happy to send a PR for this if maintainers agree on the approach. Wanted to check first since @kevo-1 mentioned it's a design decision.

Thanks, +1. This is exactly the ambiguity I'm trying to avoid for the LLM agents and MCP integrations.

I can easily open a tightly scoped PR to fix this. Instead of returning an empty iterator, I can make the missing pod scenario explicit. I'll also add tests and ensure both the trainer and optimizer K8s backends are updated to match.

Happy to follow whatever the maintainers prefer for the return format. That said, raising a specific PodNotFound exception feels like the cleanest and most debuggable approach to me.

I went ahead and opened PR #339 implementing an opt-in fail-fast behavior (strict=True) with tests, keeping default behavior unchanged. Feedback welcome on the API shape and error type.
#339