fix: handle kubernetes describe failures gracefully (#1150)

[clumsy]

When job is not found we will show State=UNKNOWN and surface the error in Msg

Test plan:
[x] added unit test
[x] torchx status sfai_kubernetes://torchx/real-namespace:bogus-job-id

AppStatus:
    State: UNKNOWN
    Num Restarts: -1
    Roles: 
    Msg: Not Found: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch.volcano.sh \"bogus-jobs-id\" not found","reason":"NotFound","details":{"name":"bogus-job-id","group":"batch.volcano.sh","kind":"jobs"},"code":404}

    Structured Error Msg: <NONE>
    UI URL: None

[clumsy]

For your consideration @kiukchung @d4l3k
Looks like we might need the same for other APIs

[clumsy]

setup_pyre.sh lacked +x so I added it here, don't think it justifies a full PR

[kiukchung]

when the job is not found describe API should return None no?

[clumsy]

Do you mean State: None? What would you like it to show @kiukchung ? I'm happy to update my fix as long as it's not an exception.

[kiukchung]

Do you mean State: None? What would you like it to show @kiukchung ? I'm happy to update my fix as long as it's not an exception.

I mean that if volcano raises an exception when the job does not exist, then torchx's describe() API should return None as per its documentation: https://meta-pytorch.org/torchx/latest/runner.html

describe(app_handle: str)) → Optional[AppDef]
  """
   Reconstructs the application (to the best extent) given the app handle. Note that the reconstructed application may not be the complete app as it was submitted via the run API. How much of the app can be reconstructed is scheduler dependent.

    Returns:

        AppDef or None if the app does not exist anymore or if the scheduler does not support describing the app handle
  """

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.27%. Comparing base (79e14da) to head (61107b6).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1151      +/-   ##
==========================================
- Coverage   91.65%   90.27%   -1.39%     
==========================================
  Files          83       83              
  Lines        6483     6599     +116     
==========================================
+ Hits         5942     5957      +15     
- Misses        541      642     +101     

Flag	Coverage Δ
unittests	90.27% <100.00%> (-1.39%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[clumsy]

Oh, that's a good point, @kiukchung

Let me update this PR then!

[clumsy]

This looks perfect, @kiukchung!
torchx status kubernetes://torchx/real-namespace:bla
produces

torchx 2025-10-23 16:39:56 ERROR    AppDef: real-namespace:bla, does not exist or has been removed from kubernetes's data plane

[kiukchung]

This looks perfect, @kiukchung! torchx status kubernetes://torchx/real-namespace:bla produces

torchx 2025-10-23 16:39:56 ERROR    AppDef: real-namespace:bla, does not exist or has been removed from kubernetes's data plane

cool! nit: can we log the caught exception for debugging purposes?

try:
  ...
except Exception as e:
   # is there a narrower exception we can catch? or make sure this is actually a "app not found"
   if not is_app_not_found_error(e):
     logger.exception("error when querying kubernetes/volcano for the job's status")
   return None

or better just let other errors throw

try:
  ...
except Exception as e:
   if is_app_not_found_error(e):
     return None
   raise # for any other exceptione, re-raise

[clumsy]

Done, @kiukchung

I agree it's cleaner

[kiukchung]

@clumsy can you rebase on top of main?