[BUG] GetOperationStatus polls not retried on IOException — orphaned server operations and "Query has been timed out due to inactivity"

[diegofanesi]

Summary

During sustained DBSQL concurrency stress tests we consistently observed queries failing with
Query has been timed out due to inactivity on the server side, paired with a TTransportException
on the client. Root cause analysis points to a single underlying gap, plus a secondary cleanup leak:

1. Root cause — transport-level IOException (stale pooled connection, TCP reset) on a
   GetOperationStatus poll is never retried on the Thrift path. DatabricksHttpRetryHandler
   only retries HTTP-status-code errors, not raw IOException, so a single transient failure
   abandons a still-running server operation.
2. Secondary — CloseOperation cleanup RPCs fail against the same broken connection during
   Statement/ResultSet close, leaving completed operations open until the server's inactivity
   timeout reaps them.

The SEA path (UseThriftClient=0) already retries this class of IOException via the SDK's
idempotent-request strategy; the Thrift path does not. The fix is to bring the Thrift path to
parity.

---

Environment

• Driver: databricks-jdbc (also reproduced against Simba JDBC 3.3.1)
• Warehouse type: DBSQL Serverless
• Workload: 20+ concurrent long-running queries (TPC-DS benchmark)
• Transport: HTTP/Thrift (DatabricksHttpTTransport)

---

Observed behaviour

Server side (system.query.history):

execution_status : FAILED
error_message    : Query has been timed out due to inactivity.
total_duration_ms: ~430 000  (query ran ~430 s after the client had already moved on)

Client side (exception propagated from pollTillOperationFinished):

com.databricks.jdbc.exception.DatabricksSQLException:
  ... TTransportException: <connection reset / No route to host / stale connection>

The operation continues running server-side for ~430 seconds after the client exception, then the
server records the inactivity failure. In a second variant (9 occurrences), a completed operation
whose CloseOperation RPC failed remained open for ~23 minutes before the server reaped it.

Note: the ~430 s / ~23 min windows are observed empirically from system.query.history; they
reflect server-side reaping behaviour, not something visible in the driver source.

---

Root cause

1 — retryRequest does not retry plain IOException (stale connection, TCP reset)

DatabricksHttpRetryHandler.java, retryRequest() (lines 130–134):

@Override
public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
    int statusCode = getErrorCodeFromException(exception);
    if (!isStatusCodeRetryable(statusCode)) {
        return false;   // ← returns false for every raw IOException
    }
    // ... only reaches here for DatabricksRetryHandlerException wrapping 429/503
}

getErrorCodeFromException (lines 251–256) only extracts a status code from
DatabricksRetryHandlerException (thrown by the process() interceptor for HTTP-level errors);
for any other IOException it returns 0:

static int getErrorCodeFromException(IOException exception) {
    if (exception instanceof DatabricksRetryHandlerException) {
        return ((DatabricksRetryHandlerException) exception).getErrCode();
    }
    return 0;
}

isStatusCodeRetryable(0) falls through to default: return false (line 298). The result: any
transport-level failure on a GetOperationStatus poll — stale pooled connection, TCP RST,
load-balancer reset — is treated as unrecoverable and not retried.

2 — The polling loop delegates retry to the HTTP layer, which then doesn't retry

DatabricksThriftAccessor.java, pollTillOperationFinished() (lines 339–364):

try {
    statusResp = getOperationStatus(statusReq, statementId);
} catch (TTransportException e) {
    // "after retries were exhausted"
    throw buildTransportFailureException(statementId.toSQLExecStatementId(), e);
}

The poll loop has no retry of its own — the comment at the catch site assumes retries already
happened at the HTTP layer. But because of gap 1 they didn't. The raw IOException surfaces from
DatabricksHttpTTransport.flush(), which wraps it as a TTransportException (lines 138–150) with no
retry:

try (CloseableHttpResponse response = httpClient.execute(request)) {
    ...
} catch (DatabricksHttpException | IOException e) {
    ...
    throw new TTransportException(TTransportException.UNKNOWN, errorMessage, e);
}

One failed status poll → TTransportException → caller abandons the operation → server runs until
the ~430 s inactivity window.

3 — closeServerOperation swallows the failure; the follow-up close also fails and the operation leaks

DatabricksStatement.java, closeServerOperation() (lines 1026–1034):

} catch (SQLException | RuntimeException e) {
    // Best-effort — don't fail the user's close for a server cleanup failure.
    // serverOperationClosed stays false so Statement.close() will retry the RPC.
    LOGGER.warn("Failed to proactively close server operation for statement {}: {}", ...);
}

If the same class of connection error hits during closeServerOperation() (called from
ResultSet.close()), the CloseOperation RPC is dropped and serverOperationClosed stays false.
Statement.close() then does retry the RPC (DatabricksStatement.java:165–166), but against the
same broken connection it fails again. Unlike the proactive close, this second failure is not
swallowed — close(boolean) is a try { … } finally { … } with no catch (lines 142–194), so the
SQLException propagates to the caller. Either way the operation is left alive server-side and is
reaped by the inactivity timeout (~23 minutes in the observed cases).

Why this only affects long-running concurrent workloads

HTTP connection pooling (PoolingHttpClientConnectionManager) reuses persistent connections
across the 200 ms poll-sleep intervals (POLL_INTERVAL default "200"). Under high concurrency,
pooled connections can become stale (server-side close of a keep-alive connection, AWS NAT gateway
timeout, load-balancer idle reset). Because polls reuse a connection every ~200 ms, Apache
HttpClient's idle-revalidation does not kick in between polls, so a stale connection is used and
throws IOException mid-flight. Per gap 1, there is no transparent reconnect-and-retry.

---

Proposed fix

Fix 1 (primary) — Retry transport-level IOException on safe polling/cleanup RPCs

GetOperationStatus and CloseOperation are read-only or idempotent from the server's perspective
(polling a running operation or closing an already-completed one causes no side effects on retry),
so a transport failure should transparently reconnect and retry. Two implementation options:

Option A (preferred) — retry at the poll-loop level, where the RPC identity and operation handle
are already known, so no request-tagging plumbing is needed:

// DatabricksThriftAccessor.pollTillOperationFinished()
try {
    statusResp = getOperationStatus(statusReq, statementId);
    transportRetries = 0; // reset on success
} catch (TTransportException e) {
    if (++transportRetries <= MAX_TRANSPORT_RETRIES) {
        LOGGER.warn("Retrying GetOperationStatus after transport failure "
            + "(attempt {}): {}", transportRetries, e.getMessage());
        // brief backoff, then continue the loop with a fresh connection
        continue;
    }
    throw buildTransportFailureException(statementId.toSQLExecStatementId(), e);
}

Option B — retry inside DatabricksHttpRetryHandler.retryRequest for safe RPCs. Note this is
not a one-line change: DatabricksHttpTTransport.flush() currently calls
httpClient.execute(request) with no HttpContext and only sees opaque serialized Thrift bytes, so
it cannot know which RPC it is flushing. This option requires new plumbing to tag each request with
its RPC name (e.g. a transport field or thread-local set before flush(), plus passing an
HttpContext into execute) so the handler can allow-list GetOperationStatus / CloseOperation.

The same cleanup path should also be applied to CloseOperation (see Fix 2) so the leak in root
cause 3 is closed as well.

Fix 2 — Make CloseOperation failures recoverable instead of best-effort-and-forget

Even with Fix 1, closeServerOperation should not depend on a single follow-up attempt against a
connection that is already broken. Options: retry the close on a fresh connection, or schedule a
deferred retry via the session's cleanup path rather than only relying on Statement.close():

// DatabricksStatement.java — closeServerOperation()
} catch (SQLException | RuntimeException e) {
    LOGGER.warn("Failed to close server operation for statement {}: {}", statementId, e);
    // Schedule a deferred retry via the session's cleanup queue rather than
    // relying on a single follow-up attempt in Statement.close().
    connection.getSession().scheduleCleanup(statementId);
}

Fix 3 (optional, narrow) — Heartbeat for slow-consumer results only

ENABLE_HEARTBEAT defaults to "0" (DatabricksJdbcUrlParams.java:252–255). Enabling heartbeat
sends periodic GetOperationStatus pings while results are consumed slowly
(ResultHeartbeatManager / DatabricksResultSet.startHeartbeatIfEnabled).

This is not a fix for the main problem, and we do not recommend flipping the default on the basis
of this issue:

• It cannot affect the dominant failure mode (transport failure during polling): the heartbeat is
  only started once a DatabricksResultSet exists, which is after the operation has finished — so
  it is never running while a still-RUNNING operation is being polled.
• It is gated by isHeartbeatEligible() (excludes SEA-inline, direct/CLOSED results, and
  PENDING/RUNNING) and has a 60 s initial delay (HEARTBEAT_INTERVAL_SECONDS default "60"), so
  it only matters for a genuinely stalled consumer holding an eligible open result set for >60 s.
• Its benefit assumes a GetOperationStatus poll resets the server's inactivity timer, which is a
  server-side assumption not verifiable from the driver.

Keep it as an opt-in mitigation for slow-consumer scenarios, not as the remedy for this issue.

---

Impact

• Operations are abandoned after a single transient HTTP failure during polling, even though the
  server-side query is still running and could be re-polled with a fresh connection.
• Completed operations are leaked to the server's inactivity timeout, holding resources for up to
  ~23 minutes per operation.
• The transport-failure mode surfaces only as a generic TTransportException, with no indication
  that the server is still carrying orphaned work.

---

Files

File	Location of gap
DatabricksHttpRetryHandler.java	retryRequest() (130–134) + getErrorCodeFromException() (251–256) — returns false/0 for all plain IOException
DatabricksThriftAccessor.java	pollTillOperationFinished() (347–349) — no retry on TTransportException; getOperationStatus() (1042) — bare RPC
DatabricksHttpTTransport.java	flush() (128, 138–150) — executes with no HttpContext; wraps IOException as TTransportException
DatabricksStatement.java	closeServerOperation() (1026–1034) — swallows close failure; close(boolean) (142–194) retries once and then propagates
DatabricksJdbcUrlParams.java	ENABLE_HEARTBEAT (252) — disabled by default (relevant only to Fix 3)

---

Behaviour difference between UseThriftClient=1 and UseThriftClient=0

Both paths share the same close-cleanup gap, but differ on the poll-retry and interrupt behaviour.

Mode A — IOException on a status poll

On the Thrift path (UseThriftClient=1), GetOperationStatus is a POST routed through
DatabricksHttpRetryHandler. As above, retryRequest() returns false for all plain
IOException, so a single stale connection immediately abandons the operation.

On the SEA path (UseThriftClient=0), status polling is a GET to
/api/2.0/sql/statements/{id}, which goes through the Databricks SDK's ApiClient. The SDK wraps
any IOException into DatabricksError("IO_ERROR", 523, e) (ApiClient.java:274–283) and applies
RequestBasedRetryStrategyPicker: GET /api/2.0/sql/statements/.* is explicitly listed as an
idempotent request, so it receives IdempotentRequestRetryStrategy. That strategy's
isRetriable():

public boolean isRetriable(DatabricksError databricksError) {
    if (RetryUtils.isCausedByTransientError(databricksError)) { return true; }
    if (isNonRetriableException(databricksError))              { return false; }
    if (isNonRetriableHttpCode(databricksError))               { return false; }
    return true;  // ← default: retry everything else, including IOException (523)
}
// NON_RETRIABLE_HTTP_CODES = {400,401,403,404,405,409,410,411,412,413,414,415,416}
// 523 is not in the set → isRetriable returns true → IOException IS retried on SEA

Result: the SEA path retries IOException on status polls; the Thrift path does not. (Verified
in databricks-sdk-java: RequestBasedRetryStrategyPicker, IdempotentRequestRetryStrategy,
ApiClient.)

Mode A — Thread interrupt

On the Thrift path, InterruptedException in the poll sleep triggers a cancelOperation RPC
(DatabricksThriftAccessor.java:357–363). That cancel is attempted; if the underlying connection
is broken it throws DatabricksHttpException (from cancelOperation, lines 164–174) which
propagates — so the client is not left unaware, though the server op may still be orphaned because
the cancel never landed.

On the SEA path, InterruptedException immediately throws DatabricksTimeoutException
without sending any cancel (DatabricksSdkClient.java:278–285) — the operation is orphaned even
faster.

Mode B — closeOperation fails

Both paths share DatabricksStatement.closeServerOperation(). The proactive close is best-effort
(swallowed); Statement.close() retries once and then propagates on failure. Mode B is identical
on both paths.

Summary

Failure scenario	Thrift (=1)	SEA (=0)
IOException on status poll	❌ Not retried, operation orphaned	✅ Retried by SDK idempotent strategy
Thread interrupt	⚠️ Cancel attempted; throws (not silent) if transport broken	❌ No cancel sent — worse
closeOperation IOException	❌ Leaked (proactive swallowed; retry propagates)	❌ Leaked (same shared path)

Switching to UseThriftClient=0 mitigates Mode A poll failures but does not fix the close leak and
makes the interrupt case worse. The proper fix is in the shared cleanup path and in the Thrift poll
retry, regardless of client.

---

Related

#1571 appears to track a sibling gap in the same method — DatabricksHttpRetryHandler.retryRequest
aborts when retryInterval == -1 and the status code isn't in ApiRetriableHttpCodes (line 142),
which would drop header-less 503/429 on the Thrift path. The mechanism is consistent with the code
here (same file/method), though we haven't re-verified the specifics of #1571 itself. A
retryRequest/poll-loop fix that also handles raw IOException for safe RPCs would be
complementary and could land in the same PR.

A couple of judgment calls I made — tell me if you'd rather go the other way:

• I made Fix 1 the primary and led with poll-loop-level retry (Option A) since it avoids the flush() plumbing problem, keeping Option B as an alternative.
• I kept OpenSession does not retry transient 503/429 without a Retry-After header (Thrift); 502/504 never retried #1571 but softened it to "appears to / haven't re-verified," since I validated the mechanism against the code but didn't fetch that issue. If you want, I can fetch OpenSession does not retry transient 503/429 without a Retry-After header (Thrift); 502/504 never retried #1571 and either firm this up or drop it.

[diegofanesi]

the fix is already applied on https://github.com/diegofanesi/databricks-jdbc. Please note: the fix affects only to thrift mode, not SEA.

[diegofanesi]

ready to merge from my end.