Skip to content

fix: retry transient read timeouts when polling dbt Cloud run status#294

Merged
jymaire merged 1 commit into
mainfrom
fix/dbt-cloud-retry-read-timeout
Jul 7, 2026
Merged

fix: retry transient read timeouts when polling dbt Cloud run status#294
jymaire merged 1 commit into
mainfrom
fix/dbt-cloud-retry-read-timeout

Conversation

@jymaire

@jymaire jymaire commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Problem

io.kestra.plugin.dbt.cloud.CheckStatus (and TriggerRun with wait: true) polls the dbt Cloud run status through AbstractDbtCloud.request, whose retry policy only retries HTTP 502/503/504. A transient read timeout during a status poll is surfaced by the core HTTP client as RuntimeException(java.net.SocketTimeoutException), so it is not retried, propagates out of the polling loop, and fails the whole task — even though the dbt Cloud run is still healthy and goes on to complete:

ERROR java.net.SocketTimeoutException: Read timed out
    at io.kestra.core.http.client.HttpClient.request(HttpClient.java:294)
    at io.kestra.plugin.dbt.cloud.AbstractDbtCloud.lambda$request$1(AbstractDbtCloud.java:94)
    at io.kestra.plugin.dbt.cloud.CheckStatus.fetchRunResponse(CheckStatus.java:214)
    at io.kestra.plugin.dbt.cloud.CheckStatus.run(CheckStatus.java:118)
    at io.kestra.plugin.dbt.cloud.TriggerRun.run(TriggerRun.java:208)

The very next attempt of the same run completed 10 total | 10 success, confirming there was no real dbt error — just a transient network read timeout on one poll.

Root cause

The retry predicate in AbstractDbtCloud.request only matched HttpClientResponseException with status 502/503/504:

(res, throwable) -> throwable instanceof HttpClientResponseException ex &&
    (code == 502 || code == 503 || code == 504)

A read timeout is not an HttpClientResponseException, so it never matched, was not retried, and failed the task.

Fix

Broaden the retry predicate (extracted into isRetriableTransientError) to also treat as transient:

  • connection-level failures — HttpClientRequestException (socket / SSL handshake);
  • read/connect timeouts — SocketTimeoutException, including when wrapped as the cause.

Genuine client errors (4xx) still fail fast — no change there. Reuses the existing maxRetries / initialDelayMs config; no new properties. Benefits every request path (CheckStatus polling, downloadArtifacts, TriggerRun).

Tests

CheckStatusRetryTest:

  • shouldRetryOnReadTimeoutAndEventuallySucceed — reproduces the reported failure (RuntimeException(SocketTimeoutException) then 200) and asserts retry + success. Verified red without the fix (SocketTimeoutException: Read timed out), green with it.
  • shouldNotRetryOnClientError — asserts a 404 fails fast (single attempt, no retry).

Full io.kestra.plugin.dbt.cloud package: 14/14 green.

Closes #295

🤖 Generated with Claude Code

The dbt Cloud request retry policy only retried HTTP 502/503/504, so a
transient read timeout — surfaced by the core HTTP client as
RuntimeException(SocketTimeoutException) — was not retried and failed the
whole CheckStatus/TriggerRun task, even though the dbt Cloud run was still
healthy and went on to complete successfully.

Broaden the retry predicate to also cover connection-level failures
(HttpClientRequestException, i.e. socket/SSL) and read/connect timeouts
(SocketTimeoutException, including when wrapped as the cause). Genuine
client errors such as 4xx still fail fast.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jymaire jymaire self-assigned this Jul 6, 2026
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

📦 Artifacts

Name Size Updated Expiration
jar 212.19 KB Jul 6, 26, 3:42:13 PM UTC Jul 13, 26, 3:42:12 PM UTC

🧪 Java Unit Tests

TestsPassed ✅SkippedFailedTime ⏱
Java Tests Report40 ran40 ✅0 ⚠️0 ❌4m 2s 331ms

🔁 Unreleased Commits

✅ No unreleased commits found.

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Tests report quick summary:

success ✅ > tests: 40, success: 40, skipped: 0, failed: 0

unfold for details
Project Status Success Skipped Failed
plugin-dbt success ✅ 40 0 0
@jymaire
jymaire merged commit a16089e into main Jul 7, 2026
7 checks passed
@jymaire
jymaire deleted the fix/dbt-cloud-retry-read-timeout branch July 7, 2026 07:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants