mirror of
https://github.com/temporalio/temporal.git
synced 2026-08-30 18:41:49 -07:00
bbc86b7eee
## Problem
A standby child workflow's `CloseExecutionTask` verifies its parent
recorded the completion, and past
`MaxLocalParentWorkflowVerificationDuration` also resends the parent
from the active cluster. That resend is a cross-cluster state sync plus
a possibly paginated history backfill — minutes of work — but the whole
call was bounded by the standby task's hard-coded **3s** `taskTimeout`.
Measured at the active cluster's history shard, the deadline arriving
there was `2.999s`. So the resend never completed.
## Change
- **Run the resend in the background**, bounded by
`ReplicationTaskApplyTimeout` — the setting that already bounds this
same work on the replication stream. The verify RPC returns immediately
and the standby task retries until the parent lands, so it never holds a
transfer-queue worker for the sync.
- The background context is **detached from the request** (gRPC cancels
that when the handler returns) and **rooted at the shard lifecycle**, so
the work stops with the shard.
- **One in-flight resend per parent**, tracked in a shard-level map, and
at most `history.parentWorkflowResendMaxInFlight` (8) concurrent resends
per shard. Callers retry while an earlier resend runs; without this the
test measured 5 full state fetches where 1 suffices. The cap bounds the
goroutines this path can create.
- **Lifted two client ceilings** so the deadline can actually propagate
— `admin.SyncWorkflowState` (was 10s) and `history.SyncWorkflowState`
(was 30s) now share a `DefaultStateSyncTimeout` backstop. This also
fixes the same 10s cap on the replication stream's
`ExecutableTaskImpl.SyncState`, where production's 5m setting was never
reachable either.
- Metrics:
`parent_workflow_resend_{attempts,skipped,limited,failures,latency}`.
Async failures reach no caller, so `_failures` is the alert signal;
`_limited` means the shard is shedding resends. The background goroutine
recovers panics, which would otherwise take down the process.
Also fixes the history-client codegen template, which hardcoded
`createContext` and silently ignored the timeout-tier field.
## Rollout
`history.enableAsyncParentWorkflowResend`, **default false**. Disabled =
the previous inline behavior, bounded by the caller's task deadline. Opt
in per cell.
## Testing
Unit tests cover the inline, async, and per-parent-dedup paths.
An xdc test (added in abae9cec, removed in b6eb6631) withholds the
parent's replication tasks so the child *must* pull it, asserts the
parent is absent from the standby, then stalls the active cluster's
`SyncWorkflowState` for 4 minutes:
```
--- PASS: TestChildPullsParentWhenParentReplicationIsWithheld (286.03s)
incoming-ctx-remaining: 4m59.999859334s (2.999s before this change)
sync-state-calls: 1 (5 without the per-parent guard)
dropped-parent-tasks: 9
```
During the stall, 4 verify RPCs reached the standby parent shard (t+0,
+50s, +101s, +169s) and exactly 1 `SyncWorkflowState` reached the active
cluster: the task retried and the guard turned the retries away.
4 minutes exceeds every deadline that previously bounded this path (3s /
10s / 30s) with ~1m headroom against the 5m setting, so the setting is
demonstrably what governs.
To reproduce: `git revert b6eb6631`, then
`go test -tags test_dep ./tests/xdc/ -run
TestVerifyChildCompletionParentResendSuite -timeout 30m`
## Known gaps
- Concurrency across *distinct* parents is unbounded (ordinary fan-out,
not amplification).
- When the parent is deleted on the source, the async path can't report
that back, so the child retries to the 15m discard instead of finishing
immediately. The `workflowNotFoundCache` TODO already in this file would
address it.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>