mirror of
https://github.com/temporalio/temporal.git
synced 2026-08-30 18:41:49 -07:00
bbc86b7eee
## Problem
A standby child workflow's `CloseExecutionTask` verifies its parent
recorded the completion, and past
`MaxLocalParentWorkflowVerificationDuration` also resends the parent
from the active cluster. That resend is a cross-cluster state sync plus
a possibly paginated history backfill — minutes of work — but the whole
call was bounded by the standby task's hard-coded **3s** `taskTimeout`.
Measured at the active cluster's history shard, the deadline arriving
there was `2.999s`. So the resend never completed.
## Change
- **Run the resend in the background**, bounded by
`ReplicationTaskApplyTimeout` — the setting that already bounds this
same work on the replication stream. The verify RPC returns immediately
and the standby task retries until the parent lands, so it never holds a
transfer-queue worker for the sync.
- The background context is **detached from the request** (gRPC cancels
that when the handler returns) and **rooted at the shard lifecycle**, so
the work stops with the shard.
- **One in-flight resend per parent**, tracked in a shard-level map, and
at most `history.parentWorkflowResendMaxInFlight` (8) concurrent resends
per shard. Callers retry while an earlier resend runs; without this the
test measured 5 full state fetches where 1 suffices. The cap bounds the
goroutines this path can create.
- **Lifted two client ceilings** so the deadline can actually propagate
— `admin.SyncWorkflowState` (was 10s) and `history.SyncWorkflowState`
(was 30s) now share a `DefaultStateSyncTimeout` backstop. This also
fixes the same 10s cap on the replication stream's
`ExecutableTaskImpl.SyncState`, where production's 5m setting was never
reachable either.
- Metrics:
`parent_workflow_resend_{attempts,skipped,limited,failures,latency}`.
Async failures reach no caller, so `_failures` is the alert signal;
`_limited` means the shard is shedding resends. The background goroutine
recovers panics, which would otherwise take down the process.
Also fixes the history-client codegen template, which hardcoded
`createContext` and silently ignored the timeout-tier field.
## Rollout
`history.enableAsyncParentWorkflowResend`, **default false**. Disabled =
the previous inline behavior, bounded by the caller's task deadline. Opt
in per cell.
## Testing
Unit tests cover the inline, async, and per-parent-dedup paths.
An xdc test (added in abae9cec, removed in b6eb6631) withholds the
parent's replication tasks so the child *must* pull it, asserts the
parent is absent from the standby, then stalls the active cluster's
`SyncWorkflowState` for 4 minutes:
```
--- PASS: TestChildPullsParentWhenParentReplicationIsWithheld (286.03s)
incoming-ctx-remaining: 4m59.999859334s (2.999s before this change)
sync-state-calls: 1 (5 without the per-parent guard)
dropped-parent-tasks: 9
```
During the stall, 4 verify RPCs reached the standby parent shard (t+0,
+50s, +101s, +169s) and exactly 1 `SyncWorkflowState` reached the active
cluster: the task retried and the guard turned the retries away.
4 minutes exceeds every deadline that previously bounded this path (3s /
10s / 30s) with ~1m headroom against the 5m setting, so the setting is
demonstrably what governs.
To reproduce: `git revert b6eb6631`, then
`go test -tags test_dep ./tests/xdc/ -run
TestVerifyChildCompletionParentResendSuite -timeout 30m`
## Known gaps
- Concurrency across *distinct* parents is unbounded (ordinary fan-out,
not amplification).
- When the parent is deleted on the source, the async path can't report
that back, so the child retries to the 15m discard instead of finishing
immediately. The `workflowNotFoundCache` TODO already in this file would
address it.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
461 lines
14 KiB
Go
461 lines
14 KiB
Go
// Code generated by cmd/tools/genrpcwrappers. DO NOT EDIT.
|
|
|
|
package admin
|
|
|
|
import (
|
|
"context"
|
|
|
|
"go.temporal.io/server/api/adminservice/v1"
|
|
"google.golang.org/grpc"
|
|
)
|
|
|
|
func (c *clientImpl) AddOrUpdateRemoteCluster(
|
|
ctx context.Context,
|
|
request *adminservice.AddOrUpdateRemoteClusterRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.AddOrUpdateRemoteClusterResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.AddOrUpdateRemoteCluster(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) AddSearchAttributes(
|
|
ctx context.Context,
|
|
request *adminservice.AddSearchAttributesRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.AddSearchAttributesResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.AddSearchAttributes(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) AddTasks(
|
|
ctx context.Context,
|
|
request *adminservice.AddTasksRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.AddTasksResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.AddTasks(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) CancelDLQJob(
|
|
ctx context.Context,
|
|
request *adminservice.CancelDLQJobRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.CancelDLQJobResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.CancelDLQJob(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) CloseShard(
|
|
ctx context.Context,
|
|
request *adminservice.CloseShardRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.CloseShardResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.CloseShard(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) DeepHealthCheck(
|
|
ctx context.Context,
|
|
request *adminservice.DeepHealthCheckRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.DeepHealthCheckResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.DeepHealthCheck(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) DeleteWorkflowExecution(
|
|
ctx context.Context,
|
|
request *adminservice.DeleteWorkflowExecutionRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.DeleteWorkflowExecutionResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.DeleteWorkflowExecution(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) DescribeCluster(
|
|
ctx context.Context,
|
|
request *adminservice.DescribeClusterRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.DescribeClusterResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.DescribeCluster(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) DescribeDLQJob(
|
|
ctx context.Context,
|
|
request *adminservice.DescribeDLQJobRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.DescribeDLQJobResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.DescribeDLQJob(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) DescribeHistoryHost(
|
|
ctx context.Context,
|
|
request *adminservice.DescribeHistoryHostRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.DescribeHistoryHostResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.DescribeHistoryHost(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) DescribeMutableState(
|
|
ctx context.Context,
|
|
request *adminservice.DescribeMutableStateRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.DescribeMutableStateResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.DescribeMutableState(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) DescribeTaskQueuePartition(
|
|
ctx context.Context,
|
|
request *adminservice.DescribeTaskQueuePartitionRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.DescribeTaskQueuePartitionResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.DescribeTaskQueuePartition(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) ForceUnloadTaskQueuePartition(
|
|
ctx context.Context,
|
|
request *adminservice.ForceUnloadTaskQueuePartitionRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.ForceUnloadTaskQueuePartitionResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.ForceUnloadTaskQueuePartition(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) GenerateLastHistoryReplicationTasks(
|
|
ctx context.Context,
|
|
request *adminservice.GenerateLastHistoryReplicationTasksRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.GenerateLastHistoryReplicationTasksResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.GenerateLastHistoryReplicationTasks(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) GetDLQMessages(
|
|
ctx context.Context,
|
|
request *adminservice.GetDLQMessagesRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.GetDLQMessagesResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.GetDLQMessages(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) GetDLQReplicationMessages(
|
|
ctx context.Context,
|
|
request *adminservice.GetDLQReplicationMessagesRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.GetDLQReplicationMessagesResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.GetDLQReplicationMessages(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) GetDLQTasks(
|
|
ctx context.Context,
|
|
request *adminservice.GetDLQTasksRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.GetDLQTasksResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.GetDLQTasks(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) GetNamespace(
|
|
ctx context.Context,
|
|
request *adminservice.GetNamespaceRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.GetNamespaceResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.GetNamespace(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) GetNamespaceReplicationMessages(
|
|
ctx context.Context,
|
|
request *adminservice.GetNamespaceReplicationMessagesRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.GetNamespaceReplicationMessagesResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.GetNamespaceReplicationMessages(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) GetReplicationMessages(
|
|
ctx context.Context,
|
|
request *adminservice.GetReplicationMessagesRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.GetReplicationMessagesResponse, error) {
|
|
ctx, cancel := c.createContextWithLargeTimeout(ctx)
|
|
defer cancel()
|
|
return c.client.GetReplicationMessages(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) GetSearchAttributes(
|
|
ctx context.Context,
|
|
request *adminservice.GetSearchAttributesRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.GetSearchAttributesResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.GetSearchAttributes(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) GetShard(
|
|
ctx context.Context,
|
|
request *adminservice.GetShardRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.GetShardResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.GetShard(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) GetTaskQueueTasks(
|
|
ctx context.Context,
|
|
request *adminservice.GetTaskQueueTasksRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.GetTaskQueueTasksResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.GetTaskQueueTasks(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) GetTaskQueueUserData(
|
|
ctx context.Context,
|
|
request *adminservice.GetTaskQueueUserDataRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.GetTaskQueueUserDataResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.GetTaskQueueUserData(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) GetWorkflowExecutionRawHistory(
|
|
ctx context.Context,
|
|
request *adminservice.GetWorkflowExecutionRawHistoryRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.GetWorkflowExecutionRawHistoryResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.GetWorkflowExecutionRawHistory(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) GetWorkflowExecutionRawHistoryV2(
|
|
ctx context.Context,
|
|
request *adminservice.GetWorkflowExecutionRawHistoryV2Request,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.GetWorkflowExecutionRawHistoryV2Response, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.GetWorkflowExecutionRawHistoryV2(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) ImportWorkflowExecution(
|
|
ctx context.Context,
|
|
request *adminservice.ImportWorkflowExecutionRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.ImportWorkflowExecutionResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.ImportWorkflowExecution(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) ListClusterMembers(
|
|
ctx context.Context,
|
|
request *adminservice.ListClusterMembersRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.ListClusterMembersResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.ListClusterMembers(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) ListClusters(
|
|
ctx context.Context,
|
|
request *adminservice.ListClustersRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.ListClustersResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.ListClusters(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) ListHistoryTasks(
|
|
ctx context.Context,
|
|
request *adminservice.ListHistoryTasksRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.ListHistoryTasksResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.ListHistoryTasks(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) ListQueues(
|
|
ctx context.Context,
|
|
request *adminservice.ListQueuesRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.ListQueuesResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.ListQueues(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) MergeDLQMessages(
|
|
ctx context.Context,
|
|
request *adminservice.MergeDLQMessagesRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.MergeDLQMessagesResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.MergeDLQMessages(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) MergeDLQTasks(
|
|
ctx context.Context,
|
|
request *adminservice.MergeDLQTasksRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.MergeDLQTasksResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.MergeDLQTasks(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) MigrateSchedule(
|
|
ctx context.Context,
|
|
request *adminservice.MigrateScheduleRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.MigrateScheduleResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.MigrateSchedule(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) PurgeDLQMessages(
|
|
ctx context.Context,
|
|
request *adminservice.PurgeDLQMessagesRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.PurgeDLQMessagesResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.PurgeDLQMessages(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) PurgeDLQTasks(
|
|
ctx context.Context,
|
|
request *adminservice.PurgeDLQTasksRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.PurgeDLQTasksResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.PurgeDLQTasks(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) ReapplyEvents(
|
|
ctx context.Context,
|
|
request *adminservice.ReapplyEventsRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.ReapplyEventsResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.ReapplyEvents(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) RebuildMutableState(
|
|
ctx context.Context,
|
|
request *adminservice.RebuildMutableStateRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.RebuildMutableStateResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.RebuildMutableState(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) RefreshWorkflowTasks(
|
|
ctx context.Context,
|
|
request *adminservice.RefreshWorkflowTasksRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.RefreshWorkflowTasksResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.RefreshWorkflowTasks(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) RemoveRemoteCluster(
|
|
ctx context.Context,
|
|
request *adminservice.RemoveRemoteClusterRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.RemoveRemoteClusterResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.RemoveRemoteCluster(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) RemoveSearchAttributes(
|
|
ctx context.Context,
|
|
request *adminservice.RemoveSearchAttributesRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.RemoveSearchAttributesResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.RemoveSearchAttributes(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) RemoveTask(
|
|
ctx context.Context,
|
|
request *adminservice.RemoveTaskRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.RemoveTaskResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.RemoveTask(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) ResendReplicationTasks(
|
|
ctx context.Context,
|
|
request *adminservice.ResendReplicationTasksRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.ResendReplicationTasksResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.ResendReplicationTasks(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) StartAdminBatchOperation(
|
|
ctx context.Context,
|
|
request *adminservice.StartAdminBatchOperationRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.StartAdminBatchOperationResponse, error) {
|
|
ctx, cancel := c.createContext(ctx)
|
|
defer cancel()
|
|
return c.client.StartAdminBatchOperation(ctx, request, opts...)
|
|
}
|
|
|
|
func (c *clientImpl) SyncWorkflowState(
|
|
ctx context.Context,
|
|
request *adminservice.SyncWorkflowStateRequest,
|
|
opts ...grpc.CallOption,
|
|
) (*adminservice.SyncWorkflowStateResponse, error) {
|
|
ctx, cancel := c.createContextWithStateSyncTimeout(ctx)
|
|
defer cancel()
|
|
return c.client.SyncWorkflowState(ctx, request, opts...)
|
|
}
|