mirror of
https://github.com/temporalio/temporal.git
synced 2026-08-30 18:41:49 -07:00
## What changed? Add a new replication task type `DeleteExecutionReplicationTask` that replicates workflow deletion from the active cluster to passive/standby clusters. Gated by feature flag `history.enableDeleteWorkflowExecutionReplication` (default: false). #### **Key changes across the replication pipeline:** 1. Proto enums: `TASK_TYPE_REPLICATION_DELETE_EXECUTION` (34), `REPLICATION_TASK_TYPE_DELETE_EXECUTION_TASK` (13) 2. Replication task is associated with a new stage in `ShardContext.DeleteWorkflowExecution`, bundled with delete visibility task. 3. ~Engine interface: added `ForceDeleteWorkflowExecution` so the task can invoke the `ForceDeleteWorkflowExecution`.~ ## Why? Today, when a user delete workflow execution in source cluster, this operation will not replicate to the standby/target clusters. When a namespace failover to a target cluster, those deleted workflow may resurrected. <details> <summary>Race condition analysis</summary> **Before this change:** 1. **Cross-cluster resurrection:** Active deletes workflow → standby untouched → failover → workflow reappears. 2. **Termination event silently dropped:** Deleting a running workflow terminates it first, generating a `HistoryReplicationTask`. But the async `CloseExecutionTask` may delete mutable state before the stream sender converts that task. The converter calls `getBranchToken()` → `NotFound` → task silently dropped. The standby never sees the termination or the deletion. **After this change:** Race 1 is fixed — `DeleteExecutionReplicationTask` explicitly tells the standby to delete. Race 2 is mitigated — even if the termination event's replication task is dropped, the delete replication task ensures the standby cleans up. - If the workflow is still running (termination not yet replicated), the `DeleteExecutionTask` reschedules itself until the workflow closes. - If the termination event arrives later, the workflow closes normally, then the delete proceeds. - If the workflow is already deleted (e.g., by retention), the task is a no-op (`NotFound` treated as success). </details> <details> <summary>Deletion paths</summary> | Path | Replication task? | |------|-------------------| | User deletes workflow (active, running or closed) | Yes | | User deletes on passive (DC forwarding ON) | Forwarded to active → yes | | User deletes on passive (no forwarding) | No — `ActiveInCluster` check skips | | Retention expiry (with or without archival) | No — stage pre-marked as processed | | Admin ForceDelete (tdbg) | No — bypasses `DeleteWorkflowExecution` | </details> ## How did you test it? - [x] built - [x] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) Before change: <img width="1507" height="163" alt="Screenshot 2026-03-26 at 11 59 51 PM" src="https://github.com/user-attachments/assets/118cc50e-b69d-468a-9e45-5f49e4e4b9d1" /> After change: <img width="1507" height="135" alt="Screenshot 2026-03-27 at 12 00 07 AM" src="https://github.com/user-attachments/assets/8ccb7a11-2cb4-48b5-af89-b4a60ddb6333" /> ## Potential risks n/a
93 lines
3.1 KiB
Go
93 lines
3.1 KiB
Go
// Code generated by protoc-gen-go-helpers. DO NOT EDIT.
|
|
package enums
|
|
|
|
import (
|
|
"fmt"
|
|
)
|
|
|
|
var (
|
|
TaskSource_shorthandValue = map[string]int32{
|
|
"Unspecified": 0,
|
|
"History": 1,
|
|
"DbBacklog": 2,
|
|
}
|
|
)
|
|
|
|
// TaskSourceFromString parses a TaskSource value from either the protojson
|
|
// canonical SCREAMING_CASE enum or the traditional temporal PascalCase enum to TaskSource
|
|
func TaskSourceFromString(s string) (TaskSource, error) {
|
|
if v, ok := TaskSource_value[s]; ok {
|
|
return TaskSource(v), nil
|
|
} else if v, ok := TaskSource_shorthandValue[s]; ok {
|
|
return TaskSource(v), nil
|
|
}
|
|
return TaskSource(0), fmt.Errorf("%s is not a valid TaskSource", s)
|
|
}
|
|
|
|
var (
|
|
TaskType_shorthandValue = map[string]int32{
|
|
"Unspecified": 0,
|
|
"ReplicationHistory": 1,
|
|
"ReplicationSyncActivity": 2,
|
|
"TransferWorkflowTask": 3,
|
|
"TransferActivityTask": 4,
|
|
"TransferCloseExecution": 5,
|
|
"TransferCancelExecution": 6,
|
|
"TransferStartChildExecution": 7,
|
|
"TransferSignalExecution": 8,
|
|
"TransferResetWorkflow": 10,
|
|
"WorkflowTaskTimeout": 12,
|
|
"ActivityTimeout": 13,
|
|
"UserTimer": 14,
|
|
"WorkflowRunTimeout": 15,
|
|
"DeleteHistoryEvent": 16,
|
|
"ActivityRetryTimer": 17,
|
|
"WorkflowBackoffTimer": 18,
|
|
"VisibilityStartExecution": 19,
|
|
"VisibilityUpsertExecution": 20,
|
|
"VisibilityCloseExecution": 21,
|
|
"VisibilityDeleteExecution": 22,
|
|
"TransferDeleteExecution": 24,
|
|
"ReplicationSyncWorkflowState": 25,
|
|
"ArchivalArchiveExecution": 26,
|
|
"StateMachineOutbound": 27,
|
|
"StateMachineTimer": 28,
|
|
"WorkflowExecutionTimeout": 29,
|
|
"ReplicationSyncHsm": 30,
|
|
"ReplicationSyncVersionedTransition": 31,
|
|
"ChasmPure": 32,
|
|
"Chasm": 33,
|
|
"ReplicationDeleteExecution": 34,
|
|
}
|
|
)
|
|
|
|
// TaskTypeFromString parses a TaskType value from either the protojson
|
|
// canonical SCREAMING_CASE enum or the traditional temporal PascalCase enum to TaskType
|
|
func TaskTypeFromString(s string) (TaskType, error) {
|
|
if v, ok := TaskType_value[s]; ok {
|
|
return TaskType(v), nil
|
|
} else if v, ok := TaskType_shorthandValue[s]; ok {
|
|
return TaskType(v), nil
|
|
}
|
|
return TaskType(0), fmt.Errorf("%s is not a valid TaskType", s)
|
|
}
|
|
|
|
var (
|
|
TaskPriority_shorthandValue = map[string]int32{
|
|
"Unspecified": 0,
|
|
"High": 1,
|
|
"Low": 10,
|
|
}
|
|
)
|
|
|
|
// TaskPriorityFromString parses a TaskPriority value from either the protojson
|
|
// canonical SCREAMING_CASE enum or the traditional temporal PascalCase enum to TaskPriority
|
|
func TaskPriorityFromString(s string) (TaskPriority, error) {
|
|
if v, ok := TaskPriority_value[s]; ok {
|
|
return TaskPriority(v), nil
|
|
} else if v, ok := TaskPriority_shorthandValue[s]; ok {
|
|
return TaskPriority(v), nil
|
|
}
|
|
return TaskPriority(0), fmt.Errorf("%s is not a valid TaskPriority", s)
|
|
}
|