mirror of
https://github.com/temporalio/temporal.git
synced 2026-08-30 18:41:49 -07:00
## What changed? Add a new replication task type `DeleteExecutionReplicationTask` that replicates workflow deletion from the active cluster to passive/standby clusters. Gated by feature flag `history.enableDeleteWorkflowExecutionReplication` (default: false). #### **Key changes across the replication pipeline:** 1. Proto enums: `TASK_TYPE_REPLICATION_DELETE_EXECUTION` (34), `REPLICATION_TASK_TYPE_DELETE_EXECUTION_TASK` (13) 2. Replication task is associated with a new stage in `ShardContext.DeleteWorkflowExecution`, bundled with delete visibility task. 3. ~Engine interface: added `ForceDeleteWorkflowExecution` so the task can invoke the `ForceDeleteWorkflowExecution`.~ ## Why? Today, when a user delete workflow execution in source cluster, this operation will not replicate to the standby/target clusters. When a namespace failover to a target cluster, those deleted workflow may resurrected. <details> <summary>Race condition analysis</summary> **Before this change:** 1. **Cross-cluster resurrection:** Active deletes workflow → standby untouched → failover → workflow reappears. 2. **Termination event silently dropped:** Deleting a running workflow terminates it first, generating a `HistoryReplicationTask`. But the async `CloseExecutionTask` may delete mutable state before the stream sender converts that task. The converter calls `getBranchToken()` → `NotFound` → task silently dropped. The standby never sees the termination or the deletion. **After this change:** Race 1 is fixed — `DeleteExecutionReplicationTask` explicitly tells the standby to delete. Race 2 is mitigated — even if the termination event's replication task is dropped, the delete replication task ensures the standby cleans up. - If the workflow is still running (termination not yet replicated), the `DeleteExecutionTask` reschedules itself until the workflow closes. - If the termination event arrives later, the workflow closes normally, then the delete proceeds. - If the workflow is already deleted (e.g., by retention), the task is a no-op (`NotFound` treated as success). </details> <details> <summary>Deletion paths</summary> | Path | Replication task? | |------|-------------------| | User deletes workflow (active, running or closed) | Yes | | User deletes on passive (DC forwarding ON) | Forwarded to active → yes | | User deletes on passive (no forwarding) | No — `ActiveInCluster` check skips | | Retention expiry (with or without archival) | No — stage pre-marked as processed | | Admin ForceDelete (tdbg) | No — bypasses `DeleteWorkflowExecution` | </details> ## How did you test it? - [x] built - [x] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) Before change: <img width="1507" height="163" alt="Screenshot 2026-03-26 at 11 59 51 PM" src="https://github.com/user-attachments/assets/118cc50e-b69d-468a-9e45-5f49e4e4b9d1" /> After change: <img width="1507" height="135" alt="Screenshot 2026-03-27 at 12 00 07 AM" src="https://github.com/user-attachments/assets/8ccb7a11-2cb4-48b5-af89-b4a60ddb6333" /> ## Potential risks n/a
75 lines
2.6 KiB
Go
75 lines
2.6 KiB
Go
// Code generated by protoc-gen-go-helpers. DO NOT EDIT.
|
|
package enums
|
|
|
|
import (
|
|
"fmt"
|
|
)
|
|
|
|
var (
|
|
ReplicationTaskType_shorthandValue = map[string]int32{
|
|
"Unspecified": 0,
|
|
"NamespaceTask": 1,
|
|
"HistoryTask": 2,
|
|
"SyncShardStatusTask": 3,
|
|
"SyncActivityTask": 4,
|
|
"HistoryMetadataTask": 5,
|
|
"HistoryV2Task": 6,
|
|
"SyncWorkflowStateTask": 7,
|
|
"TaskQueueUserData": 8,
|
|
"SyncHsmTask": 9,
|
|
"BackfillHistoryTask": 10,
|
|
"VerifyVersionedTransitionTask": 11,
|
|
"SyncVersionedTransitionTask": 12,
|
|
"DeleteExecutionTask": 13,
|
|
}
|
|
)
|
|
|
|
// ReplicationTaskTypeFromString parses a ReplicationTaskType value from either the protojson
|
|
// canonical SCREAMING_CASE enum or the traditional temporal PascalCase enum to ReplicationTaskType
|
|
func ReplicationTaskTypeFromString(s string) (ReplicationTaskType, error) {
|
|
if v, ok := ReplicationTaskType_value[s]; ok {
|
|
return ReplicationTaskType(v), nil
|
|
} else if v, ok := ReplicationTaskType_shorthandValue[s]; ok {
|
|
return ReplicationTaskType(v), nil
|
|
}
|
|
return ReplicationTaskType(0), fmt.Errorf("%s is not a valid ReplicationTaskType", s)
|
|
}
|
|
|
|
var (
|
|
NamespaceOperation_shorthandValue = map[string]int32{
|
|
"Unspecified": 0,
|
|
"Create": 1,
|
|
"Update": 2,
|
|
}
|
|
)
|
|
|
|
// NamespaceOperationFromString parses a NamespaceOperation value from either the protojson
|
|
// canonical SCREAMING_CASE enum or the traditional temporal PascalCase enum to NamespaceOperation
|
|
func NamespaceOperationFromString(s string) (NamespaceOperation, error) {
|
|
if v, ok := NamespaceOperation_value[s]; ok {
|
|
return NamespaceOperation(v), nil
|
|
} else if v, ok := NamespaceOperation_shorthandValue[s]; ok {
|
|
return NamespaceOperation(v), nil
|
|
}
|
|
return NamespaceOperation(0), fmt.Errorf("%s is not a valid NamespaceOperation", s)
|
|
}
|
|
|
|
var (
|
|
ReplicationFlowControlCommand_shorthandValue = map[string]int32{
|
|
"Unspecified": 0,
|
|
"Resume": 1,
|
|
"Pause": 2,
|
|
}
|
|
)
|
|
|
|
// ReplicationFlowControlCommandFromString parses a ReplicationFlowControlCommand value from either the protojson
|
|
// canonical SCREAMING_CASE enum or the traditional temporal PascalCase enum to ReplicationFlowControlCommand
|
|
func ReplicationFlowControlCommandFromString(s string) (ReplicationFlowControlCommand, error) {
|
|
if v, ok := ReplicationFlowControlCommand_value[s]; ok {
|
|
return ReplicationFlowControlCommand(v), nil
|
|
} else if v, ok := ReplicationFlowControlCommand_shorthandValue[s]; ok {
|
|
return ReplicationFlowControlCommand(v), nil
|
|
}
|
|
return ReplicationFlowControlCommand(0), fmt.Errorf("%s is not a valid ReplicationFlowControlCommand", s)
|
|
}
|