## Problem
A standby child workflow's `CloseExecutionTask` verifies its parent
recorded the completion, and past
`MaxLocalParentWorkflowVerificationDuration` also resends the parent
from the active cluster. That resend is a cross-cluster state sync plus
a possibly paginated history backfill — minutes of work — but the whole
call was bounded by the standby task's hard-coded **3s** `taskTimeout`.
Measured at the active cluster's history shard, the deadline arriving
there was `2.999s`. So the resend never completed.
## Change
- **Run the resend in the background**, bounded by
`ReplicationTaskApplyTimeout` — the setting that already bounds this
same work on the replication stream. The verify RPC returns immediately
and the standby task retries until the parent lands, so it never holds a
transfer-queue worker for the sync.
- The background context is **detached from the request** (gRPC cancels
that when the handler returns) and **rooted at the shard lifecycle**, so
the work stops with the shard.
- **One in-flight resend per parent**, tracked in a shard-level map, and
at most `history.parentWorkflowResendMaxInFlight` (8) concurrent resends
per shard. Callers retry while an earlier resend runs; without this the
test measured 5 full state fetches where 1 suffices. The cap bounds the
goroutines this path can create.
- **Lifted two client ceilings** so the deadline can actually propagate
— `admin.SyncWorkflowState` (was 10s) and `history.SyncWorkflowState`
(was 30s) now share a `DefaultStateSyncTimeout` backstop. This also
fixes the same 10s cap on the replication stream's
`ExecutableTaskImpl.SyncState`, where production's 5m setting was never
reachable either.
- Metrics:
`parent_workflow_resend_{attempts,skipped,limited,failures,latency}`.
Async failures reach no caller, so `_failures` is the alert signal;
`_limited` means the shard is shedding resends. The background goroutine
recovers panics, which would otherwise take down the process.
Also fixes the history-client codegen template, which hardcoded
`createContext` and silently ignored the timeout-tier field.
## Rollout
`history.enableAsyncParentWorkflowResend`, **default false**. Disabled =
the previous inline behavior, bounded by the caller's task deadline. Opt
in per cell.
## Testing
Unit tests cover the inline, async, and per-parent-dedup paths.
An xdc test (added in abae9cec, removed in b6eb6631) withholds the
parent's replication tasks so the child *must* pull it, asserts the
parent is absent from the standby, then stalls the active cluster's
`SyncWorkflowState` for 4 minutes:
```
--- PASS: TestChildPullsParentWhenParentReplicationIsWithheld (286.03s)
incoming-ctx-remaining: 4m59.999859334s (2.999s before this change)
sync-state-calls: 1 (5 without the per-parent guard)
dropped-parent-tasks: 9
```
During the stall, 4 verify RPCs reached the standby parent shard (t+0,
+50s, +101s, +169s) and exactly 1 `SyncWorkflowState` reached the active
cluster: the task retried and the guard turned the retries away.
4 minutes exceeds every deadline that previously bounded this path (3s /
10s / 30s) with ~1m headroom against the 5m setting, so the setting is
demonstrably what governs.
To reproduce: `git revert b6eb6631`, then
`go test -tags test_dep ./tests/xdc/ -run
TestVerifyChildCompletionParentResendSuite -timeout 30m`
## Known gaps
- Concurrency across *distinct* parents is unbounded (ordinary fan-out,
not amplification).
- When the parent is deleted on the source, the async path can't report
that back, so the child retries to the 15m discard instead of finishing
immediately. The `workflowNotFoundCache` TODO already in this file would
address it.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
This centralizes the health checker into `health.SignalAggregator`. This
standardizes how we will do health checks by using latency based
quantiles and error ratios.
The goal is to be able to use this for any spot we want to do health
detection.
Main idea is you have your overall settings and thresholds for latency
metrics and error ratios, then you can have "groups" of specific keys
that have their own tracking for latency and error ratios. For grpc
health checking, these keys will be the endpoints. For something like
history the keys will be the specific functions.
Doing this also sets us up better for per namespace health settings.
This is 1 of 3 prs. This one is mainly for just testing that everything
works as expected. Then next one will be cleaning up and consolidating
all the health signals in the health package. Then the last one will be
applying
## Why?
We need more strict settings/thresholds for endpoints we consider to be
critical like StartWorkflowExecution, SignalWorkflowExecution,
StartActivityExecution, etc...
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Currently, if you pass in bad settings (e.g. buffer size <= 0), there
will be panics. This isn't a newly introduced problem, but it is an
existing one that will be addressed in future PRs.
> Part 1 of a planned 5-PR series building toward replication stream
namespace isolation (a restructuring of #10147): read buffer → reader
group → lane protocol → isolation manager → sender isolation. This PR
stands on its own; follow-ups will be opened as each lands review.
**Next in series: #11302** (reader group).
## What changed?
A shard-scoped read-through buffer over the tip of the replication task
queue, sitting inside the ack manager's `GetReplicationTasksIter`. Every
replication stream sender on a shard — one per remote cluster, with one
iterator per priority lane — scans the same queue, so each new task page
was previously read from persistence once per scanner. With the buffer,
overlapping tip scans share one persistence read; only readers below the
buffered range (deep catch-up, lanes that have lagged out of coverage)
fall through to persistence.
Coverage is a contiguous task-id interval established by persistence
pages: within it the buffer is authoritative, so absence of a task means
the range holds none. An empty persistence page with a non-empty
continuation token (legal, e.g. under Cassandra paging) is NOT
authoritative — the fetch keeps paging until tasks arrive or the token
runs out. Rows are immutable and the queue is append-only, so there is
no invalidation; eviction just shrinks coverage from the front. Reads
are bounded by the shard's exclusive-high read watermark, so covered
ranges are stable once established.
**Ownership:** the buffer stores SERIALIZED rows and deserializes per
serve, so every reader receives fresh task structs it exclusively owns —
exactly what a persistence read would have produced. This matters
because downstream converters mutate tasks in place (e.g.
`SyncVersionedTransitionTask` equivalents get IDs assigned via
`AddTasks`); handing multiple senders pointers to shared structs would
be a data race. The serve-time deserialization replaces the persistence
read the reader would otherwise have done, so it is not added cost
relative to the unbuffered path. Serialize/deserialize failures are
never silent: both are error-logged (they indicate a bug — e.g. a task
type missing serializer support — or broken persistence data); a
serialize failure serves the page uncached, a deserialize failure drops
the buffer's coverage entirely and falls back to persistence.
Capacity is `ReplicationStreamReadBufferSize` tasks per shard (default 0
= disabled); disabling at runtime releases the buffered rows. The buffer
holds slim queue rows (task metadata) for every priority — event
payloads only enter the pipeline at send-time conversion — so memory
cost is a few hundred bytes per row.
Observability (to drive future sizing/sharing decisions):
`replication_stream_read_buffer_hits` / `_misses` count pages served
from memory vs. fetched from persistence while the buffer is enabled
(misses are counted only after a successful fetch), and
`replication_stream_read_buffer_miss_lag` records, for misses below the
buffered range, how far below coverage the read began (in task ids).
Small lag values mean a larger buffer would convert those misses to
hits; large values mean readers deep in backlog, where no tip buffer
helps.
## Why?
A standalone win for the code as it is today, with no dependency on the
rest of this series: any multi-cluster mesh already pays `remote
clusters × priority lanes` read amplification on the queue tip, and the
buffer collapses those overlapping scans into one persistence read per
page. It additionally unlocks the later PRs in the series: per-namespace
isolation lanes multiply the number of concurrent scanners, and with the
buffer their tip scans become in-memory filter passes instead of extra
persistence load.
## How did you test it?
- [x] built
- [x] added new unit test(s) — pass-through when disabled, memory
serving for second readers and partial overlaps, contiguous coverage
extension, below-coverage fall-through without cache disturbance, gap
restart at a newer tip, front eviction, truncated-page authority bounds,
hit/miss/lag metric emission, per-reader ownership of served rows,
runtime disable releasing state, empty-page-with-token continuation in
`GetReplicationTasksIter`, and a `-race` concurrent-readers stress test
over a moving tip
- [x] covered by existing tests — the xdc isolation test later in the
series runs with the buffer enabled
## Potential risks
Default-off. The main correctness surface is coverage bookkeeping
(serving a range the buffer isn't authoritative for); the
coverage-interval design plus the truncated-fetch and empty-page-token
tests target exactly that. Serve-time deserialization guarantees no
cross-reader object sharing, and round-trip failures are loud (error
logs) rather than silently degrading. Memory is strictly bounded by the
row-count cap and released on disable.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes replication task loading and coverage bookkeeping in the ack
manager; default-off but incorrect coverage could skip or mis-serve
tasks when the buffer is enabled.
>
> **Overview**
> Adds a **shard-scoped read-through buffer** on the replication task
queue tip so overlapping scans from every remote cluster and priority
lane can share one persistence read instead of amplifying reads per
scanner.
>
> **`readBuffer`** (`read_buffer.go`) tracks contiguous task-id
coverage, stores serialized slim queue rows, and deserializes per serve
so each reader gets owned task structs (downstream code mutates tasks in
place). Capacity is **`ReplicationStreamReadBufferSize`** (default **0**
= disabled); disabling at runtime clears buffered state.
>
> **`GetReplicationTasksIter`** in the ack manager routes reads through
the buffer and tightens persistence paging: empty pages with a
continuation token keep paging until tasks arrive or the token is empty;
truncated pages only extend coverage through the last returned task id.
>
> New metrics: **`replication_stream_read_buffer_hits`**, **`_misses`**,
and **`_miss_lag`** for tuning buffer size and observing catch-up
behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
5836b28b30. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
## What changed?
Add dynamic config to flip skip persistence optimization.
## Why?
Allow backwards compatible rollout of skip persistence flag.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Adds two limits on the memory held by in-flight buffer for paginated
RespondWorkflowTaskCompleted requests:
- Process-wide limit on the total bytes buffered across all workflows on
a history host. Dynamic config is
`WorkflowTaskCompletionBufferTotalSizeLimit`
- Per-namespace share of that process-wide limit (expressed as a ratio),
so a single namespace can't consume the entire host budget.
`WorkflowTaskCompletionBufferNamespaceRatio`
When buffering a page would push either limit over the top, the page is
rejected with the existing transient buffer-lost signal and the
in-progress buffer is dropped.
## Why?
Pagination lets one workflow task ship large volume of commands split
across several requests, which the server holds in memory until the
final page arrives. Without a ceiling, many concurrent large completions
or one heavy namespace could exhaust a history host's memory. The
process-wide limit bounds total exposure, and the per-namespace share
keeps one namespace from starving the others.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
* Gated behind the pagination feature flag (off by default), so there's
no effect until pagination is enabled for a namespace.
* The limit is enforced per process; a namespace that stays over its
share will keep hitting buffer-lost and retrying until its in-flight
buffers drain. This is recoverable but could show up as retry churn if a
limit is set too low.
## What changed?
Adds a per-namespace **percentage** control for routing new
workflow-triggered Nexus operations to the CHASM implementation, on top
of the existing boolean `nexusoperation.enableChasmWorkflowOperations`
flag.
- New dynamic config
`nexusoperation.chasmWorkflowOperationsRolloutPercent` (int, 0–100),
**default 0**.
- The HSM→CHASM creation decision is centralized in one predicate,
`nexusoperation.UseChasmForWorkflow(enabled, rolloutPercent,
namespaceName, workflowID)`: an operation is created on the CHASM tree
only when the boolean flag is on **AND** the workflow falls within the
rollout percentage. Membership is decided by the shared `RolloutAccepts`
helper hashing `namespace + workflowID` (same key shape as the Scheduler
CHASM rollout), so a given workflow deterministically lands on the same
implementation across all of its operations and dialing the percentage
up is monotonic.
- **Both** framework-decision sites use the same predicate:
- **Live creation** — the CHASM `ScheduleNexusOperation` command handler
(`handleScheduleCommand`). Out-of-rollout → `ErrCommandNotSupported`, so
the operation is created on the HSM tree as before.
- **Reset / replication rebuild** —
`MutableStateRebuilder.applyChasmEvent`, on the
`NexusOperationScheduled` create event. Sharing the exact predicate is
required: otherwise a reset could flip an out-of-rollout workflow's
operations onto CHASM.
- **Cancelation is intentionally not gated** by this predicate. It is
routed by the tree that already owns the operation, so an operation
created while the flag/percentage was higher can still be canceled after
a downgrade.
## Why?
The per-namespace boolean is too coarse for a safe migration: turning it
on sends *all* of a namespace's new Nexus operations to CHASM at once. A
percentage control lets us dial CHASM adoption up gradually *within* a
namespace and roll back by dialing down — the de-risking control our
rollout plan requires. The default of 0 makes the boolean alone a no-op
until the percentage is explicitly raised, so enabling the flag can't
cause an all-at-once cutover.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Low, and gated behind the default. Default 0 means behavior is unchanged
for any namespace that hasn't set the percentage — enabling the boolean
alone now routes nothing to CHASM until the percentage is raised. The
one behavioral consequence: any test/config that enables the boolean
expecting CHASM must also set the percentage (done for the tests in the
repo); no production config force-enables the boolean today.
## What changed?
This adds configurable latency thresholds for persistence health checks
so that we can dynamically change what percentiles we want to look at,
as well as what values we want to alert on.
This also adds enforceability as a concept to the health checks so that
we can have them reported in a safer way when testing.
These latencies are computed using the same tdigest based library we use
in history health checks, which will allow us to get much more useful
percentile values than the current average only reporting.
## Why?
So we can test out different percentile settings and evaluate good
defaults.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Changing health checks is always risky, but this adds a softer version
of them so it should make things safer ultimately.
## What changed?
Foundational half of CHASM Nexus operation support in workflow reset /
replication. Brings the shared event-reapply / cherry-pick path (used by
workflow reset **and** replication conflict resolution) to handle
CHASM-backed Nexus operations, and adds the plumbing both halves need.
- `ndc/workflow_resetter.go` — `reapplyEvents` probes HSM, then falls
back to the CHASM workflow registry, using an explicit
`cherryPickOutcome` (applied / skipped / fallback).
- rest of the changes are related to plumbing of CHASM workflow registry
and enableChasmWorkflowOperations dynamic config
## Why?
Reset (and replication conflict resolution) reapply post-reset-point /
divergent-branch events to the surviving run. Without this, a
CHASM-backed Nexus operation event that must be reapplied is dropped or
mis-routed.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- Changes `ndc.NewEventsReapplier`'s signature. External consumer
saas-temporal calls it, so companion PR temporalio/saas-temporal
(matching branch `gokhan/nexus-reset-reapply`) updates that call site;
the crossrepo check validates them together.
## What changed?
This adds configurable latency thresholds for history health checks so
that we can dynamically change what percentiles we want to look at, as
well as what values we want to alert on.
This also adds enforceability as a concept to the health checks so that
we can have them reported in a safer way when testing.
## Why?
So we can test out different percentile settings and evaluate good
defaults.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Changing health checks is always risky, but this adds a softer version
of them so it should make things safer ultimately.
…fecycle events
Add common/events: a pluggable events.Handler for emitting structured
("wide") events, with a typed Encoder (incl. Any for whole-object
values), a global event registry (NewEventDef) with startup
duplicate-name validation, a default handler that logs each event as one
line keyed by the event type, and a noop handler. Wired through the
server via WithCustomEventHandler, catalog validation at bootstrap, a
per-service fx provider, and a GetEventHandler() accessor on
ShardContext.
Define two events on the framework:
- NamespaceLifecycle: a generic, phase-discriminated namespace event
(stable identity fields + a nested "details" object). Emitters supply
the phases.
- ReplicationLifecycle: traces a replication task sent -> executing ->
applied across sync_workflow_state / sync_versioned_transition /
verify_versioned_transition, emitted at the stream sender, the passive
executables, and the ndc workflow-state replicator (post-apply
mutable-state summary, no extra read).
## What changed?
^
## Why?
Improve lifecycle observability
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Stephan Behnke <stephanos@users.noreply.github.com>
## What changed?
Adds server-side support for paginating `RespondWorkflowTaskCompleted`
requests, gated behind the new per-namespace dynamic config
`history.enableWorkflowTaskCompletionPagination` (default off).
- **Buffering on the workflow context**
(`service/history/workflow/context.go`): a new `TaskCompletionBuffer`
holds intermediate pages keyed by page number, scoped to a workflow task
identity `(scheduledEventID, attempt)`. New `WorkflowContext` methods
`AppendTaskCompletionPage` and `GetMergedTaskCompletionPages` buffer
intermediate pages and concatenate them (in order, pages
`0..finalPageNumber-1`) ahead of the final page's commands. The buffer
is reclaimed by `Clear()` and dropped by
`reconcileTaskCompletionBuffer()` when the workflow task is no longer
current (timed out, failed, completed, or workflow closed).
- **Handler integration**
(`service/history/api/respondworkflowtaskcompleted/api.go`): when
pagination is enabled, intermediate pages (`intermediate_page=true`) are
buffered and acknowledged without persisting; the final page merges the
buffered commands before continuing through the normal completion path.
When pagination is disabled, any paginated request is rejected with
`FailedPrecondition`.
- **Limits**: a per-workflow-task buffer size limit
(`history.workflowTaskCompletionBufferSizeLimit`, default 40 MiB) and a
hard limit of 1024 pages as a sanity check. Overflow fails the workflow
task with `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. A
lost/incomplete buffer returns a transient `WorkflowTaskBufferLost`
error so the SDK can resend.
- **Capability + config plumbing**: surfaces
`WorkflowTaskCompletionPagination` in `NamespaceInfo.Capabilities`
(`service/frontend/namespace_handler.go`, `service/frontend/service.go`)
and wires the new history configs (`service/history/configs/config.go`).
- **Metrics** (`common/metrics/metric_defs.go`):
`workflow_task_completion_paginated_bytes` (histogram of total
completion wire size) and `workflow_task_completion_buffer_lost`
(counter).
## Why?
A single RespondWorkflowTaskCompleted request can carry more commands
than the gRPC max message size allows, which currently blocks workflows
that emit a large number of commands in one task. Pagination lets the
SDK split a completion into multiple pages that the server reassembles.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [x] added new unit test(s)
(`service/history/workflow/context_test.go`,
`service/history/api/respondworkflowtaskcompleted/api_test.go`)
- [x] added new functional test(s)
(`tests/workflow_completion_pagination_test.go`)
## Potential risks
- Feature is fully gated by
`history.enableWorkflowTaskCompletionPagination` (default off),
## What changed?
- Replace the hardcoded `shrinkPredicateMaxPendingKeys = 3` with a new
dynamic config `history.queueShrinkPredicateMaxPendingKeys` (default
10), wired into all history queues.
- Emit a new `queue_predicate_resolution_loss` counter, tagged by
`reason`, in the two places a slice gives up exact predicate resolution:
when the shrink is skipped (`reason=max_pending_keys`) and when the
predicate falls back to the universal predicate
(`reason=predicate_size`).
## Why?
The shrink threshold was a magic constant with no way to tune it in
production, and there was no signal when a slice loses resolution and
starts reprocessing extra tasks. The metric makes both cases observable.
## How did you test it?
- [x] built
- [x] covered by existing tests
## What
Changes `system.enableCancelActivityWorkerCommand` from a global dynamic
config setting to a namespace-scoped setting.
## Why
To control rollout per namespace.
## How did you test it?
Updated unit tests.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
Removed the `EnableDeleteWorkflowExecutionReplication` dynamic config
flag and its history config wiring. Delete workflow execution
replication tasks are now always generated for eligible active global
namespaces.
Also removed test overrides for the flag and deleted the disabled-flag
XDC test case.
## Why?
The rollout is completed and clean up the feature flag.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Delete execution replication can no longer be disabled dynamically.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
New `MaximumEventBatchSizeInBytes` dynamic config, disabled by default.
When set, EventStore rolls the current in-memory batch and starts a new
one before appending an event that would push the cumulative serialized
size over the threshold. A single oversized event still gets its own
batch. If enabled things will break because of the incorrect event batch
id references, so there is a big warning.
## Why?
This is an initial step for support of workflow tasks completion
requests larger than gRPC limit. Today, for such large requests the size
of the batch could easily exceed the configured tx limit
system.transactionSizeLimit. Because of that, we have to generate
smaller batches, which this PR enables to do.
There are many places in the code, where we assume that the event lands
in the same batch as the corresponding WorkflowTaskCompleted event. So
currently, if MaximumEventBatchSizeInBytes is set to >0, things will
break. But adding this early on allows to find the issues in the code
faster, by setting MaximumEventBatchSizeInBytes to a small value.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
This change doesn't have any effect unless MaximumEventBatchSizeInBytes
is tuned from the default value of 0. The config comment explicitly says
that this is experimental and things will break if it is enabled.
## What changed?
Added support for Nexus workflow update completion callbacks via CHASM.
This allows a Nexus caller to be notified when a workflow update
completes by attaching completion callbacks to the update request.
## Why?
Nexus operations that target workflow updates need a way to receive
completion notifications. Without this, a Nexus caller that sends an
update has no async mechanism to learn when the update finishes.
Completion callbacks enable the same async notification pattern that
already exists for workflow-level Nexus operations.
## How did you test it?
- [ ] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Touches speculative workflow updates, they are always hard to reason
about. Tried to compensate with lots of test coverage.
Note: Needs this API PR
https://github.com/temporalio/api/pull/742/changes
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **High Risk**
> Touches workflow update state machine and mutable state event handling
to persist/trigger per-update callbacks, including close/retry/reset
paths, which is complex and can affect correctness of update outcomes
and callback delivery.
>
> **Overview**
> Adds **workflow update completion callbacks** via CHASM so Nexus
callers can register callbacks on `UpdateWorkflowExecution` and have
them fired on update completion or workflow close.
>
> This introduces a `WorkflowUpdate` CHASM component with new
`UpdateState` protobuf (including persisted `rejection_failure`), stores
update callbacks under `Workflow.Updates`, and extends callback
processing to handle *update-level* callbacks on update completion,
rejection (including reset/reapply), and on run transitions
(retry/timeout/continue-as-new) where update callbacks must fire even if
workflow-level callbacks are inherited.
>
> It also adds dynamic config gates/limits
(`EnableWorkflowUpdateCallbacks`, `MaxCallbacksPerUpdateID`), updates
`DescribeWorkflow` to surface update callback triggers, extends mutable
state/history builder APIs to carry per-update callback options in
`WorkflowExecutionOptionsUpdated`, and adds `Update.AttachCallbacks`
logic to persist/flush callbacks (including buffering while `stateSent`,
request-id dedup, and stricter validation requiring `request_id` when
callbacks are present).
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
4484fee104. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: long-nt-tran <long.tran@temporal.io>
## What changed?
### **High level**
With https://github.com/temporalio/api/pull/761 to add the linking on
the Signal and Signal-with-Start responses, This PR adds logic from the
server that:
* Adds `requestID` from Signal and Signal-with-Start requests to the
CHASM workflow tree under a new map field `IncomingSignals`, and event
store, so these requestIDs stay in buffer
* Return a backlink in the response that references the `requestID`
* On buffer flush to the DB transaction, attach these `requestID` to a
concrete `eventID`, which would allow users to later know which event
correlated w/ this request. We will wire the concrete event ID to the
signal request IDs stored in the workflow component CHASM tree
(`IncomingSignals` map)
> [!NOTE]
> Feature is gated behind a new dynamicconfig
`EnableCHASMSignalBacklinks`, which implicitly is only checked if
`EnableChasm` is enabled.
## Why?
This will enable the caller of the signal to have a backlink to the
cross-namespace signal invoked, which will become more relevant for
Nexus SDK ergonomics.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
In functional tests, I augmented existing tests for Signal and
Signal-with-Start to:
* Ensure that backlink is returned via the responses
* Later use `DescribeWorkflow` to ensure that we get a concrete EventID
(mapped when buffer flushed)
* Multiple signals with the same `requestID` gets de-duped
```
$ go test ./tests/ -run TestLinksTestSuite
ok go.temporal.io/server/tests 1.486s
```
```
$ go test ./tests/ -run 'TestNexusWorkflowTestSuite' -count=1
ok go.temporal.io/server/tests 4.714s
```
## Potential risks
Need to test end-to-end to see that the link shows up correctly in the
Web UI.
Feature is gated behind dynamicconfig since it requires CHASM-based
workflow to be enabled.
## What changed?
Add fairness mechanism to history rps rate limiter. Each namespace will
have a history RPS limit quota per history host based on this formula
NamespaceQuota = FrontendRPSLimit * HistoryShardsInHost/TotalShards *
Multiplier.
This multiplier is configurable through dynamic config. If a namespace's
request rate exceeds this quota, those extra requests will be
deprioritized. If history rps rate limiter starts to thorttle requests,
these deprioritized requests will be throttled first. This will prevent
unbalance load from a namespace affecting other namespaces in a history
host.
## Why?
To prevent noisy neighbour issues in history rps rate limiter.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
When gracefully shutting down, close inbound streams, not just outbound.
## Why?
This avoids unexpected connection reset on the sender side.
Flag enabled (default — new behavior):
1. Evict from membership + drain wait
2. handler.Stop() ← stops StreamReceiverMonitor (closes inbound streams
+ outbound),
replicationTaskFetcherFactory, shardController, eventNotifier
3. server.GracefulStop() ← stream handler goroutines already unblocked →
clean drain
4. visibilityManager.Close()
Flag disabled (current behavior):
1. Evict from membership + drain wait
2. handler.controller.Stop() ← only stops the shard controller (inbound
streams left open)
3. server.GracefulStop() ← stream handler goroutines still blocked →
falls back to Stop()
4. handler.Stop() ← remainder of handler teardown
5. visibilityManager.Close()
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches history service shutdown sequencing and replication stream
lifecycle, which can affect rolling restarts and cross-cluster
replication behavior if the new ordering has unintended side effects.
>
> **Overview**
> Improves history service graceful shutdown by **optionally closing
inbound replication streams** (in addition to outbound) so remote
senders are signaled to stop and stream handler goroutines can drain
cleanly.
>
> Introduces new dynamic config
`history.enableCloseInboundReplicationStreamOnShutdown` (default
`true`), threads it through history `Config`, and uses it to (a) stop
inbound streams in `StreamReceiverMonitor.Stop()` and (b) reorder
`service.go` shutdown to call `handler.Stop()` before
`server.GracefulStop()` when enabled; adds unit tests covering both
enabled/disabled behaviors.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
8aab9eeee6. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Pulled out the DeepHealthCheck into its own file. Added a fixed delay
from startup during which DeepHealthCheck will see NOT_SERVING from the
local health server as SERVING
## Why?
During history service scale-up/scale-downs, individual history hosts
can be detected as "NOT_SERVING" because the shard initialization hasn't
happened yet. This is an expected behavior from the history service pod
and does not indicate an error, but DeepHealthCheck currently treats it
as one.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
This change makes the DeepHealthCheck less likely to expose a problem
that causes an individual history service to reprovision faster than the
suppression interval. The standard HealthCheck for historyservice is
unaffected.
---------
Co-authored-by: Stephen Stanton <stephenstanton10@gmail.com>
## What changed?
- Add per namespace rate limiter on history service
## Why?
- Noisy neighbor protection. Provide know for preventing one namespace
from consuming all available history host rps.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Summary
- Extends `CheckTaskQueueVersionMembership` response with two new
fields: `is_version_active_or_draining` (bool) and `revision_number`
(int64). Matching populates both from its deployment data.
- Reactivation signals are **skipped** when matching reports the target
version as CURRENT/RAMPING/DRAINING; otherwise they are sent with a
**deterministic UUID v5 RequestId** derived from `revision_number`.
- Replaces the old TTL-based `ReactivationSignalCache` with a per-pod
**revision-based dedup LRU** on the worker-deployment client: each entry
records the highest revision this pod has successfully signaled for a
given version, so older or equal signals are skipped.
## What changed on the wire (matching → history)
`CheckTaskQueueVersionMembershipResponse` now has two new flat fields
(no wrapper message):
```proto
bool is_version_active_or_draining = 2; // true when status is CURRENT/RAMPING/DRAINING
int64 revision_number = 3; // from WorkerDeploymentVersionData.revision_number; 0 if unknown / legacy
```
Matching's `CheckTaskQueueVersionMembership` fills both via the helper
`worker_versioning.IsVersionActiveOrDraining(deploymentData, dep, build)
(bool, int64)`.
Naming choice — we picked `is_version_active_or_draining` (negative
polarity) rather than something like `supports_reactivation` so the
proto zero value (`false`) maps to the safe default ("send the signal").
Old matching binaries and runtime "version not found" both produce the
zero value, and history correctly falls through.
## Where `revision_number` flows
- **Matching**: populates the response field from the version's tracked
revision.
- **History-side helper/caches**:
`ValidateVersioningOverrideAndGetReactivationEligibility` returns
`(isVersionActiveOrDraining bool, revisionNumber int64, err)`.
`VersionMembershipAndReactivationStatusCache` stores both.
- **History signaler plumbing**: `VersionReactivationSignalerFn`,
`ReactivateVersionWorkflowIfPinned`, and all five call sites
(`startworkflow`, `signalwithstartworkflow`, `updateworkflowoptions`,
`resetworkflow`, `multioperation`) carry `revisionNumber int64`.
`resetworkflow.validatePostResetOperationInputs` returns parallel slices
`([]bool, []int64, error)` for per-operation inputs.
- **Signal RequestId**: `ClientImpl.SignalVersionReactivation` composes
`requestID = uuid.NewSHA1(uuid.NameSpaceOID,
[]byte("reactivation-signal:" + revisionNumber)).String()` — a
deterministic UUID v5 derived from the revision alone. Cassandra's
`signal_requested set<uuid>` column requires UUID-formatted RequestIds.
## Why revision-based dedup
History is sharded on `(namespaceID, workflowID)`. N concurrent
`StartWorkflow` calls pinned to the same drained version fan out across
potentially every history pod in the fleet. Before this PR each pod
independently fired a reactivation signal at the version workflow,
producing up to N `WorkflowExecutionSignaled` events — directly at odds
with the version workflow's design (it intentionally keeps history
minimal and CaNs aggressively, see `version_workflow.go:68-74`).
Per-pod caches alone can't fix this because they don't coordinate. What
we need is a **cluster-wide-deterministic dedup key** so all pods
converge on the same value for the same reactivation cycle. The
version's `revision_number` — incremented in `syncTaskQueuesAsync` on
every status change — is exactly that signal. Every pod reads the same
revision from matching, every pod composes the same UUID RequestId, and
Temporal's built-in `mutableState.pendingSignalRequestedIDs` dedup (see
`service/history/api/signalworkflow/api.go:40`) collapses concurrent
signals into exactly one event on the version workflow.
The per-pod map is a local optimization on top of that: it prevents a
single pod from re-sending the same-or-older-revision signal once it has
successfully sent one, cutting RPC volume.
## How the new caches look
### 1. `VersionMembershipAndReactivationStatusCache` (read-side,
per-pod)
Caches matching's `CheckTaskQueueVersionMembership` response so repeated
pinned-override validations on the same task queue don't re-hit
matching.
- **Key**: `(namespaceID, taskQueue, taskQueueType, deploymentName,
buildID)`
- **Value**: `(isMember bool, isVersionActiveOrDraining bool,
revisionNumber int64)`
- **Eviction**: `VersionMembershipCacheTTL` (1s default; 5s in
functional tests).
### 2. `highestRevSignaledToVersionWf` (write-side dedup, per-pod)
A field on `ClientImpl` in `service/worker/workerdeployment/client.go`.
For each target version workflow, stores the highest revision this pod
has successfully signaled. Subsequent calls at the same-or-lower
revision skip the RPC.
- **Key**: `reactivationVersionKey{namespaceID, deploymentName,
buildID}`
- **Value**: `int64` (highest revision successfully signaled)
- **Eviction**: LRU, bounded by `VersionReactivationSignalCacheMaxSize`.
The previous TTL-based `ReactivationSignalCache` module (in
`common/worker_versioning/`) has been deleted along with its provider
and `VersionReactivationSignalCacheTTL` config.
## Backwards/forwards compatibility
- **Old matching → new history**: old binaries don't set
`is_version_active_or_draining` or `revision_number`; both default to
proto zero values. `false` on the active bool → history falls through →
signal fires (safe default). `revisionNumber = 0` flows through as-is.
- **New matching → old history**: new fields on the response are ignored
by old history → identical to pre-PR behavior.
- **New matching → new history**: signal fires only when the version is
not active/draining; cross-pod fires converge on one UUID RequestId and
fold into one `WorkflowExecutionSignaled` event.
## Test plan
- [x] Unit tests for `IsVersionActiveOrDraining` covering all status
cases (CURRENT, RAMPING, DRAINING, DRAINED, INACTIVE, UNSPECIFIED), new
vs. old format, deleted and not-found versions.
- [x] Unit tests for
`ValidateVersioningOverrideAndGetReactivationEligibility` (cache
hit/miss, RPC with/without eligibility, Unimplemented fallback).
- [x] Unit tests for the per-pod dedup on
`ClientImpl.SignalVersionReactivation`: same-rev dedups, newer-rev
fires, older-rev skipped, different version isolated, signal-failure
allows retry.
- [x] Unit test for RequestId format (UUID v5, deterministic across
calls with the same revision).
- [x] Functional tests (all pass on SQLite and cass-es):
- `TestStartWorkflowExecution_ReactivateVersionOnPinned`
-
`TestStartWorkflowExecution_ReactivateVersionOnPinned_WithConflictPolicy`
- `TestSignalWithStartWorkflowExecution_ReactivateVersionOnPinned`
- `TestUpdateWorkflowExecutionOptions_ReactivateVersionOnPinned`
- `TestResetWorkflowExecution_ReactivateVersionOnPinned`
(The four `TestReactivationSignalCache_Deduplication_*` functional tests
from an earlier iteration were deleted — their coverage moved to unit
tests.)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes matching↔history API and reactivation signaling semantics by
skipping signals for active/draining versions and introducing
revision-based dedup via deterministic RequestIds; issues could affect
version workflow state transitions or signal fan-out during upgrades.
>
> **Overview**
> Matching’s `CheckTaskQueueVersionMembershipResponse` is extended with
`should_skip_reactivation` and `revision_number`, and matching now
populates both from per-task-queue deployment data.
>
> History-side versioning validation is refactored to return and cache
reactivation eligibility + revision, and reactivation signaling paths
(`StartWorkflow`, `SignalWithStart`, `UpdateWorkflowExecutionOptions`,
`ResetWorkflow`, multi-op) now **skip signals** when matching reports
the version as *CURRENT/RAMPING/DRAINING*.
>
> The old TTL-based `ReactivationSignalCache` is removed
(configs/metrics/providers updated), and the worker-deployment client
now performs **revision-based per-pod dedup** plus receiver-side dedup
by sending signals with a deterministic UUIDv5-like `RequestId` derived
from `revision_number`. Tests are updated/added to cover status
evaluation, new plumbing, and dedup behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
a1ec5e93db. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## What changed?
Added completion callback support to standalone activities:
- Callback lifecycle: When a standalone activity reaches a terminal
state (completed, failed, canceled, terminated, timed out), it now fires
any registered Nexus completion callbacks — the same mechanism workflows
already use.
- Frontend validation: Callback URL length, header size, and endpoint
allowlist validation is now shared between workflows and standalone
activities via callbacks.ValidateCallbacks, refactored out of the
workflow handler.
- Describe response: DescribeActivityExecution now returns callback
state (CallbackInfo) including trigger, status, attempt count, and
failure details.
- Start response: StartActivityExecutionResponse now includes a
Link_Activity_ identifying the started (or reused) activity.
- Config: Renamed MaxCHASMCallbacksPerWorkflow to
MaxCallbacksPerExecution since it now applies to both workflows and
standalone activities.
## Why?
The v2 scheduler needs to start standalone activities on a schedule and
be notified when they complete, so it can track action results, handle
overlap policies, and support pause-on-failure. Workflows already have
this via CompletionCallbacks + Nexus callback delivery. This PR gives
standalone activities the same capability, reusing the existing callback
library rather than reimplementing the
delivery/retry/backoff logic. This is a prerequisite for the scheduler's
Invoker to call StartActivityExecution with a callback pointing back to
the Scheduler component.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## What changed?
New outbound task type (`WorkerCommandsTask`) that carries worker
commands to be dispatched to workers via Nexus. Uses the generic
`WorkerCommand` proto (not cancel-activity-specific), so this task type
can carry any future command types.
Suggested review order: proto changes → `worker_commands_task.go` →
`task_generator.go` → `workflow_task_completed_handler.go`
Key pieces:
- **Proto**: `TASK_TYPE_WORKER_COMMANDS` enum, `WorkerCommandsTask` in
`OutboundTaskInfo` with `repeated WorkerCommand`.
- **Task definition**: `worker_commands_task.go` — implements outbound
`Task` and `HasDestination` interfaces.
- **Task creation** (`workflow_task_completed_handler.go`,
`task_generator.go`): When `RequestCancelActivityTask` is processed for
a started activity whose worker has a control queue, collects a
`CancelActivityCommand` with the activity's task token. Commands are
batched by destination control queue and flushed as one
`WorkerCommandsTask` per queue at the end of WFT processing.
- **Serialization**: `task_serializers.go` for persistence
round-tripping.
Dispatch is a no-op here — handled in #9233. Gated by dynamic config
`EnableCancelActivityWorkerCommand` (default: off).
## Why?
To support proactive activity cancellation without waiting for
heartbeat. This is the task creation leg of the flow.
1. [#9231] Store `worker_control_task_queue` in `ActivityInfo` at
activity start.
2. **[This PR]** On `RequestCancelActivityTask`, batch commands by
control queue into `WorkerCommandsTask` outbound tasks.
3. [#9233] Dispatch each task as a Nexus `ExecuteCommands` operation to
the worker, with a 3-attempt retry cap.
4. [SDK] Worker receives the cancel command and cancels the running
activity.
Gated by dynamic config `EnableCancelActivityWorkerCommand` (default:
off).
## How did you test it?
**Unit tests** cover task generation, command batching (including
multi-queue batching), task serialization round-tripping, and the
feature-flag-off path.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
Implement a rate limiter to limit workflow ID reuse.
## Why?
Some patterns can create large number of worklow exeuction with same
IThis can cause some issues in the persistence partition.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
Ported command handler for Nexus "cancel" command from HSM to CHASM.
CHASM migration.
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Roey Berman <roey.berman@gmail.com>
## What changed?
Add a new replication task type `DeleteExecutionReplicationTask` that
replicates workflow deletion from the active cluster to passive/standby
clusters. Gated by feature flag
`history.enableDeleteWorkflowExecutionReplication` (default: false).
#### **Key changes across the replication pipeline:**
1. Proto enums: `TASK_TYPE_REPLICATION_DELETE_EXECUTION` (34),
`REPLICATION_TASK_TYPE_DELETE_EXECUTION_TASK` (13)
2. Replication task is associated with a new stage in
`ShardContext.DeleteWorkflowExecution`, bundled with delete visibility
task.
3. ~Engine interface: added `ForceDeleteWorkflowExecution` so the task
can invoke the `ForceDeleteWorkflowExecution`.~
## Why?
Today, when a user delete workflow execution in source cluster, this
operation will not replicate to the standby/target clusters. When a
namespace failover to a target cluster, those deleted workflow may
resurrected.
<details>
<summary>Race condition analysis</summary>
**Before this change:**
1. **Cross-cluster resurrection:** Active deletes workflow → standby
untouched → failover → workflow reappears.
2. **Termination event silently dropped:** Deleting a running workflow
terminates it first, generating a `HistoryReplicationTask`. But the
async `CloseExecutionTask` may delete mutable state before the stream
sender converts that task.
The converter calls `getBranchToken()` → `NotFound` → task silently
dropped. The standby never sees the termination or the deletion.
**After this change:**
Race 1 is fixed — `DeleteExecutionReplicationTask` explicitly tells the
standby to delete.
Race 2 is mitigated — even if the termination event's replication task
is dropped, the delete replication task ensures the standby cleans up.
- If the workflow is still running (termination not yet replicated), the
`DeleteExecutionTask` reschedules itself until the workflow closes.
- If the termination event arrives later, the workflow closes normally,
then the delete proceeds.
- If the workflow is already deleted (e.g., by retention), the task is a
no-op (`NotFound` treated as success).
</details>
<details>
<summary>Deletion paths</summary>
| Path | Replication task? |
|------|-------------------|
| User deletes workflow (active, running or closed) | Yes |
| User deletes on passive (DC forwarding ON) | Forwarded to active → yes
|
| User deletes on passive (no forwarding) | No — `ActiveInCluster` check
skips |
| Retention expiry (with or without archival) | No — stage pre-marked as
processed |
| Admin ForceDelete (tdbg) | No — bypasses `DeleteWorkflowExecution` |
</details>
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
Before change:
<img width="1507" height="163" alt="Screenshot 2026-03-26 at 11 59
51 PM"
src="https://github.com/user-attachments/assets/118cc50e-b69d-468a-9e45-5f49e4e4b9d1"
/>
After change:
<img width="1507" height="135" alt="Screenshot 2026-03-27 at 12 00
07 AM"
src="https://github.com/user-attachments/assets/8ccb7a11-2cb4-48b5-af89-b4a60ddb6333"
/>
## Potential risks
n/a
## What changed?
Deleted `"system.enableNexus"`.
## Why?
Nexus has been GA since Dec 2024.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add chasm logical task type dynamic config filter and standby task
discard delay dynamic config flag.
## Why?
Standby tasks have a configurable dynamic config (per task type) that
specifies the timeout for discarding the task and running the
post-discard function. Currently, this is not configurable for CHASM
logical tasks. By default, CHASM tasks need a higher discard timeout
since they are not regenerated from pending tasks in MS, which can lead
to abandoned tasks. For tasks that can be safely dispatched to Matching,
they can be configured with a lower value. (See Standalone activity
tasks).
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Fix a race condition in GetWorkflowExecutionHistory that caused SDK
workers to receive incomplete history and fail with "premature end of
stream". On the last page of a paginated `GetWorkflowExecutionHistory`
response, re-query mutable state to detect events that were committed to
the DB between the first and last page fetches. If `freshNextEventId >
continuationToken.NextEventId`, the gap is fetched from persistence and
appended to the response before transient/speculative events are added.
The continuation token is updated with the fresh boundary so
`appendTransientTasks` validates against the correct `NextEventId`. If
the re-query itself fails, the request returns an error so the client
retries.
Also adds a `nil` check in `ValidateTransientWorkflowTaskEvents`,
preventing a possible nil-pointer dereference.
## Why?
A speculative WFT is converted to normal (e.g., by an incoming signal),
committing 1–2 new events. The continuation token from page 1 points to
NextEventId=8; the DB range [6, 8) on page 2 returns only events 6–7,
missing the newly-committed events 8 and 9. `appendTransientTasks` finds
no transient tasks (speculative was committed), so the assembled history
is missing 2 events.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
- The re-query on the last history page adds one extra GetMutableState
RPC per paginated `GetWorkflowExecutionHistory` call. This is bounded to
final-page responses only and the existing path already made this call
inside `appendTransientTasks`, so the net overhead is one additional
call specifically when a gap is detected.
- Returning an error when the fresh mutable-state re-query fails changes
the previous behavior of silently continuing. Clients will retry, which
is correct, but retry storms are possible if persistence is consistently
unavailable mitigated by the client's existing backoff.
## What changed?
Add a ExecutionQueueScheduler to serialize tasks from a busy workflow
execution. Currently, all workflow tasks go through FIFO scheduler. We
have a per-workflow-execution lock that must be aquired in each history
task. But if a single execution has large number of tasks, each of these
tasks compete for this lock and create large number of retries. By
Adding this new scheduler, we can serialize task processing from such
busy workflows. These tasks are routed to a new WorkflowQueueScheduler
when lock contention is detected in a workflow. This will create a new
queue for that workflow. Additional tasks for this workflow will be then
routed to this new queue. This queue will be cleaned up after a few
seconds of inactivity from that workflow execution. We have added a
ExecutionAwareScheduler which will manage this routing of workflow tasks
to either FIFO scheduler or this new WQ Scheduler.
```
Task → InterleavedWeightedRoundRobinScheduler
↓
ExecutionAwareScheduler
↓
┌─────┴─────┐
↓ ↓
FIFOScheduler ExecutionQueueScheduler
(normal path) (contended executions)
```
This new scheduler is only enabled when
history.taskSchedulerEnableExecutionQueueScheduler is enabled. The
number of workflow queues created in this scheduler will be controlled
by config history.taskSchedulerWorkflowQueueSchedulerQueueSize.
Tasks are routed to FIFOScheduler(Like the way it was before this
change) if number of queues reaches this value.
A new set of goroutines is spawned for each queue in this new scheduler.
This is fine here as we don’t expect more than a few hundred hot
workflows per history host. This simplifies the design for this
scheduler.
## Why?
To reduce workflow lock contention and wasted history CPU when tasks are
competing for workflow lock.
## Benchmark Results
Collected by running 2,000 parallel activities from a single workflow
with a 5ms lock timeout to trigger contention. EQS queue concurrency =
2, max queues = 500.
### Task Routing & Failures
| Metric | **EQS ENABLED (C=2)** | **EQS DISABLED** | Improvement |
| --- | --- | --- | --- |
| EQS Tasks Submitted | 1,953 | 0 | - |
| EQS Tasks Completed | 1,940 | 0 | - |
| EQS Tasks Failed | 13 | 0 | - |
| EQS Tasks Aborted | 0 | 0 | - |
| EQS Submit Rejected | 0 | 0 | - |
| FIFO Tasks Completed | 65 | 2,051 | - |
| **Total Failures** | **66** | **6,382** | **97x fewer failures** |
### End-to-End Task Latency (task_latency_queue)
| Percentile | **EQS ENABLED (C=2)** | **EQS DISABLED** | Improvement |
| --- | --- | --- | --- |
| Average | 426ms | 908ms | **2.1x faster** |
| P90 | 668ms | **1,599ms** | **2.4x faster** |
| P99 | 729ms | **3,982ms** | **5.5x faster** |
| Max | 736ms | **6,773ms** | **9.2x faster** |
### Runtime
| Metric | **EQS ENABLED (C=2)** | **EQS DISABLED** | Improvement |
| --- | --- | --- | --- |
| Test Time | 4.6s | 8.9s | **1.9x faster** |
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Added `TEMPORAL_TEST_DATA_ENCODING` to change DataBlob encoding from
"proto3" to "json".
## Why?
Observability and debugging. The ability to see payloads decoded in
debugger and OTEL traces is valuable.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
End users should never use this. The env var name therefore includes
`TEST_`.
## What changed?
- activate drained/inactive versions to draining when they get a
workflow started on it
- also added a history cache, per history node, so that we don't bombard
our version workflows with signals that shall change the drainage status
of these workflows.
## Why?
- versioning correctness, in the sense that if someone were to move a
workflow on to a version that is drained, the drainage status should be
updated to draining (since it now has one open workflow working on it)
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- Sure, this change is lowkey risky. Would appreciate a thorough review.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches history start/reset/update flows and adds asynchronous
signaling into worker-deployment workflows; mis-wiring or cache/config
issues could cause missing or excessive reactivation signals and
unexpected version state churn under load.
>
> **Overview**
> When workflows are **pinned to a specific deployment version**,
history now triggers a fire-and-forget `reactivate-version` signal to
the corresponding worker-deployment *version workflow* so versions in
`DRAINED/INACTIVE` transition back to `DRAINING`.
>
> This wiring is applied across start paths (`StartWorkflowExecution`
incl. conflict handling, `SignalWithStart`, multi-op start), option
changes (`UpdateWorkflowExecutionOptions` after persistence), and
`ResetWorkflowExecution` post-reset operations. A new per-history-node
`ReactivationSignalCache` (TTL/max-size + metrics tags) deduplicates
signals, and a new dynamic config flag
(`history.enableVersionReactivationSignals`) plus cache settings
control/limit load.
>
> Worker-deployment adds `Client.SignalVersionReactivation` and the
version workflow gains a version-gated handler for `reactivate-version`
to update drainage/status and sync summaries; extensive functional tests
cover reactivation and cache dedup behavior.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
15a5ac69b2. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Carly de Frondeville <cdefrondeville@berkeley.edu>
## What changed?
Set TargetVersionChanged instead of SuggestCaN when TargetVersionChanged
https://github.com/temporalio/api/pull/709
## Why?
Setting SuggestContinueAsNew=true for Pinned workflows whenever their is
a new Target Version available for that workflow causes Pinned workflows
to hit that condition much more frequently than they expect. Users who
are currently doing: if workflow_info.suggestContinueAsNew{ do
continue-as-new } in their Pinned workflow code would need to change
that code to protect themselves from running into an infinite-CaN-loop,
because the default CaN behavior for a Pinned workflow is to stay
Pinned.
We should not force users to protect themselves from such a situation.
Because upgrading on continue-as-new is opt-in, receiving the suggestion
to continue-as-new-onto-new-target-version should be opt-in as well. If
people are forced to check the new suggest-continue-as-new-reasons field
to "opt out," that is unsafe, because inevitably some people will forget
to do so or misunderstand, and then get hit by this unexpected footgun.
Much safer and still ergonomical to let upgrade-on-can be opt-in on both
fronts, as proposed here. With this change, the people who are currently
doing if workflow_info.suggestContinueAsNew{ do continue-as-new } won't
see any change in semantics, regardless of their versioning behavior.
People who consciously know that they want to do upgrade-on-can /
Trampolining will have to change their CaN options anyway, so it's easy
enough to teach them to pay attention to this new
TargetWorkerDeploymentVersionChanged flag.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches workflow task started event generation/persistence and
versioning-related signaling, which can affect worker behavior and
history compatibility; changes are gated by dynamic config and covered
by tests.
>
> **Overview**
> Stops using `SuggestContinueAsNew` (and its reason tags) to signal
pinned workflows that a newer target worker deployment version exists,
and instead introduces an explicit
`TargetWorkerDeploymentVersionChanged` boolean on `WorkflowTaskStarted`
events and persisted `WorkflowExecutionInfo`.
>
> Adds namespace dynamic config `EnableSendTargetVersionChanged`
(default on) and a new metric `workflow_target_version_changed_count`
emitted when this flag is set; updates the workflow task state machine,
mutable state plumbing/mocks, proto/pb persistence, and functional tests
accordingly. Also bumps `go.temporal.io/api` to pick up the new event
attribute.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
3886491826. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Disable Suggest-ContinueAsNew-on-new target version#9222
## Why?
Users of Pinned workflows who listen to SuggestContinueAsNew are at risk
of continuing-as-new way too frequently.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Small, well-scoped behavior gating behind a new dynamic config flag
with default-off behavior; main risk is unintended change in CaN
recommendation behavior when the flag is enabled per-namespace.
>
> **Overview**
> Adds a new namespace-level dynamic config flag,
`system.enableSuggestCaNOnNewTargetVersion`, to **disable by default**
suggesting Continue-As-New to pinned workflows when a newer target
worker deployment version becomes available.
>
> History’s workflow task started event logic now only emits the
`SuggestContinueAsNew` recommendation/reason for target-version changes
when this flag is enabled, and the versioning v3 integration tests are
updated to cover both enabled and disabled behavior.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
9fd0483efb. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
<!-- BUGBOT_STATUS --><sup><a
href="https://cursor.com/dashboard?tab=bugbot">Cursor Bugbot</a>
reviewed your changes and found no issues for commit
<u>9fd0483</u></sup><!-- /BUGBOT_STATUS -->
## What changed?
Enabled the behavior by default.
## Why?
It has been supported for a SDK and server versions already and was
originally put in temporarily.
## What changed?
Cache the result of `GetTaskQueueUserData` that history makes to
Matching when an activity wants to start a deployment version
transition.
## Why?
This protects the matching root partition from being hammered by
requests when a lot of AutoUpgrade workflows want to start
activity-initiated transitions. Activity initiated transitions happen in
one of the following cases:
1) Target version changed while activity was backlogged.
2) Target version changed while activity was in retry backoff
3) Target version changed in some edge cases involving parallel
activities
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
The cache can potentially increase history mem usage but there are knobs
to adjust size and ttl.
## What changed?
This PR extends the raw history optimization to pass raw history bytes
from History Service → Matching Service → Frontend without
deserialization in Matching Service.
**Key changes:**
1. History Service:
- When `SendRawHistoryBetweenInternalServices` is enabled, sets
`RawHistoryBytes` (field 21) with raw proto-encoded history batches
2. Matching Service:
- Passes raw history bytes through to frontend via
`PollWorkflowTaskQueueResponseWithRawHistory`
- Uses wire-compatible proto messages so gRPC auto-deserializes
`[][]byte` → `History` on the client side
3. Frontend:
- Receives raw history in `RawHistory` field (auto-deserialized by gRPC)
- Processes search attributes for raw history since it bypasses history
service's normal processing
4. Proto definitions:
- Added `raw_history_bytes` (field 21) to
`RecordWorkflowTaskStartedResponse`
- Added `PollWorkflowTaskQueueResponseWithRawHistory` message with
wire-compatible layout
- Added `raw_history` (field 22) to `PollWorkflowTaskQueueResponse`
## Why?
When `history.sendRawHistoryBetweenInternalServices` is enabled, the
previous implementation only avoided deserialization from persistence →
History Service. However, Matching Service was still deserializing
history events (via gRPC auto-deserialization) and re-serializing them
when forwarding to Frontend.
This change eliminates that unnecessary serialization/deserialization
cycle in Matching Service by:
1. Having History Service send raw bytes directly
2. Having Matching Service forward these raw bytes without parsing
3. Having Frontend receive the bytes which gRPC auto-deserializes
This reduces CPU usage in Matching Service for workflows with large
histories.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s) (`tests/workflow_task_test.go`)
## Potential risks
SendRawHistoryBetweenInternalServices must be disabled when rolling back
from this version to an older version.
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## What changed?
Change the EnableTransitionHistory dynamic config from a global setting
to a namespace-scoped setting, allowing it to be configured per
namespace.
Also update the default value from false to true.
## Why?
To better control the feature.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
no risk.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## What changed?
Pause replication streams when the scheduler is under pressure.
## Why?
This allows us to apply backpressure to replication streams when we
cannot keep up with the load. The schedulers are shared amongst streams
so just keep tracking of tasked tasks is not enough. We use a timer to
notify us if a submit is taking too long rather than recording the time
after the fact so that the backpressure is more reactive.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Backpressure for replication streams**
>
> - Add slow-submission flow control: `stream_receiver` tracks
per-priority slow submission timestamps and
`stream_receiver_flow_controller` pauses when within
`ReplicationReceiverSlowSubmissionWindow` (in addition to outstanding
task count)
> - New dynamic configs:
`history.ReplicationReceiverSubmissionLatencyThreshold`,
`history.ReplicationReceiverSlowSubmissionWindow`,
`history.EnableReplicationReceiverSlowSubmissionFlowControl`, wired
through `configs.Config`
> - Refactor `StreamReceiver`: split scheduler selection into
`getTaskSchedulerPriority`/`getTaskScheduler`, measure `Submit` latency,
and feed `lastSlowSubmission` into flow control signals
> - Tests: expand flow controller tests to cover slow-submission window
logic; adjust stream receiver tests and minor robustness checks
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
9e6ec6ed81. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Update replication task rate limiter priority based on active/standby
state
## Why?
The replication task should have higher priority than standby task
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
There is a task processing priority change requires review
## What changed?
Enable RateLimitedScheduler in all cases. Pass NoopRatelimiter when
TaskSchedulerEnableRateLimiter is disabled. Also disable
TaskSchedulerEnableRateLimiterShadowMode when
TaskSchedulerEnableRateLimiter disabled. Shadow mode is not relevant
when NoopRatelimiter is used.
## Why?
It makes it easy to inject different TaskSchedulerRateLimiter without
modifying these configs.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Keep the total number and the size of the external payloads per the
workflow execution
## Why?
We are working on building the support for external payloads in SDK,
which are stored outside of Temporal. We'd like to be able to show the
total size and the number of external payloads in the given workflow
execution.
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
N/A
## What changed?
- WISOTT
- Also added a cache per history host so that we don't overburden
matching with these calls.
- Also added a whole new unit test testing the function
`ValidateVersioningOverride`
- TODO in a follow-up PR: add metrics for this cache. Doing this as a
follow-up in the interest of time but have it tracked in JIRA.
## Why?
- Versioning correctness.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- I don't think this is risky
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Validate versioning overrides by ensuring pinned versions exist in the
task queue via matching RPC with a per-host cache, refactor validation
into history APIs, and add configs and tests.
>
> - **Worker Versioning / Validation**:
> - Add `ValidateVersioningOverride(ctx, ...)` to verify pinned versions
exist in a task queue via `matching.CheckTaskQueueVersionMembership`.
> - Introduce per-host `versionMembershipCache` to cache membership
results; reject with `FailedPrecondition` if not present.
> - Remove frontend/batcher inline override validation; perform it in
history layer (start, signal-with-start, update options, reset
post-ops).
> - **History Service Wiring**:
> - Thread `matchingClient` and `versionMembershipCache` through history
engine, starter, multi-op, signal-with-start, reset, and
update-workflow-options APIs.
> - Mark batch UpdateWorkflowOptions non-retryable for "Pinned version
is not present in the task queue".
> - **Config / Dynamic Config**:
> - Add `history.versionMembershipCacheTTL` and
`history.versionMembershipCacheMaxSize`; provide cache in `fx` with
lifecycle management.
> - **Testing**:
> - Add unit tests for `ValidateVersioningOverride` covering cache
hits/misses and v0.31/v0.32 paths.
> - Extend functional tests to assert membership checks, cache behavior,
batch update failures, and reset with post-reset options.
> - Test helpers: `TestVars.WithDeploymentSeries`, `WithBuildID`, and
utilities to ensure versions are present via matching RPC.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
d8b755273f. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
- The `enableChasm` dynamic config flag is now applied per-namespace.
## Why?
- For this release, we only want CHASM enabled in canary codepaths, for
both callbacks and scheduler.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- This reduces the risk by reducing `enableChasm`'s blast radius.
However, we'll need to be wary of writing tests that rely on
`enableChasm` being set to `false` (current default), since in the
future it'll be enabled everywhere.
## What changed?
Integrated chasm/lib/callback into MutableState and updated
callback-related APIs to use CHASM implementation. Added integration
tests following the scheduler pattern.
## Why?
Porting callback functionality from HSM to CHASM as part of the ongoing
HSM-to-CHASM migration effort.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
This will need to be tested more, but is not being turned on with this
PR
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Integrates CHASM-based callbacks into mutable state and history APIs,
adds configs and error types, updates routing/state transitions, and
includes migration tests.
>
> - **CHASM Callback Integration**:
> - Add CHASM callback module (FX wiring, HTTP caller provider, request
routing) and metrics shared with HSM implementation.
> - Implement `ToAPICallback`, invocation flow changes, and refined
error wrapping.
> - **History/MutableState**:
> - Expose `ChasmEnabled`, `ChasmWorkflowComponent{,ReadOnly}`; use
CHASM for adding/processing completion callbacks when enabled.
> - Describe API now builds callback info from both CHASM and HSM trees.
> - Continue-as-new/retry path aggregates callbacks from both
implementations.
> - **State Machine/Executors**:
> - Switch to `queues/errors` types; move `NamespaceIDAndDestination` to
`queues/common`.
> - Adjust transitions: generate `BackoffTask` with scheduled time;
include destination on reschedule; no tasks on success/fail.
> - Simplify retry result (policy passed separately) and validation
signatures.
> - **Config**:
> - Add `EnableCHASMCallbacks` and `MaxCHASMCallbacksPerWorkflow`; wire
through history configs.
> - **Tests**:
> - Add migration and CHASM-enabled functional tests; update unit tests
to new error/types and behaviors.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
e3b51a931d. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Roey Berman <roey@temporal.io>
## What changed?
- Add a new flag that controls whether clusters setup replication
streams to each other
- Feature flag to gate the new flag for compatibility
- Functional test to vet the changes
## Why?
Optimization to avoid excessive network activity when we only want
namespace replication.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
---------
Co-authored-by: Claude <noreply@anthropic.com>
## What changed?
- Bump stamp on workflow task retry
- Propagate stamp on matching workflow task forwarding.
- Add functional test
- Refactor test interceptors
## Why?
Workflow tasks can pile up on passive side due to stamps not
incrementing, meaning they never become stale and never become eligible
for invalidation. This ensures tasks can be invalidated due to
staleness.
We need to ensure we forward the stamp in matching, otherwise prior to
dispatch the task will be deemed invalid as the stamp does not match
what is stored in mutable state.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
