## What changed?
Skip unbuildable replication tasks on stream sender instead of blocking
the stream. The skip is logged and new metric ReplicationTaskSendSkipped
added.
## Why?
When the replication stream sender cannot build ("convert") a task, it
retries and, once the retry budget is exhausted, returns an error that
tears the stream down. On reconnect the sender resumes from the same
watermark, hits the same unbuildable task, and blocks the whole shard's
stream indefinitely. This change skips the task to unblock and logs, add
a metric for the skipped task so that it could be investigated later.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
Added more state-effecting API calls to the forwarding list.
## Why?
The intent is for passive to always forward those APIs it cannot handle
locally.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes which cross-cluster RPCs are auto-forwarded for global
namespaces; misconfiguration could leave state changes on passive
clusters or alter failover behavior for new API types.
>
> **Overview**
> Expands **selected-apis-forwarding** so passive clusters forward
additional **state-effecting** RPCs to the namespace’s active cluster.
>
> **Workflow APIs** newly whitelisted: `UpdateWorkflowExecution`,
`PauseWorkflowExecution`, `UnpauseWorkflowExecution`,
`ResetWorkflowExecution`, and `ExecuteMultiOperation`. Policy comments
now describe forwarding as state-effecting APIs and point readers at
`selectedAPIsForwardingRedirectionPolicyWhitelistedAPIs` instead of an
inline list.
>
> Tests add **`TestSelectedAPIs`** to assert the full whitelist
(workflow, standalone activity, and Nexus operation APIs). The
non-whitelisted API case is fixed by suffixing API names with
`_notwhitelisted` and using a global namespace whose active cluster is
remote, so forwarding tests no longer accidentally exercise whitelisted
names.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
07fed6a663. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What
Emits `source_cluster`, `source_shard`, and `source_task_id` on the
`executing` and `applied` phases of the `replication_lifecycle` wide
event, not just `sent`.
## Why
`source_task_id` was already on the wire
(`ReplicationTask.source_task_id`, populated at every
`raw_task_converter.go` conversion site) and already stored on the task
as `ExecutableTaskImpl.taskID` — which is also the watermark the
receiver acks back. It was simply never emitted on the passive side, so
a `sent` event on the active cluster could not be joined to the matching
apply.
Observability only: no proto change, no wire-format change, no version
gating.
## Details
- `common/wideevents`: added `SourceCluster` and `SourceShard`; moved
`SourceTaskID` out of the sent-only group so all three are emitted
phase-independently, guarded so they are absent when unset.
- `common/wideevents/context.go`: `ReplicationTaskOrigin{ShardID,
TaskID}` carries source provenance to the NDC `applied` emitter, which
sits below the engine boundary and has no task in scope. Diagnostic only
— never affects control flow.
- Emitters populate the fields for `sync_versioned_transition`,
`verify_versioned_transition`, and `sync_workflow_state`.
**The join key is `(source_cluster, source_shard, source_task_id)` — the
id alone is not unique.** Task ids are allocated per source shard from
overlapping ranges. On a 4-shard xdc run with 6 workflows, 4 of 19
distinct `source_task_id`s appeared on more than one source shard, so
this matters in any multi-shard deployment. The passive side gets the
shard directly from `SourceShardKey()` (the stream receiver is created
per source-shard pair); no reversal of `generateShardIDs` is involved,
and that reversal would be ambiguous anyway when shard counts differ
between clusters.
`source_task_id` is intentionally absent on the two on-demand paths —
the `SyncState` resend and child-completion verification — which
re-fetch the artifact from the source, so the triggering task is not the
one whose payload was applied. Emitting its id would create a false
join. `source_shard` and `source_cluster` are still set on the resend
path, since the source shard is known there. In queries, `applied AND
source_task_id IS NULL` therefore isolates resend-originated applies.
## Test
`go build ./...`, plus `common/wideevents`,
`service/history/replication`, and `service/history/ndc` unit tests.
`TestReplicationLifecycleFieldSetLocked` pins the new field set.
Verified end to end on a two-cluster, 4-shard xdc harness (scaffolding
not committed), asserting on every captured event that `source_shard` is
present and non-zero, that it spans multiple distinct shards (so the
assertion cannot pass on a constant), and that every passive
`(source_shard, source_task_id)` pair matches a `sent` event the active
side actually emitted on that shard:
```
source_shard present on all 71 replication_lifecycle events
source_shard spans 3 distinct shards
4 sent source_task_ids appear on more than one source_shard (of 19 distinct ids)
(source_shard, source_task_id) joined to a sent event on 48 passive events
```
Note there is no CI coverage that the emitters populate the fields —
only that the payload can. Happy to add a unit test on payload
construction if preferred.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
Adds `NamespaceLifecycle` wide events on the two paths that decide when
a namespace handover can complete. Also threads `ShardID` and an
`EventLogger` into `HandoverTrackerParams` so the tracker can attribute
an event to its shard.
**Shard handover tracker** (`service/history/shard/`) — each shard holds
its own replication watermark for a namespace in handover, and the
handover cannot complete until every shard's watermark has been acked by
the target:
| Phase | When |
|---|---|
| `shard_handover_watermark_set` | the shard takes or advances a
watermark — `reason` is `added` or `updated` |
| `shard_handover_watermark_removed` | the shard drops it —
`deleted_from_db` separates a namespace deletion from a normal exit out
of handover |
Both fire only on an actual mutation of the tracker map. Notably, the
non-handover branch of `UpdateHandoverState` runs for *every* namespace
notification on *every* shard; it now guards on presence before
deleting, so it emits only on a real removal rather than on every
notification.
**`WaitHandover`** (`service/worker/migration/`) — one event, emitted on
the way out:
| Phase | When |
|---|---|
| `shard_handover_incomplete` | the wait ended with shards still behind
|
Carries `not_ready_count`, `total_shards`,
`missing_handover_info_count`, `max_lagging_tasks{,_shard_id}`,
`elapsed_seconds`, `exit_reason`, and a `lagging_shards` list of
`{shard_id, lagging_tasks}` capped at 64 (`not_ready_count` is always
the true count).
A handover that completes leaves no laggards and emits nothing, so the
happy path — nearly all of them — is silent. The wait is capped at
`maximumHandoverTimeoutSeconds` (30) with `MaximumAttempts: 1`, so this
is at most one event per handover.
`exit_reason` separates the wait being killed while shards were still
behind from a failed `GetReplicationStatus`. The activity never returns
on its own in the former case: the SDK cancels the activity context off
the heartbeat, the next `GetReplicationStatus` fails, `WaitHandover`
returns, and the deferred summary runs.
A not-ready shard with `lagging_tasks == 0` is the missing-handover-info
case (that shard's namespace cache hasn't picked up the handover yet);
any other not-ready shard is behind by `lagging_tasks`.
## Why?
When a handover stalls, the only signal today is `Wait handover not
ready`, which reports counts plus the single worst shard, once a second
for the life of the wait. That is not enough to name the shards actually
holding it up, or to tell a shard that never took a watermark from one
whose watermark is not being acked. These events make both attributable,
and the summary form costs one row per failed handover instead of one
log line per second.
Emission is a no-op unless a `LoggerProvider` is configured, so there is
no cost for deployments that have not opted in.
## How did you test it?
- [x] covered by new unit tests
- [x] built
New tests pin the behavior that keeps these off the hot path:
- `TestHandoverWatermarkRemovedOnlyWhenTracked` — the per-notification
branch emits nothing when there is no watermark to remove.
- `TestHandoverWatermarkEvents` — set/update/remove, and that a repeat
notification at the same version emits nothing.
- `TestHandoverWatermarkPendingThenResolved` — the unacquired-shard
sentinel is reported as `pending`, and resolving it on acquire emits
nothing.
- `TestEmitHandoverLagSummarySilentWhenReady` — a completed handover
emits nothing.
- `TestCheckHandoverOnceSnapshotIsLastPollOnly` — the snapshot is
overwritten per poll, so laggards from an earlier poll cannot leak into
the summary.
- `TestEmitHandoverLagSummaryNamesLaggingShards` / `ExitReason` /
`Truncates` — payload contents, exit-reason classification, and the
per-shard cap.
`go test ./service/history/shard/... ./service/worker/migration/...`
passes; `go vet` and `gofmt` clean.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
- `WithRequestID` now applies to `UpdateComponent`, enabling
execution-level idempotency guarding via request ID.
- When a request ID is passed as part of an `UpdateComponent` call (via
API handler), it is persisted upon successful updateFn call. If it is
already present, instead, `UpdateComponent` fails with a
`FailedPrecondition`.
- When a request ID is not passed in, a generated ID is still created
for error tracing purposes, but it is not written to mutable state.
- On transaction close, mutable state will sweep the oldest RequestIDs
(with an `attach_time`) upon hitting the configured limit.
- This will sweep both entries below a configurable max age, as well as
past a certain hard length limit.
## Why?
- Scheduler's `UpdateSchedule` and `PatchSchedule` are implemented as
handlers that persist a signal in V1. V1 signals provide idempotency via
their request IDs. Scheduler V2 doesn't make use of signals, so instead,
it must record request IDs explicitly.
- We reuse the existing map within mutable state.
- We *must* fail with an explicit error (`FailedPrecondition`) instead
of simply returning a zero value (as Signals would on repeated
successful requests). This is because `UpdateComponent` can apply to API
models that include response values (which we don't record, therefore,
we can't return on subsequent calls).
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Fred Tzeng <fred.tzeng@temporal.io>
A `sent` event records that a task went out, not *what it carried* — and
that can't be recovered afterwards. `LastUpdateVersionedTransition`
stamps are overwritten by later transitions, and the replication
progress cache that supplies the lower bound of both the state diff and
the event range is in-memory and per `(run, target)`.
Seven sent-only fields:
- `target_cluster` — scopes the bounds, since progress is cached per
target
- `priority` — the stream the task went out on
- `artifact_kind` — snapshot vs mutation
- `exclusive_start_versioned_transition` — the mutation's exclusive
lower bound
- `hydrated_versioned_transition` — the versioned transition of the
state actually serialized
- `shipped_first_event_id` / `shipped_last_event_id` — the event ids
actually carried
Note `hydrated_versioned_transition` is `>=` the task's own versioned
transition whenever the sender advanced past the queued task, so the
existing `transition_count` describes the queue entry while this
describes the payload. Likewise `shipped_*_event_id` are the events
carried, distinct from `first_event_id`/`next_event_id`, which are the
task's own range.
`priority` is only known to the sender: it is assigned per send loop,
and only the low-priority stream is rate limited, while flow control,
trackers and ack watermarks are all per priority. Without it a backed-up
stream can't be attributed.
Also populates `failover_version`/`transition_count` on the
`sync_versioned_transition` applied event; the `!= 0` guard dropped
them, leaving sent→applied uncorrelatable.
All values are read from the task at the existing emit site, so no
interfaces change and the send path is untouched. Everything stays
behind `history.emitReplicationLifecycleEvents`.
Verified end-to-end against a live two-cluster replication stream.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
Add process-lifetime baselines to `objectleak`, ignore tiny pointer-free
allocations that the runtime cannot track individually, and allow
asynchronous cleanup the full settle timeout.
## Why?
Make object leak reports actionable by distinguishing process-lifetime
objects from leaks and avoiding runtime-induced false positives.
Replaces #11313.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
Use the `jobID` parameter in the `deterministicRequestID` function to
allow multiple batch operations to signal the same workflow.
## Why?
Two batch operations that send the same signal to the same workflow/run
ID will only send one signal, the second will be de-duped in the signal
logic because the request ID was hashed from workflow id / run id /
signal name. Including the batch job ID gives proper hashing to prevent
these separate signals from being de-duped
## How did you test it?
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
NA, this is a bug fix.
## What changed?
Only use 4-core ARM runners instead of 8-core; and increase shard size
for test sharding.
## Why?
8-core runners have - at least during peak hours - very long
provisioning time (several minutes). By using 4-core we reduce that
time, but risk OOM kills as they have less memory. To counter that, we
increase the number of test shards so that fewer tests are run per
shard.
## What changed?
Extract generic JUnit XML file reading and writing into
`tools/common/junit`.
## Why?
Centralizing format-level JUnit handling.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sean Kane <spkane31@gmail.com>
## What changed?
When trying to clean up some PRs for worker callbacks, I noticed some
unnecessary duplication and cruft in our testcases for completion
handlers.
Essentially it's this:
<img width="567" height="216" alt="image"
src="https://github.com/user-attachments/assets/c64b759c-c5be-4fc3-a346-92cf77e1f6db"
/>
Rather than having N places where we construct a `completionHandler` and
several implementations of spinning up an `httptest` webserver, we now
just have a single `newNexusCompletionHandler` function that does both.
## Why?
Remove boilerplate, making tests more concise and easier to read.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
If there was some subtle behavioral change this could undermine our
testing.
## What changed?
Flip the default of `history.enableHostLevelEventsCache` from `false` to
`true`, so history shards share a single host-level events cache instead
of each allocating a shard-level one.
## Why?
We have been using host level history cache for a while. We can now
enable it in code by default.
## How did you test it?
- [x] built
- [x] covered by existing tests
## What changed?
Updated the activity code to use a CHASM context API that was added
after the original code was written.
## Why?
Prevent this pattern from being copied to other libraries.
> **Part 2 of a 5-PR series** building to replication stream namespace
isolation (a restructuring of #10147): read buffer → reader group → lane
protocol → isolation manager → sender isolation.
> #11263 (read buffer) has merged, so this PR's diff is now standalone
against `main`. · **Next in series: #11303** (lane wire protocol +
receiver routing).
## What changed?
Refactors the stream sender's per-priority `QueueReaderState` arithmetic
— catch-up begin watermark lookup, reader-state construction from
`SyncReplicationState` acks, and failover watermark selection — into a
`replicationReaderGroup` type behind a new
`EnableReplicationReaderGroup` dynamic config flag (default off).
Scope-index mapping is centralized in `priorityScopeIndex`, which
tolerates both the single-stack (1 scope) and tiered (3 scope) persisted
formats. With the flag off, behavior is byte-for-byte the pre-refactor
logic — including the strict "exactly 3 scopes" check (states with more
scopes fall back to the overall watermark). Only the flag-on path
tolerates >3 scopes, which later PRs in this series use for per-lane
cursors. The flag routes both the recv side (reader-state persistence)
and the send side (catch-up begin lookup); changing it restarts
replication streams. The dynamic config is read once in the constructor
so all derived state sees one consistent snapshot.
## Why?
Groundwork for replication stream namespace isolation (later PRs in this
series), which extends the persisted reader state with additional
per-lane scopes. Pulling the existing scope arithmetic into one place
first makes the later change a pure extension instead of a rewrite, and
gives the sender a single point that understands the persisted scope
layout.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s) — `replication_reader_group_test.go` covers
reader-ID derivation, catch-up watermark lookup for all scope formats,
reader-state construction (tiered and single-stack, including
mismatched-mode errors), failover watermark selection, and the exact
3-vs-4-scope boundary in both flag positions
- [x] added an equivalence test —
`TestRecvSyncReplicationState_ReaderGroupEquivalence` runs the same ack
through both flag branches and asserts the persisted `QueueReaderState`
and failover watermark are identical
## Potential risks
The refactored path is flag-gated and default-off; the default path is
byte-for-byte the previous logic (pinned by the equivalence test), so
risk is limited to enabling the flag. The flag-change stream restart
mirrors the existing tiered-stack flag behavior.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches replication stream sender cursor and failover watermark logic,
which is core multi-cluster infrastructure, but the new path is
default-off and equivalence-tested against the legacy path.
>
> **Overview**
> Extracts the stream sender's per-priority `QueueReaderState`
arithmetic into a new `replicationReaderGroup`, gated by
**`EnableReplicationReaderGroup`** (default off).
>
> Catch-up watermark lookup, reader-state construction from
`SyncReplicationState` acks, and failover watermark selection now live
behind this abstraction. Scope-index mapping is centralized in
`priorityScopeIndex`, which keeps the legacy exactly-3-scopes behavior
when the flag is off and only tolerates >3 scopes on the new path (for
upcoming namespace-isolation lanes).
>
> Flag flips restart replication streams, matching the existing
tiered-stack flag pattern. Equivalence tests pin that both paths persist
identical reader state and failover watermarks.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
bd89ff0903. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
## What changed
`RPCFactory` and the SDK client factory close the gRPC connections they
own on shutdown, as fx stop hooks. Removes the four gRPC connection
ignores from the leak test.
Also bumps `auto-scaled-workers` to pick up
temporalio/temporal-auto-scaled-workers#108, without which the SDK
connection stays open.
## Why?
Nothing released these connections, so every connection's goroutines and
the membership resolver watching for changes outlived the cluster.
Clients from `NewClient` share the system client's ref-counted
connection, so the SDK closes it only once the last one is closed.
## How did you test it?
- [x] built
- [x] covered by existing tests
`make leak-test` at the CI settings reports `no unexpected goroutines`.
Each change was verified to fail the leak test when reverted.
## What changed?
`parentclosepolicy.Processor` and `scanner.Scanner` keep a reference to
the SDK workers they start and stop them on shutdown, and the batch
activity closes the SDK client it creates per execution. The SDK worker
goleak ignores go with them.
## Why?
Neither component kept a reference before, so nothing could stop the
workers and their pollers ran until the process exited. The batcher
leaked a client reference per activity execution, so that one
accumulated per server process rather than per cluster.
Part of removing the TODO ignores in `tests/leakcheck/leak_test.go` (see
#11322).
## How did you test it?
- [x] built
- [x] covered by existing tests
## What changed?
Add a `__temporal_system_payload = "true"` metadata field to outer
`Payload` returned by system nexus operations when the operations
protobuf result embeds a nested `commonpb.Payload(s)` field.
## Why?
System Nexus operation results are serialized into a single outer
`Payload`. When the underlying proto message itself contains a nested
`Payload(s)` field, that nested payload's bytes get hidden inside the
outer payload's opaque `Data`. Downstream consumers (payload
codec/visitor logic) have no way to know they need to unwrap the outer
payload before they can reach and process what's inside.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## What changed?
Introduce history.ReplicationStreamMaxLifetime (with a companion jitter
coefficient) so the stream receiver gracefully recycles each replication
stream after a bounded, jittered interval. When the lifetime elapses the
receiver stops the stream and the stream receiver monitor reopens a
fresh one, which can land on a different (non-draining) connection.
## Why?
This lets proxies such as Envoy gracefully drain connections that would
otherwise be pinned open indefinitely by endless replication streams,
and bounds the number of outbound connections a history pod accumulates.
Disabled by default (0). Receiver-side only; mirrors the existing
livenessMonitor. Jitter is clamped to [0,1] to guard against a
misconfigured dynamic value panicking backoff.Jitter.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed
`versioninfo.Caller.Call` closes the response body on every path, not
just the 200 one. The two goroutines reported in the leak test is from
this.
## Why?
The `defer` sat below the status check, so a non-200 returned with the
body open. The transport sets `DisableKeepAlives`, so its read and write
loops stay pinned waiting for a body nothing will close — one connection
leaked per failed check, for the process lifetime.
## How did you test it?
- [x] built
- [x] covered by existing tests
Fixes#11289
## What changed?
`RPCFactory.CreateLocalFrontendGRPCConnection` and
`CreateRemoteFrontendGRPCConnection` now cache the `grpc.ClientConn`
they dial instead of dialing a fresh one on every call. The local
connection is cached with `sync.OnceValue` (mirroring
`CreateLocalFrontendHTTPClient`, which already did this); the remote
connection is cached in a map keyed by `rpcAddress`, since it's dialed
with an address that varies per call.
## Why?
`OperatorHandlerImpl`/`AdminHandler`'s
`{Add,Remove,List/Get}SearchAttributes` and `AddOrUpdateRemoteCluster`
handlers call
`clientFactory.NewLocalFrontendClientWithTimeout`/`NewRemoteAdminClientWithTimeout`
on every RPC, and neither method backing them ever cached or closed the
underlying connection. Each call dialed a brand new `grpc.ClientConn`
that was never reclaimed, so goroutines and memory grew roughly 1:1 with
call count and were never released short of a process restart. Fixing
the caching at the `RPCFactory` level fixes all 8 call sites without
touching the handlers or the `Factory` interface.
## How did you test it?
- [x] built
- [x] covered by existing tests (`go test ./common/rpc/...
./common/testing/nettest/... ./client/... ./service/frontend/...`,
including `-race`)
## Potential risks
Connections are now held for the process lifetime instead of being
dialed per call, matching the caching `client.Bean` already does for the
clients built on top of these same connections.
`AddOrUpdateRemoteCluster` connections are now keyed by `rpcAddress` and
never evicted; this only grows with the (small, admin-controlled) set of
distinct remote cluster addresses ever passed to that RPC, which was
already effectively cached forever by `client.Bean` once a remote
cluster is added.
---------
Co-authored-by: Prathyush PV <prathyush.pv@temporal.io>
## What changed?
Enable the `activity.enableStandalone` dynamic config on the release
server in the mixed-brain test.
## Why?
omes `throughput_stress` auto-enables standalone-activity load when the
namespace reports support. The current server (1.32) defaults it on, so
omes sends SAA calls; the previous release (v1.31.2) defaults it off, so
calls routed to the release frontend were rejected (`Unimplemented:
"Standalone activity is disabled"`), stalling workflows until the run
timed out. Enabling the DC on the release server matches a real
rolling-upgrade cluster and lets the test exercise SAA across both
versions instead of failing on every branch.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
When a workflow is deleted on the source cluster while a
`SYNC_VERSIONED_TRANSITION` task is in flight, applying the task on the
target fails with NotFound. HandleErr then runs a best-effort cleanup by
building an ExecutableDeleteExecutionTask, but returned
deletionTask.Execute() raw. If the workflow is already gone in the
target cluster too, that delete also returns NotFound, which is
retryable, so the task retried until the retry policy expired and was
then sent to the DLQ.
Route the cleanup through deletionTask.HandleErr, which already maps
NotFound to nil. This matches the two other cleanup paths in the package
(`ExecutableTaskImpl.DeleteWorkflow` and the delete task's own
HandleErr), and matches the intent stated in the comment above it: the
cleanup is optional because the deletion replicates on its own.
## Why?
bug fix for cases we should not DLQ the task
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
Apply Callback Validator to Workflow Update Callbacks
## Why?
We should consistently apply the validator in all cases.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes request validation on a user-facing API path; invalid
callbacks are rejected earlier, which may surface errors for previously
accepted bad URLs but improves security consistency with start-workflow.
>
> **Overview**
> **Workflow update** requests with `completion_callbacks` now run
through the same **callback validator** used for workflow start,
including multi-op update paths that call
`prepareUpdateWorkflowRequest`.
>
> `prepareUpdateWorkflowRequest` takes `ctx` and namespace so validation
can enforce allowed callback URLs and normalize Nexus callback headers
before the request reaches history.
>
> Tests add unit coverage for invalid variants and header normalization,
plus functional tests that configure `callback.AllowedAddresses` and
reject disallowed callback URLs on update.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
d699d6de4e. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Adds `shardinfo_immediate_queue_backlog_age`, the time-based counterpart
to `shardinfo_immediate_queue_lag`, emitted from
`emitShardInfoMetricsLogs` right beside it with the same scope and tags.
Immediate task keys carry no timestamp, so unlike the scheduled lag the
age can't be derived from the queue state — instead the oldest task at
the ack frontier (the same `minTaskKey` the count derives its lag from)
is read with a single `BatchSize=1` query, off the shard read lock.
## Why?
Timer queues already report backlog as a duration, but immediate
(transfer/visibility/outbound) queues only had a count. A count can't
distinguish 2k tasks accumulated in seconds from 2k accumulated over
hours, which is exactly what matters when triaging a backlog.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
## Summary
Forward `VersioningOverride` from schedule actions into workflow start
requests for both the legacy and CHASM schedulers. Schedule validation
now rejects structurally invalid overrides before they are persisted.
## Root cause and impact
Both schedulers manually built `StartWorkflowExecutionRequest` and
omitted `VersioningOverride`, so schedules accepted and retained the
setting but started workflows using ordinary version routing.
## Problem
A standby child workflow's `CloseExecutionTask` verifies its parent
recorded the completion, and past
`MaxLocalParentWorkflowVerificationDuration` also resends the parent
from the active cluster. That resend is a cross-cluster state sync plus
a possibly paginated history backfill — minutes of work — but the whole
call was bounded by the standby task's hard-coded **3s** `taskTimeout`.
Measured at the active cluster's history shard, the deadline arriving
there was `2.999s`. So the resend never completed.
## Change
- **Run the resend in the background**, bounded by
`ReplicationTaskApplyTimeout` — the setting that already bounds this
same work on the replication stream. The verify RPC returns immediately
and the standby task retries until the parent lands, so it never holds a
transfer-queue worker for the sync.
- The background context is **detached from the request** (gRPC cancels
that when the handler returns) and **rooted at the shard lifecycle**, so
the work stops with the shard.
- **One in-flight resend per parent**, tracked in a shard-level map, and
at most `history.parentWorkflowResendMaxInFlight` (8) concurrent resends
per shard. Callers retry while an earlier resend runs; without this the
test measured 5 full state fetches where 1 suffices. The cap bounds the
goroutines this path can create.
- **Lifted two client ceilings** so the deadline can actually propagate
— `admin.SyncWorkflowState` (was 10s) and `history.SyncWorkflowState`
(was 30s) now share a `DefaultStateSyncTimeout` backstop. This also
fixes the same 10s cap on the replication stream's
`ExecutableTaskImpl.SyncState`, where production's 5m setting was never
reachable either.
- Metrics:
`parent_workflow_resend_{attempts,skipped,limited,failures,latency}`.
Async failures reach no caller, so `_failures` is the alert signal;
`_limited` means the shard is shedding resends. The background goroutine
recovers panics, which would otherwise take down the process.
Also fixes the history-client codegen template, which hardcoded
`createContext` and silently ignored the timeout-tier field.
## Rollout
`history.enableAsyncParentWorkflowResend`, **default false**. Disabled =
the previous inline behavior, bounded by the caller's task deadline. Opt
in per cell.
## Testing
Unit tests cover the inline, async, and per-parent-dedup paths.
An xdc test (added in abae9cec, removed in b6eb6631) withholds the
parent's replication tasks so the child *must* pull it, asserts the
parent is absent from the standby, then stalls the active cluster's
`SyncWorkflowState` for 4 minutes:
```
--- PASS: TestChildPullsParentWhenParentReplicationIsWithheld (286.03s)
incoming-ctx-remaining: 4m59.999859334s (2.999s before this change)
sync-state-calls: 1 (5 without the per-parent guard)
dropped-parent-tasks: 9
```
During the stall, 4 verify RPCs reached the standby parent shard (t+0,
+50s, +101s, +169s) and exactly 1 `SyncWorkflowState` reached the active
cluster: the task retried and the guard turned the retries away.
4 minutes exceeds every deadline that previously bounded this path (3s /
10s / 30s) with ~1m headroom against the 5m setting, so the setting is
demonstrably what governs.
To reproduce: `git revert b6eb6631`, then
`go test -tags test_dep ./tests/xdc/ -run
TestVerifyChildCompletionParentResendSuite -timeout 30m`
## Known gaps
- Concurrency across *distinct* parents is unbounded (ordinary fan-out,
not amplification).
- When the parent is deleted on the source, the async path can't report
that back, so the child retries to the 15m discard instead of finishing
immediately. The `workflowNotFoundCache` TODO already in this file would
address it.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
Bump `go.temporal.io/api` from the pseudo-version
`v1.63.5-0.20260804201935-e54fd69950e1` to the tagged release `v1.63.5`,
in both the root module and `tests/mixedbrain`.
## Why?
Cloud releases may only ship tagged versions of `go.temporal.io/api`.
The v1.63.5 tag points at `e54fd699`, the same api-go commit the
pseudo-version already referenced, so no api code changes are included.
## How did you test it?
- [x] covered by existing tests
## What changed?
- Fence Backfiller tasks with a persisted task sequence value instead of
comparing task execution time with the backfill HWM.
- Accept unnumbered tasks created by an older binary and restore task
numbering when a new binary executes one.
- Add lifecycle coverage and CHASM test support for firing due persisted
pure tasks.
## Why?
`LastProcessedTime` tracks progress through the requested schedule
range, while a task's `ScheduledTime` controls when that task runs.
Comparing the two can either keep an already-processed historical task
valid or reject a forward-dated task before it runs.
## How did you test it?
- [x] covered by existing tests
- [x] added new unit test(s)
## Potential risks
The immediate task `N` created with a Backfiller is scheduled and
executed in one transaction. Mixed-version risk begins with delayed task
`N+1`.
If `N+1` is handled by a 160 binary:
- For a forward-dated backfill, the old HWM validator can remove `N+1`
before execution, leaving the Backfiller alive with no task to make
further progress.
- For a historical backfill, the old validator can accept `N+1` again
after it has already advanced the HWM, so duplicate execution remains
possible during rollout or rollback.
## What changed?
- Check all non-drained version queues before marking a partition
drained.
- Scaler Describe calls only inspect loaded partitions and do not
refresh queue liveness.
## Why?
- `AllActive` missed unloaded per-version backlog
- Loading unloaded partitions from the scaler caused reload cycles.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Scale-down may wait until an unloaded partition is loaded by real
traffic. Explicit version queues may still be loaded when their
partition is already loaded.
## What changed?
Consume the per-partition backlog counts that the server now delivers in
ClientPartitionCounts.
## Why?
To weight pollers toward partitions with more backlog, so pollers aren't
trapped on empty partitions while others hold a backlog.
When backlogCounts are nil or incomplete, or when backlogCap = 0, pick
the partition weighted by fewest outstanding pollers as we do now.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes how poll and task writes are routed across partitions in the
matching client—a core throughput/latency path—but retains explicit
fallbacks when backlog metadata is missing or stale.
>
> **Overview**
> The matching client load balancer now uses **per-partition backlog**
from `ClientPartitionCounts` (trailer) instead of only uniform random or
fewest-poller heuristics.
>
> **Poll (read) path:** When `backlogCap > 0` and backlog counts cover
all read partitions with at least one positive backlog, partition choice
is **weighted by decoded backlog plus a floor**
(`readPartitionWeightFloor`) so pollers favor partitions with work while
empty partitions still get some traffic. Otherwise behavior stays
**fewest outstanding pollers**.
>
> **Write path:** `PickWritePartition` chooses partitions with
probability proportional to **gap below `backlogCap`**; it falls back to
uniform random when cap is zero, counts are incomplete, or every
partition is at/above cap.
>
> `parsePartitionCounts` now populates `BacklogCap` and `BacklogCount`
from the server response. Unit tests cover weighted distribution,
incomplete backlog fallback, and write gap behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
aa555d8977. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: David Reiss <dnr@dnr.im>
Co-authored-by: David Reiss <david@temporal.io>
## What changed?
Replace the gods treemap backing fairTaskReader.outstandingTasks with a
tidwall/btree (consistent with matcher_data.go and the evictedAcks
cache), and rewrite mergeTasksLocked around the btree's copy-on-write
support: snapshot the outstanding tasks, merge in the newly read/written
tasks, trim to the lowest batchSize loaded tasks, and install the
trimmed tree.
## Why?
The previous limited-copy logic tracked loaded tasks and acks in
separate paths and needed a special-cased "evict acks above readLevel"
pass. The new code keeps loaded tasks and acks in one tree and chops
everything above the cut uniformly, so no ack ever sits above a dropped
task. Because readLevel is now the highest tracked level (loaded or
ack), it no longer collapses toward the ack level, which removes the
readLevel/atEnd churn behind the previously-observed stuck-reader state.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
This code path has historically been a bit tough to get right, I've
tried referencing previous bugs we've found in it and made sure we're
not introducing some new regressions, but hard to say for 100%
certainty.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
Add TemporalNamespaceDivision group by column allowlist
## Why?
allow aggregation across archetypes for system workflow usage.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What changed
- Add a global `matching.enableWorkerDeploymentVersionDemotionSignal`
dynamic config, defaulting to `false` for OSS.
- Preserve the existing signal-based demotion path when the config is
`true`.
- Restore the legacy `SyncWorkerDeploymentVersion` update path when the
config is `false`.
- Record the dynamic-config decision deterministically with workflow
versioning and `MutableSideEffect`.
- Cover SetCurrent and SetRamping demotions under both modes.
- Replay the complete Worker Deployment history corpus with the config
both disabled and enabled.
Temporal Cloud configures this value to `true`. The OSS default is
intended to change to `true` in v1.33, after existing Version workflows
have had time to Continue-As-New onto code that registers the demotion
signal handler.
## Why
#9973 changed version demotion from a synchronous update to a
fire-and-forget signal. Long-running Worker Deployment Version workflows
that had started before the signal handler was available could ignore
that signal, leaving versions stuck in `Draining`.
Cloud has already repaired affected Version workflows through forced
Continue-As-New and should keep the signal path enabled. OSS v1.31 does
not contain #9973, so v1.32 should default to the legacy update path and
avoid introducing this failure mode during upgrade. The main idea, with
OSS, would be to keep this feature disabled for now (default value of
the new dynamic config value introduced is false) and then turn the knob
on to true in the next OSS release.
Existing histories remain deterministic:
- OSS histories without `commit-routing-first` continue replaying the
update path.
- Cloud histories that already contain the `demote-version` signal but
lack `version-demotion-signal-dynamic-config` continue replaying the
signal path, even if the current config is `false`. (this should never
happen since I shall be doing a global rollout of this dynamic config
for cloud very soon)
- Workflows with no recorded demotion decision use and record the
current dynamic-config value when their first demotion occurs.
## User impact
OSS users upgrading to v1.32 retain the pre-#9973 demotion behavior by
default. Cloud retains the currently deployed signal behavior. Existing
OSS and Cloud workflow histories replay without nondeterminism under
either current config value.
## Validation
- `go test -tags test_dep ./service/worker/workerdeployment`
- `go test -tags test_dep ./service/worker/workerdeployment/replaytester
-run '^TestReplays$' -count=1`
#11421 adds the freshly generated histories that prove the signal-based
demotion command path.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **High Risk**
> Changes worker deployment routing/demotion behavior and workflow
determinism rules; mis-toggling the config or replay mismatches could
leave deployment versions stuck in Draining or cause nondeterminism on
long-running workflows.
>
> **Overview**
> Adds global dynamic config
**`matching.enableWorkerDeploymentVersionDemotionSignal`** (default
**`false`** for OSS) so Worker Deployment workflows can choose between
**fire-and-forget demotion signals** and the legacy
**`SyncWorkerDeploymentVersion`** activity path when changing current or
ramping versions.
>
> The deployment workflow records the choice deterministically via
**`workflow.GetVersion`** checkpoints (`commit-routing-first`,
`version-demotion-signal-dynamic-config`) and **`MutableSideEffect`** on
the config value. Histories that already recorded signal-based demotion
keep the signal path on replay even if the config is off; OSS histories
without those markers stay on the update path.
>
> **`fx.go`** wires the config getter into **`Workflow`**. Replay tests
run the full history corpus with the flag both enabled and disabled.
Workflow tests cover SetCurrent and SetRamping demotion for each mode.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
4e9c27b75f. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
add a functional test for time-skipping behavior during workflow pause
and unpause
## Why?
make sure time-skipping works correctly in this scenario:
1. we **allow** setting and storing the time skipping config when paused
2. time **doesn't skip** when paused and **resumes skipping** when
unpaused
3. but if a user sets a time point to disable time skipping in the
options and also pauses the execution (refers to time skipping timer)
this timer **can** still fire and turn off time skipping
## What changed?
This centralizes the health checker into `health.SignalAggregator`. This
standardizes how we will do health checks by using latency based
quantiles and error ratios.
The goal is to be able to use this for any spot we want to do health
detection.
Main idea is you have your overall settings and thresholds for latency
metrics and error ratios, then you can have "groups" of specific keys
that have their own tracking for latency and error ratios. For grpc
health checking, these keys will be the endpoints. For something like
history the keys will be the specific functions.
Doing this also sets us up better for per namespace health settings.
This is 1 of 3 prs. This one is mainly for just testing that everything
works as expected. Then next one will be cleaning up and consolidating
all the health signals in the health package. Then the last one will be
applying
## Why?
We need more strict settings/thresholds for endpoints we consider to be
critical like StartWorkflowExecution, SignalWorkflowExecution,
StartActivityExecution, etc...
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Currently, if you pass in bad settings (e.g. buffer size <= 0), there
will be panics. This isn't a newly introduced problem, but it is an
existing one that will be addressed in future PRs.
## What changed?
- Increment the CHASM scheduler conflict token when pause-on-failure
changes the persisted paused state and notes.
- Add a regression test covering a stale Describe token used by Update
after the automatic pause commits.
## Why?
A token-protected Update could previously replace the entire schedule
using a token captured before pause-on-failure, silently clearing the
automatic pause and its explanatory notes. Invalidating the token keeps
optimistic concurrency behavior aligned with the fields Update replaces
and with the V1 scheduler.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
```
go test -tags test_dep ./chasm/lib/scheduler -run 'Test(HandleNexusCompletion_PauseOnFailure|PauseOnFailureInvalidatesConflictToken)$' -count=1
```
## Potential risks
Token-protected updates already in flight when pause-on-failure commits
will now fail with a conflict-token mismatch and must be retried.
Updates that intentionally omit the conflict token remain unconditional.
## Migration: give same-time pending starts unique identities
### Description
`convertBufferedStartsLegacyToCHASM` derives each migrated start's
`RequestId` and `WorkflowId` as pure functions of its timestamps plus
batch-constant inputs (namespace, schedule, conflict token, base
workflow id). Two pending starts with the same `NominalTime` and
`ActualTime` therefore get identical identities. The loop index was
available but unused.
### User experience
Two or more pending actions can legitimately share a nominal/actual time
(e.g. under `ALLOW_ALL` overlap, or overlapping backfill/trigger
requests). When such a schedule is migrated V1→V2, the collision
silently loses actions: the shared `WorkflowId` means only one workflow
starts (`REJECT_DUPLICATE`), and the shared `RequestId` makes completion
routing and dedup treat the pair as one — the invoker keys
`CompletedStarts`/`FailedStarts`/retries by request ID, so the second
start is indistinguishable from the first. No error is surfaced.
### How it occurs
`GenerateRequestID` = `sched-<backfillID>-<sha1(ns, sched, token,
nominalMs, actualMs)>` and `GenerateWorkflowID` =
`<base>-<nominalSecond>`. For a conversion batch everything except the
timestamps is constant, so equal timestamps → identical IDs.
### How it's fixed
Disambiguate with the per-action loop index, at each identity's natural
seam:
- **Request ID** — the index rides in the existing `backfillID` tag
(`"migrated-0"`, `"migrated-1"`, …). That tag is the literal prefix of
`sched-<tag>-<uuid>`, so the IDs differ without touching the hash;
`convertRunningWorkflowsToBufferedStarts` already uses the same
mechanism with the run ID. `GenerateRequestID` itself is unchanged, so
no other caller's IDs move.
- **Workflow ID** — suffixed only past the first start (`i > 0`). Unlike
the request ID this one is user-visible, and dedup against an action the
V1 scheduler had already started depends on it matching, so the common
case of a single pending action keeps the ID that both V1 and native V2
would give it.
Both apply only to regenerated (empty) IDs; identities carried over from
V1 are preserved.
### Test
- `TestSameTimePendingStartsReceiveUniqueIdentities` — fails before,
passes after. Also pins the first start's workflow ID to the undecorated
`GenerateWorkflowID` output.
- `TestMigratedStartsPreserveExistingIdentities` — V1-supplied
identities are neither regenerated nor suffixed.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed
Adds a freshly generated Worker Deployment replay corpus containing 20
Deployment workflow histories and 11 Version workflow histories.
Two Deployment workflow histories exercise the signal-based
version-demotion path: they record the `commit-routing-first` version
marker at version `0`, initiate the `demote-version` external workflow
signal, and record successful external signaling. The corresponding
Version workflow histories record receipt of those signals.
## Why
The existing replay corpus did not contain Deployment workflow histories
that emitted `demote-version`. These fixtures give us real signal-path
histories to use when validating workflow compatibility changes around
OSS upgrades and the demotion gate.
There are no production code changes in this PR.
## Historical context
PR #9973 initially added `run_1776382947`, containing 21 Deployment
workflow histories and 11 Version workflow histories. That corpus
included two Deployment histories sending `demote-version` and two
Version histories receiving it, together with the new workflow version
markers.
Later in the same PR's branch history, commit
[`4e41ba1`](4e41ba19a9)
deleted that corpus and replaced it with `run_1780926905`, containing 19
Deployment workflow histories and 11 Version workflow histories. The
replacement corpus that ultimately merged contains no `demote-version`
events or `commit-routing-first`/`demote-version-signal` markers; its
`SetCurrent` histories use `SyncWorkerDeploymentVersion`.
PR #9973 was squash-merged, so `4e41ba1` is visible in the PR branch
history but is not preserved as an individual commit on `main`. Because
the replay harness validates determinism and workflow counts without
asserting the presence of specific commands or markers, the loss of
signal-path coverage was not detected. This PR restores that coverage
with histories generated from the current signal path.
## Validation
- `GOWORK=off go test -tags test_dep
./service/worker/workerdeployment/replaytester -run '^TestReplays$'
-count=1`
- Verified all 31 compressed histories with `gzip -t`
- Verified generated counts: 20 Deployment workflows and 11 Version
workflows
- Adversarial fixture review passed
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Test-only replay fixture additions with no runtime or service logic
changes.
>
> **Overview**
> Adds a new replay fixture set under `testdata/v2/run_1785941282` with
**20** Worker Deployment workflow histories and **11** Worker Version
histories (plus `expected_counts.txt` for the replay harness).
>
> Unlike the corpus that landed after PR #9973, this set includes
histories that exercise the **signal-based version demotion** path:
Deployment workflows record `commit-routing-first` at version `0`, emit
the `demote-version` external signal, and log successful signaling;
matching Version workflows record receiving those signals.
>
> **No production code changes**—only test data to restore determinism
replay coverage for demotion-related workflow compatibility.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
a8aae3fa63. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
1. Test fix `tests/activity_api_batch_{cancel,terminate}_test.go`: both
`*_ExcludesNonRunning` tests waited only for two activities to match
`ActivityType` before starting the batch, but the just-completed one may
still be indexed as `ExecutionStatus='Running'`. New shared helper
`waitForRunningFilterToSettle` also waits on the `Running` count,
mirroring the query the batcher counts with in
`adjustQueryBatchTypeEnum`.
2. `isNonRetryableError` treats `serviceerror.FailedPrecondition` as
non-retryable for
`TERMINATE_ACTIVITY`/`CANCEL_ACTIVITY`/`DELETE_ACTIVITY`. Every
reachable FailedPrecondition on those paths is permanent — a terminal
status, or an already-recorded terminate/cancel request with a different
request ID — and processTaskWithRetries retries in place with no
backoff.
## Why?
TestActivityBatchCancel_ExcludesNonRunning was a top CI breaker in the
2026-08-04 Flaky Tests Report.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s) — the two existing ones are the
subject of the fix
## Potential risks
- Activity batch targets in a terminal state now fail immediately rather
than after N attempts. They are still counted as failures, so
`DescribeBatchOperation` totals are unchanged.
## What changed?
Cap the failure kept in LastFailureDetails when an attempt will be
retried, for parity with MutableStateImpl.truncateRetryableActivityFailure
of workflow activities.
## Why?
This is for parity between workflow activity and CHASM based activity
(currently standalone activity) implementation. The workflow activity
limits the retryable failure data kept in the mutable state, so similar
behavior is added for standalone activity. Non-retryable failures are
left intact since they are reported back to the caller, again similar to
workflow activity. It uses the same dynamic config that is used in
workflow activities size limit.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Fred Tzeng <41805201+fretz12@users.noreply.github.com>
## What changed?
Include LastDeploymentVersion in ActivityExecutionInfo returned by
buildActivityExecutionInfo.
## Why?
TransitionStarted persists the poller’s deployment version for the
started activity,
but buildActivityExecutionInfo never assigns the LastDeploymentVersion
field.
Thus Describe always reports nil even when the state contains the value.
Although
the LastDeploymentVersion field is currently not used/set for standalone
activity,
but can be/is only set for workflow activity, this change fixes a
potential bug in
the future when deployment version gets set for standalone activity too.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
- Do not reset heartbeats by default
- Honor `reset_heartbeat` flag
## Why?
- Parity with WFA
- This product behavior makes sense: a user with a long-running activity
using exponential backoff on attempt 10 may wish to reset the attempt
counter in order that the next retry backoff is short, and yet preserve
their checkpointed progress.
## How did you test it?
- [x] modified existing functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes activity reset semantics and persisted state for heartbeat
checkpoints across immediate and deferred reset paths; behavior is
well-covered by tests but affects long-running activity retry/reset
workflows.
>
> **Overview**
> Activity reset now **rewinds the attempt counter** but **keeps
persisted heartbeat checkpoint details by default**, matching
workflow-activity behavior. Clearing heartbeats is **opt-in** via
`reset_heartbeat` / `ResetHeartbeat` on reset APIs.
>
> CHASM activity state adds `reset_should_clear_heartbeat` for resets
requested while a worker is still running; clearing runs when the
attempt yields (same deferred pattern as `restore_original_options`).
Immediate reset paths (`reset`, `resetKeepPaused`) and
cancel-on-reset-request clear that deferred flag only when the flag is
set.
>
> Standalone activity reset forwarding no longer forces `ResetHeartbeat:
true`; it passes the client request. Model/events add
`ResetClearingHeartbeat`; parity and functional tests cover keep vs
clear for scheduled and started activities.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
09fb7a0bd1. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
> Part 1 of a planned 5-PR series building toward replication stream
namespace isolation (a restructuring of #10147): read buffer → reader
group → lane protocol → isolation manager → sender isolation. This PR
stands on its own; follow-ups will be opened as each lands review.
**Next in series: #11302** (reader group).
## What changed?
A shard-scoped read-through buffer over the tip of the replication task
queue, sitting inside the ack manager's `GetReplicationTasksIter`. Every
replication stream sender on a shard — one per remote cluster, with one
iterator per priority lane — scans the same queue, so each new task page
was previously read from persistence once per scanner. With the buffer,
overlapping tip scans share one persistence read; only readers below the
buffered range (deep catch-up, lanes that have lagged out of coverage)
fall through to persistence.
Coverage is a contiguous task-id interval established by persistence
pages: within it the buffer is authoritative, so absence of a task means
the range holds none. An empty persistence page with a non-empty
continuation token (legal, e.g. under Cassandra paging) is NOT
authoritative — the fetch keeps paging until tasks arrive or the token
runs out. Rows are immutable and the queue is append-only, so there is
no invalidation; eviction just shrinks coverage from the front. Reads
are bounded by the shard's exclusive-high read watermark, so covered
ranges are stable once established.
**Ownership:** the buffer stores SERIALIZED rows and deserializes per
serve, so every reader receives fresh task structs it exclusively owns —
exactly what a persistence read would have produced. This matters
because downstream converters mutate tasks in place (e.g.
`SyncVersionedTransitionTask` equivalents get IDs assigned via
`AddTasks`); handing multiple senders pointers to shared structs would
be a data race. The serve-time deserialization replaces the persistence
read the reader would otherwise have done, so it is not added cost
relative to the unbuffered path. Serialize/deserialize failures are
never silent: both are error-logged (they indicate a bug — e.g. a task
type missing serializer support — or broken persistence data); a
serialize failure serves the page uncached, a deserialize failure drops
the buffer's coverage entirely and falls back to persistence.
Capacity is `ReplicationStreamReadBufferSize` tasks per shard (default 0
= disabled); disabling at runtime releases the buffered rows. The buffer
holds slim queue rows (task metadata) for every priority — event
payloads only enter the pipeline at send-time conversion — so memory
cost is a few hundred bytes per row.
Observability (to drive future sizing/sharing decisions):
`replication_stream_read_buffer_hits` / `_misses` count pages served
from memory vs. fetched from persistence while the buffer is enabled
(misses are counted only after a successful fetch), and
`replication_stream_read_buffer_miss_lag` records, for misses below the
buffered range, how far below coverage the read began (in task ids).
Small lag values mean a larger buffer would convert those misses to
hits; large values mean readers deep in backlog, where no tip buffer
helps.
## Why?
A standalone win for the code as it is today, with no dependency on the
rest of this series: any multi-cluster mesh already pays `remote
clusters × priority lanes` read amplification on the queue tip, and the
buffer collapses those overlapping scans into one persistence read per
page. It additionally unlocks the later PRs in the series: per-namespace
isolation lanes multiply the number of concurrent scanners, and with the
buffer their tip scans become in-memory filter passes instead of extra
persistence load.
## How did you test it?
- [x] built
- [x] added new unit test(s) — pass-through when disabled, memory
serving for second readers and partial overlaps, contiguous coverage
extension, below-coverage fall-through without cache disturbance, gap
restart at a newer tip, front eviction, truncated-page authority bounds,
hit/miss/lag metric emission, per-reader ownership of served rows,
runtime disable releasing state, empty-page-with-token continuation in
`GetReplicationTasksIter`, and a `-race` concurrent-readers stress test
over a moving tip
- [x] covered by existing tests — the xdc isolation test later in the
series runs with the buffer enabled
## Potential risks
Default-off. The main correctness surface is coverage bookkeeping
(serving a range the buffer isn't authoritative for); the
coverage-interval design plus the truncated-fetch and empty-page-token
tests target exactly that. Serve-time deserialization guarantees no
cross-reader object sharing, and round-trip failures are loud (error
logs) rather than silently degrading. Memory is strictly bounded by the
row-count cap and released on disable.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes replication task loading and coverage bookkeeping in the ack
manager; default-off but incorrect coverage could skip or mis-serve
tasks when the buffer is enabled.
>
> **Overview**
> Adds a **shard-scoped read-through buffer** on the replication task
queue tip so overlapping scans from every remote cluster and priority
lane can share one persistence read instead of amplifying reads per
scanner.
>
> **`readBuffer`** (`read_buffer.go`) tracks contiguous task-id
coverage, stores serialized slim queue rows, and deserializes per serve
so each reader gets owned task structs (downstream code mutates tasks in
place). Capacity is **`ReplicationStreamReadBufferSize`** (default **0**
= disabled); disabling at runtime clears buffered state.
>
> **`GetReplicationTasksIter`** in the ack manager routes reads through
the buffer and tightens persistence paging: empty pages with a
continuation token keep paging until tasks arrive or the token is empty;
truncated pages only extend coverage through the last returned task id.
>
> New metrics: **`replication_stream_read_buffer_hits`**, **`_misses`**,
and **`_miss_lag`** for tuning buffer size and observing catch-up
behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
5836b28b30. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
## What changed?
Reverts #9277 (which removed the shared RPCFactory internode connection
cache), then reimplements the restored cache as a small typed map: the
getter re-dials any connection that has been shut down, and a periodic
sweep drops shut-down entries. Because connections are shared again, the
history connection pool also revalidates its own cached entry and
re-dials when a sibling pool has closed it, a failed dial is no longer
cached, and the redundant closes that follow the first no longer log a
warning.
## Why?
After #9277 each downstream client held its own gRPC connection per
host. Low-traffic clients (e.g. the standalone-activity / Nexus `Start*`
APIs) don't keep their connection busy, so gRPC's 30-min idle timeout
closes it and every sparse call pays a fresh mTLS dial (~50-100ms) — a
large p50/p99 regression. Sharing the connection lets the busy main
client keep it warm.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
## Description
Adds a validation guard for schedule creation, affecting V1 and V2
schedules. Returns a 4XX error if fails. A killswitch is added in case
this runs afoul of some behaviour change.
Affects both Update and Create codepaths.
## Testing
Committed before the fix. The load-bearing failure is the last one —
with a malformed spec, `CreateSchedule` ran past validation into
`createScheduleWorkflow` and hit the V1 backend, proving the request
reached persistence:
```
--- FAIL: .../interval_with_mismatched_signs An error is expected but got nil
--- FAIL: .../interval_with_nanos_at_1e9 An error is expected but got nil
--- FAIL: .../phase_with_nanos_at_1e9 An error is expected but got nil
--- FAIL: TestCreateUpdateSchedule_RejectsMalformedIntervalDuration/CreateSchedule
Unexpected call to *namespace.MockRegistry.GetNamespaceID([test-namespace])
```
All five valid-boundary rows and all three semantic controls (`interval
is too small`, `phase is negative`, `phase cannot be greater than
Interval`) passed before the fix and still pass, with unchanged
messages.
Scope is deliberately limited to durations; `StartTime`/`EndTime`
protobufs (SCH-058) were closed as no-action and are untouched.
Source: SCH-057, Schedule V2 Bug Review (P1, Confirmed, Supported).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
- Apply InitialPatch pause and unpause state before generator and
backfiller work is armed.
- Add CHASM test-engine coverage for initial pause and unpause creation.
- Give the CHASM test engine a default local namespace entry.
## Why?
CHASM-backed schedules previously ignored InitialPatch.Pause and
InitialPatch.Unpause, unlike workflow-backed schedules.
