## Problem
A standby child workflow's `CloseExecutionTask` verifies its parent
recorded the completion, and past
`MaxLocalParentWorkflowVerificationDuration` also resends the parent
from the active cluster. That resend is a cross-cluster state sync plus
a possibly paginated history backfill — minutes of work — but the whole
call was bounded by the standby task's hard-coded **3s** `taskTimeout`.
Measured at the active cluster's history shard, the deadline arriving
there was `2.999s`. So the resend never completed.
## Change
- **Run the resend in the background**, bounded by
`ReplicationTaskApplyTimeout` — the setting that already bounds this
same work on the replication stream. The verify RPC returns immediately
and the standby task retries until the parent lands, so it never holds a
transfer-queue worker for the sync.
- The background context is **detached from the request** (gRPC cancels
that when the handler returns) and **rooted at the shard lifecycle**, so
the work stops with the shard.
- **One in-flight resend per parent**, tracked in a shard-level map, and
at most `history.parentWorkflowResendMaxInFlight` (8) concurrent resends
per shard. Callers retry while an earlier resend runs; without this the
test measured 5 full state fetches where 1 suffices. The cap bounds the
goroutines this path can create.
- **Lifted two client ceilings** so the deadline can actually propagate
— `admin.SyncWorkflowState` (was 10s) and `history.SyncWorkflowState`
(was 30s) now share a `DefaultStateSyncTimeout` backstop. This also
fixes the same 10s cap on the replication stream's
`ExecutableTaskImpl.SyncState`, where production's 5m setting was never
reachable either.
- Metrics:
`parent_workflow_resend_{attempts,skipped,limited,failures,latency}`.
Async failures reach no caller, so `_failures` is the alert signal;
`_limited` means the shard is shedding resends. The background goroutine
recovers panics, which would otherwise take down the process.
Also fixes the history-client codegen template, which hardcoded
`createContext` and silently ignored the timeout-tier field.
## Rollout
`history.enableAsyncParentWorkflowResend`, **default false**. Disabled =
the previous inline behavior, bounded by the caller's task deadline. Opt
in per cell.
## Testing
Unit tests cover the inline, async, and per-parent-dedup paths.
An xdc test (added in abae9cec, removed in b6eb6631) withholds the
parent's replication tasks so the child *must* pull it, asserts the
parent is absent from the standby, then stalls the active cluster's
`SyncWorkflowState` for 4 minutes:
```
--- PASS: TestChildPullsParentWhenParentReplicationIsWithheld (286.03s)
incoming-ctx-remaining: 4m59.999859334s (2.999s before this change)
sync-state-calls: 1 (5 without the per-parent guard)
dropped-parent-tasks: 9
```
During the stall, 4 verify RPCs reached the standby parent shard (t+0,
+50s, +101s, +169s) and exactly 1 `SyncWorkflowState` reached the active
cluster: the task retried and the guard turned the retries away.
4 minutes exceeds every deadline that previously bounded this path (3s /
10s / 30s) with ~1m headroom against the 5m setting, so the setting is
demonstrably what governs.
To reproduce: `git revert b6eb6631`, then
`go test -tags test_dep ./tests/xdc/ -run
TestVerifyChildCompletionParentResendSuite -timeout 30m`
## Known gaps
- Concurrency across *distinct* parents is unbounded (ordinary fan-out,
not amplification).
- When the parent is deleted on the source, the async path can't report
that back, so the child retries to the 15m discard instead of finishing
immediately. The `workflowNotFoundCache` TODO already in this file would
address it.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
Bump `go.temporal.io/api` from the pseudo-version
`v1.63.5-0.20260804201935-e54fd69950e1` to the tagged release `v1.63.5`,
in both the root module and `tests/mixedbrain`.
## Why?
Cloud releases may only ship tagged versions of `go.temporal.io/api`.
The v1.63.5 tag points at `e54fd699`, the same api-go commit the
pseudo-version already referenced, so no api code changes are included.
## How did you test it?
- [x] covered by existing tests
## What changed?
- Fence Backfiller tasks with a persisted task sequence value instead of
comparing task execution time with the backfill HWM.
- Accept unnumbered tasks created by an older binary and restore task
numbering when a new binary executes one.
- Add lifecycle coverage and CHASM test support for firing due persisted
pure tasks.
## Why?
`LastProcessedTime` tracks progress through the requested schedule
range, while a task's `ScheduledTime` controls when that task runs.
Comparing the two can either keep an already-processed historical task
valid or reject a forward-dated task before it runs.
## How did you test it?
- [x] covered by existing tests
- [x] added new unit test(s)
## Potential risks
The immediate task `N` created with a Backfiller is scheduled and
executed in one transaction. Mixed-version risk begins with delayed task
`N+1`.
If `N+1` is handled by a 160 binary:
- For a forward-dated backfill, the old HWM validator can remove `N+1`
before execution, leaving the Backfiller alive with no task to make
further progress.
- For a historical backfill, the old validator can accept `N+1` again
after it has already advanced the HWM, so duplicate execution remains
possible during rollout or rollback.
## What changed?
- Check all non-drained version queues before marking a partition
drained.
- Scaler Describe calls only inspect loaded partitions and do not
refresh queue liveness.
## Why?
- `AllActive` missed unloaded per-version backlog
- Loading unloaded partitions from the scaler caused reload cycles.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Scale-down may wait until an unloaded partition is loaded by real
traffic. Explicit version queues may still be loaded when their
partition is already loaded.
## What changed?
Consume the per-partition backlog counts that the server now delivers in
ClientPartitionCounts.
## Why?
To weight pollers toward partitions with more backlog, so pollers aren't
trapped on empty partitions while others hold a backlog.
When backlogCounts are nil or incomplete, or when backlogCap = 0, pick
the partition weighted by fewest outstanding pollers as we do now.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes how poll and task writes are routed across partitions in the
matching client—a core throughput/latency path—but retains explicit
fallbacks when backlog metadata is missing or stale.
>
> **Overview**
> The matching client load balancer now uses **per-partition backlog**
from `ClientPartitionCounts` (trailer) instead of only uniform random or
fewest-poller heuristics.
>
> **Poll (read) path:** When `backlogCap > 0` and backlog counts cover
all read partitions with at least one positive backlog, partition choice
is **weighted by decoded backlog plus a floor**
(`readPartitionWeightFloor`) so pollers favor partitions with work while
empty partitions still get some traffic. Otherwise behavior stays
**fewest outstanding pollers**.
>
> **Write path:** `PickWritePartition` chooses partitions with
probability proportional to **gap below `backlogCap`**; it falls back to
uniform random when cap is zero, counts are incomplete, or every
partition is at/above cap.
>
> `parsePartitionCounts` now populates `BacklogCap` and `BacklogCount`
from the server response. Unit tests cover weighted distribution,
incomplete backlog fallback, and write gap behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
aa555d8977. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: David Reiss <dnr@dnr.im>
Co-authored-by: David Reiss <david@temporal.io>
## What changed?
Replace the gods treemap backing fairTaskReader.outstandingTasks with a
tidwall/btree (consistent with matcher_data.go and the evictedAcks
cache), and rewrite mergeTasksLocked around the btree's copy-on-write
support: snapshot the outstanding tasks, merge in the newly read/written
tasks, trim to the lowest batchSize loaded tasks, and install the
trimmed tree.
## Why?
The previous limited-copy logic tracked loaded tasks and acks in
separate paths and needed a special-cased "evict acks above readLevel"
pass. The new code keeps loaded tasks and acks in one tree and chops
everything above the cut uniformly, so no ack ever sits above a dropped
task. Because readLevel is now the highest tracked level (loaded or
ack), it no longer collapses toward the ack level, which removes the
readLevel/atEnd churn behind the previously-observed stuck-reader state.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
This code path has historically been a bit tough to get right, I've
tried referencing previous bugs we've found in it and made sure we're
not introducing some new regressions, but hard to say for 100%
certainty.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
Add TemporalNamespaceDivision group by column allowlist
## Why?
allow aggregation across archetypes for system workflow usage.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What changed
- Add a global `matching.enableWorkerDeploymentVersionDemotionSignal`
dynamic config, defaulting to `false` for OSS.
- Preserve the existing signal-based demotion path when the config is
`true`.
- Restore the legacy `SyncWorkerDeploymentVersion` update path when the
config is `false`.
- Record the dynamic-config decision deterministically with workflow
versioning and `MutableSideEffect`.
- Cover SetCurrent and SetRamping demotions under both modes.
- Replay the complete Worker Deployment history corpus with the config
both disabled and enabled.
Temporal Cloud configures this value to `true`. The OSS default is
intended to change to `true` in v1.33, after existing Version workflows
have had time to Continue-As-New onto code that registers the demotion
signal handler.
## Why
#9973 changed version demotion from a synchronous update to a
fire-and-forget signal. Long-running Worker Deployment Version workflows
that had started before the signal handler was available could ignore
that signal, leaving versions stuck in `Draining`.
Cloud has already repaired affected Version workflows through forced
Continue-As-New and should keep the signal path enabled. OSS v1.31 does
not contain #9973, so v1.32 should default to the legacy update path and
avoid introducing this failure mode during upgrade. The main idea, with
OSS, would be to keep this feature disabled for now (default value of
the new dynamic config value introduced is false) and then turn the knob
on to true in the next OSS release.
Existing histories remain deterministic:
- OSS histories without `commit-routing-first` continue replaying the
update path.
- Cloud histories that already contain the `demote-version` signal but
lack `version-demotion-signal-dynamic-config` continue replaying the
signal path, even if the current config is `false`. (this should never
happen since I shall be doing a global rollout of this dynamic config
for cloud very soon)
- Workflows with no recorded demotion decision use and record the
current dynamic-config value when their first demotion occurs.
## User impact
OSS users upgrading to v1.32 retain the pre-#9973 demotion behavior by
default. Cloud retains the currently deployed signal behavior. Existing
OSS and Cloud workflow histories replay without nondeterminism under
either current config value.
## Validation
- `go test -tags test_dep ./service/worker/workerdeployment`
- `go test -tags test_dep ./service/worker/workerdeployment/replaytester
-run '^TestReplays$' -count=1`
#11421 adds the freshly generated histories that prove the signal-based
demotion command path.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **High Risk**
> Changes worker deployment routing/demotion behavior and workflow
determinism rules; mis-toggling the config or replay mismatches could
leave deployment versions stuck in Draining or cause nondeterminism on
long-running workflows.
>
> **Overview**
> Adds global dynamic config
**`matching.enableWorkerDeploymentVersionDemotionSignal`** (default
**`false`** for OSS) so Worker Deployment workflows can choose between
**fire-and-forget demotion signals** and the legacy
**`SyncWorkerDeploymentVersion`** activity path when changing current or
ramping versions.
>
> The deployment workflow records the choice deterministically via
**`workflow.GetVersion`** checkpoints (`commit-routing-first`,
`version-demotion-signal-dynamic-config`) and **`MutableSideEffect`** on
the config value. Histories that already recorded signal-based demotion
keep the signal path on replay even if the config is off; OSS histories
without those markers stay on the update path.
>
> **`fx.go`** wires the config getter into **`Workflow`**. Replay tests
run the full history corpus with the flag both enabled and disabled.
Workflow tests cover SetCurrent and SetRamping demotion for each mode.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
4e9c27b75f. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
add a functional test for time-skipping behavior during workflow pause
and unpause
## Why?
make sure time-skipping works correctly in this scenario:
1. we **allow** setting and storing the time skipping config when paused
2. time **doesn't skip** when paused and **resumes skipping** when
unpaused
3. but if a user sets a time point to disable time skipping in the
options and also pauses the execution (refers to time skipping timer)
this timer **can** still fire and turn off time skipping
## What changed?
This centralizes the health checker into `health.SignalAggregator`. This
standardizes how we will do health checks by using latency based
quantiles and error ratios.
The goal is to be able to use this for any spot we want to do health
detection.
Main idea is you have your overall settings and thresholds for latency
metrics and error ratios, then you can have "groups" of specific keys
that have their own tracking for latency and error ratios. For grpc
health checking, these keys will be the endpoints. For something like
history the keys will be the specific functions.
Doing this also sets us up better for per namespace health settings.
This is 1 of 3 prs. This one is mainly for just testing that everything
works as expected. Then next one will be cleaning up and consolidating
all the health signals in the health package. Then the last one will be
applying
## Why?
We need more strict settings/thresholds for endpoints we consider to be
critical like StartWorkflowExecution, SignalWorkflowExecution,
StartActivityExecution, etc...
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Currently, if you pass in bad settings (e.g. buffer size <= 0), there
will be panics. This isn't a newly introduced problem, but it is an
existing one that will be addressed in future PRs.
## What changed?
- Increment the CHASM scheduler conflict token when pause-on-failure
changes the persisted paused state and notes.
- Add a regression test covering a stale Describe token used by Update
after the automatic pause commits.
## Why?
A token-protected Update could previously replace the entire schedule
using a token captured before pause-on-failure, silently clearing the
automatic pause and its explanatory notes. Invalidating the token keeps
optimistic concurrency behavior aligned with the fields Update replaces
and with the V1 scheduler.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
```
go test -tags test_dep ./chasm/lib/scheduler -run 'Test(HandleNexusCompletion_PauseOnFailure|PauseOnFailureInvalidatesConflictToken)$' -count=1
```
## Potential risks
Token-protected updates already in flight when pause-on-failure commits
will now fail with a conflict-token mismatch and must be retried.
Updates that intentionally omit the conflict token remain unconditional.
## Migration: give same-time pending starts unique identities
### Description
`convertBufferedStartsLegacyToCHASM` derives each migrated start's
`RequestId` and `WorkflowId` as pure functions of its timestamps plus
batch-constant inputs (namespace, schedule, conflict token, base
workflow id). Two pending starts with the same `NominalTime` and
`ActualTime` therefore get identical identities. The loop index was
available but unused.
### User experience
Two or more pending actions can legitimately share a nominal/actual time
(e.g. under `ALLOW_ALL` overlap, or overlapping backfill/trigger
requests). When such a schedule is migrated V1→V2, the collision
silently loses actions: the shared `WorkflowId` means only one workflow
starts (`REJECT_DUPLICATE`), and the shared `RequestId` makes completion
routing and dedup treat the pair as one — the invoker keys
`CompletedStarts`/`FailedStarts`/retries by request ID, so the second
start is indistinguishable from the first. No error is surfaced.
### How it occurs
`GenerateRequestID` = `sched-<backfillID>-<sha1(ns, sched, token,
nominalMs, actualMs)>` and `GenerateWorkflowID` =
`<base>-<nominalSecond>`. For a conversion batch everything except the
timestamps is constant, so equal timestamps → identical IDs.
### How it's fixed
Disambiguate with the per-action loop index, at each identity's natural
seam:
- **Request ID** — the index rides in the existing `backfillID` tag
(`"migrated-0"`, `"migrated-1"`, …). That tag is the literal prefix of
`sched-<tag>-<uuid>`, so the IDs differ without touching the hash;
`convertRunningWorkflowsToBufferedStarts` already uses the same
mechanism with the run ID. `GenerateRequestID` itself is unchanged, so
no other caller's IDs move.
- **Workflow ID** — suffixed only past the first start (`i > 0`). Unlike
the request ID this one is user-visible, and dedup against an action the
V1 scheduler had already started depends on it matching, so the common
case of a single pending action keeps the ID that both V1 and native V2
would give it.
Both apply only to regenerated (empty) IDs; identities carried over from
V1 are preserved.
### Test
- `TestSameTimePendingStartsReceiveUniqueIdentities` — fails before,
passes after. Also pins the first start's workflow ID to the undecorated
`GenerateWorkflowID` output.
- `TestMigratedStartsPreserveExistingIdentities` — V1-supplied
identities are neither regenerated nor suffixed.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed
Adds a freshly generated Worker Deployment replay corpus containing 20
Deployment workflow histories and 11 Version workflow histories.
Two Deployment workflow histories exercise the signal-based
version-demotion path: they record the `commit-routing-first` version
marker at version `0`, initiate the `demote-version` external workflow
signal, and record successful external signaling. The corresponding
Version workflow histories record receipt of those signals.
## Why
The existing replay corpus did not contain Deployment workflow histories
that emitted `demote-version`. These fixtures give us real signal-path
histories to use when validating workflow compatibility changes around
OSS upgrades and the demotion gate.
There are no production code changes in this PR.
## Historical context
PR #9973 initially added `run_1776382947`, containing 21 Deployment
workflow histories and 11 Version workflow histories. That corpus
included two Deployment histories sending `demote-version` and two
Version histories receiving it, together with the new workflow version
markers.
Later in the same PR's branch history, commit
[`4e41ba1`](4e41ba19a9)
deleted that corpus and replaced it with `run_1780926905`, containing 19
Deployment workflow histories and 11 Version workflow histories. The
replacement corpus that ultimately merged contains no `demote-version`
events or `commit-routing-first`/`demote-version-signal` markers; its
`SetCurrent` histories use `SyncWorkerDeploymentVersion`.
PR #9973 was squash-merged, so `4e41ba1` is visible in the PR branch
history but is not preserved as an individual commit on `main`. Because
the replay harness validates determinism and workflow counts without
asserting the presence of specific commands or markers, the loss of
signal-path coverage was not detected. This PR restores that coverage
with histories generated from the current signal path.
## Validation
- `GOWORK=off go test -tags test_dep
./service/worker/workerdeployment/replaytester -run '^TestReplays$'
-count=1`
- Verified all 31 compressed histories with `gzip -t`
- Verified generated counts: 20 Deployment workflows and 11 Version
workflows
- Adversarial fixture review passed
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Test-only replay fixture additions with no runtime or service logic
changes.
>
> **Overview**
> Adds a new replay fixture set under `testdata/v2/run_1785941282` with
**20** Worker Deployment workflow histories and **11** Worker Version
histories (plus `expected_counts.txt` for the replay harness).
>
> Unlike the corpus that landed after PR #9973, this set includes
histories that exercise the **signal-based version demotion** path:
Deployment workflows record `commit-routing-first` at version `0`, emit
the `demote-version` external signal, and log successful signaling;
matching Version workflows record receiving those signals.
>
> **No production code changes**—only test data to restore determinism
replay coverage for demotion-related workflow compatibility.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
a8aae3fa63. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
1. Test fix `tests/activity_api_batch_{cancel,terminate}_test.go`: both
`*_ExcludesNonRunning` tests waited only for two activities to match
`ActivityType` before starting the batch, but the just-completed one may
still be indexed as `ExecutionStatus='Running'`. New shared helper
`waitForRunningFilterToSettle` also waits on the `Running` count,
mirroring the query the batcher counts with in
`adjustQueryBatchTypeEnum`.
2. `isNonRetryableError` treats `serviceerror.FailedPrecondition` as
non-retryable for
`TERMINATE_ACTIVITY`/`CANCEL_ACTIVITY`/`DELETE_ACTIVITY`. Every
reachable FailedPrecondition on those paths is permanent — a terminal
status, or an already-recorded terminate/cancel request with a different
request ID — and processTaskWithRetries retries in place with no
backoff.
## Why?
TestActivityBatchCancel_ExcludesNonRunning was a top CI breaker in the
2026-08-04 Flaky Tests Report.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s) — the two existing ones are the
subject of the fix
## Potential risks
- Activity batch targets in a terminal state now fail immediately rather
than after N attempts. They are still counted as failures, so
`DescribeBatchOperation` totals are unchanged.
## What changed?
Cap the failure kept in LastFailureDetails when an attempt will be
retried, for parity with MutableStateImpl.truncateRetryableActivityFailure
of workflow activities.
## Why?
This is for parity between workflow activity and CHASM based activity
(currently standalone activity) implementation. The workflow activity
limits the retryable failure data kept in the mutable state, so similar
behavior is added for standalone activity. Non-retryable failures are
left intact since they are reported back to the caller, again similar to
workflow activity. It uses the same dynamic config that is used in
workflow activities size limit.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Fred Tzeng <41805201+fretz12@users.noreply.github.com>
## What changed?
Include LastDeploymentVersion in ActivityExecutionInfo returned by
buildActivityExecutionInfo.
## Why?
TransitionStarted persists the poller’s deployment version for the
started activity,
but buildActivityExecutionInfo never assigns the LastDeploymentVersion
field.
Thus Describe always reports nil even when the state contains the value.
Although
the LastDeploymentVersion field is currently not used/set for standalone
activity,
but can be/is only set for workflow activity, this change fixes a
potential bug in
the future when deployment version gets set for standalone activity too.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
- Do not reset heartbeats by default
- Honor `reset_heartbeat` flag
## Why?
- Parity with WFA
- This product behavior makes sense: a user with a long-running activity
using exponential backoff on attempt 10 may wish to reset the attempt
counter in order that the next retry backoff is short, and yet preserve
their checkpointed progress.
## How did you test it?
- [x] modified existing functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes activity reset semantics and persisted state for heartbeat
checkpoints across immediate and deferred reset paths; behavior is
well-covered by tests but affects long-running activity retry/reset
workflows.
>
> **Overview**
> Activity reset now **rewinds the attempt counter** but **keeps
persisted heartbeat checkpoint details by default**, matching
workflow-activity behavior. Clearing heartbeats is **opt-in** via
`reset_heartbeat` / `ResetHeartbeat` on reset APIs.
>
> CHASM activity state adds `reset_should_clear_heartbeat` for resets
requested while a worker is still running; clearing runs when the
attempt yields (same deferred pattern as `restore_original_options`).
Immediate reset paths (`reset`, `resetKeepPaused`) and
cancel-on-reset-request clear that deferred flag only when the flag is
set.
>
> Standalone activity reset forwarding no longer forces `ResetHeartbeat:
true`; it passes the client request. Model/events add
`ResetClearingHeartbeat`; parity and functional tests cover keep vs
clear for scheduled and started activities.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
09fb7a0bd1. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
> Part 1 of a planned 5-PR series building toward replication stream
namespace isolation (a restructuring of #10147): read buffer → reader
group → lane protocol → isolation manager → sender isolation. This PR
stands on its own; follow-ups will be opened as each lands review.
**Next in series: #11302** (reader group).
## What changed?
A shard-scoped read-through buffer over the tip of the replication task
queue, sitting inside the ack manager's `GetReplicationTasksIter`. Every
replication stream sender on a shard — one per remote cluster, with one
iterator per priority lane — scans the same queue, so each new task page
was previously read from persistence once per scanner. With the buffer,
overlapping tip scans share one persistence read; only readers below the
buffered range (deep catch-up, lanes that have lagged out of coverage)
fall through to persistence.
Coverage is a contiguous task-id interval established by persistence
pages: within it the buffer is authoritative, so absence of a task means
the range holds none. An empty persistence page with a non-empty
continuation token (legal, e.g. under Cassandra paging) is NOT
authoritative — the fetch keeps paging until tasks arrive or the token
runs out. Rows are immutable and the queue is append-only, so there is
no invalidation; eviction just shrinks coverage from the front. Reads
are bounded by the shard's exclusive-high read watermark, so covered
ranges are stable once established.
**Ownership:** the buffer stores SERIALIZED rows and deserializes per
serve, so every reader receives fresh task structs it exclusively owns —
exactly what a persistence read would have produced. This matters
because downstream converters mutate tasks in place (e.g.
`SyncVersionedTransitionTask` equivalents get IDs assigned via
`AddTasks`); handing multiple senders pointers to shared structs would
be a data race. The serve-time deserialization replaces the persistence
read the reader would otherwise have done, so it is not added cost
relative to the unbuffered path. Serialize/deserialize failures are
never silent: both are error-logged (they indicate a bug — e.g. a task
type missing serializer support — or broken persistence data); a
serialize failure serves the page uncached, a deserialize failure drops
the buffer's coverage entirely and falls back to persistence.
Capacity is `ReplicationStreamReadBufferSize` tasks per shard (default 0
= disabled); disabling at runtime releases the buffered rows. The buffer
holds slim queue rows (task metadata) for every priority — event
payloads only enter the pipeline at send-time conversion — so memory
cost is a few hundred bytes per row.
Observability (to drive future sizing/sharing decisions):
`replication_stream_read_buffer_hits` / `_misses` count pages served
from memory vs. fetched from persistence while the buffer is enabled
(misses are counted only after a successful fetch), and
`replication_stream_read_buffer_miss_lag` records, for misses below the
buffered range, how far below coverage the read began (in task ids).
Small lag values mean a larger buffer would convert those misses to
hits; large values mean readers deep in backlog, where no tip buffer
helps.
## Why?
A standalone win for the code as it is today, with no dependency on the
rest of this series: any multi-cluster mesh already pays `remote
clusters × priority lanes` read amplification on the queue tip, and the
buffer collapses those overlapping scans into one persistence read per
page. It additionally unlocks the later PRs in the series: per-namespace
isolation lanes multiply the number of concurrent scanners, and with the
buffer their tip scans become in-memory filter passes instead of extra
persistence load.
## How did you test it?
- [x] built
- [x] added new unit test(s) — pass-through when disabled, memory
serving for second readers and partial overlaps, contiguous coverage
extension, below-coverage fall-through without cache disturbance, gap
restart at a newer tip, front eviction, truncated-page authority bounds,
hit/miss/lag metric emission, per-reader ownership of served rows,
runtime disable releasing state, empty-page-with-token continuation in
`GetReplicationTasksIter`, and a `-race` concurrent-readers stress test
over a moving tip
- [x] covered by existing tests — the xdc isolation test later in the
series runs with the buffer enabled
## Potential risks
Default-off. The main correctness surface is coverage bookkeeping
(serving a range the buffer isn't authoritative for); the
coverage-interval design plus the truncated-fetch and empty-page-token
tests target exactly that. Serve-time deserialization guarantees no
cross-reader object sharing, and round-trip failures are loud (error
logs) rather than silently degrading. Memory is strictly bounded by the
row-count cap and released on disable.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes replication task loading and coverage bookkeeping in the ack
manager; default-off but incorrect coverage could skip or mis-serve
tasks when the buffer is enabled.
>
> **Overview**
> Adds a **shard-scoped read-through buffer** on the replication task
queue tip so overlapping scans from every remote cluster and priority
lane can share one persistence read instead of amplifying reads per
scanner.
>
> **`readBuffer`** (`read_buffer.go`) tracks contiguous task-id
coverage, stores serialized slim queue rows, and deserializes per serve
so each reader gets owned task structs (downstream code mutates tasks in
place). Capacity is **`ReplicationStreamReadBufferSize`** (default **0**
= disabled); disabling at runtime clears buffered state.
>
> **`GetReplicationTasksIter`** in the ack manager routes reads through
the buffer and tightens persistence paging: empty pages with a
continuation token keep paging until tasks arrive or the token is empty;
truncated pages only extend coverage through the last returned task id.
>
> New metrics: **`replication_stream_read_buffer_hits`**, **`_misses`**,
and **`_miss_lag`** for tuning buffer size and observing catch-up
behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
5836b28b30. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
## What changed?
Reverts #9277 (which removed the shared RPCFactory internode connection
cache), then reimplements the restored cache as a small typed map: the
getter re-dials any connection that has been shut down, and a periodic
sweep drops shut-down entries. Because connections are shared again, the
history connection pool also revalidates its own cached entry and
re-dials when a sibling pool has closed it, a failed dial is no longer
cached, and the redundant closes that follow the first no longer log a
warning.
## Why?
After #9277 each downstream client held its own gRPC connection per
host. Low-traffic clients (e.g. the standalone-activity / Nexus `Start*`
APIs) don't keep their connection busy, so gRPC's 30-min idle timeout
closes it and every sparse call pays a fresh mTLS dial (~50-100ms) — a
large p50/p99 regression. Sharing the connection lets the busy main
client keep it warm.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
## Description
Adds a validation guard for schedule creation, affecting V1 and V2
schedules. Returns a 4XX error if fails. A killswitch is added in case
this runs afoul of some behaviour change.
Affects both Update and Create codepaths.
## Testing
Committed before the fix. The load-bearing failure is the last one —
with a malformed spec, `CreateSchedule` ran past validation into
`createScheduleWorkflow` and hit the V1 backend, proving the request
reached persistence:
```
--- FAIL: .../interval_with_mismatched_signs An error is expected but got nil
--- FAIL: .../interval_with_nanos_at_1e9 An error is expected but got nil
--- FAIL: .../phase_with_nanos_at_1e9 An error is expected but got nil
--- FAIL: TestCreateUpdateSchedule_RejectsMalformedIntervalDuration/CreateSchedule
Unexpected call to *namespace.MockRegistry.GetNamespaceID([test-namespace])
```
All five valid-boundary rows and all three semantic controls (`interval
is too small`, `phase is negative`, `phase cannot be greater than
Interval`) passed before the fix and still pass, with unchanged
messages.
Scope is deliberately limited to durations; `StartTime`/`EndTime`
protobufs (SCH-058) were closed as no-action and are untouched.
Source: SCH-057, Schedule V2 Bug Review (P1, Confirmed, Supported).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
- Apply InitialPatch pause and unpause state before generator and
backfiller work is armed.
- Add CHASM test-engine coverage for initial pause and unpause creation.
- Give the CHASM test engine a default local namespace entry.
## Why?
CHASM-backed schedules previously ignored InitialPatch.Pause and
InitialPatch.Unpause, unlike workflow-backed schedules.
## What changed?
Removed the UpdateOptionsAllowedAfterDeferredRestoreSupersededByCancel
functional test and added focused unit coverage verifying that
TransitionCancelRequested clears ResetRestoreOptions.
Also corrected the nearby comment claiming that UpdateOptions was
permitted in RESET_REQUESTED when no restore was pending.
## Why?
PR #11394 disallowed UpdateOptions in CANCEL_REQUESTED and
RESET_REQUESTED. PR #11358 later added a conflicting test expecting an
update to succeed after cancel superseded a deferred restore.
The test was removed instead of changed to expect an error because that
result is already covered by UpdateWhileCancelRequestedFails. It also
could not verify whether the deferred restore flag was cleared: both a
cleared and stale flag produce the same FailedPrecondition while the
activity is CANCEL_REQUESTED.
## How did you test it?
- [X] built
- [] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Update sqlparser to v0.1.0. It's actually no-op since it points to the
same commit.
## Why?
Use tagged version instead of commit.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
* Deduplicate cancel retries before terminal-state validation
* Added tests covering terminal-state retries and run-qualified retries
after operation ID reuse.
## Why?
A delayed retry can arrive after the original activity has closed.
Deduplication must still recognize that retry, and callers must pin
cancellation to a run_id so activity ID reuse cannot
redirect the request to a replacement execution.
See also: https://github.com/temporalio/temporal/pull/11344
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Moved activity failure retry classification into a shared helper used by
standalone activities, workflow activities, and workflow retries.
Standalone activities now use the same retry classification as workflow
activities for failures reported through
`RespondActivityTaskFailedById`. This includes retryable
`ServerFailure`s, worker-reported start-to-close and heartbeat timeouts,
and otherwise unrecognized failure variants. Schedule-to-start and
schedule-to-close timeouts remain non-retryable.
## Why?
Standalone activities previously treated non-application failures as
non-retryable. Sharing the existing workflow retry classifier keeps
retry behavior consistent across SAA, WFA, and workflow retries.
---------
Co-authored-by: Fred Tzeng <fred.tzeng@temporal.io>
## What changed?
Added user_metadata summary and details size validation to
StartActivityExecution, using the existing namespace-specific limits and
matching standalone Nexus operation behavior.
## Why?
Standalone activities previously persisted user_metadata without
enforcing its configured size limits. This made the limits inconsistent
across top-level executions and allowed oversized metadata to reach
persistence.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Moved `r.Lock()`/`defer r.Unlock()` in `ReaderImpl.AppendSlices` above
the `r.slices.Back()` ordering check, matching the lock-before-read
pattern already used by `MergeSlices` and `ClearSlices`.
## Why?
`AppendSlices` is called from the queue's `processEventLoop` while the
reader's own `eventLoop` mutates `r.slices` under the mutex in
`loadAndSubmitTasks`. Reading `Back()` without the lock races on
`container/list` internals — confirmed with the race detector.
## How did you test it?
- [x] built
- [x] added new unit test(s)
`TestAppendSlices_RaceWithLockedMutation` fails under `-race` on
unpatched `main` (data race: `AppendSlices` → `Back()` vs. locked
`MoveToBack`) and passes after this change.
Commands run:
- `go test -race -tags test_dep -count=1 -run
'TestReaderSuite/TestAppendSlices_RaceWithLockedMutation$'
./service/history/queues/` — fails on unpatched code, passes with the
fix
- `go test -race -tags test_dep -count=1 ./service/history/queues/` —
full package passes
- `make lint-code` — 0 issues
- `make fmt-imports` — clean
Fixes#11352
## Potential risks
Lock is now held slightly longer (covers the cheap `Back()` read + range
compare in addition to `PushBack`). Same pattern as `MergeSlices`; no
API or persistence behavior change.
## What changed?
Reject SAA unpause requests unless an unpause transition is possible.
## Why?
A no-op unpause could be recorded and later retried, obscuring invalid
state transitions.
Context: follow-up to [standalone activity operator-request
idempotency](https://github.com/temporalio/temporal/commit/d8b84b8aac).
## How did you test it?
- [x] covered by existing tests
- [x] added new functional test(s)
## Potential risks
- Clients that unpause non-paused activities now receive
`FailedPrecondition` instead of a successful no-op.
- This also affects clients unpausing activities that *were*
legitimately paused: the request-ID dedup that makes the rejection
retry-safe only works when the client supplies `RequestId`. When it's
omitted, the server mints a fresh UUID per attempt, so a retry never
matches `LastUnpauseRequestId`. Concretely: a client unpauses a `PAUSED`
activity, the mutation commits, the RPC times out on the way back
(activity is now `SCHEDULED`), the client retries — on `main` that retry
was a benign no-op, now it returns `FailedPrecondition`. Worth
confirming the SDK/CLI populate `RequestId` before this ships.
- The same `UnpauseActivityExecution` RPC against a workflow-owned
activity still silently no-ops when the activity isn't paused
(`service/history/api/unpauseactivity/api.go:120`), so behavior now
diverges by activity kind (standalone vs. workflow-owned) for the same
RPC.
- Separately, `UpdateActivityExecutionOptions` now also rejects with
`FailedPrecondition` while a deferred `Reset(RestoreOriginalOptions)` is
pending (previously it would silently apply and then be clobbered when
the deferred restore landed). Callers that previously succeeded here now
fail while the restore is pending.
---------
Co-authored-by: Dan Davison <dandavison7@gmail.com>
## What
Replace the hand-rolled `cloneTaskQueueStats` with `common.CloneProto`
so new `TaskQueueStats` fields propagate automatically. Also add
`RateLimitingActive` to the two struct literals in
`splitTaskQueueStatsByRampPercentage`.
## Why
[#10944](https://github.com/temporalio/temporal/pull/10944) added
`RateLimitingActive` to `TaskQueueStats` but missed updating these
hand-rolled struct constructions. The field was silently zeroed, causing
`DescribeTaskQueue` and `DescribeWorkerDeploymentVersion` to always
report `false`.
## How did you test it?
- **Unit**: tests for `cloneTaskQueueStats` and
`splitTaskQueueStatsByRampPercentage` verifying the field survives both
transforms.
- **Functional**: `TestDescribeTaskQueue_RateLimitingActive` — sets a 1
RPS API rate limit, drives traffic, and asserts `RateLimitingActive ==
true` in the `DescribeTaskQueue` response end-to-end.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
`cherryPickHSMEvent` treats `hsm.ErrStateMachineNotFound` as skippable
again instead of routing the event to the CHASM tree, restoring the
behavior from before #10986. That is the whole behavior change — one
line.
- `service/history/ndc/workflow_resetter.go` — the routing change, plus
comment cleanup on the surrounding cherry-pick helpers.
- `service/history/ndc/workflow_resetter_test.go` — updated the existing
`state machine not found` case, and added
`TestReapplyEventsHSMNotFoundDoesNotConsultChasm`, which asserts the
event is skipped, no error surfaces, and CHASM is **never consulted**.
- `tests/nexus_workflow_test.go` — the post-reset assertion in
`TestNexusOperationAsyncCompletion` now branches on the rail:
`RequireHistoryEvent` on HSM, `RequireNoHistoryEvent` on CHASM. Both
resets still run on both rails, so the
`RESET_REAPPLY_EXCLUDE_TYPE_NEXUS` coverage and the "reset itself still
succeeds" assertion are kept on the CHASM rail.
## Why?
**Full background, root cause, and the shape of the real fix are
captured in #11384.** In short: reapply cannot distinguish "the CHASM
tree owns this Nexus operation" from "no tree owns it", because CHASM
answers a missing operation with a bare `serviceerror.NotFound`. Reapply
is fail-fast and `BackfillWorkflow` commits only on a clean return, so
one such event discards an entire replication batch — including
completions already applied earlier in the same batch.
This was diagnosed against an MCN handover failure in
`TestReconfigureMCNReplicaWithBenchGoOnTestEnv`, where one Nexus
operation's completion was applied at batch index 0 and then discarded
80 times because an unrelated operation at index 8 existed in neither
tree.
The reset path is affected too, not only replication, which rules out
the narrower fix of gating the fallback on `!isReset`. Details in
#11384.
**This PR is the workaround, not the fix.** It restores pre-#10986 skip
semantics so the failures stop; #11384 tracks doing it properly.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
Verified locally on sqlite:
| scope | result |
|---|---|
| `service/history/ndc` unit tests | green |
| Nexus + Callbacks + Reset + Workflow functional suites | 648 PASS / 0
FAIL |
| XDC Nexus state replication, all 3 variants incl. `Chasm` | 19 PASS /
0 FAIL |
| `TestNexusOperationAsyncCompletion`, both HSM and CHASM rails | PASS |
Both new assertions are mutation-checked rather than merely green.
Restoring the fallback:
- fails the unit test on an unexpected `ChasmEnabled()` call, and
- fails the CHASM functional test at the post-reset
`RequireNoHistoryEvent`.
Across the three functional test files touched by CHASM Nexus work since
#10986 (`nexus_workflow_test.go`, `nexus_standalone_test.go`,
`xdc/nexus_state_replication_test.go`), exactly one assertion depended
on the fallback — the one now branched on the rail.
`TestNexusOperationSurvivesResetCrossTree`, both cancellation cross-tree
tests, and `TestNexusOperationChasmReplicatedWithMixedFlag` all pass
unchanged.
## Review iteration
This went through several rounds of review, which changed the test
strategy more than the fix. Worth knowing if you're re-reading after an
earlier round:
- The functional-test guard started as a top-of-test `Skip`, became a
mid-test early `return`, and is now a per-rail branch on the single
assertion that actually differs. The earlier forms dropped passing CHASM
coverage (completion-token validation, the async-completion happy path,
and the exclude-types block) as collateral.
- The new unit test started as a nine-subtest loop over every Nexus
event type, then two subtests over `isReset`, and is now a single case.
Both loops were vacuous: `chasmworkflow.Registry.Register` dedupes by Go
type so only one definition ever registered, and `reapplyEvents` reads
`isReset` only in the hardcoded `CancelRequested` / `Terminated` cases,
which a Nexus event never reaches.
- Several rounds went into comment accuracy around what is and isn't
reachable after this change. Those comments are now short and defer to
#11384 rather than restating the analysis in four places.
## Potential risks
**This does not itself demonstrate the MCN handover failure is fixed.**
The failure is a replication branch-fork under handover, which no local
test constructs; validation needs the bench-go repro (~1 in 2–3 hit
rate). Local results show the fix restores pre-#10986 skip semantics
without collateral damage, nothing more.
Reset reapply of a Nexus completion for a CHASM-tree operation is
silently skipped again — the bug #10986 fixed. Unreachable while CHASM
Nexus operations are not rolled out, but the feature cannot ship until
#11384 is addressed. The CHASM-rail assertion in
`TestNexusOperationAsyncCompletion` pins that regression, so it will
flip to `RequireHistoryEvent` as part of the real fix rather than being
forgotten.
Two related defects are left in place deliberately, since neither is
reachable once the fallback is gone: CHASM's bare `NotFound` on a
missing operation, and `cherryPickChasmEvent` returning
`serviceerror.Internal` when CHASM is disabled. Both are tracked in
#11384, along with the `TODO(follow-up)` about completion-token
resolution that this PR removes from the code.
---------
Co-authored-by: Chris Smith <aChrisSmith@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
Do not permit `UpdateOptions` in `CANCEL_REQUESTED` or `RESET_REQUESTED`
## Why?
- Hard to define semantics: if `UpdateOptions` lands after
`Reset(restore_original_options)` then should the update be silently
overridden when honoring the reset on attempt end?
- We opt to simplify the combinatorial possibilities now and retain the
possibility of evolving the API to allow it in the future.
- It is unclear whether these transitions should be allowed.
## How did you test it?
- [x] covered by existing tests
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes public API semantics for activities with pending cancel/reset;
clients that relied on mid-flight option updates will now get
FailedPrecondition.
>
> **Overview**
> **`UpdateActivityExecutionOptions` is no longer allowed** while an
activity is in **`CANCEL_REQUESTED`** or **`RESET_REQUESTED`**. Those
statuses are now treated like other non-updatable states and return
**`FailedPrecondition`** with the same message pattern as terminal
statuses.
>
> This replaces the prior behavior where options could still be updated
on a running attempt with a pending cancel, and where updates during
**`RESET_REQUESTED`** could bump the attempt stamp and re-issue timeout
tasks. Standalone activity tests were flipped to expect refusal and to
assert timeouts and run state stay unchanged.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
b4c607a9a8. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Added a missing `d.syncSummary()` call in
`syncVersionDataToComputeStatus`, so it now notifies the parent
Deployment workflow after pulling a compute status from WCI.
## Why?
Without this, the pull only updates the Version workflow's own state.
The Deployment workflow (which
`ListWorkerDeployments`/`DescribeWorkerDeployment` actually read from)
is not updated, so `computeStatus` can stay permanently missing from the
API even when the data is available.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
`Activity.HandleFailed` now treats a `RespondActivityTaskFailed` request
with an omitted `Failure` as retryable, matching WFA behavior.
Previously a nil failure was treated as non-retryable and the SAA closed
as `FAILED`.
## Why?
This achieves parity for SAA with workflow activities.
## How did you test it?
- [x] covered by existing tests
- [x] added new functional test(s)
## Potential risks
This is a behavioral change, but was a bug in the original
implementation
## What changed?
- Emit schedule-to-start latency metric when SAA starts
- The metric distribution (which is per-task queue) will now contain
data points from both SAA and WFA. This is reasonable because there's
nothing about SAA that implies that its time in matching backlog should
have a different distribution.
## Why?
- Required metric; WFA parity
## How did you test it?
- [X] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Adds observability only on the activity-start path after a successful
transition; no auth, persistence, or matching behavior changes. Shared
metric name with workflow tasks is filtered by operation/tags in tests.
>
> **Overview**
> **Standalone activities (SAA)** now emit
`task_schedule_to_start_latency` when History accepts an activity task
start in `HandleStarted`, matching workflow-embedded activity (WFA)
behavior and filling a required metrics gap.
>
> Latency is **started time minus attempt dispatch time** (via
`dispatchTimeForAttempt`), not raw schedule time—so retries measure
backlog from the current attempt’s dispatch, excluding prior attempts
and backoff.
>
> Samples use the same per-task-queue partition scope as WFA
(`HistoryRecordActivityTaskStartedScope`, activity task type,
`MetricsBreakdownByTaskQueue` → real task queue name vs `__omitted__`).
Idempotent `RecordActivityTaskStarted` replays do not record again.
>
> Unit tests in `activity_test` assert sample count and latency for
first start and retry; functional parity tests cover SAA vs WFA for
first attempt and retry across task-queue breakdown settings.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
e702b65ccd. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Added an explicit precondition check for standalone activity
cancellation responses. RespondActivityTaskCanceled now requires the
activity to be in CANCEL_REQUESTED and returns the established
ErrActivityTaskNotCancelRequested error otherwise.
## Why?
Standalone activities previously exposed an internal invalid-transition
error when a worker reported cancellation without a prior cancellation
request. Workflow activities return a stable InvalidArgument API error
for the same scenario.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
See API change https://github.com/temporalio/api/pull/846
## What changed?
- Drop `reset_attempts` and `reset_heartbeat` from
`UnpauseActivityExecution`
## Why?
- We have so far been unable to assign desirable and consistent
semantics to them during implementation: for example if
`Unpause[resetAttempts]` is received during retry backoff it is unclear
whether to honor the remaining delay time, because this is how Unpause
usually behaves, or dispatch immediately, because this is how Reset
behaves.
- No known user demand
- They are confusing: they mix `Unpause` and `Reset` functionality in a
confusing way
- They can be added later
## How did you test it?
- [x] covered by existing tests
## Breaking changes
- This API has always been rejected by the server. When server starts to
accept it, an old client could submit these options and they would be
ignored. Operator API is not GA.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes activity unpause scheduling semantics for the execution API
and drops reset-on-unpause behavior that was only partially implemented;
low user impact if the API was not GA and had no known callers.
>
> **Overview**
> Aligns the server with the **UnpauseActivityExecution** API change:
**`reset_attempts`** and **`reset_heartbeat`** are no longer part of
unpause for standalone (CHASM) activities.
>
> **CHASM activity unpause** no longer resets attempt count, retry
interval, or heartbeat state on unpause, and always considers the
pending retry backoff when scheduling dispatch (the branch that skipped
that when `reset_attempts` was set is removed). Workflow-embedded
unpause forwarding via **`UnpauseActivityExecution`** no longer passes
those fields to the legacy **`UnpauseActivity`** history call (jitter
and identity only).
>
> **`go.temporal.io/api`** is bumped to the revision that removes the
fields from **`UnpauseActivityExecutionRequest`**.
>
> **Tests** are updated so unpause helpers no longer take a reset flag;
reset-on-unpause coverage stays on legacy **`UnpauseActivity`** only
(execution API skips that case). Standalone tests for
**`UnpauseWithResetAttempts`** and **`UnpauseWithResetHeartbeat`** on
**`UnpauseActivityExecution`** are removed.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
7a41d507ca. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Exposed RetryState on standalone activity execution outcomes and
persisted it in activity state. Retry evaluation now records terminal
reasons including retry policy not set, cancellation requested,
non-retryable failure, maximum attempts reached, and timeout.
## Why?
Standalone activities previously collapsed retry decisions into a
boolean, preventing callers from distinguishing why an activity stopped
retrying. This brings standalone activity behavior and observability
into parity with workflow activities.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
During a rolling upgrade, activities closed by an older server may
return RETRY_STATE_UNSPECIFIED. Existing closed activities also remain
unspecified because retry state was not previously persisted. Older
clients safely ignore the new protobuf field.
---------
Co-authored-by: Dan Davison <dandavison7@gmail.com>
## What changed?
- Persist payload sent with attempt failure as last heartbeat details
- Emit metrics associated with that codepath for WFA parity
## Why?
- The first is a relatively bad bug: an activity attempt should be able
to have the latest checkpoint data sent in and persisted with a
retryable failure, but SAA was not persisting it
- SAA vs WFA metrics parity
## How did you test it?
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes activity failure and retry persistence where workers rely on
checkpoint data; scope is narrow with new SAA/WFA parity tests, but
incorrect handling could affect retries or observability.
>
> **Overview**
> Standalone activities (SAA) now **persist `LastHeartbeatDetails` from
`RespondActivityTaskFailed`** before deciding whether to retry or fail
terminally. Previously that checkpoint lived only on the terminal
`TransitionFailed` path, so **retryable failures dropped the worker’s
final progress payload**.
>
> Heartbeat handling on failure now mirrors a normal heartbeat: update
last-heartbeat state (details, recorded time, count) and record metrics
via a shared **`emitHeartbeatMetrics`** helper used by
**`RecordHeartbeat`** as well.
>
> Trace drivers and parity tests gain **`HasHeartbeatDetails`** on
failed-respond events, plus coverage that WFA and SAA expose the same
stored heartbeat details (including terminal SAA failures).
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
a013d1627f. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Updated the CHASM scheduler's DescribeSchedule response to resolve the
effective catchup window using the namespace's scheduler tweakables.
DescribeSchedule now reports:
- nil, zero, or negative values as `DefaultCatchupWindow`
- positive values below the minimum as `MinCatchupWindow`
- values at or above the minimum unchanged
The resolution is applied to a cloned schedule, leaving persisted
scheduler state unchanged.
## Why?
DescribeSchedule previously used a hard-coded one-year default and did
not consistently report the same effective catchup window used during
schedule processing.
Using the existing catchup-window resolver keeps DescribeSchedule
consistent with runtime behavior and namespace-specific dynamic
configuration.
## V1 and V2 behavior
V1 and V2 currently differ when the configured catchup window is zero or
negative:
- V1 treats zero or negative values as below the minimum and resolves
them to `MinCatchupWindow`.
- V2 treats zero or negative values as unset and resolves them to
`DefaultCatchupWindow`.
- Both implementations clamp positive values below the minimum to
`MinCatchupWindow`.
This PR changes only the CHASM/V2 DescribeSchedule path and does not
modify V1 behavior.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
Commands run:
- `go test -tags test_dep ./chasm/lib/scheduler`
- `go test -tags test_dep ./tests -run
'TestScheduleCHASM/TestDescribeCatchupWindowAfterCreateAndUpdate'
-count=1`
- `make lint-code`
`testBasics` already performs create→describe and update→describe,
including the unset/default case, but it is shared by V1 and V2. Since
zero/negative semantics currently differ between the implementations,
adding those cases there would break V1 coverage. A focused CHASM-only
functional test was added instead.
The CHASM-only test covers create with an unset catchup window, followed
by updates with zero, negative, positive-below-minimum, and
above-minimum values.
Local testing:
1. http://localhost:3000/ + UI side override to allow <10 secs
2. Create schedule with catchup window 0 secs. Load it and verify it
shows as 10 secs.
3. DC changes to switch to CHASM. Repeat(2) to verify its set as 1 year.
## Potential risks
DescribeSchedule now returns the effective catchup window rather than
the raw persisted value for non-positive and below-minimum values. This
matches the value used by CHASM schedule processing.
### V1 and V2 migration
This PR changes only the CHASM DescribeSchedule response. It does not
normalize the persisted schedule policy or change migration payloads, so
the existing migration behavior remains:
- **V1 → V2:** V1 eagerly normalizes zero or negative values to an
explicit `MinCatchupWindow`. Migration copies that positive duration, so
V2 continues using the minimum.
- **V2 → V1:** V2 persists the original zero or negative value and
treats it as unset/default at runtime. Migration currently copies that
raw value. V1 then resolves it to `MinCatchupWindow`, potentially
changing the effective behavior from the V2 default to the V1 minimum.
- **Unset:** The target implementation resolves the unset value using
its own default. Behavior could change if the source and target defaults
differ.
- **Positive below minimum:** The target implementation applies its own
minimum. Behavior could change if the source and target minimums differ.
- **At or above minimum:** The explicit value is preserved across
migration.
Resolving or persisting the effective catchup window during migration is
outside the scope of this PR.
## What changed?
- Port SAA/WFA `CompleteById` tests to declarative framework
## Why?
- We will gain additional test assertions when the declarative tests are
wired up to the spec (model)
- Easier to read and reason about the tests: 225 LOC reduction
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Changes are limited to test vocabulary, test drivers, and parity
tests; no production activity completion logic is modified.
>
> **Overview**
> Extends the activity parity **trace model** with `CompleteByID`
(`RespondCompletedByIDType`) and teaches the workflow-activity and
standalone-activity drivers to issue `RespondActivityTaskCompletedById`
when that event appears in a trace.
>
> Replaces two long, hand-written parity tests
(`TestCompleteByID_BeforeAnyWorkerStarts` and
`TestCompleteByID_WhilePaused`) with a single table-driven
`TestCompleteByID` that drives the same scenarios via traces
(`CompleteByID` alone, or `Pause` then `CompleteByID`) for both WFA and
SAA. Standalone activity still asserts `LastStartedTime` is set after
force-complete without a worker poll.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
79ecf3099e16db51d593b71237164770064116d8. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Enable standalone activity start delay by default. Remove unnecessary
test overrides.
## Why?
Start delay to be enabled by default for standalone activities GA
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed
- Allow the S3 visibility archiver query parser to accept
`WorkflowType`.
- Keep `WorkflowTypeName` as a deprecated compatibility alias.
- Rename the parsed query field to `workflowType` and update S3
visibility archiver tests.
- Fix the S3 parser StartTime test assertion and StartTime operator
error message.
## Why
- S3 visibility archiver queries only accepted `WorkflowTypeName`, while
filestore, gcloud, and non-archived visibility records use
`WorkflowType`. This keeps old queries working while accepting the
standard field name.
- Fix https://github.com/temporalio/temporal/issues/7821
### Motivation
- The change that updated MaxDeployments behavior and error text caused
flakiness in `main` by making `TestNamespaceDeploymentsLimit` unstable
under shared test namespaces.
- Revert and re-disable the affected functional test to restore stable
test behavior while the underlying flakiness is investigated.
### Description
- Restore the previous worker-deployment limit error wording in
`service/worker/workerdeployment/client.go` (reverting the wording
change that exposed "worker deployments" in the message).
- Re-disable the unstable functional test by adding `s.T().Skip()` in
`tests/worker_deployment_test.go` for `TestNamespaceDeploymentsLimit`
and restore its prior flow/assertions that expect the original error
message.
- Adjust test assertions in
`TestCreateWorkerDeployment_MaxDeploymentsLimit` to match the restored
error text (`"reached maximum deployments in namespace"`).
- Modified files: `service/worker/workerdeployment/client.go` and
`tests/worker_deployment_test.go`.
### Testing
- Ran the targeted functional test with `go test -tags test_dep ./tests
-run 'TestWorkerDeploymentSuite/TestNamespaceDeploymentsLimit$'
-count=1`, which completed successfully.
- Ran `make lint-code`, which failed due to a network error downloading
`golangci-lint` (HTTP 403 from `proxy.golang.org`).
------
[Codex
Task](https://chatgpt.com/codex/cloud/tasks/task_b_6a6cedda33c08324a2863a637b5c00ca)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> User-visible error string reversion and test skip/assertion alignment
only; no limit logic changes.
>
> **Overview**
> Reverts namespace **max deployment** limit errors to say **"reached
maximum deployments in namespace"** instead of **"worker deployments"**,
in both `CreateWorkerDeployment` and auto-create via poll paths in
`client.go`.
>
> **`TestNamespaceDeploymentsLimit`** is skipped again (shared namespace
/ visibility flake) with a TODO on poller error messaging; the in-test
flow is simplified to shared helpers when re-enabled.
**`TestCreateWorkerDeployment_MaxDeploymentsLimit`** now expects the
restored error substring.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
63a7cf00ac. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What
Move ListWorkers, DescribeWorker, and CountWorkers from
`VisibilityAPIToPriority` to `APIToPriority`.
Changed priority to P3 to be aligned with other status Querying APIs.
## Why
Today, these APIs are served by matching and not visibility.
## How did you test it?
Updated unit tests
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
In the CHASM (V2) scheduler, treat a non-positive (zero or negative)
schedule
catchup window the same as unset: return the default catchup window
instead of
clamping up to the minimum. Only a positive value below the minimum is
clamped up.
Change is in `chasm/lib/scheduler/spec_processor.go` (`catchupWindow`).
## Why?
Previously only a `nil` catchup window fell back to the default; a zero
or
negative value slipped through to `max(cw, MinCatchupWindow)` and was
silently
clamped up to the minimum. A non-positive value is effectively
"unset/invalid"
and should resolve to the default.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
Added a component level functional test (Generator) for now.
Ideally, we'd add a server-level test that creates schedules with
different catchup window values and verifies the result via
DescribeSchedule. However, that doesn't currently validate the intended
behavior because the describe logic overwrites the catchup window in
some cases (see:
https://github.com/temporalio/temporal/blob/main/chasm/lib/scheduler/scheduler.go#L697-L699).
Will work on this fix next as it needs more plumbing and also add this
specific test in the next PR.
## Potential risks
Behavior change for any schedule that explicitly sets a catchup window
<= 0:
it now resolves to DefaultCatchupWindow instead of MinCatchupWindow.
## What changed?
SAA terminal timeout failures now chain the previous attempt’s failure
as their cause.
## Why?
When retries ended in a timeout, SDK users could see only the timeout
and not the application failure that drove the retries. Preserving the
cause exposes the useful underlying error and matches Workflow Activity
behavior.
## How did you test it?
- [x] added new unit test(s)
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes activity timeout failure protobuf shape and terminal outcome
logic on a user-visible error path; scope is limited to SAA timeout
handling with strong test coverage.
>
> **Overview**
> **Standalone activity (SAA) terminal timeouts now set `Failure.Cause`
to the last attempt’s stored failure** (typically the application error
that triggered retries), so clients see the underlying error via
`TimeoutError.Unwrap()` instead of only the timeout wrapper—aligned with
workflow-embedded activities.
>
> `TransitionTimedOut` reads `priorAttemptFailure` from
`LastFailureDetails` **before** recording the current timeout, then
passes it into schedule-to-start/close outcome failures and sets `Cause`
on start-to-close and heartbeat terminal failures. When a per-attempt
timeout exhausts the schedule-to-close retry window
(`RETRY_STATE_TIMEOUT`), the final schedule-to-close outcome still
chains that prior failure even though the per-attempt timeout was
written to attempt state first.
>
> Coverage adds a state-machine unit test for the retry-window path and
SAA/WFA parity tests (including a check that **retryable** timeouts do
not chain causes on `LastFailure`). Test helpers use a stable
`TestFailure` application failure type for assertions.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
89ae0251ee. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
- Workflow task completion buffer overflow now fails the WFT with
`WORKFLOW_TASK_FAILED_CAUSE_REQUEST_TOO_LARGE` instead of
`PAYLOADS_TOO_LARGE`.
- `DescribeNamespace` reports
`NamespaceInfo.Limits.workflow_task_completion_size_limit_error`,
sourced from `history.workflowTaskCompletionBufferSizeLimit`.
API PR: https://github.com/temporalio/api/pull/838
## Why?
`PAYLOADS_TOO_LARGE` is misleading here, no single payload is oversized,
the request total is. Exposing the limit lets SDKs page under it instead
of discovering it by failing.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
No
## What changed?
Propagated the activity retry policy into the
PollActivityTaskQueueResponse created for eagerly dispatched activities.
Added a unit-test assertion verifying that eager activity tasks contain
the retry policy from the original schedule command.
## Why?
The eager response construction omitted RetryPolicy, causing SDKs to
report an undefined retry policy in activity context even though the
workflow scheduled the activity with one.
## How did you test it?
- built
- run locally and tested manually
- added new checks to existing unit test
- Verified manually using the TypeScript SDK retry-policy test against
the patched local server.
## What changed?
This PR refactors the validation logic used for SANO from a collection
of loose functions, into methods on an unexported `validator` type.
(Similar to how `chasm/lib/callback/validator.go` is structured.)
The same checks have all been preserved, although I did fix up one error
message string to be consistent with others.
## Why?
The motivation for this refactoring is to make it easier to land the
"worker callbacks" feature. That will require expanding the validation
checks, and bundling all the dependent parameters on the `type validator
struct` is cleaner than needing to wire through a new parameter at every
callsite.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
I am relying on GenAI in its assertion that the existing validation
checks are essentially identical with these changes. Worst case
scenario, this alerts which types of SANO requests are accepted or
rejected.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
- Deduplicate cancel and pause retries before terminal-state validation.
- Added tests covering terminal-state deduplication and run-qualified
retries after activity ID reuse.
## Why?
A delayed mutation retry can arrive after the original activity has
closed. Deduplication must still recognize that retry, and callers must
pin mutations to a run_id so activity ID reuse cannot redirect the
request to a replacement execution.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
