## What changed?
`GetWorkflowExecutionHistory` and `GetWorkflowExecutionHistoryReverse`
now check `branch_token` in the page token against the token in mutable
state.
## Why?
To confirm if it is still the correct branch after conflict resolution.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Adds metrics for how often and by how much queue slices
fail to narrow their predicate, and how large persisted queue state
actually is.
- `queue_slice_pending_keys` — histogram, recorded on every narrowing
attempt (declined or
succeeded). This is the distribution
`queueShrinkPredicateMaxPendingKeys` should be sized against.
- `shard_info_size` / `queue_state_size` — histograms recorded when a
shard record is actually
written, giving whole-record and per-category size.
- `queue_state_size_total` / `queue_slice_count_total` — counters paired
with the histograms above
(and with the existing `queue_slice_count`), so an exact bytes-per-slice
ratio is possible.
- `queue_slice_count` gains a `task_category` tag (previously untagged
beyond `operation`).
These are only metrics changes - no behavior changes.
## Why?
A slice only narrows its predicate below
`queueShrinkPredicateMaxPendingKeys` (10) pending
namespaces; above that it stays universal and re-reads the whole range
every time. Raising that
threshold safely requires knowing the pending-key distribution and the
persisted size.
This PR is the baseline for evaluating a follow-on encoding.
There are two counters because this server's tally-backed Prometheus
reporter doesn't preserve the
true recorded value when a histogram flushes — it replays each sample as
its bucket's upper bound,
so a histogram's `_sum` has no more precision than its buckets.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] run locally and tested manually (`queue_predicate_resolution_loss`
confirmed live against a local server under forced
narrowing-decline conditions)
## Potential risks
## What changed?
- emit remote cluster and namespace replication lifecycle records under
`namespace_lifecycle`
- retain compatibility aliases for specialized event-name constants with
TODO cleanup
- update tests and shared envelope documentation
## Why?
Update schema event names to match namespace lifecycle schema which
currently exist so they are properly interpreted.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Refactor NamespaceRateLimitInterceptor with functions to consume N
tokens:
- removed `tokens` overwrite argument as it's never used
- added functions to consume N tokens
The changes itself in this PR is no-op since it's introducing new
functions to the interface.
## Why?
Added flexibility to wrap `NamespaceRateLimitInterceptor`.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed
The completion (callback) handler's request-scoped logger carried only
the namespace, even though a richer one was built just above it.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Go 1.27 prerequisite that applies the `slicesbackward` Go fixer for
reverse iteration.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What
Treat `NotFound` from the Version workflow delete update as success, so
the Deployment workflow can clean up its stale version reference.
## Why
When a Version workflow is already closed or its history is gone, the
delete update returns `NotFound`, blocking the Deployment workflow from
removing the reference. Fixes#11539.
## How did you test it?
Unit test covering the `NotFound` → success path and verifying other
History errors still propagate.
## What changed?
Added Temporal Nexus attributes to spans.
## Why?
Domain attributes make Nexus traces more useful.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Every task produced by `GenerateMigrationTasks` is now low priority end
to end:
- Added a `Priority` field to `HistoryReplicationTask`,
`SyncActivityTask` and `SyncHSMTask`.
- Round-tripped `Priority` through the replication task serializer for
those three, plus `SyncVersionedTransitionTask` (its `Priority` field
existed but was never persisted). The proto
`ReplicationTaskInfo.Priority` field already existed, so no proto
change.
- `StreamSenderImpl.getTaskPriority` now honors `Priority` on all five
replication task types, defaulting to high when unspecified.
- `GenerateMigrationTasks` stamps `TASK_PRIORITY_LOW` on the
`HistoryReplicationTask`, the sync activity tasks and the `SyncHSMTask`
it returns (`SyncWorkflowStateTask` and `SyncVersionedTransitionTask`
already set it).
## Why?
`GenerateMigrationTasks` is only reachable through force replication
(`GenerateLastHistoryReplicationTasks`, called by the
migration/force-replication workflow and by tdbg). That traffic is bulk
backfill and should not compete with live replication. Only
`SyncWorkflowStateTask` was actually being treated as low priority;
everything else fell through to `TASK_PRIORITY_HIGH` in the stream
sender, and `SyncVersionedTransitionTask`'s low priority was silently
dropped on write.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
`TestTaskGeneratorImpl_GenerateMigrationTasks` now asserts
`TASK_PRIORITY_LOW` on every returned task and every task equivalent.
`service/history/replication`, `common/persistence/serialization` and
`service/history/tasks` pass.
Note: `service/history/workflow` has a pre-existing failure in
`TestTaskRefresherSuite/TestRefreshSubStateMachineTasks` that reproduces
on unmodified `main` at 39fc2c45e and is unrelated to this change.
## Potential risks
- `SyncVersionedTransitionTask.Priority` is now persisted where it was
previously dropped. Tasks written before this change still deserialize
with `TASK_PRIORITY_UNSPECIFIED`.
- `getTaskPriority` defaults to high on `UNSPECIFIED` for the
newly-handled types, so normal (non-force) replication keeps its current
priority. Only tasks explicitly stamped low move to the low priority
stream.
- Force replication tasks now share the low priority stream and its rate
limiting with sync-state traffic, so a large force replication may
progress more slowly than before — which is the intent.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
Adds server support for WorkflowQuery-backed Nexus Operations
## Why?
Part of effort to expose all Temporal primitives as Nexus Operations
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## TODO:
- [x] https://github.com/temporalio/api/pull/842
## Summary
- capture post-transaction mutable state for fresh `NotFound` snapshots
and `IsFirstSync` creation
- report successful verify history repairs as `outcome=backfilled`
- include the repaired event range and `new_run_id` in the verify
applied event
- add regression coverage for fresh zombie applies and
non-current-branch backfills
## Testing
- `go test ./service/history/ndc ./service/history/replication
./common/wideevents -count=1`
## What changed?
Adds `testcontext.EnsureRemaining` and has `await` use it so long await
calls can request additional test-scoped context time while still
respecting the test context cap.
## Why?
Await calls can need more time than the default test context has left
(esp after the environment setup). Extending the test timeout in this
way allows for (1) stuck tests to fail earlier than the default test
timeout and (2) legitimately longer running tests to pass without
manually tweaking the test timeout.
---------
Co-authored-by: Sean Kane <sean.kane@temporal.io>
## What changed?
- Clear an activity’s timer-task status when it is unpaused so timeout
tasks are regenerated.
- Make ResetActivity with keepPaused=false fully unpause both scheduled
and running activities, including clearing pause metadata.
- Add and strengthen unit and functional coverage for unpause,
reset-unpause, timer regeneration, and keepPaused=true.
## Why?
Timeout tasks can fire while an activity is paused and be discarded.
Previously, the activity’s timer-task status still indicated that those
tasks existed, preventing them from being recreated after unpause and
potentially making the timeout ineffective.
ResetActivity also bypassed normal unpause handling, and running
activities returned early without clearing their paused state.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Unpausing now invalidates existing timeout tasks and recreates the next
applicable timer during transaction close. This is correct, but a
behavioral change. Stale queued tasks may still be processed and
discarded through the existing stamp validation.
## What changed?
Adds `namespace_lifecycle` start and finish events for the namespace
handover, force replication, and catchup system workflows.
The events carry workflow identity and the core operation inputs.
Finished events classify the result as succeeded, canceled, or failed.
Force replication reports its cumulative verified workflow count and
emits only one start and one finish across a continue-as-new chain.
Emission uses one shared activity, the existing
`system.emitNamespaceLifecycleEvents` gate, disconnected cleanup for
cancellation, and workflow versioning for replay compatibility. Existing
shard handover events are unchanged.
## Why?
These system workflows currently have no consistent operation-level
event pair, which makes it difficult to correlate a namespace migration
request with its final outcome.
## How did you test it?
- [x] covered by existing tests
- [x] added new unit test(s)
`go test -tags test_dep ./common/wideevents ./service/worker/migration`
`make fmt-imports`
`make lint-code` reports no issues introduced by this change; the
repository-wide target still reports existing findings on current
`main`.
## Potential risks
The terminal event is best effort and cannot run after server-side
workflow termination or workflow run timeout because those outcomes do
not execute workflow cleanup.
## What changed?
Makes Standalone Activity conflict updates idempotent by recording the
`requestID` when attaching callbacks or links and recognizing duplicate
request IDs. I needed to add a dedicated CHASM error for
## Why?
This prevents a successful attachment whose response was lost from
failing on retry or duplicating/replacing callbacks and links.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
A blocked `ChasmEngine.pollComponent` now returns a
`ShardOwnershipLostError` as soon as its shard moves off this host — by
adding a `select` case on the shard's lifecycle context — instead of
blocking until the request context deadline.
## Why?
Follow-up to #10860 (requested in review): `pollComponent` had the same
gap as the `GetWorkflowExecutionHistory` long poll — nothing in its
`select` was tied to shard lifecycle, so a poll in flight when its shard
moved stalled until timeout. This lets the caller redirect to the new
owner immediately.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
### What changed
Sets gobreaker's OnStateChange hook on the outbound queue circuit
breaker pool, logging every transition.
### Why
Obtain more details for debugging curcuit breaker in production.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
- Added a separate `namespace_replication_lifecycle` wide event with
`created`, `received`, `processed`, and `dlqed` phases.
- Included namespace/task identity, source and target clusters, source
task ID, retry attempt count, deterministic task fingerprint, and the
serialized namespace replication task.
- Included the successful `CreateNamespaceRequest` or resolved
`UpdateNamespaceRequest` as `persistence_request` on `processed`;
duplicate, stale, and skipped tasks omit it.
- Passed receiver-side diagnostic metadata through a typed context so
the existing `TaskExecutor.Execute` and create/update handler signatures
remain unchanged.
- Added the dedicated, default-off
`system.emitNamespaceReplicationLifecycleEvents` dynamic-config gate,
checked explicitly at both the processor and processed-event emitter.
- Preserved the namespace replication queue message ID as
`source_task_id` when reading tasks.
## Why?
Namespace CRUD events describe user-visible namespace mutations, but do
not show whether the resulting namespace replication task was queued,
received, applied, retried, or sent to the DLQ. These events provide
that transport and processing audit trail without additional persistence
reads.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
Commands:
```text
go test -tags test_dep ./common/wideevents ./common/namespace/nsreplication ./service/worker/replicator ./service/frontend
make GOLANGCI_LINT_FIX=false GOLANGCI_LINT_BASE_REV=origin/main lint-code
```
Local two-cluster testing covered create, update, and failover after
rebasing onto `origin/main`. Each task produced linked `created ->
received -> processed` events, and `processed` contained the expected
persistence request. With the dynamic-config flag off, namespace
replication still completed and neither cluster emitted a matching
lifecycle event.
## What changed
Adds logs tags for failures on the Nexus frontend path.
## Why
Have more details to correlate issues with requests.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed
Adds a few Nexus-specific log tags to the handler-side frontend logger.
## Why
Mainly for the request ID to debug Nexus calls across namespaces better.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- Generator execution clamps its processed time range to
`LastProcessedTime`, but `computeFutureActionTimes` (used by both
`Describe` and `UpdateFutureActionTimes`) only floored its starting time
at `max(now, UpdateTime)`, ignoring the watermark.
- This let Describe/`FutureActionTimes` advertise occurrences at or
before `LastProcessedTime` that the generator had already processed or
silently skipped.
- Floors the starting time at `LastProcessedTime` as well, so every
advertised future action is guaranteed to be strictly after the
generator's high water mark.
- Reuses `common/util.MaxTime` for this floor and for the existing
`UpdateTime`-vs-`LastProcessedTime` clamp in
`GeneratorTaskHandler.Execute`, instead of hand-rolled `if X.After(Y) {
Y = X }` checks.
## Test plan
- Added
`TestGeneratorTask_FutureActionTimesRespectLastProcessedTimeWatermark`
in `chasm/lib/scheduler/generator_tasks_test.go`, which pushes
`LastProcessedTime` ahead of "now" and asserts all advertised future
times are strictly after it. Verified it fails without the fix and
passes with it.
- Built on a new scheduler-specific `chasmtest` helper
(`newSchedulerTestEngine` + `updateScheduler`/`readScheduler` in
`helper_test.go`), adapted from PR 0 of the `sch-readable` stack, rather
than the `newTestEnv` rapid harness, so the watermark mutation and task
execution cross the same transaction/read boundaries as production. Left
out the parts of that PR not applicable here (frontend-client plumbing,
the generic side-effect-task firing helper), since
`GeneratorTaskHandler` is a pure task handler that doesn't touch the
frontend client.
- `go test -tags test_dep ./chasm/lib/scheduler/...` passes.
## The bug
`applyUpdatesToSubStateMachine` shadowed the variable it meant to
populate:
```go
var existing V // outer: never assigned
if existing, ok := pendingInfos[key]; ok { // := declares a NEW existing, scoped to the if
...
ms.approximateSize -= existing.Size() + getSizeOfKey(key) // inner one — correct here
}
val := updated
if sanitizeFn != nil {
val = common.CloneProto(updated)
sanitizeFn(existing, val) // outer one — always nil
}
```
`:=` only requires *one* new variable on the left (`ok`), and the `if`
init statement is its own scope, so `existing` is redeclared there
rather than assigned. It compiles because both variables are used, and
`go vet` does not check shadowing by default.
`sanitizeFn` has therefore always received `current == nil`.
## What this activates
Two of the five `sanitizeFn` implementations depend on `current`; the
other three (timers, request cancels, signals) ignore it or are `nil`,
so they are unaffected.
**Activities.** `getActivityTimerTaskStatus` short-circuits to
`TimerTaskStatusNone` when `current` is nil, so every applied activity
update cleared the *entire* timer task mask rather than only the bits
whose deadline moved. The deadline comparison added in #11565 has never
taken effect — that PR is currently a no-op.
**Child executions.** `if current != nil { incoming.Clock =
current.Clock }` never ran. `Clock` is a local shard vector clock that
`sanitizeChildExecutionInfo` strips before replicating, so the incoming
copy always arrives nil — the guard exists precisely to restore the
local value. Without it, every replicated update to an existing child
execution erased it.
## Risk
Both are restorations of intended behavior, not new behavior, and both
previously failed in the safe direction:
- An over-cleared activity mask regenerates a timer task that is already
pending. Duplicates are dropped at execution
(`processSingleActivityTimeoutTask` re-derives the sequence and fires
what expired), so the symptom was extra timer-queue writes, not missed
timeouts.
- A nil child clock is tolerated by callers — see the comment in
`recordchildworkflowcompleted/api.go`: *"it should be fine e.g. that
ci.Clock is nil"*.
The child execution change is a **no-op unless the local cluster
recorded a clock by starting the child itself**, which for a passive
cluster means after a failover. On a standby that never started the
child, both sides are nil.
For activities, preserved bits suppress recreation of a timer task, so
it is worth being explicit about why that is safe on the passive side:
an expired-but-unresolved standby timer returns `ErrTaskRetry` and stays
in the queue rather than being consumed, so there is nothing to
recreate. The one path that does drop a task, past
`StandbyTaskMissingEventsDiscardDelay`, logs a warning and increments
`task_errors_discarded`, so it cannot happen silently.
## Tests
Two new tests at the `applyUpdatesToSubStateMachines` level, one per
activated `sanitizeFn`. **Both fail with the shadowing restored**
(verified: `expected: 13, actual: 0` for the mask; `local child
execution clock was not carried over` for the clock).
The gap they close: the existing `getActivityTimerTaskStatus` tests call
the decision function directly with a non-nil `current`, so no unit test
could observe the *caller* passing nil. Nine passing tests, zero
coverage of the wiring.
Two pre-existing tests needed updating, both informative:
- `TestApplyMutation` / `TestApplySnapshot` failed with *"Unexpected
call to IsVersionFromSameCluster"* — that mock was never needed because
the nil short-circuit made the cluster check unreachable. The gap is
itself evidence the path was dead.
- `verifyActivityInfos` asserted `s.Equal(int32(TimerTaskStatusNone),
actual.TimerTaskStatus)`, encoding the bug and blocking any correct fix.
Replaced with the invariant that actually matters:
```go
s.Zero(actual.TimerTaskStatus&^originStatus, "TimerTaskStatus gained a
bit that was not set locally")
```
Applying an update may carry over or drop locally set bits, but must
never introduce one that was not already set — claiming a timer task
exists when none does is the direction that loses timers. That holds
regardless of which bits a given case preserves, so it will not need
rewriting as the mask logic evolves.
## Related, tracked separately
While auditing whether the mask can be trusted, one path was found that
moves a deadline without invalidating the mask: unpausing an activity.
Paused activities are excluded from the timer sequence, so a timeout
task firing during a long pause is dropped as an invalid task, yet
`unpauseActivityInfo` leaves the bit set. That is pre-existing and
active-side, independent of this PR, and is being tracked as its own
change.
It is worth noting here because this PR removes the passive side's
accidental repair for that class of stale bit: today's blanket wipe
clears any stale bit on the next replicated activity update.
---
`go build ./service/...`, `go vet`, and golangci-lint (repo config,
`--new-from-rev=origin/main`) clean; `./service/history/workflow/...
./service/history/ndc/...` pass across 3 consecutive runs.
Two pre-existing flakes are skipped in those runs and unrelated to this
change (both reproduce on unmodified `main`):
`TestTaskRefresherSuite/TestRefreshSubStateMachineTasks`
(nanosecond-differing HSM deadlines, map iteration order — fails 7/12 on
main) and
`TestMutableStateSuite/*/TestApplyWorkflowExecutionOptionsUpdatedEvent_TimeSkippingConfig`
(wall-clock resolution, ~1/30).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
## What changed and why?
1. Edge case: If fast-forward completes during a workflow transaction
but when time skipping checks at close transaction this fast-forward
time is before ms.Now(), time skipping should still get disabled. Now we
don't look at time points that are in the past and it is a bug.
2. Time-skipping propagation: Always propagate the time-skipping
configuration and fast-forward state to the next run and other
executions, regardless of whether time skipping is currently active.
This ensures that read APIs (for example, Describe and PollFastForward)
continue to return meaningful information instead of nil.
- Otherwise, in an edge case where fast-forward completes in the first
run and the user polls after the second run has become the current run
of the workflow execution, the poll API would return a NotFound error
instead of a completed poll result.
- Similarly, the Describe API would return a nil configuration instead
of the original configuration that should have been propagated.
3. Trivial changes:
- simiplify parameter of `propagateTimeSkippingToNextRun`
- unify UT names of timeskipping_test.go
## What changed?
Validate the retry delay is a valid proto duration.
## Why?
Prevent malformed retry delays from poisoning Nexus completions.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Potentially users could have been sending an invalid proto duration,
unclear how exactly, and now we would fail their request.
## What changed?
1. Binds httpCaller after setting httpClient
2. Uses TransitionStarted.Possible for complete-before-start
3. Records request time for nexus operation cancel
## Why?
1. Binds httpCaller after setting httpClient
When clusterID isn't set, a non-nil httpCaller with a nil receiver gets
passed to `nexusrpc.NewHTTPClient`. Since the httpCaller is non-nil, it
will skip the check that sets the nil httpCaller to the default caller.
2. Uses TransitionStarted.Possible for complete-before-start
Matches HSM, for if a start response is lost, and a completion lands
while while the operation is in BACKING_OFF.
3. Records request time for nexus operation cancel
This just seems like it wasn't being recorded.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Captured `ExecutionState.Status` before calling `GetReleaseFn()(nil)` in
`updateWithStart.Invoke` to avoid a data race with concurrent goroutines
that may acquire the lock and modify `ExecutionState` after it is
released.
Added a regression test that confirms the race under `-race`.
Fixes#11600
## Why?
`Invoke` in `service/history/api/multioperation/api.go` releases the
workflow lock at line 196 via `workflowLease.GetReleaseFn()(nil)`, then
reads `workflowLease.GetMutableState().GetExecutionState().Status` at
line 201 — after the lock is released. Any concurrent goroutine waiting
on `Lock()` for the same workflow (e.g. a signal, terminate, or another
update) can acquire the lock and modify `ExecutionState` between lines
196 and 201, creating a data race.
This matches the pattern noted in the `Updater` struct itself:
> WARNING: any references to mutable state data *have to* be copied to
avoid data races when used outside the workflow lease.
## How did you test it?
- [x] added new unit test(s)
The test spawns a concurrent writer that modifies
`ExecutionState.Status` after the lock is released.
**Before fix:**
```
go test -race -tags test_dep -count=1 \
-run TestUpdateWithStartSuite/TestInvoke_CompletedUpdate_StatusCapturedBeforeRelease \
./service/history/api/multioperation/
WARNING: DATA RACE
Read at ... api.go:201
--- FAIL
```
**After fix:**
```
ok go.temporal.io/server/service/history/api/multioperation
```
## Potential risks
Minimal - single line moved before the release call. No API or
persistence behavior change.
## What changed?
Batch operations now target Paused executions in addition to Running
ones. The filter auto-appended by adjustQueryBatchTypeEnum changed from
ExecutionStatus='Running' to ExecutionStatus='Running' OR
ExecutionStatus='Paused', affecting all workflow batch types
(terminate/signal/cancel/update-options) and activity batch types
(unpause/update-options/reset/terminate/cancel).
## Why?
A paused execution is still non-terminal. Users issuing a batch
terminate/signal/cancel reasonably expect it to apply to paused targets,
but the old Running-only filter silently skipped them.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
- The filter now references ExecutionStatus='Paused' on every batch
operation. This is ok as it's an additive clause to the visibilty query
- This would be changing behavior for batch callers as paused activities
are now affected.
## What
- Don't send scale down signal when task is matched from backlog even if
the poll wait time is high.
- Do not send scale up signal when task queue is rate limited.
## Why
- When a task comes from DB backlog, the poll wait time reflects DB read
path latency, not excess pollers — the -1 is not appropriate. Instead we
want to apply the normal scale up check.
- Similarly, when dispatch is bottlenecked by a task queue rate limit,
scaling up pollers won't help.
## How did you test it?
Unit tests
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
1. perf improvement: no time-skipping task regen on full refresh
2. new functional test: claim Nexus HSM timers are part of in-fight
nexus operation, and won’t be skipped, add functional test
3. new functional test: add a functional test to verify time skipping
won't change retention time
4. trivial bug fix: time-skipping: task regen didn’t read
EnableWorkflowExecutionTimeoutTimer
## Why?
- correctness related No. 4 though it is an edgy case
- perf related No.1
- test coverage No.2,3
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Wrapped the frontend Nexus dispatch routes with the shared OpenTelemetry
HTTP handler.
## Why?
Nexus HTTP requests need an inbound server span to connect the caller
trace.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Removed blanket timestamp wrapping
- cron/retry/start child wf use virtual time directly
## Why?
there was a double-shifting for CaN but not for other cases
it shall be fixed unifying the clock used by all virtual time
propagating cases, and this PR chooses to unify in a way that all cases
propagate use virtual time
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What
Adds a `--schedule-id` flag to `tdbg schedule migrate status`. When set,
it looks up one specific schedule instead of the namespace-wide V1-vs-V2
counts, and reports whether that schedule is currently V1
(workflow-backed), V2 (CHASM), or caught in a migration sentinel state.
## Why
During schedule migration triage, an operator often needs to know the
status of *one specific* schedule, not aggregate counts. There was no
way to answer that without manually poking `execution describe` and
reasoning about CHASM node internals by hand.
## How it works
It always probes **both** sides — the V1 workflow ID
(`temporal-sys-scheduler:<id>`) and the V2/CHASM business ID (`<id>`) —
via the same `DescribeMutableState` RPC that `tdbg execution describe`
uses, regardless of which ID form was passed in. It deliberately does
not short-circuit based on the input's shape (e.g. its prefix) without
confirming against the server — an unconfirmed inference is
confusing/untrustworthy in a diagnostic tool.
This lets it flag both kinds of migration sentinel:
- a **CHASM-side sentinel** `Scheduler` component reserving the ID
during a V1→V2 migration
- a **V1-side `DummyWorkflow`** reserving the workflow ID during a V2→V1
rollback
Output leads with a plain-language headline describing the schedule's
current, authoritative form (written for someone with no prior knowledge
of the V1/V2 migration internals), followed by a note about any sentinel
found on the other side, a details table of both sides' raw status, and
the exact `execution describe` invocations to inspect each side further.
### Example output (V2→V1 rollback in flight)
```
Schedule "foo" is a V2 (CHASM) schedule.
Additionally, a placeholder ("sentinel") V1 workflow exists at workflow ID "temporal-sys-scheduler:foo",
reserving that ID while a V2→V1 rollback is in progress. This is expected during rollback and requires no action.
Details:
V1 (workflow-backed) [workflow ID temporal-sys-scheduler:foo]: sentinel (V2→V1 rollback placeholder)
V2 (CHASM) [business ID foo]: genuine
Inspect further:
V1: tdbg execution describe --workflow-id temporal-sys-scheduler:foo -n <namespace>
V2: tdbg execution describe --workflow-id foo --archetype scheduler.scheduler -n <namespace>
```
## Testing
- `go test ./tools/tdbg/... -run TestScheduleStatus -v` — new tests
cover: prefixed input, genuine V1, genuine V2, V1 genuine + CHASM
sentinel, V1 sentinel + genuine V2, not-found on both sides, and an
unexpected/inconsistent-state fallback. Existing aggregate-count tests
are unaffected.
- `go build ./tools/...`, `go vet ./tools/tdbg/...`
- Manually checked `tdbg schedule migrate status --help` output.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What changed?
- Adds `phase=error` to the existing `replication_lifecycle` wide event;
no new event type or table.
- Covers state-based replication (`SyncVersionedTransition`,
`VerifyVersionedTransition`, and `SyncWorkflowState`) plus standby
transfer, timer, and outbound queue failures.
- Captures sender, passive execution/apply, verification,
recovery/refetch, namespace refresh, Nack, DLQ, and history-branch
cleanup boundaries.
- Uses one shared error builder with small sender, executable-task, NDC,
and standby-queue adapters.
- Records workflow identity, source task identity, target context,
operation, error, attempt/priority, disposition/recovery, and extensible
diagnostics in `details`.
- Identifies apply provenance as `apply_artifact_source=task_payload` or
`sync_state_refetch`.
- Remains gated by `history.emitReplicationLifecycleEvents` (default
off).
## Why?
Replication failures span the sender, passive executor, recovery loop,
and apply layer. Recording these boundaries in the existing lifecycle
event makes the path of a workflow or replication task directly
traceable without adding another event schema.
## Example traces
The examples below are abridged records captured from the two-cluster
XDC test. Events for the same task correlate on `source_cluster`,
`source_shard`, and `source_task_id`; workflow identity is present on
every record.
A state task that fails on the passive cluster and is written to the
DLQ:
```json
{"phase":"sent","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"priority":"High"}
{"phase":"error","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"details":{"operation":"passive_task_execution","error":"failed to apply replication task","error_type":"serviceerror.InvalidArgument","apply_artifact_source":"task_payload","attempt":1,"priority":"High","target_cluster":"standby"}}
{"phase":"error","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"details":{"operation":"task_execution","error":"failed to apply replication task","terminal":true,"priority":"High","target_cluster":"standby"}}
{"phase":"error","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"details":{"operation":"dlq_write","disposition":"dlq","terminal":true,"priority":"High","target_cluster":"standby","target_shard":1}}
```
A verification task that detects missing state, refetches it, and then
verifies successfully:
```json
{"phase":"sent","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"priority":"High"}
{"phase":"executing","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"attempt":1}
{"phase":"applied","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"outcome":"resend_needed"}
{"phase":"error","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"details":{"operation":"standby_verification","error":"missing mutable state, resend","error_type":"serviceerror.SyncState","recovery_action":"sync_state","priority":"High","target_cluster":"standby"}}
{"phase":"applied","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"outcome":"applied","details":{"apply_artifact_source":"sync_state_refetch"}}
{"phase":"executing","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"attempt":1}
{"phase":"applied","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"outcome":"verified"}
```
## How was it tested?
- `go test -tags test_dep ./common/wideevents
./service/history/replication ./service/history/ndc ./service/history
./tests/testcore`
- Changed-lines `golangci-lint`: 0 issues.
- Temporary, uncommitted two-cluster XDC tests forced passive task
failures, DLQ handling, standby verification, and SyncState
refetch/recovery with lifecycle events both enabled and disabled.
- A tiered-processing run confirmed concrete `High` priority on sent and
error records.
## Risks
- Enabling the dynamic config increases event volume; retries and
recovery can produce several error phases for one source task.
- `details.operation`, `details.disposition`, and
`details.recovery_action` distinguish those boundaries.
- Emission is best effort and does not change replication error
propagation, retry, or recovery behavior.
## What changed?
Instrumented local frontend, CHASM callback, and external Nexus
operation HTTP clients with the shared OpenTelemetry transport.
Legacy HSM callbacks and cross-cluster forwarding are intentionally out
of scope.
## Why?
Outbound Nexus HTTP calls need to carry trace context so callbacks and
internal frontend calls remain connected to their originating spans.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## Motivation
`TimerSequenceID` (~48 bytes) is heap-allocated and returned as a
`*TimerSequenceID` from 5 getter functions: `getUserTimerTimeout`,
`getActivityScheduleToStartTimeout`,
`getActivityScheduleToCloseTimeout`, `getActivityStartToCloseTimeout`,
and `getActivityHeartbeatTimeout`. These are called for every pending
timer/activity during `LoadAndSortUserTimers()` and
`LoadAndSortActivityTimers()` — a hot path in every workflow task.
Callers were already dereferencing the pointer before appending to the
value-type `[]TimerSequenceID` slice, so the existing code was
allocating on the heap only to immediately copy to the stack.
## Changes
- `TimerSequenceID` is now returned by value with a `(TimerSequenceID,
bool)` tuple
- Idiomatic Go pattern (same as map access) replaces nil-check sentinel
- All 5 getter methods updated + call sites and tests adapted
## Impact
Eliminates one heap allocation per getter call on the timer-sorting hot
path.
## Tests
- `service/history/workflow` (1075 tests): ✅ passed
---------
Co-authored-by: Prathyush PV <prathyush.pv@temporal.io>
Add nil check for `request.History` before accessing
`request.History.Data` in `serializeAppendRawHistoryNodesRequest` to
prevent a panic when the caller passes a nil DataBlob.
Co-authored-by: Prathyush PV <prathyush.pv@temporal.io>
# What changed?
`tdbg` now registers the **archival** task category (`5`) in its
task-category registry (`tools/tdbg/app.go`), so `tdbg dlq
list/merge/purge --dlq-type 5` works. Previously `merge`/`purge`
rejected it with `unknown dlq category 5`. Because the `--dlq-type` help
text is generated from the registry, archival now appears there too.
Adds a regression test to `TestDLQCommand_V2`.
## Why?
`tdbg` built its registry with `tasks.NewDefaultTaskCategoryRegistry()`,
which omits `CategoryArchival`. The server registers archival
conditionally in `TaskCategoryRegistryProvider` (`temporal/fx.go`) when
archival is enabled, but `tdbg` is a client tool with no access to the
cluster's archival config, so it rejected `--dlq-type 5` client-side —
even though archival tasks are written to the history task DLQ and the
server's `AdminService.MergeDLQTasks` accepts the category. That left
archival DLQ tasks unrecoverable via the standard tool (operators had to
call the RPC directly). Registering the category unconditionally in
`tdbg` is safe: with archival disabled there is simply no archival DLQ
to operate on.
Fixes#11586.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Signed-off-by: Tihomir Surdilovic <tihomir@temporal.io>
Co-authored-by: Prathyush PV <prathyush.pv@temporal.io>
## What changed?
Adds structured `namespace_lifecycle` wide events for namespace
mutations.
### Namespace mutations
- `namespace_registered` is emitted after successful registration.
- `namespace_updated` is emitted after successful namespace updates and
includes full `before` / `after` snapshots, requested values, and
`requested_fields` identifying what the caller explicitly set.
- Local-to-global promotion is distinguished by `is_promotion` and
`promote_namespace_requested`.
- Active-cluster failover is distinguished by `is_failover` and the
active-cluster transition in `before` / `after`.
- Deprecation is represented by the `Registered` to `Deprecated` state
transition.
- Workflow-rule creation and deletion include the affected rule ID and
detail, plus force-scan and request-ID data when supplied.
- `namespace_renamed` is emitted by the delete-namespace worker when the
namespace is renamed to its tombstone name. This is the observable
deletion point because namespace deletion is local and is not replicated
as a namespace operation.
The snapshots cover namespace info, configuration, archival settings,
replication topology/state, failover versions/history, custom
search-attribute aliases, bad binaries, and workflow-rule IDs. Request
security tokens are not captured.
### Dynamic configuration
All namespace lifecycle emission is gated by the new global
dynamic-config setting:
```yaml
system.emitNamespaceLifecycleEvents:
- value: true
```
The setting defaults to `false` and is evaluated dynamically by
frontend, history, and worker producers. In addition to the new
namespace mutation events, the gate covers the existing handover-related
namespace lifecycle events:
- `shard_handover_watermark_set`
- `shard_handover_watermark_removed`
- `shard_handover_incomplete`
This PR does not introduce or change those handover event payloads; it
only makes their emission follow the same namespace lifecycle flag.
## Why?
Existing RPC metrics and logs do not provide a structured, field-level
record of namespace control-plane changes. These events provide an
attributable and queryable view of what changed, including promotion
versus failover, requested versus persisted values, rule mutations, and
delete-pipeline renames. The shared gate lets operators enable the
complete namespace lifecycle signal consistently across services.
## How did you test?
- [x] Unit tests with `-tags test_dep` for frontend emission and
disabled gating, history handover gating, delete-namespace rename
emission, migration incomplete-handover gating, dynamic config, and
common wide-event payloads.
- [x] `make lint-code` (`0 issues`).
- [x] Full local two-cluster E2E using
`config/development-cluster-a.yaml` and
`config/development-cluster-b.yaml` with a JSON event logger.
- With the flag enabled, validated register, ordinary update,
workflow-rule create/delete, deprecate, delete rename, promotion,
cluster-list update, and failover from cluster A to B.
- Also verified that the existing handover producers remain functional
when enabled: all 16 shard watermark additions and removals on cluster B
and a forced 32-shard incomplete-handover event on cluster A.
- With the flag disabled, repeated all producer paths and confirmed zero
emitted bytes. The failure-only incomplete-handover path was rerun after
disabling the flag and left both event-log counts unchanged.
- Confirmed live dynamic-config enable/disable behavior without
restarting either cluster.
### Abridged failover event
```json
{
"event_name": "namespace_lifecycle",
"phase": "namespace_updated",
"details": {
"before": {
"active_cluster": "cluster-a",
"failover_version": 1
},
"after": {
"active_cluster": "cluster-b",
"failover_version": 2
},
"requested": {
"active_cluster": "cluster-b"
},
"requested_fields": ["active_cluster"],
"is_failover": true,
"is_promotion": false
}
}
```
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
To editorialize, the presence and difficulty of spotting these bugs
makes me want to refactor this code more, but I'm going to defer that
for now. Anyway, checked the V1 code's equivalent and also checked that
the integration-test catches the problem before applying fix.
## LLM Summary
- include an existing deferred `BUFFER_ONE` start in overlap resolution
- keep the earliest buffered occurrence and reject later arrivals
- retain catchup-window expiry for an already-deferred occurrence
## Problem
`BUFFER_ONE` permits at most one pending occurrence while an action
workflow is running. CHASM represents buffered-start lifecycle with
`Attempt`:
- `0`: newly enqueued and not processed;
- `-1`: processed but deferred while another workflow is running;
- `1+`: executing or retrying.
Before this change, `InvokerProcessBufferTask` passed only `Attempt ==
0` starts to overlap resolution. This produced the following sequence:
1. A workflow is running.
2. Occurrence A becomes due under `BUFFER_ONE`.
3. A is retained and marked deferred with `Attempt == -1`.
4. Occurrence B becomes due before the workflow closes.
5. Processing ignores A and presents only B to the shared overlap
resolver.
6. The resolver sees no occupied one-element buffer and retains B as
well.
The invoker can therefore contain both A and B, violating `BUFFER_ONE`
and potentially executing an unexpected workflow.
## Fix
Include a deferred start in the pending overlap set when its effective
overlap policy resolves to `BUFFER_ONE`:
```go
return start.Attempt == 0 ||
(start.Attempt == -1 &&
scheduler.resolveOverlapPolicy(start.GetOverlapPolicy()) ==
enumspb.SCHEDULE_OVERLAP_POLICY_BUFFER_ONE)
```
This lets the existing shared V1 `ProcessBuffer` logic see the occupied
buffer, retain the earliest occurrence, and reject later arrivals. The
effective policy is resolved because a buffered start may store
`UNSPECIFIED` and inherit the schedule current policy.
The special handling is limited to `BUFFER_ONE`; other deferred starts
should not be broadly reprocessed merely because a new occurrence
arrived.
## How this was found
The Schedule V1-to-CHASM production-history replay initially reported
5,355 V1 actions versus 4,961 CHASM actions, suggesting CHASM had
skipped 394 workflows.
Tracing the first independently different action showed the opposite:
CHASM emitted one extra occurrence. Because that workflow did not exist
in V1 history, the replay had no completion event to apply to it. It
remained running in simulated CHASM state and blocked hundreds of later
starts. The apparent 394-action deficit was a downstream alternate-chain
cascade, not 394 independent defects.
Inspecting state at the first difference revealed one running workflow,
one deferred `BUFFER_ONE` start (`Attempt == -1`), one new start
(`Attempt == 0`), and only the new start participating in overlap
resolution. A focused unit test reproduced that exact state.
After the fix, the representative converged to 5,355 actions on both
sides with identical workflow identities; only observation-time
differences remained.
## Impact
The direct product impact is one additional retained and potentially
executed workflow. This is high severity because `BUFFER_ONE` explicitly
bounds pending work, and an unexpected workflow may perform externally
visible or non-idempotent actions.
## Testing
- `TestProcessBufferTask_BufferOneKeepsExistingDeferredStart` verifies
that the first deferred occurrence occupies the buffer and the later
occurrence is rejected.
- `TestProcessBufferTask_BufferOneDropsDeferredStartPastCatchupWindow`
verifies that a deferred occurrence is still dropped when its catchup
deadline has expired.
```sh
go test -tags test_dep ./chasm/lib/scheduler \
-run "^TestProcessBufferTask_BufferOne" -count=1
```
## Summary
- `computeFutureActionTimes` computed `count = min(RemainingActions,
recentActionCount)` and passed it directly to `make()` as a slice
capacity. A negative `RemainingActions` (e.g. corrupted/invalid
persisted or operator-supplied schedule state) produced a negative
capacity and panicked with `makeslice: cap out of range` instead of
degrading gracefully.
- Clamp `count` to zero so invalid negative action counts no longer
panic scheduler reads or tasks.
- Added a shared `newTestEngineContext` helper for
CHASM-test-engine-backed scheduler tests and reused it in the existing
tests that were each hand-rolling the same registry/engine setup.
## What changed?
Fixed bookkeeping of mutable state approximate size on the two paths
that mutate an `ActivityInfo` without adding its bytes to the counter:
`AddActivityTaskStartedEvent` (start), and the heartbeat handler, where
`RetryLastWorkerIdentity` now moves into `UpdateActivityProgress`.
## Why?
Both sites mutate the pointer `GetActivityInfo` returned, but don't
alter the approximate size.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
The risk should be low. We will hit the mutable state size limit faster,
but it would be proper accounting.
## What changed?
Validate links on callbacks consistently.
## Why?
Some links on callbacks for some requests are not being validated
properly
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
`SetReaderWatermark` takes whether the slice a batch came from still has
tasks to load, and only counts the read toward the reader stuck attempt
total when it does.
## Why?
A read that drained its slice made progress, but the counter incremented
on those too. A shard that keeps generating tasks creates a new slice
per notification and several can cover one fire time second, so a reader
draining each of them in a single read still looked stuck.
Counting only reads that left tasks behind measures the tasks wedged in
the window instead, which is what blocks everything ordered after them.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
## What changed?
- Shows Bayesian commit suspects directly below the overall stats.
- Separates synthetic test-runner timeouts from test failures and
reports affected artifacts.
- Shows final-retry test failures as affected-artifact counts rather
than workflow-run rates.
## Why?
The synthetic `testrunner.TotalTimeout` event was reported both as a
flaky test and as a CI breaker, while the former CI-breaker percentage
mixed artifact and workflow-run units.
## How did you test it?
- `go test -tags test_dep ./tools/flakereport ./tools/common/github`
- `make lint-code`
## Potential risks
This changes report categorization and presentation only; raw
`failures.json` retains synthetic timeout records as timeout events.
The "overdue scan hit per-namespace check cap" warning previously only
logged the namespace and the cap value, giving no way to tell which
schedule the scan stopped at or how much progress it had made before
hitting the cap. Add the schedule ID it stopped on, how many schedules
were checked this pass, and how many anomalies had already been found,
so the log line is actionable on its own.
## What changed?
adds some logs
## Why?
Trying to debug some odd results
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
very low, feature is turned off by default, log line only
