## What changed?
`handleStartOperationError` (`components/nexusoperations/executors.go`), `callErrorToFailure`, and `newInvocationResult` (`chasm/lib/nexusoperation/task_handler_helpers.go`) all unwrap `callErr` into a typed `serviceerror.ServiceError` via `errors.As`/`errors.AsType`, then pass the *original wrapped* `callErr` — not the unwrapped `serviceErr` — to `common.IsRetryableRPCError`.
Fix: pass the already-unwrapped `serviceErr` to `IsRetryableRPCError` at all three call sites — in both the workflow-based (`components/nexusoperations`) and CHASM standalone (`chasm/lib/nexusoperation`) Nexus operation state machines. These were the only call sites of `IsRetryableRPCError` outside its own definition/tests.
## Why?
`IsRetryableRPCError` only recognizes a service error via a direct (non-unwrapping) type assertion or a gRPC status; neither sees through wrapping. So whenever the transport wraps the service error (e.g. `net/http.Client.Do` wraps `RoundTripper` errors in `*url.Error`), a transient error like `Unavailable` is always misclassified as non-retryable, permanently failing the Nexus operation instead of retrying it.
Observed in practice: a Nexus `cancel` operation dispatched during a brief window where the internal service resolver had zero available frontend members failed with `Unavailable: no frontend host to route request to`. That error was wrapped by the HTTP round-tripper, misclassified as non-retryable, and permanently failed the operation — with ~19 of its 20-minute `scheduleToCloseTimeout` still unused, while sibling operations dispatched moments later succeeded normally. A single retry would have resolved it.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Low risk: the change only affects the retryability decision for the `serviceerror` branch of Nexus start-operation error handling, and only for errors that were previously misclassified (wrapped service errors). Correctly-classified cases (direct/unwrapped service errors, gRPC status errors) are unaffected since `serviceErr` and `callErr` resolve to the same classification for those.
## What changed?
Capture panic in `ServiceErrorInterceptor`
## Why?
`ServiceErrorInterceptor` is a top-level interceptor for frontend,
history and matching. Capturing panics to make sure unhandled panics
won't crash the service.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed
Kept dynamic data out of Nexus log messages and moved it to tags. Added
a review guideline requiring static logger messages and structured tags
for all dynamic content.
## Why
Ensures that Nexus logs are aggregatable.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
Added wide events for the child workflow resend lifecycle, including:
- scheduled
- started
- succeeded
- source not found
- failed
- deduplicated
- limited by the host-level concurrency cap
## Why?
Child workflow resends run asynchronously, making missing-child
replication issues difficult to diagnose from request logs alone.
These events make it possible to trace scheduler admission, remote state
synchronization, verification, and terminal outcomes while keeping
parent and child resend events consistent.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
## What changed?
- Added an opt-in recovery path for a child workflow missing from a
standby cluster.
- After the existing resend delay, VerifyFirstWorkflowTaskScheduled
asynchronously fetches the child state from the active cluster, applies
it locally, and verifies it again.
- Added deduplication, per-shard concurrency limits, metrics, namespace
checks, and transition-history gating.
Corrected the discard-time source check to verify the child workflow
rather than the parent.
- Moved the reusable in-flight resend tracker into the shared
workflowresend package.
- Updated the existing XDC parent-child test to assert that the missing
child and its first workflow task are restored.
## Why?
Cross-shard replication may deliver the parent’s
ChildWorkflowExecutionStarted event before the child workflow reaches
the standby cluster. Previously, verification repeatedly returned
NotFound and eventually discarded the standby task, leaving the child
missing.
This adds the child-side symmetric recovery behavior to the parent
resend implemented in #11424 .
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
### Rollout
`history.enableChildWorkflowResend` default to `false`.
### Known
- Enabling this feature introduces additional cross-cluster state-sync
traffic. It is disabled by default and protected by deduplication and a
per-shard concurrency limit.
- Regular replication may race with state sync; duplicate application is
treated as success.
- The resend delay and replication timeout should remain below the
standby task discard delay so recovery has time to complete.
## What changed?
Capture any panics in Visibility query converter.
## Why?
Visibility query converter is complex, and at times might make
assumptions that might not hold (due to bugs in the store query
converter implementation for example). Capturing at top level, and
returning an error instead.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed
- Add `worker_deployment_name` and `worker_build_id` labels to Workflow
Task completion and failure metrics.
- Add the split labels to Activity success, failure, cancellation,
timeout, and completion-latency metrics.
- Add the split labels to Workflow Task and Activity schedule-to-start
latency and poll-time task-dispatch latency.
- Continue emitting the existing combined `worker_version` label on
task-dispatch latency for compatibility.
- Reuse `metrics.breakdownByBuildID` as the task-queue-scoped
cardinality gate for the new labels.
## Which metrics changed
Every metric below now emits both `worker_deployment_name` and
`worker_build_id` when `metrics.breakdownByBuildID` is enabled and
deployment attribution is available. When the gate is disabled or
attribution is unavailable—including unversioned tasks and timeouts
before a worker starts—both labels remain present with empty values.
### Workflow Task outcomes
- `workflow_tasks_completed`
- `failed_workflow_tasks`
### Activity outcomes
- `activity_success`
- `activity_fail`
- `activity_task_fail`
- `activity_cancel`
- `activity_timeout`
- `activity_task_timeout`
### Activity latency
- `activity_end_to_end_latency` (deprecated; use
`activity_start_to_close_latency` instead)
- `activity_start_to_close_latency`
- `activity_schedule_to_close_latency`
### Task-routing latency
- `task_schedule_to_start_latency`
- `task_dispatch_latency` (continues to emit the existing combined
`worker_version` label as well)
## Why
- To improve user-experience for worker-versioning by having more
insights
- Worker Deployment name and build ID need to be independently
filterable. Emitting them separately avoids requiring consumers to parse
the combined `worker_version` value. This PR preserves `worker_version`
on `task_dispatch_latency` for compatibility and does not remove or
deprecate it.
## How each metric was tested
The functional tests run real versioned Workflow and Activity workers
with `metrics.breakdownByBuildID` enabled. They assert that emitted
server metrics contain the actual Worker Deployment name and build
ID—not merely that the label keys exist.
| Functional scenario | Metrics verified | What is asserted |
|---|---|---|
| Workflow and Activity task dispatch |
`task_schedule_to_start_latency`, `task_dispatch_latency` | Both
Workflow and Activity task series contain real deployment/build values
when enabled. Disabled and unversioned cases emit empty values. |
| Successful Workflow and Activity | `workflow_tasks_completed`,
`activity_success`, `activity_end_to_end_latency`,
`activity_start_to_close_latency`, `activity_schedule_to_close_latency`
| Successful completion series contain real deployment/build values. |
| Terminal Activity failure | `activity_task_fail`, `activity_fail`,
`activity_end_to_end_latency`, `activity_start_to_close_latency`,
`activity_schedule_to_close_latency` | Both the failed attempt and
terminal-failure series retain real worker attribution. |
| Activity cancellation | `activity_cancel` | The cancellation series is
attributed to the worker that started the Activity. |
| Terminal Activity timeout | `activity_task_timeout`,
`activity_timeout` | Both attempt-level and terminal timeout series
retain the started worker’s deployment/build values. |
| Failed Workflow Task | `failed_workflow_tasks` | A real versioned
Workflow Task is polled and failed; the resulting series contains the
poller’s deployment/build values. |
### Local end-to-end validation
The PR server was also run locally with the Worker Deployment versioning
canary and a `bench-go` workload.
Prometheus was scraped directly to verify that:
- real `worker_deployment_name` and `worker_build_id` values were
emitted;
- versioned and unversioned series were both accepted without
inconsistent-label errors;
- `activity_success`, `activity_task_fail`,
`activity_end_to_end_latency`, `activity_start_to_close_latency`, and
`activity_schedule_to_close_latency` carried the expected real
deployment/build values.
The remaining failure, cancellation, timeout, and failed-Workflow-Task
paths are covered by the functional tests above.
NEW: Also tested each of these 13 metric families in a cloud test cell
with sample metrics pasted here:
https://grafana.tmprl-internal.cloud/d/shdb6gf/new-dashboard?orgId=1&from=2026-08-27T00:00:00.000Z&to=2026-08-27T23:59:59.000Z&timezone=utc
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Broad metrics surface area across History and Matching with
cardinality gated by config; deployment attribution on timeouts and
eager starts changes which tag values appear on existing metric names.
>
> **Overview**
> Adds **`worker_deployment_name`** and **`worker_build_id`** to History
and Matching metrics so worker versioning can be filtered without
parsing the combined **`worker_version`** label.
**`metrics.breakdownByBuildID`** (task-queue scoped) controls whether
those tags carry real values or empty strings;
**`task_dispatch_latency`** still emits **`worker_version`** for
compatibility.
>
> **Workflow tasks:** Completion and failure counters
(`workflow_tasks_completed`, `failed_workflow_tasks`) now go through
shared helpers that attach versioning behavior plus deployment tags from
the poller’s **`DeploymentOptions`**.
>
> **Workflow activities:** Success, failure, cancel, timeout, and
related latency metrics get the same split labels via
**`VersioningMetricContext`** on respond paths (from request deployment
options), timer-driven timeouts (from **`LastDeploymentVersion`** when
an attempt had started), and schedule-to-start latency on
**`RecordActivityTaskStarted`**. Eager activities started during WFT
completion now record the completing worker’s deployment on the started
event.
>
> **Standalone activities (CHASM):** Completion/timeout/cancel paths use
**`completionMetricsHandler`**, which always includes the deployment
label keys with empty values until standalone versioning exists—keeping
Prometheus label-set parity with workflow-embedded activities.
>
> **Other:** **`AddActivityTaskStartedEvent`** clears stored deployment
when the poller is unversioned; Matching **`task_dispatch_latency`**
adds the new tags alongside **`worker_version`**.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
1c60b05f06. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What
This fixes the Schedule invariant scanner's false-positives coming from
replication. Due to carelessness it was firing on the passive side
because I forgot to filter this out, and for a while during
post-replication disconnection, the task processing will cease. Also
adds a small check for Described schedules to filter out visibility
drift.
## How
- Adds a guard for only checking active NS
- Adds a describe check for the next fire time, so that
visibility-delayed schedules are excluded
## Risks:
- That I make a mistake and break the scanner
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
1. Object leak reports now includes sorted heap addresses for unexpected
retained objects.
2. The leak test can optionally capture a raw heap dump with
`LEAK_HEAP_DUMP=1`.
## Why?
Local debugging; it helps identify the runtime root retaining a reported
object.
## Potential risks
Raw heap diagnostics can be large and may contain sensitive process
data. They are only captured after an object leak failure when
`LEAK_HEAP_DUMP=1` is explicitly set.
**What**
Handle terminal errors from asynchronous task-queue delete propagation
without marking propagation complete, so the Version workflow stays open
until cleanup succeeds.
**Why**
Retryable activity failures retry indefinitely, but a terminal error can
escape. The current code ignores that error and allows the Version
workflow to complete without confirming task-queue cleanup.
Note: We don't know how this non retryable case can happen. We observed
1 setup in this state; so this PR exports a metric for us to observe
when it happens.
**How did you test it?**
Workflow test covering a non-retryable cleanup failure and verifying the
Version workflow remains open.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
**1. Delete execution replication tasks carry a failover version.**
Stamped with the source cluster's
failover version at generation (`shard/context_impl.go:1013`), carried
in the existing
`ReplicationTaskInfo.version` (no proto change), and skipped on apply
when older than the target's
namespace failover version (`executable_delete_execution_task.go:130`).
Still applied: unversioned
tasks (queued pre-upgrade), tasks newer than the target's namespace
entry, and deletions synthesized
by versioned-transition tasks.
**2. `DeleteWorkflowExecution` is rejected on a cluster passive for the
workflow**
(`workflow_handler.go:2513`), with the usual `NamespaceNotActive`. In
the frontend, because the history
path is shared with replication apply, which must delete on passive
clusters — that path, the
delete-namespace worker, and admin force-delete are untouched.
## Why?
An unversioned delete task generated before a failover kept being
applied afterwards, deleting a run
the new active cluster owns.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
```
temporal --address :7233 workflow -n global-ns delete -w pay-invoice-0 -r 019fcf5b-996e-7416-9a2d-4ffb0ae7fc13 --grpc-meta xdc-redirection=false
WARNING: Deleting Workflow Executions in a global Namespace removes them from all replicas. Requests sent to a passive cluster are forwarded to the active cluster by default; to target the passive cluster directly, specify `--grpc-meta xdc-redirection=false`.
Delete Workflow "pay-invoice-0" with Run ID "019fcf5b-996e-7416-9a2d-4ffb0ae7fc13"? y/N y
Error: failed to delete workflow: Namespace: global-ns is active in cluster: cluster-b, while current cluster cluster-a is a standby cluster.
```
## Potential risks
- A deletion issued just before a failover is dropped on targets;
cleanup waits for the new active
cluster's retention timer. Delayed, not leaked — intended trade-off.
- API behavior change: deleting against a passive cluster now fails,
including batch delete
(`batcher/activities.go:704`). No killswitch; admin force-delete is the
escape hatch.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed
Adds `TestLogger.StartCapture` / `StopCapture` to records log calls and
make them queryable.
## Why
We want ability to verify certain logs where emitted to verify
observability. See #11689 for first use case.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Go 1.27 prerequisite that applies the `errorsastype` Go fixer and its
required error-interface updates.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
Back `testSequentialTaskQueue` with an unbounded slice instead of a
3000-capacity channel, so `Add` no longer blocks.
## Why?
`common/tasks` deadlocked and timed out after 15m on main ([run
32773080605](https://github.com/temporalio/temporal/actions/runs/32773080605)).
`SequentialScheduler.Submit` calls `Add` from inside `PutOrDo`'s
callback, which runs under the shard write lock. Once the test queue's
channel filled, the submitter blocked in `Add` while holding that lock,
and the only worker was blocked on the same lock in `RemoveIf`. Holding
the lock across the add is deliberate — it keeps the add atomic against
`RemoveIf`'s empty-check — so the invariant is that `Add` must not
block, and only the test queue violated it.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
Adds generic `await.Rcv` and `await.Snd` helpers that bound blocking
channel operations by the test context and report closure or
cancellation clearly.
Replaces both previous helpers and existing unsafe channel rcv/snd
across tests.
## What changed?
- Moves `activity.linkValidator` into `common/links`.
- Moves `callback.Validator` into `common/callbacks`
In addition, this PR performs some minor refactorings for consistency
and clarity.
- Moved some `links.Validator`-specific tests from
`chasm/lib/activity/validator_test.go` elsewhere.
- Introduced a `callbacks.ValidatorConfig` to bundle all of the specific
settings. (Since we'll need to wire 3+ more parameters when updating the
`callbacks.Validator` to support worker callbacks.)
> The singular package names `common/link` or `common/callback` would be
more consistent. But `common/links` already existed, there are other
pluralized ones like `common/enums` or `common/headers`. And IMHO, the
plural seems a little more applicable since the validations are only on
groupings of links or callbacks.
## Why?
The `activity.linkValidator` and `callback.Validator` types are great,
but they aren't able to be used as across other CHASM components as
easily. Moreover, `callback.Validator` uses types that are exposed from
the CHASM `callback` package, it will lead to circular dependencies in
the future. (I'm hitting this now in PRs for landing worker callbacks.)
Moving the `commonpb` protobuf validation into `common/` means we can
better separate the the distinction between validation logic and the
CHASM executions that rely on it.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
This should just be a standard refactoring. There should not be any new
validation checks enabled on codepaths where they weren't already
present. (Or in test cases, we initialize fields of
`callback.ValidatorConfig` that weren't used before.)
## What changed?
Fixes a build error, seemingly introduced when multiple changes were
merged automatically after approval.
## Why?
Because a broken build stops the flow of spice. And the spice must flow.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None. But there might be other issues, I'm curious to see if there are
any other issues flagged by CI/CD.
## What changed
All three warnings in `ConvertNexusLinksToProtoLinks` interpolated the
link type into the message, and two also embedded the link URL. Moved
both into tags and kept the message static.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
ForkHistoryBranchResponse now carries BaseBranchToken, and the two reset
paths rebuild through it when the store set it. A store that doesn't
(Cassandra, SQL) leaves it nil and both call sites fall back to the
token the caller already had, so behavior is unchanged everywhere else.
## Why?
ForkHistoryBranch can modify the base branch token in ways the caller
may not have visibility into. This PR fixes it so that the changed token
is returned to the caller.
## How did you test it?
- [ ] built
- [ ] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
`GetWorkflowExecutionHistory` and `GetWorkflowExecutionHistoryReverse`
now check `branch_token` in the page token against the token in mutable
state.
## Why?
To confirm if it is still the correct branch after conflict resolution.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Adds metrics for how often and by how much queue slices
fail to narrow their predicate, and how large persisted queue state
actually is.
- `queue_slice_pending_keys` — histogram, recorded on every narrowing
attempt (declined or
succeeded). This is the distribution
`queueShrinkPredicateMaxPendingKeys` should be sized against.
- `shard_info_size` / `queue_state_size` — histograms recorded when a
shard record is actually
written, giving whole-record and per-category size.
- `queue_state_size_total` / `queue_slice_count_total` — counters paired
with the histograms above
(and with the existing `queue_slice_count`), so an exact bytes-per-slice
ratio is possible.
- `queue_slice_count` gains a `task_category` tag (previously untagged
beyond `operation`).
These are only metrics changes - no behavior changes.
## Why?
A slice only narrows its predicate below
`queueShrinkPredicateMaxPendingKeys` (10) pending
namespaces; above that it stays universal and re-reads the whole range
every time. Raising that
threshold safely requires knowing the pending-key distribution and the
persisted size.
This PR is the baseline for evaluating a follow-on encoding.
There are two counters because this server's tally-backed Prometheus
reporter doesn't preserve the
true recorded value when a histogram flushes — it replays each sample as
its bucket's upper bound,
so a histogram's `_sum` has no more precision than its buckets.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] run locally and tested manually (`queue_predicate_resolution_loss`
confirmed live against a local server under forced
narrowing-decline conditions)
## Potential risks
## What changed?
- emit remote cluster and namespace replication lifecycle records under
`namespace_lifecycle`
- retain compatibility aliases for specialized event-name constants with
TODO cleanup
- update tests and shared envelope documentation
## Why?
Update schema event names to match namespace lifecycle schema which
currently exist so they are properly interpreted.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Refactor NamespaceRateLimitInterceptor with functions to consume N
tokens:
- removed `tokens` overwrite argument as it's never used
- added functions to consume N tokens
The changes itself in this PR is no-op since it's introducing new
functions to the interface.
## Why?
Added flexibility to wrap `NamespaceRateLimitInterceptor`.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Go 1.27 prerequisite that applies the `slicesbackward` Go fixer for
reverse iteration.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What
Treat `NotFound` from the Version workflow delete update as success, so
the Deployment workflow can clean up its stale version reference.
## Why
When a Version workflow is already closed or its history is gone, the
delete update returns `NotFound`, blocking the Deployment workflow from
removing the reference. Fixes#11539.
## How did you test it?
Unit test covering the `NotFound` → success path and verifying other
History errors still propagate.
## What changed?
Added Temporal Nexus attributes to spans.
## Why?
Domain attributes make Nexus traces more useful.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Every task produced by `GenerateMigrationTasks` is now low priority end
to end:
- Added a `Priority` field to `HistoryReplicationTask`,
`SyncActivityTask` and `SyncHSMTask`.
- Round-tripped `Priority` through the replication task serializer for
those three, plus `SyncVersionedTransitionTask` (its `Priority` field
existed but was never persisted). The proto
`ReplicationTaskInfo.Priority` field already existed, so no proto
change.
- `StreamSenderImpl.getTaskPriority` now honors `Priority` on all five
replication task types, defaulting to high when unspecified.
- `GenerateMigrationTasks` stamps `TASK_PRIORITY_LOW` on the
`HistoryReplicationTask`, the sync activity tasks and the `SyncHSMTask`
it returns (`SyncWorkflowStateTask` and `SyncVersionedTransitionTask`
already set it).
## Why?
`GenerateMigrationTasks` is only reachable through force replication
(`GenerateLastHistoryReplicationTasks`, called by the
migration/force-replication workflow and by tdbg). That traffic is bulk
backfill and should not compete with live replication. Only
`SyncWorkflowStateTask` was actually being treated as low priority;
everything else fell through to `TASK_PRIORITY_HIGH` in the stream
sender, and `SyncVersionedTransitionTask`'s low priority was silently
dropped on write.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
`TestTaskGeneratorImpl_GenerateMigrationTasks` now asserts
`TASK_PRIORITY_LOW` on every returned task and every task equivalent.
`service/history/replication`, `common/persistence/serialization` and
`service/history/tasks` pass.
Note: `service/history/workflow` has a pre-existing failure in
`TestTaskRefresherSuite/TestRefreshSubStateMachineTasks` that reproduces
on unmodified `main` at 39fc2c45e and is unrelated to this change.
## Potential risks
- `SyncVersionedTransitionTask.Priority` is now persisted where it was
previously dropped. Tasks written before this change still deserialize
with `TASK_PRIORITY_UNSPECIFIED`.
- `getTaskPriority` defaults to high on `UNSPECIFIED` for the
newly-handled types, so normal (non-force) replication keeps its current
priority. Only tasks explicitly stamped low move to the low priority
stream.
- Force replication tasks now share the low priority stream and its rate
limiting with sync-state traffic, so a large force replication may
progress more slowly than before — which is the intent.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
Adds server support for WorkflowQuery-backed Nexus Operations
## Why?
Part of effort to expose all Temporal primitives as Nexus Operations
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## TODO:
- [x] https://github.com/temporalio/api/pull/842
## Summary
- capture post-transaction mutable state for fresh `NotFound` snapshots
and `IsFirstSync` creation
- report successful verify history repairs as `outcome=backfilled`
- include the repaired event range and `new_run_id` in the verify
applied event
- add regression coverage for fresh zombie applies and
non-current-branch backfills
## Testing
- `go test ./service/history/ndc ./service/history/replication
./common/wideevents -count=1`
## What changed?
Adds `testcontext.EnsureRemaining` and has `await` use it so long await
calls can request additional test-scoped context time while still
respecting the test context cap.
## Why?
Await calls can need more time than the default test context has left
(esp after the environment setup). Extending the test timeout in this
way allows for (1) stuck tests to fail earlier than the default test
timeout and (2) legitimately longer running tests to pass without
manually tweaking the test timeout.
---------
Co-authored-by: Sean Kane <sean.kane@temporal.io>
## What changed?
Adds `namespace_lifecycle` start and finish events for the namespace
handover, force replication, and catchup system workflows.
The events carry workflow identity and the core operation inputs.
Finished events classify the result as succeeded, canceled, or failed.
Force replication reports its cumulative verified workflow count and
emits only one start and one finish across a continue-as-new chain.
Emission uses one shared activity, the existing
`system.emitNamespaceLifecycleEvents` gate, disconnected cleanup for
cancellation, and workflow versioning for replay compatibility. Existing
shard handover events are unchanged.
## Why?
These system workflows currently have no consistent operation-level
event pair, which makes it difficult to correlate a namespace migration
request with its final outcome.
## How did you test it?
- [x] covered by existing tests
- [x] added new unit test(s)
`go test -tags test_dep ./common/wideevents ./service/worker/migration`
`make fmt-imports`
`make lint-code` reports no issues introduced by this change; the
repository-wide target still reports existing findings on current
`main`.
## Potential risks
The terminal event is best effort and cannot run after server-side
workflow termination or workflow run timeout because those outcomes do
not execute workflow cleanup.
### What changed
Sets gobreaker's OnStateChange hook on the outbound queue circuit
breaker pool, logging every transition.
### Why
Obtain more details for debugging curcuit breaker in production.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
- Added a separate `namespace_replication_lifecycle` wide event with
`created`, `received`, `processed`, and `dlqed` phases.
- Included namespace/task identity, source and target clusters, source
task ID, retry attempt count, deterministic task fingerprint, and the
serialized namespace replication task.
- Included the successful `CreateNamespaceRequest` or resolved
`UpdateNamespaceRequest` as `persistence_request` on `processed`;
duplicate, stale, and skipped tasks omit it.
- Passed receiver-side diagnostic metadata through a typed context so
the existing `TaskExecutor.Execute` and create/update handler signatures
remain unchanged.
- Added the dedicated, default-off
`system.emitNamespaceReplicationLifecycleEvents` dynamic-config gate,
checked explicitly at both the processor and processed-event emitter.
- Preserved the namespace replication queue message ID as
`source_task_id` when reading tasks.
## Why?
Namespace CRUD events describe user-visible namespace mutations, but do
not show whether the resulting namespace replication task was queued,
received, applied, retried, or sent to the DLQ. These events provide
that transport and processing audit trail without additional persistence
reads.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
Commands:
```text
go test -tags test_dep ./common/wideevents ./common/namespace/nsreplication ./service/worker/replicator ./service/frontend
make GOLANGCI_LINT_FIX=false GOLANGCI_LINT_BASE_REV=origin/main lint-code
```
Local two-cluster testing covered create, update, and failover after
rebasing onto `origin/main`. Each task produced linked `created ->
received -> processed` events, and `processed` contained the expected
persistence request. With the dynamic-config flag off, namespace
replication still completed and neither cluster emitted a matching
lifecycle event.
## What changed
Adds logs tags for failures on the Nexus frontend path.
## Why
Have more details to correlate issues with requests.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed
Adds a few Nexus-specific log tags to the handler-side frontend logger.
## Why
Mainly for the request ID to debug Nexus calls across namespaces better.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What
- Don't send scale down signal when task is matched from backlog even if
the poll wait time is high.
- Do not send scale up signal when task queue is rate limited.
## Why
- When a task comes from DB backlog, the poll wait time reflects DB read
path latency, not excess pollers — the -1 is not appropriate. Instead we
want to apply the normal scale up check.
- Similarly, when dispatch is bottlenecked by a task queue rate limit,
scaling up pollers won't help.
## How did you test it?
Unit tests
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
Wrapped the frontend Nexus dispatch routes with the shared OpenTelemetry
HTTP handler.
## Why?
Nexus HTTP requests need an inbound server span to connect the caller
trace.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Adds `phase=error` to the existing `replication_lifecycle` wide event;
no new event type or table.
- Covers state-based replication (`SyncVersionedTransition`,
`VerifyVersionedTransition`, and `SyncWorkflowState`) plus standby
transfer, timer, and outbound queue failures.
- Captures sender, passive execution/apply, verification,
recovery/refetch, namespace refresh, Nack, DLQ, and history-branch
cleanup boundaries.
- Uses one shared error builder with small sender, executable-task, NDC,
and standby-queue adapters.
- Records workflow identity, source task identity, target context,
operation, error, attempt/priority, disposition/recovery, and extensible
diagnostics in `details`.
- Identifies apply provenance as `apply_artifact_source=task_payload` or
`sync_state_refetch`.
- Remains gated by `history.emitReplicationLifecycleEvents` (default
off).
## Why?
Replication failures span the sender, passive executor, recovery loop,
and apply layer. Recording these boundaries in the existing lifecycle
event makes the path of a workflow or replication task directly
traceable without adding another event schema.
## Example traces
The examples below are abridged records captured from the two-cluster
XDC test. Events for the same task correlate on `source_cluster`,
`source_shard`, and `source_task_id`; workflow identity is present on
every record.
A state task that fails on the passive cluster and is written to the
DLQ:
```json
{"phase":"sent","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"priority":"High"}
{"phase":"error","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"details":{"operation":"passive_task_execution","error":"failed to apply replication task","error_type":"serviceerror.InvalidArgument","apply_artifact_source":"task_payload","attempt":1,"priority":"High","target_cluster":"standby"}}
{"phase":"error","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"details":{"operation":"task_execution","error":"failed to apply replication task","terminal":true,"priority":"High","target_cluster":"standby"}}
{"phase":"error","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"details":{"operation":"dlq_write","disposition":"dlq","terminal":true,"priority":"High","target_cluster":"standby","target_shard":1}}
```
A verification task that detects missing state, refetches it, and then
verifies successfully:
```json
{"phase":"sent","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"priority":"High"}
{"phase":"executing","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"attempt":1}
{"phase":"applied","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"outcome":"resend_needed"}
{"phase":"error","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"details":{"operation":"standby_verification","error":"missing mutable state, resend","error_type":"serviceerror.SyncState","recovery_action":"sync_state","priority":"High","target_cluster":"standby"}}
{"phase":"applied","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"outcome":"applied","details":{"apply_artifact_source":"sync_state_refetch"}}
{"phase":"executing","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"attempt":1}
{"phase":"applied","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"outcome":"verified"}
```
## How was it tested?
- `go test -tags test_dep ./common/wideevents
./service/history/replication ./service/history/ndc ./service/history
./tests/testcore`
- Changed-lines `golangci-lint`: 0 issues.
- Temporary, uncommitted two-cluster XDC tests forced passive task
failures, DLQ handling, standby verification, and SyncState
refetch/recovery with lifecycle events both enabled and disabled.
- A tiered-processing run confirmed concrete `High` priority on sent and
error records.
## Risks
- Enabling the dynamic config increases event volume; retries and
recovery can produce several error phases for one source task.
- `details.operation`, `details.disposition`, and
`details.recovery_action` distinguish those boundaries.
- Emission is best effort and does not change replication error
propagation, retry, or recovery behavior.
## What changed?
Instrumented local frontend, CHASM callback, and external Nexus
operation HTTP clients with the shared OpenTelemetry transport.
Legacy HSM callbacks and cross-cluster forwarding are intentionally out
of scope.
## Why?
Outbound Nexus HTTP calls need to carry trace context so callbacks and
internal frontend calls remain connected to their originating spans.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Add nil check for `request.History` before accessing
`request.History.Data` in `serializeAppendRawHistoryNodesRequest` to
prevent a panic when the caller passes a nil DataBlob.
Co-authored-by: Prathyush PV <prathyush.pv@temporal.io>
## What changed?
Adds structured `namespace_lifecycle` wide events for namespace
mutations.
### Namespace mutations
- `namespace_registered` is emitted after successful registration.
- `namespace_updated` is emitted after successful namespace updates and
includes full `before` / `after` snapshots, requested values, and
`requested_fields` identifying what the caller explicitly set.
- Local-to-global promotion is distinguished by `is_promotion` and
`promote_namespace_requested`.
- Active-cluster failover is distinguished by `is_failover` and the
active-cluster transition in `before` / `after`.
- Deprecation is represented by the `Registered` to `Deprecated` state
transition.
- Workflow-rule creation and deletion include the affected rule ID and
detail, plus force-scan and request-ID data when supplied.
- `namespace_renamed` is emitted by the delete-namespace worker when the
namespace is renamed to its tombstone name. This is the observable
deletion point because namespace deletion is local and is not replicated
as a namespace operation.
The snapshots cover namespace info, configuration, archival settings,
replication topology/state, failover versions/history, custom
search-attribute aliases, bad binaries, and workflow-rule IDs. Request
security tokens are not captured.
### Dynamic configuration
All namespace lifecycle emission is gated by the new global
dynamic-config setting:
```yaml
system.emitNamespaceLifecycleEvents:
- value: true
```
The setting defaults to `false` and is evaluated dynamically by
frontend, history, and worker producers. In addition to the new
namespace mutation events, the gate covers the existing handover-related
namespace lifecycle events:
- `shard_handover_watermark_set`
- `shard_handover_watermark_removed`
- `shard_handover_incomplete`
This PR does not introduce or change those handover event payloads; it
only makes their emission follow the same namespace lifecycle flag.
## Why?
Existing RPC metrics and logs do not provide a structured, field-level
record of namespace control-plane changes. These events provide an
attributable and queryable view of what changed, including promotion
versus failover, requested versus persisted values, rule mutations, and
delete-pipeline renames. The shared gate lets operators enable the
complete namespace lifecycle signal consistently across services.
## How did you test?
- [x] Unit tests with `-tags test_dep` for frontend emission and
disabled gating, history handover gating, delete-namespace rename
emission, migration incomplete-handover gating, dynamic config, and
common wide-event payloads.
- [x] `make lint-code` (`0 issues`).
- [x] Full local two-cluster E2E using
`config/development-cluster-a.yaml` and
`config/development-cluster-b.yaml` with a JSON event logger.
- With the flag enabled, validated register, ordinary update,
workflow-rule create/delete, deprecate, delete rename, promotion,
cluster-list update, and failover from cluster A to B.
- Also verified that the existing handover producers remain functional
when enabled: all 16 shard watermark additions and removals on cluster B
and a forced 32-shard incomplete-handover event on cluster A.
- With the flag disabled, repeated all producer paths and confirmed zero
emitted bytes. The failure-only incomplete-handover path was rerun after
disabling the flag and left both event-log counts unchanged.
- Confirmed live dynamic-config enable/disable behavior without
restarting either cluster.
### Abridged failover event
```json
{
"event_name": "namespace_lifecycle",
"phase": "namespace_updated",
"details": {
"before": {
"active_cluster": "cluster-a",
"failover_version": 1
},
"after": {
"active_cluster": "cluster-b",
"failover_version": 2
},
"requested": {
"active_cluster": "cluster-b"
},
"requested_fields": ["active_cluster"],
"is_failover": true,
"is_promotion": false
}
}
```
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
Validate links on callbacks consistently.
## Why?
Some links on callbacks for some requests are not being validated
properly
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
`SetReaderWatermark` takes whether the slice a batch came from still has
tasks to load, and only counts the read toward the reader stuck attempt
total when it does.
## Why?
A read that drained its slice made progress, but the counter incremented
on those too. A shard that keeps generating tasks creates a new slice
per notification and several can cover one fire time second, so a reader
draining each of them in a single read still looked stuck.
Counting only reads that left tasks behind measures the tasks wedged in
the window instead, which is what blocks everything ordered after them.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
