## What changed?
- Moves `activity.linkValidator` into `common/links`.
- Moves `callback.Validator` into `common/callbacks`
In addition, this PR performs some minor refactorings for consistency
and clarity.
- Moved some `links.Validator`-specific tests from
`chasm/lib/activity/validator_test.go` elsewhere.
- Introduced a `callbacks.ValidatorConfig` to bundle all of the specific
settings. (Since we'll need to wire 3+ more parameters when updating the
`callbacks.Validator` to support worker callbacks.)
> The singular package names `common/link` or `common/callback` would be
more consistent. But `common/links` already existed, there are other
pluralized ones like `common/enums` or `common/headers`. And IMHO, the
plural seems a little more applicable since the validations are only on
groupings of links or callbacks.
## Why?
The `activity.linkValidator` and `callback.Validator` types are great,
but they aren't able to be used as across other CHASM components as
easily. Moreover, `callback.Validator` uses types that are exposed from
the CHASM `callback` package, it will lead to circular dependencies in
the future. (I'm hitting this now in PRs for landing worker callbacks.)
Moving the `commonpb` protobuf validation into `common/` means we can
better separate the the distinction between validation logic and the
CHASM executions that rely on it.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
This should just be a standard refactoring. There should not be any new
validation checks enabled on codepaths where they weren't already
present. (Or in test cases, we initialize fields of
`callback.ValidatorConfig` that weren't used before.)
## What changed?
When `dispatchForExistingWorkflow` finds `currentRunID == ""`, it treats
the missing current execution record as a deletion issued by the user,
and handles the target run accordingly:
- **Running target:** return an internal invariant error. Replication
tasks are ordered, so a deletion cannot overtake the close event — a
running run with no current record is conceptually impossible. The task
retries and is eventually sent to the DLQ.
- **Closed target:** apply the target as a zombie with
`UpdateWorkflowModeBypassCurrent`, leaving the current record missing.
The current record is never re-established here, because that would
resurrect a user-deleted workflow.
If the task carries a new run (continue-as-new / cron / retry
successor), it is **not dropped**: when it is not already present
locally it is persisted as a zombie via bypass-current, so **no history
is lost** — but it never becomes the current run. A zombie is not a dead
end: when the successor's own close event later replicates, `ZOMBIE ->
COMPLETED` is a valid transition and it converges like any other run
(intermediate events keep it a zombie, so they never hit the
running-invariant error). While open, a zombie is invisible to
visibility, so it never shows up as a stray running workflow.
For example, given `r1 -> r2 -> r3` (continued-as-new chain) followed by
deletion of the current run `r3`, a later replication update of `r1` may
carry `r2` or `r3` as its new run. Either way the successor is persisted
as a zombie, never as current, so the user's deletion intent is
preserved while its history is retained. The current record is
(re)established only by a separately replicated new/reset run through
its own new-workflow path.
## Why?
Cross-cluster deletion removes the passive cluster's current execution
record while closed run rows can survive until retention.
Re-establishing a current record from a later replication task would
resurrect a deliberately deleted workflow, so the passive cluster
preserves the deletion intent: closed historical runs converge as
zombies, and only a
separately replicated new/reset run may establish a new current record.
Carried successors are still persisted (as zombies) so their history is
never lost, and they close normally once their own close events
replicate.
## How did you test it?
- [x] built
- [x] run locally and tested manually — reproduced the original
duplicate-run failure with
`r1 -> r2 -> r3`, deleted `r3` on both clusters, then reset `r1`
- [x] covered by existing tests
- [x] added unit tests — missing-current dispatch for the closed,
rebuilt, and impossible
running cases, plus a carried new run in both the persist-if-absent and
skip-if-already-present paths
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What changed?
The `commonpb.Callback`-variant of `commonpb.Callback_Internal` is
unused and should be removed entirely. This PR removes the remaining
unnecessary instances of that type.
(My actual motivation is really to avoid a larger diff later, since
introducing Worker-variant callbacks will start returning errors when
you try to attach an `Internal`-variant callback.)
However, changing `Callback_Internal` to `Callback_Nexus` changed the
behavior of `TestDedupLinksFromCallbacks`. After scratching my head for
a while, I add a doc comment to clarify exactly what the function does,
and then updated the tests to be easier to read and understand.
## Why?
The call to `dedupLinksFromCallbacks(...)` in the testcase did _not_
dedupe the links attached to `callbacks[0]` because it was the
`commonpb.Callback_Internal` variant. (Relying on a quirk of the
function only filtering callbacks from Nexus-variant callbacks.)
I kept that behavior in, but added a couple more test scenarios to
provide better coverage and clarify the expected behavior.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None.
## What changed?
ForkHistoryBranchResponse now carries BaseBranchToken, and the two reset
paths rebuild through it when the store set it. A store that doesn't
(Cassandra, SQL) leaves it nil and both call sites fall back to the
token the caller already had, so behavior is unchanged everywhere else.
## Why?
ForkHistoryBranch can modify the base branch token in ways the caller
may not have visibility into. This PR fixes it so that the changed token
is returned to the caller.
## How did you test it?
- [ ] built
- [ ] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
`GetWorkflowExecutionHistory` and `GetWorkflowExecutionHistoryReverse`
now check `branch_token` in the page token against the token in mutable
state.
## Why?
To confirm if it is still the correct branch after conflict resolution.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Adds metrics for how often and by how much queue slices
fail to narrow their predicate, and how large persisted queue state
actually is.
- `queue_slice_pending_keys` — histogram, recorded on every narrowing
attempt (declined or
succeeded). This is the distribution
`queueShrinkPredicateMaxPendingKeys` should be sized against.
- `shard_info_size` / `queue_state_size` — histograms recorded when a
shard record is actually
written, giving whole-record and per-category size.
- `queue_state_size_total` / `queue_slice_count_total` — counters paired
with the histograms above
(and with the existing `queue_slice_count`), so an exact bytes-per-slice
ratio is possible.
- `queue_slice_count` gains a `task_category` tag (previously untagged
beyond `operation`).
These are only metrics changes - no behavior changes.
## Why?
A slice only narrows its predicate below
`queueShrinkPredicateMaxPendingKeys` (10) pending
namespaces; above that it stays universal and re-reads the whole range
every time. Raising that
threshold safely requires knowing the pending-key distribution and the
persisted size.
This PR is the baseline for evaluating a follow-on encoding.
There are two counters because this server's tally-backed Prometheus
reporter doesn't preserve the
true recorded value when a histogram flushes — it replays each sample as
its bucket's upper bound,
so a histogram's `_sum` has no more precision than its buckets.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] run locally and tested manually (`queue_predicate_resolution_loss`
confirmed live against a local server under forced
narrowing-decline conditions)
## Potential risks
## What changed?
- emit remote cluster and namespace replication lifecycle records under
`namespace_lifecycle`
- retain compatibility aliases for specialized event-name constants with
TODO cleanup
- update tests and shared envelope documentation
## Why?
Update schema event names to match namespace lifecycle schema which
currently exist so they are properly interpreted.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Refactor NamespaceRateLimitInterceptor with functions to consume N
tokens:
- removed `tokens` overwrite argument as it's never used
- added functions to consume N tokens
The changes itself in this PR is no-op since it's introducing new
functions to the interface.
## Why?
Added flexibility to wrap `NamespaceRateLimitInterceptor`.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed
The completion (callback) handler's request-scoped logger carried only
the namespace, even though a richer one was built just above it.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Go 1.27 prerequisite that applies the `slicesbackward` Go fixer for
reverse iteration.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What
Treat `NotFound` from the Version workflow delete update as success, so
the Deployment workflow can clean up its stale version reference.
## Why
When a Version workflow is already closed or its history is gone, the
delete update returns `NotFound`, blocking the Deployment workflow from
removing the reference. Fixes#11539.
## How did you test it?
Unit test covering the `NotFound` → success path and verifying other
History errors still propagate.
## What changed?
Added Temporal Nexus attributes to spans.
## Why?
Domain attributes make Nexus traces more useful.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Every task produced by `GenerateMigrationTasks` is now low priority end
to end:
- Added a `Priority` field to `HistoryReplicationTask`,
`SyncActivityTask` and `SyncHSMTask`.
- Round-tripped `Priority` through the replication task serializer for
those three, plus `SyncVersionedTransitionTask` (its `Priority` field
existed but was never persisted). The proto
`ReplicationTaskInfo.Priority` field already existed, so no proto
change.
- `StreamSenderImpl.getTaskPriority` now honors `Priority` on all five
replication task types, defaulting to high when unspecified.
- `GenerateMigrationTasks` stamps `TASK_PRIORITY_LOW` on the
`HistoryReplicationTask`, the sync activity tasks and the `SyncHSMTask`
it returns (`SyncWorkflowStateTask` and `SyncVersionedTransitionTask`
already set it).
## Why?
`GenerateMigrationTasks` is only reachable through force replication
(`GenerateLastHistoryReplicationTasks`, called by the
migration/force-replication workflow and by tdbg). That traffic is bulk
backfill and should not compete with live replication. Only
`SyncWorkflowStateTask` was actually being treated as low priority;
everything else fell through to `TASK_PRIORITY_HIGH` in the stream
sender, and `SyncVersionedTransitionTask`'s low priority was silently
dropped on write.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
`TestTaskGeneratorImpl_GenerateMigrationTasks` now asserts
`TASK_PRIORITY_LOW` on every returned task and every task equivalent.
`service/history/replication`, `common/persistence/serialization` and
`service/history/tasks` pass.
Note: `service/history/workflow` has a pre-existing failure in
`TestTaskRefresherSuite/TestRefreshSubStateMachineTasks` that reproduces
on unmodified `main` at 39fc2c45e and is unrelated to this change.
## Potential risks
- `SyncVersionedTransitionTask.Priority` is now persisted where it was
previously dropped. Tasks written before this change still deserialize
with `TASK_PRIORITY_UNSPECIFIED`.
- `getTaskPriority` defaults to high on `UNSPECIFIED` for the
newly-handled types, so normal (non-force) replication keeps its current
priority. Only tasks explicitly stamped low move to the low priority
stream.
- Force replication tasks now share the low priority stream and its rate
limiting with sync-state traffic, so a large force replication may
progress more slowly than before — which is the intent.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
Adds server support for WorkflowQuery-backed Nexus Operations
## Why?
Part of effort to expose all Temporal primitives as Nexus Operations
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## TODO:
- [x] https://github.com/temporalio/api/pull/842
## Summary
- capture post-transaction mutable state for fresh `NotFound` snapshots
and `IsFirstSync` creation
- report successful verify history repairs as `outcome=backfilled`
- include the repaired event range and `new_run_id` in the verify
applied event
- add regression coverage for fresh zombie applies and
non-current-branch backfills
## Testing
- `go test ./service/history/ndc ./service/history/replication
./common/wideevents -count=1`
## What changed?
- Clear an activity’s timer-task status when it is unpaused so timeout
tasks are regenerated.
- Make ResetActivity with keepPaused=false fully unpause both scheduled
and running activities, including clearing pause metadata.
- Add and strengthen unit and functional coverage for unpause,
reset-unpause, timer regeneration, and keepPaused=true.
## Why?
Timeout tasks can fire while an activity is paused and be discarded.
Previously, the activity’s timer-task status still indicated that those
tasks existed, preventing them from being recreated after unpause and
potentially making the timeout ineffective.
ResetActivity also bypassed normal unpause handling, and running
activities returned early without clearing their paused state.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Unpausing now invalidates existing timeout tasks and recreates the next
applicable timer during transaction close. This is correct, but a
behavioral change. Stale queued tasks may still be processed and
discarded through the existing stamp validation.
## What changed?
Adds `namespace_lifecycle` start and finish events for the namespace
handover, force replication, and catchup system workflows.
The events carry workflow identity and the core operation inputs.
Finished events classify the result as succeeded, canceled, or failed.
Force replication reports its cumulative verified workflow count and
emits only one start and one finish across a continue-as-new chain.
Emission uses one shared activity, the existing
`system.emitNamespaceLifecycleEvents` gate, disconnected cleanup for
cancellation, and workflow versioning for replay compatibility. Existing
shard handover events are unchanged.
## Why?
These system workflows currently have no consistent operation-level
event pair, which makes it difficult to correlate a namespace migration
request with its final outcome.
## How did you test it?
- [x] covered by existing tests
- [x] added new unit test(s)
`go test -tags test_dep ./common/wideevents ./service/worker/migration`
`make fmt-imports`
`make lint-code` reports no issues introduced by this change; the
repository-wide target still reports existing findings on current
`main`.
## Potential risks
The terminal event is best effort and cannot run after server-side
workflow termination or workflow run timeout because those outcomes do
not execute workflow cleanup.
## What changed?
Makes Standalone Activity conflict updates idempotent by recording the
`requestID` when attaching callbacks or links and recognizing duplicate
request IDs. I needed to add a dedicated CHASM error for
## Why?
This prevents a successful attachment whose response was lost from
failing on retry or duplicating/replacing callbacks and links.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
A blocked `ChasmEngine.pollComponent` now returns a
`ShardOwnershipLostError` as soon as its shard moves off this host — by
adding a `select` case on the shard's lifecycle context — instead of
blocking until the request context deadline.
## Why?
Follow-up to #10860 (requested in review): `pollComponent` had the same
gap as the `GetWorkflowExecutionHistory` long poll — nothing in its
`select` was tied to shard lifecycle, so a poll in flight when its shard
moved stalled until timeout. This lets the caller redirect to the new
owner immediately.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
### What changed
Sets gobreaker's OnStateChange hook on the outbound queue circuit
breaker pool, logging every transition.
### Why
Obtain more details for debugging curcuit breaker in production.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
- Added a separate `namespace_replication_lifecycle` wide event with
`created`, `received`, `processed`, and `dlqed` phases.
- Included namespace/task identity, source and target clusters, source
task ID, retry attempt count, deterministic task fingerprint, and the
serialized namespace replication task.
- Included the successful `CreateNamespaceRequest` or resolved
`UpdateNamespaceRequest` as `persistence_request` on `processed`;
duplicate, stale, and skipped tasks omit it.
- Passed receiver-side diagnostic metadata through a typed context so
the existing `TaskExecutor.Execute` and create/update handler signatures
remain unchanged.
- Added the dedicated, default-off
`system.emitNamespaceReplicationLifecycleEvents` dynamic-config gate,
checked explicitly at both the processor and processed-event emitter.
- Preserved the namespace replication queue message ID as
`source_task_id` when reading tasks.
## Why?
Namespace CRUD events describe user-visible namespace mutations, but do
not show whether the resulting namespace replication task was queued,
received, applied, retried, or sent to the DLQ. These events provide
that transport and processing audit trail without additional persistence
reads.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
Commands:
```text
go test -tags test_dep ./common/wideevents ./common/namespace/nsreplication ./service/worker/replicator ./service/frontend
make GOLANGCI_LINT_FIX=false GOLANGCI_LINT_BASE_REV=origin/main lint-code
```
Local two-cluster testing covered create, update, and failover after
rebasing onto `origin/main`. Each task produced linked `created ->
received -> processed` events, and `processed` contained the expected
persistence request. With the dynamic-config flag off, namespace
replication still completed and neither cluster emitted a matching
lifecycle event.
## What changed
Adds logs tags for failures on the Nexus frontend path.
## Why
Have more details to correlate issues with requests.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed
Adds a few Nexus-specific log tags to the handler-side frontend logger.
## Why
Mainly for the request ID to debug Nexus calls across namespaces better.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## The bug
`applyUpdatesToSubStateMachine` shadowed the variable it meant to
populate:
```go
var existing V // outer: never assigned
if existing, ok := pendingInfos[key]; ok { // := declares a NEW existing, scoped to the if
...
ms.approximateSize -= existing.Size() + getSizeOfKey(key) // inner one — correct here
}
val := updated
if sanitizeFn != nil {
val = common.CloneProto(updated)
sanitizeFn(existing, val) // outer one — always nil
}
```
`:=` only requires *one* new variable on the left (`ok`), and the `if`
init statement is its own scope, so `existing` is redeclared there
rather than assigned. It compiles because both variables are used, and
`go vet` does not check shadowing by default.
`sanitizeFn` has therefore always received `current == nil`.
## What this activates
Two of the five `sanitizeFn` implementations depend on `current`; the
other three (timers, request cancels, signals) ignore it or are `nil`,
so they are unaffected.
**Activities.** `getActivityTimerTaskStatus` short-circuits to
`TimerTaskStatusNone` when `current` is nil, so every applied activity
update cleared the *entire* timer task mask rather than only the bits
whose deadline moved. The deadline comparison added in #11565 has never
taken effect — that PR is currently a no-op.
**Child executions.** `if current != nil { incoming.Clock =
current.Clock }` never ran. `Clock` is a local shard vector clock that
`sanitizeChildExecutionInfo` strips before replicating, so the incoming
copy always arrives nil — the guard exists precisely to restore the
local value. Without it, every replicated update to an existing child
execution erased it.
## Risk
Both are restorations of intended behavior, not new behavior, and both
previously failed in the safe direction:
- An over-cleared activity mask regenerates a timer task that is already
pending. Duplicates are dropped at execution
(`processSingleActivityTimeoutTask` re-derives the sequence and fires
what expired), so the symptom was extra timer-queue writes, not missed
timeouts.
- A nil child clock is tolerated by callers — see the comment in
`recordchildworkflowcompleted/api.go`: *"it should be fine e.g. that
ci.Clock is nil"*.
The child execution change is a **no-op unless the local cluster
recorded a clock by starting the child itself**, which for a passive
cluster means after a failover. On a standby that never started the
child, both sides are nil.
For activities, preserved bits suppress recreation of a timer task, so
it is worth being explicit about why that is safe on the passive side:
an expired-but-unresolved standby timer returns `ErrTaskRetry` and stays
in the queue rather than being consumed, so there is nothing to
recreate. The one path that does drop a task, past
`StandbyTaskMissingEventsDiscardDelay`, logs a warning and increments
`task_errors_discarded`, so it cannot happen silently.
## Tests
Two new tests at the `applyUpdatesToSubStateMachines` level, one per
activated `sanitizeFn`. **Both fail with the shadowing restored**
(verified: `expected: 13, actual: 0` for the mask; `local child
execution clock was not carried over` for the clock).
The gap they close: the existing `getActivityTimerTaskStatus` tests call
the decision function directly with a non-nil `current`, so no unit test
could observe the *caller* passing nil. Nine passing tests, zero
coverage of the wiring.
Two pre-existing tests needed updating, both informative:
- `TestApplyMutation` / `TestApplySnapshot` failed with *"Unexpected
call to IsVersionFromSameCluster"* — that mock was never needed because
the nil short-circuit made the cluster check unreachable. The gap is
itself evidence the path was dead.
- `verifyActivityInfos` asserted `s.Equal(int32(TimerTaskStatusNone),
actual.TimerTaskStatus)`, encoding the bug and blocking any correct fix.
Replaced with the invariant that actually matters:
```go
s.Zero(actual.TimerTaskStatus&^originStatus, "TimerTaskStatus gained a
bit that was not set locally")
```
Applying an update may carry over or drop locally set bits, but must
never introduce one that was not already set — claiming a timer task
exists when none does is the direction that loses timers. That holds
regardless of which bits a given case preserves, so it will not need
rewriting as the mask logic evolves.
## Related, tracked separately
While auditing whether the mask can be trusted, one path was found that
moves a deadline without invalidating the mask: unpausing an activity.
Paused activities are excluded from the timer sequence, so a timeout
task firing during a long pause is dropped as an invalid task, yet
`unpauseActivityInfo` leaves the bit set. That is pre-existing and
active-side, independent of this PR, and is being tracked as its own
change.
It is worth noting here because this PR removes the passive side's
accidental repair for that class of stale bit: today's blanket wipe
clears any stale bit on the next replicated activity update.
---
`go build ./service/...`, `go vet`, and golangci-lint (repo config,
`--new-from-rev=origin/main`) clean; `./service/history/workflow/...
./service/history/ndc/...` pass across 3 consecutive runs.
Two pre-existing flakes are skipped in those runs and unrelated to this
change (both reproduce on unmodified `main`):
`TestTaskRefresherSuite/TestRefreshSubStateMachineTasks`
(nanosecond-differing HSM deadlines, map iteration order — fails 7/12 on
main) and
`TestMutableStateSuite/*/TestApplyWorkflowExecutionOptionsUpdatedEvent_TimeSkippingConfig`
(wall-clock resolution, ~1/30).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
## What changed and why?
1. Edge case: If fast-forward completes during a workflow transaction
but when time skipping checks at close transaction this fast-forward
time is before ms.Now(), time skipping should still get disabled. Now we
don't look at time points that are in the past and it is a bug.
2. Time-skipping propagation: Always propagate the time-skipping
configuration and fast-forward state to the next run and other
executions, regardless of whether time skipping is currently active.
This ensures that read APIs (for example, Describe and PollFastForward)
continue to return meaningful information instead of nil.
- Otherwise, in an edge case where fast-forward completes in the first
run and the user polls after the second run has become the current run
of the workflow execution, the poll API would return a NotFound error
instead of a completed poll result.
- Similarly, the Describe API would return a nil configuration instead
of the original configuration that should have been propagated.
3. Trivial changes:
- simiplify parameter of `propagateTimeSkippingToNextRun`
- unify UT names of timeskipping_test.go
## What changed?
Validate the retry delay is a valid proto duration.
## Why?
Prevent malformed retry delays from poisoning Nexus completions.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Potentially users could have been sending an invalid proto duration,
unclear how exactly, and now we would fail their request.
## What changed?
Captured `ExecutionState.Status` before calling `GetReleaseFn()(nil)` in
`updateWithStart.Invoke` to avoid a data race with concurrent goroutines
that may acquire the lock and modify `ExecutionState` after it is
released.
Added a regression test that confirms the race under `-race`.
Fixes#11600
## Why?
`Invoke` in `service/history/api/multioperation/api.go` releases the
workflow lock at line 196 via `workflowLease.GetReleaseFn()(nil)`, then
reads `workflowLease.GetMutableState().GetExecutionState().Status` at
line 201 — after the lock is released. Any concurrent goroutine waiting
on `Lock()` for the same workflow (e.g. a signal, terminate, or another
update) can acquire the lock and modify `ExecutionState` between lines
196 and 201, creating a data race.
This matches the pattern noted in the `Updater` struct itself:
> WARNING: any references to mutable state data *have to* be copied to
avoid data races when used outside the workflow lease.
## How did you test it?
- [x] added new unit test(s)
The test spawns a concurrent writer that modifies
`ExecutionState.Status` after the lock is released.
**Before fix:**
```
go test -race -tags test_dep -count=1 \
-run TestUpdateWithStartSuite/TestInvoke_CompletedUpdate_StatusCapturedBeforeRelease \
./service/history/api/multioperation/
WARNING: DATA RACE
Read at ... api.go:201
--- FAIL
```
**After fix:**
```
ok go.temporal.io/server/service/history/api/multioperation
```
## Potential risks
Minimal - single line moved before the release call. No API or
persistence behavior change.
## What changed?
Batch operations now target Paused executions in addition to Running
ones. The filter auto-appended by adjustQueryBatchTypeEnum changed from
ExecutionStatus='Running' to ExecutionStatus='Running' OR
ExecutionStatus='Paused', affecting all workflow batch types
(terminate/signal/cancel/update-options) and activity batch types
(unpause/update-options/reset/terminate/cancel).
## Why?
A paused execution is still non-terminal. Users issuing a batch
terminate/signal/cancel reasonably expect it to apply to paused targets,
but the old Running-only filter silently skipped them.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
- The filter now references ExecutionStatus='Paused' on every batch
operation. This is ok as it's an additive clause to the visibilty query
- This would be changing behavior for batch callers as paused activities
are now affected.
## What
- Don't send scale down signal when task is matched from backlog even if
the poll wait time is high.
- Do not send scale up signal when task queue is rate limited.
## Why
- When a task comes from DB backlog, the poll wait time reflects DB read
path latency, not excess pollers — the -1 is not appropriate. Instead we
want to apply the normal scale up check.
- Similarly, when dispatch is bottlenecked by a task queue rate limit,
scaling up pollers won't help.
## How did you test it?
Unit tests
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
1. perf improvement: no time-skipping task regen on full refresh
2. new functional test: claim Nexus HSM timers are part of in-fight
nexus operation, and won’t be skipped, add functional test
3. new functional test: add a functional test to verify time skipping
won't change retention time
4. trivial bug fix: time-skipping: task regen didn’t read
EnableWorkflowExecutionTimeoutTimer
## Why?
- correctness related No. 4 though it is an edgy case
- perf related No.1
- test coverage No.2,3
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Wrapped the frontend Nexus dispatch routes with the shared OpenTelemetry
HTTP handler.
## Why?
Nexus HTTP requests need an inbound server span to connect the caller
trace.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Removed blanket timestamp wrapping
- cron/retry/start child wf use virtual time directly
## Why?
there was a double-shifting for CaN but not for other cases
it shall be fixed unifying the clock used by all virtual time
propagating cases, and this PR chooses to unify in a way that all cases
propagate use virtual time
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
- Adds `phase=error` to the existing `replication_lifecycle` wide event;
no new event type or table.
- Covers state-based replication (`SyncVersionedTransition`,
`VerifyVersionedTransition`, and `SyncWorkflowState`) plus standby
transfer, timer, and outbound queue failures.
- Captures sender, passive execution/apply, verification,
recovery/refetch, namespace refresh, Nack, DLQ, and history-branch
cleanup boundaries.
- Uses one shared error builder with small sender, executable-task, NDC,
and standby-queue adapters.
- Records workflow identity, source task identity, target context,
operation, error, attempt/priority, disposition/recovery, and extensible
diagnostics in `details`.
- Identifies apply provenance as `apply_artifact_source=task_payload` or
`sync_state_refetch`.
- Remains gated by `history.emitReplicationLifecycleEvents` (default
off).
## Why?
Replication failures span the sender, passive executor, recovery loop,
and apply layer. Recording these boundaries in the existing lifecycle
event makes the path of a workflow or replication task directly
traceable without adding another event schema.
## Example traces
The examples below are abridged records captured from the two-cluster
XDC test. Events for the same task correlate on `source_cluster`,
`source_shard`, and `source_task_id`; workflow identity is present on
every record.
A state task that fails on the passive cluster and is written to the
DLQ:
```json
{"phase":"sent","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"priority":"High"}
{"phase":"error","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"details":{"operation":"passive_task_execution","error":"failed to apply replication task","error_type":"serviceerror.InvalidArgument","apply_artifact_source":"task_payload","attempt":1,"priority":"High","target_cluster":"standby"}}
{"phase":"error","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"details":{"operation":"task_execution","error":"failed to apply replication task","terminal":true,"priority":"High","target_cluster":"standby"}}
{"phase":"error","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048581,"details":{"operation":"dlq_write","disposition":"dlq","terminal":true,"priority":"High","target_cluster":"standby","target_shard":1}}
```
A verification task that detects missing state, refetches it, and then
verifies successfully:
```json
{"phase":"sent","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"priority":"High"}
{"phase":"executing","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"attempt":1}
{"phase":"applied","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"outcome":"resend_needed"}
{"phase":"error","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"details":{"operation":"standby_verification","error":"missing mutable state, resend","error_type":"serviceerror.SyncState","recovery_action":"sync_state","priority":"High","target_cluster":"standby"}}
{"phase":"applied","task_type":"sync_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"outcome":"applied","details":{"apply_artifact_source":"sync_state_refetch"}}
{"phase":"executing","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"attempt":1}
{"phase":"applied","task_type":"verify_versioned_transition","workflow_id":"example-workflow","source_cluster":"active","source_shard":1,"source_task_id":1048595,"outcome":"verified"}
```
## How was it tested?
- `go test -tags test_dep ./common/wideevents
./service/history/replication ./service/history/ndc ./service/history
./tests/testcore`
- Changed-lines `golangci-lint`: 0 issues.
- Temporary, uncommitted two-cluster XDC tests forced passive task
failures, DLQ handling, standby verification, and SyncState
refetch/recovery with lifecycle events both enabled and disabled.
- A tiered-processing run confirmed concrete `High` priority on sent and
error records.
## Risks
- Enabling the dynamic config increases event volume; retries and
recovery can produce several error phases for one source task.
- `details.operation`, `details.disposition`, and
`details.recovery_action` distinguish those boundaries.
- Emission is best effort and does not change replication error
propagation, retry, or recovery behavior.
## Motivation
`TimerSequenceID` (~48 bytes) is heap-allocated and returned as a
`*TimerSequenceID` from 5 getter functions: `getUserTimerTimeout`,
`getActivityScheduleToStartTimeout`,
`getActivityScheduleToCloseTimeout`, `getActivityStartToCloseTimeout`,
and `getActivityHeartbeatTimeout`. These are called for every pending
timer/activity during `LoadAndSortUserTimers()` and
`LoadAndSortActivityTimers()` — a hot path in every workflow task.
Callers were already dereferencing the pointer before appending to the
value-type `[]TimerSequenceID` slice, so the existing code was
allocating on the heap only to immediately copy to the stack.
## Changes
- `TimerSequenceID` is now returned by value with a `(TimerSequenceID,
bool)` tuple
- Idiomatic Go pattern (same as map access) replaces nil-check sentinel
- All 5 getter methods updated + call sites and tests adapted
## Impact
Eliminates one heap allocation per getter call on the timer-sorting hot
path.
## Tests
- `service/history/workflow` (1075 tests): ✅ passed
---------
Co-authored-by: Prathyush PV <prathyush.pv@temporal.io>
## What changed?
Adds structured `namespace_lifecycle` wide events for namespace
mutations.
### Namespace mutations
- `namespace_registered` is emitted after successful registration.
- `namespace_updated` is emitted after successful namespace updates and
includes full `before` / `after` snapshots, requested values, and
`requested_fields` identifying what the caller explicitly set.
- Local-to-global promotion is distinguished by `is_promotion` and
`promote_namespace_requested`.
- Active-cluster failover is distinguished by `is_failover` and the
active-cluster transition in `before` / `after`.
- Deprecation is represented by the `Registered` to `Deprecated` state
transition.
- Workflow-rule creation and deletion include the affected rule ID and
detail, plus force-scan and request-ID data when supplied.
- `namespace_renamed` is emitted by the delete-namespace worker when the
namespace is renamed to its tombstone name. This is the observable
deletion point because namespace deletion is local and is not replicated
as a namespace operation.
The snapshots cover namespace info, configuration, archival settings,
replication topology/state, failover versions/history, custom
search-attribute aliases, bad binaries, and workflow-rule IDs. Request
security tokens are not captured.
### Dynamic configuration
All namespace lifecycle emission is gated by the new global
dynamic-config setting:
```yaml
system.emitNamespaceLifecycleEvents:
- value: true
```
The setting defaults to `false` and is evaluated dynamically by
frontend, history, and worker producers. In addition to the new
namespace mutation events, the gate covers the existing handover-related
namespace lifecycle events:
- `shard_handover_watermark_set`
- `shard_handover_watermark_removed`
- `shard_handover_incomplete`
This PR does not introduce or change those handover event payloads; it
only makes their emission follow the same namespace lifecycle flag.
## Why?
Existing RPC metrics and logs do not provide a structured, field-level
record of namespace control-plane changes. These events provide an
attributable and queryable view of what changed, including promotion
versus failover, requested versus persisted values, rule mutations, and
delete-pipeline renames. The shared gate lets operators enable the
complete namespace lifecycle signal consistently across services.
## How did you test?
- [x] Unit tests with `-tags test_dep` for frontend emission and
disabled gating, history handover gating, delete-namespace rename
emission, migration incomplete-handover gating, dynamic config, and
common wide-event payloads.
- [x] `make lint-code` (`0 issues`).
- [x] Full local two-cluster E2E using
`config/development-cluster-a.yaml` and
`config/development-cluster-b.yaml` with a JSON event logger.
- With the flag enabled, validated register, ordinary update,
workflow-rule create/delete, deprecate, delete rename, promotion,
cluster-list update, and failover from cluster A to B.
- Also verified that the existing handover producers remain functional
when enabled: all 16 shard watermark additions and removals on cluster B
and a forced 32-shard incomplete-handover event on cluster A.
- With the flag disabled, repeated all producer paths and confirmed zero
emitted bytes. The failure-only incomplete-handover path was rerun after
disabling the flag and left both event-log counts unchanged.
- Confirmed live dynamic-config enable/disable behavior without
restarting either cluster.
### Abridged failover event
```json
{
"event_name": "namespace_lifecycle",
"phase": "namespace_updated",
"details": {
"before": {
"active_cluster": "cluster-a",
"failover_version": 1
},
"after": {
"active_cluster": "cluster-b",
"failover_version": 2
},
"requested": {
"active_cluster": "cluster-b"
},
"requested_fields": ["active_cluster"],
"is_failover": true,
"is_promotion": false
}
}
```
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
Fixed bookkeeping of mutable state approximate size on the two paths
that mutate an `ActivityInfo` without adding its bytes to the counter:
`AddActivityTaskStartedEvent` (start), and the heartbeat handler, where
`RetryLastWorkerIdentity` now moves into `UpdateActivityProgress`.
## Why?
Both sites mutate the pointer `GetActivityInfo` returned, but don't
alter the approximate size.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
The risk should be low. We will hit the mutable state size limit faster,
but it would be proper accounting.
## What changed?
Validate links on callbacks consistently.
## Why?
Some links on callbacks for some requests are not being validated
properly
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
`SetReaderWatermark` takes whether the slice a batch came from still has
tasks to load, and only counts the read toward the reader stuck attempt
total when it does.
## Why?
A read that drained its slice made progress, but the counter incremented
on those too. A shard that keeps generating tasks creates a new slice
per notification and several can cover one fire time second, so a reader
draining each of them in a single read still looked stuck.
Counting only reads that left tasks behind measures the tasks wedged in
the window instead, which is what blocks everything ordered after them.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
The "overdue scan hit per-namespace check cap" warning previously only
logged the namespace and the cap value, giving no way to tell which
schedule the scan stopped at or how much progress it had made before
hitting the cap. Add the schedule ID it stopped on, how many schedules
were checked this pass, and how many anomalies had already been found,
so the log line is actionable on its own.
## What changed?
adds some logs
## Why?
Trying to debug some odd results
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
very low, feature is turned off by default, log line only
…s skipped (#10539)"
This reverts commit b11795a993.
## What changed?
Revert skip mutable state transaction if chasm nodes unchanged in
transaction.
## Why?
Needs more testing.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Problem
State-based replication blanket-cleared an activity's `TimerTaskStatus`
whenever a replicated update changed the attempt, version or stamp:
```go
if current == nil || ms.ShouldResetActivityTimerTaskMask(current, incoming) {
incoming.TimerTaskStatus = TimerTaskStatusNone
}
```
Every cleared bit lets the next task refresh regenerate a timer task.
`CreateNextActivityTimer` only ever creates the *earliest* timer in the
sequence and bails if that one already exists — so the cost is at most
one task per refresh, but which timer it is depends on the activity's
state.
**Between retry attempts, that earliest timer is schedule-to-close.**
Clearing the started state removes start-to-close and heartbeat from the
sequence, and when the user doesn't set `ScheduleToStartTimeout` it is
[normalized](https://github.com/temporalio/temporal/blob/main/chasm/lib/activity/validator.go)
to the schedule-to-close duration — anchored on `ScheduledTime`, which
advances every retry, so it is always *later* than schedule-to-close
anchored on the fixed `FirstScheduledTime`.
So the passive side regenerates schedule-to-close once per replicated
retry, and because that deadline never moves, **every duplicate carries
an identical `VisibilityTimestamp`**. A concrete case: a 31-day activity
with a flat 10s backoff reached attempt 242,037 — on the order of 242k
tasks piled on a single instant, essentially all of which re-derive the
sequence, find nothing expired, and return `errNoTimerFired`.
The duplicates are dropped at execution, so **there is no correctness
impact**. It is a timer-queue hotspot.
## Approach
Compare the four timer deadlines before and after the update, and keep a
bit only when its deadline is unchanged:
```go
return current.TimerTaskStatus &^
getActivityTimerDeadlines(current).changedMask(getActivityTimerDeadlines(incoming))
```
**Why deadlines are the right and sufficient signal.** An
`ActivityTimeoutTask` is a wake-up at a point in time:
`processSingleActivityTimeoutTask` re-derives the whole sequence from
current mutable state and fires whatever expired, explicitly without
consulting the task's attempt or stamp (*"Note: we don't need to check
activity Stamps"*). A pending task whose deadline did not move is still
correct no matter what else changed; one whose deadline moved is useless
no matter what stayed put.
Attempt and stamp are only ever proxies for "some deadline probably
moved", so they are deliberately not consulted. A retry needs no special
case: clearing the started state removes start-to-close and heartbeat
outright, and schedule-to-start's anchor advances — while
schedule-to-close survives on its untouched `FirstScheduledTime`.
This also fixes a case **the active side still has**: for mutable state
predating `FirstScheduledTime`, schedule-to-close falls back to the
`ScheduledTime` anchor, which a retry does move. Keying off the attempt
preserves that bit and leaves a task pointing at the old, earlier
instant; comparing deadlines clears it.
A cross-cluster version change still resets everything — that is about
task provenance, not deadlines.
The four deadline getters are split into free functions with one-line
method wrappers so the comparison reuses `timerSequenceImpl`'s existing
math rather than duplicating it.
## Scope
Limited to **state-based replication**. The event-based `SyncActivity`
path in `ndc/activity_state_replicator.go` keeps its existing blanket
reset and is untouched — it passes a synthetic `ActivityInfo` carrying
only version and attempt, which cannot support a deadline comparison,
and it is the older path.
## Tests
9 cases. The ones carrying the change:
- `Retry_KeepsOnlyScheduleToClose` — a real retry transition (attempt
bumped, started state cleared, schedule time advanced); start-to-close
and heartbeat bits drop, schedule-to-close survives.
- `Retry_LegacyAnchor_ClearsScheduleToClose` — same retry with
`FirstScheduledTime` nil; the moved deadline is detected and the bit
clears.
- `AttemptChangedWithoutDeadlineMove_KeepsMask` — attempt alone decides
nothing.
- `ClearsOnlyMovedDeadlines` / `TimerDisappears` — shorten or remove
`HeartbeatTimeout`; only that bit drops.
- `UnrelatedOptionChanged_KeepsMask` — stamp bumped, no deadline
affected, mask fully preserved (the old code wiped it).
`go build ./service/...` and `go vet` clean;
`./service/history/workflow/... ./service/history/ndc/...` pass across
repeated runs.
## Two pre-existing flakes (not from this change)
Both reproduce identically on unmodified `main` and neither touches
`ActivityInfo.TimerTaskStatus`. Skipped in the runs above and left alone
here:
- `TestTaskRefresherSuite/TestRefreshSubStateMachineTasks` — **7/12
failures on unmodified main.** HSM timer infos get deadlines differing
by a single nanosecond, so grouping depends on map iteration order.
Worth a separate fix; it currently fails more often than it passes.
-
`TestMutableStateSuite/*/TestApplyWorkflowExecutionOptionsUpdatedEvent_TimeSkippingConfig`
— 1/30 on clean tree, 1/30 with this change. Asserts a renewed
`TargetTime` differs from the initial one; fails when both land on the
same clock reading.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
## What changed?
Add `subqueue-id` tag to logs: all task reader logs (with a tagged
logger) and several db logs where it was missing.
## Why?
Better debugability.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Update execution's LastRunningClock when in ID reuse case and create
new execution as current.
- Two CHASM specific changes:
1. CHASM executions now consume a number from shard clock when updating
LastRunningClock to guarantee uniqueness.
2. CHASM engine methods now block until shard is acquired and ready to
serve traffic.
## Why?
- The start execution flow attempts two creation, first as branchNew and
a second one as current. However the mutable state snapshot is prepared
at the first creation time and reused for the second attempt. If the
previous run is closed after the snapshot is prepared, then the previous
run's lastRunningClock will be larger than the new run's
lastRunningClock. This will cause standby cluster to treat the new run
as the older one and put it in zombie state.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Three fixes to priTaskReader, basically analogous to #11048:
- Drop tasks that are read below the current ack level.
- Don't setReadLevelAfterGap if read level has moved since the read that
it was based on.
- Don't allow ack level to move backwards (besides logging a
softassert).
## Why?
These fix the behavior in the case of read/write races where a write
passes tasks to the reader (the bypass optimization) while the reader is
reading, and then the reader reads tasks that are already loaded. The
first fix is the main one: if we add tasks below the ack level, we'll
ack them and then the ack level can move downwards. The second fix
prevents the read level from moving backwards if we did a read at the
end of the queue that raced with a write. If it let the read level move
backwards, the ack level could also move backwards. The third one just
makes the desired behavior (the ack level is monotonic) enforced. With
the above two fixes it shouldn't be hit anymore (it was hit rarely in
those race situations).
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
- [x] verified logic in TLA+ model (not checked in yet)
## What
Skip the special `versioning attribution logic` when computing stats for
sticky queues. Otherwise, we end up with 0 value for add and dispatch
rates. This breaks poller autoscaling for sticky queues since 0/0 = NaN,
and NaN > threshold is always false, so no +1 scaling signals are ever
emitted.
Background on version attribution logic
----------------------------------------
For a versioned deployment, when tasks arrive, they are always added to
the unversioned queue. Then at dispatch time, they are routed to the
versioned queue. So the unversioned queue stats includes tasks that
belong to versioned queue. When reporting stats for a queue, we need to
remove the versioned queue stats from unversioned queue stats to avoid
double counting.
Why this is not applicable for sticky queue (any unversioned queue)
--------------------------------------------------------------------
Sticky queues are not versioned; they only have the unversioned physical
queue. So the above logic is not applicable. It was querying the stats
for same queue twice (build_id = "") and subtracting them -- resulting
in a value of 0.
## How did you test it?
Unit tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
Skip unbuildable replication tasks on stream sender instead of blocking
the stream. The skip is logged and new metric ReplicationTaskSendSkipped
added.
## Why?
When the replication stream sender cannot build ("convert") a task, it
retries and, once the retry budget is exhausted, returns an error that
tears the stream down. On reconnect the sender resumes from the same
watermark, hits the same unbuildable task, and blocks the whole shard's
stream indefinitely. This change skips the task to unblock and logs, add
a metric for the skipped task so that it could be investigated later.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
