Change the V1 scheduler version ceiling from read-once-and-locked to
re-evaluated on every tweakables MutableSideEffect. The effective ceiling
only ratchets tighter within a run (a looser or unset value never raises it),
and the recorded version never decreases mid-run: a lowered ceiling is
retained as a floor and downgrades on the next continue-as-new. This lets an
emergency tighten take effect without downgrading a running workflow.
- determineVersion: tighten-only ceiling + retained-version floor + nil-guard,
returning the recorded ceiling alongside the version.
- TweakablePolicies.VersionCeiling / VersionCeilingSet distinguish a real
ceiling of zero from histories that predate the field.
- SchedulerV1VersionCeiling doc documents the v1 floor, migration-pause, and
tighten-only semantics.
## What changed?
`CHASMToLegacyStartScheduleArgs` (the CHASM-to-V1 rollback conversion)
appends trigger-derived `BufferedStarts` after the regular pending ones
unconditionally, without sorting by due time. Sort the combined list by
`ActualTime` after appending, mirroring the sort already applied to
`RecentActions` a few lines above in the same function.
## Why?
V1's `processWatcherResult` (and the buffer-processing code generally)
assumes `BufferedStarts[0]` is always the earliest-due pending start --
it has no equivalent of CHASM's `Attempt` field to reorder around, so it
never needs to search for the right entry, unlike CHASM's own
`invoker.go`.
That assumption isn't guaranteed across a CHASM-to-V1 rollback:
- `convertBackfillersCHASMToLegacy` builds `triggerStarts` from
manual-trigger backfillers by iterating a Go map, whose iteration order
is randomized -- with more than one pending trigger, their relative
order isn't stable across calls.
- `triggerStarts` are appended after the regular buffered starts
regardless of their own due time. A manual trigger queued (and not yet
drained) before rollback could have a due time earlier than an
already-pending regular buffered start, landing it in the wrong position
in the resulting V1 list.
Found while reviewing the "legacy path doesn't use deferred starts, so
`BufferedStarts[0]` is always the next pending start" invariant this
comment documents (`service/worker/scheduler/workflow.go`) -- true for
V1 running natively, but not rigorously guaranteed for state arriving
via rollback.
## How did you test it?
- [x] Added
`TestCHASMToLegacyStartScheduleArgs_BufferedStartsSortedByActualTime`,
constructing a manual trigger whose due time predates an already-pending
regular buffered start; verified it fails without the fix (`the
earlier-due manual trigger must sort first`) and passes with it.
- [x] `go test ./chasm/lib/scheduler/...`
- [x] `make lint-code` (golangci-lint, 0 new issues)
## Potential risks
Low. This only reorders an in-memory slice being constructed fresh for a
rollback's `StartScheduleArgs` -- it doesn't change what's already been
recorded to history, and the sort key (`ActualTime`) is the same field
V1 already uses everywhere else to mean "due time."
## What changed?
- Record a buffered start's desired time when a refresh (non-long-poll
watch) observes the prior action complete, not just when the long-poll
watcher does.
- Gate the new state mutation behind a new version,
`RefreshCompletionDesiredTime` (14) -- `BufferedStarts[0].DesiredTime`
flows into the continue-as-new `Input`, which is replay-checked history,
so this can't be applied unconditionally to histories recorded below the
gate. 13 is already claimed by the in-flight `MigrationHandoffFixes`
work, so this is numbered 14; `CurrentTweakablePolicies.Version` is left
at `TriggerImmediatelyTimestamp` (12), so this PR does not itself
activate anything. `processWatcherResult` branches strictly on the
version (old codepath unchanged, new codepath gated) rather than folding
the version check into a boolean expression, so it's visually obvious
the old path is untouched on replay.
- Only backdate `DesiredTime` on the refresh path when the prior
action's `CloseTime` is genuinely after the next start's own due time --
i.e. it was actually blocked waiting on the prior action -- **and** the
start's own resolved overlap policy actually waits for a running
workflow to finish at all. A start resolved to `ALLOW_ALL` is never
blocked by a running workflow (`processBuffer` starts it regardless of
`isRunning`), so backdating it to an unrelated close time would
understate its real delay. This check is shared with `ProcessBuffer` via
a new `IgnoresRunningWorkflow` helper in `buffer.go`, so the two places
that need to agree on "does this policy wait for a running workflow"
can't drift apart.
- `refreshWorkflows` calls the backdate logic once per tracked execution
in `RunningWorkflows`. If a run still has multiple tracked executions
(e.g. `ALLOW_ALL` runs inherited from before a
pre-`DontTrackOverlapping` version ceiling was lifted), only move the
recorded close time forward -- a later-processed but earlier-closing
execution must not overwrite a genuinely later close already recorded
earlier in the same pass, or the reported delay understates how long the
start was actually blocked.
- The whole backdate decision is extracted into a pure, directly
unit-testable function, `shouldBackdateDesiredTime`.
## Why?
`processWatcherResult` only set `DesiredTime` when `long` was true (the
long-poll path). When a refresh discovered the prior action had
completed instead, `DesiredTime` stayed unset, so `ScheduleActionDelay`
fell back to the scheduled time instead of the prior action's close time
-- inflating the reported delay for back-to-back buffered actions.
Review then surfaced two follow-on correctness gaps in the fix itself:
it didn't account for `ALLOW_ALL` starts that were never actually
blocked, and it could pick the wrong close time when refreshing multiple
tracked executions in one pass.
### Summary
Combines #11134 and #11427 as two distinct fixes affecting V1 schedules
and requiring a version bump. they're joined together. Shipping them
separately would require two separate version-bump deploys for the
"same" version number. This merges both behavioral changes under one
shared v13:
- **`RefreshBeforeMigrationCheck`** (from #11134): This fixes a problem
that was preventing v1->v2 migration from ever succeeding under default
configuration
- **`PreserveMigratedStartIDs`** (from #11427): Try and keep the
requestIDs from when workflows are started under when a rollback occurs.
- Adds a third guard while in this space: Guards against late migrations
that occur when a transient error bounces a v1>v2 migration and the
schedule goes back to sleep and then attempts to migrate again.
Following #11134's two-phase-rollout rationale, this PR only teaches the
scheduler to *understand* v13 for safe replay/rollback: both fixes are
gated behind `hasMinVersion(13)`, but `CurrentTweakablePolicies.Version`
stays at
`TriggerImmediatelyTimestamp` (12). A follow-up deploy bumps `Version`
to 13 to activate both fixes at once — a single activation instead of
two.
#### Details
- `service/worker/scheduler/workflow.go`: adds
`RefreshBeforeMigrationCheck`
and `PreserveMigratedStartIDs`, both `= 13`, with a shared doc comment;
adds
a `// TODO` on `CurrentTweakablePolicies.Version` pointing at the
follow-up
activation deploy; ports both fixes' logic unchanged (gated on the
respective constant).
- `service/worker/scheduler/workflow_test.go`: ports all three new tests
from
the two source PRs
(`TestAutoMigrateReconcilesRunningWorkflowBeforeCheck`,
`TestMigratedBufferedStartPreservesIdempotencyIDs`,
`TestMigratedBufferedStartUsesLegacyIDsAtOldVersion`) plus the
`TestStart`
`RequestId` assertion.
`TestMigratedBufferedStartPreservesIdempotencyIDs`
now force-sets `CurrentTweakablePolicies.Version` (mirroring the other
two
version-forcing tests), since `Version` no longer defaults to 13 here.
- `service/worker/scheduler/testdata/replay_migration_v1_to_v2.json.gz`
and
`tests/schedule_migration_v1_to_v2_callback_compat_test.go`: brought in
verbatim from #11134.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
`go test -tags test_dep ./service/worker/scheduler/...` passes,
including
`TestReplays` against the copied fixture and all three new/updated unit
tests. `go build`/`go vet` pass for `./service/worker/scheduler/...` and
`./tests/...`.
## Potential risks
Moderately high risk as this is touching the Schedule V1 code. A problem
with nondeterminism could affect schedules quite badly.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: liam-lowe <56076876+liam-lowe@users.noreply.github.com>
Co-authored-by: alex.stanfield <13949480+chaptersix@users.noreply.github.com>
Co-authored-by: Stephan Behnke <stephanos@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: michaely520 <michaely520@users.noreply.github.com>
Co-authored-by: Feiyang Xie <feiyang3cat@outlook.com>
Co-authored-by: Kannan <rkannan82@users.noreply.github.com>
Co-authored-by: Fred Tzeng <41805201+fretz12@users.noreply.github.com>
Co-authored-by: Lakshay <54310363+Lakshaymiddha@users.noreply.github.com>
Co-authored-by: samm <sam.mathis@temporal.io>
Co-authored-by: Quinn Klassen <klassenq@gmail.com>
Co-authored-by: Will Duan <xinw.duan@gmail.com>
Co-authored-by: Qian Chen <qyc5937@gmail.com>
Co-authored-by: Prathyush PV <prathyush.pv@temporal.io>
Co-authored-by: Sean Kane <sean.kane@temporal.io>
Co-authored-by: mavemuri <74267563+mavemuri@users.noreply.github.com>
Co-authored-by: Rodrigo Zhou <rodrigo.zhou@temporal.io>
Co-authored-by: Brian VanLoo <brian.vanloo@gmail.com>
Co-authored-by: akbala <akbala@gmail.com>
Co-authored-by: Dan Davison <dan.davison@temporal.io>
Co-authored-by: Chris Smith <chrsmith@users.noreply.github.com>
## What changed?
Adds `worker.schedulerV1VersionCeiling`, a per-namespace dynamic config
that clamps the V1 scheduler workflow's recorded
`TweakablePolicies.Version` to `min(current, ceiling)`.
This will artificially gate the execution's functionality, to enable
backwards-compatibility with historical server versions (in a
cross-version multi-cluster setup).
## Why?
In a cross-version multi-cluster topology, a newer cluster can write
scheduler history an older rollback peer cannot replay after a failover
plus rollback. Clamping the recorded version to a configured ceiling
lets the newer cluster emit history the older cluster can replay.
## How did you test it?
- [x] built
- [x] added new unit test(s)
**What**
Handle terminal errors from asynchronous task-queue delete propagation
without marking propagation complete, so the Version workflow stays open
until cleanup succeeds.
**Why**
Retryable activity failures retry indefinitely, but a terminal error can
escape. The current code ignores that error and allows the Version
workflow to complete without confirming task-queue cleanup.
Note: We don't know how this non retryable case can happen. We observed
1 setup in this state; so this PR exports a metric for us to observe
when it happens.
**How did you test it?**
Workflow test covering a non-retryable cleanup failure and verifying the
Version workflow remains open.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
**1. Delete execution replication tasks carry a failover version.**
Stamped with the source cluster's
failover version at generation (`shard/context_impl.go:1013`), carried
in the existing
`ReplicationTaskInfo.version` (no proto change), and skipped on apply
when older than the target's
namespace failover version (`executable_delete_execution_task.go:130`).
Still applied: unversioned
tasks (queued pre-upgrade), tasks newer than the target's namespace
entry, and deletions synthesized
by versioned-transition tasks.
**2. `DeleteWorkflowExecution` is rejected on a cluster passive for the
workflow**
(`workflow_handler.go:2513`), with the usual `NamespaceNotActive`. In
the frontend, because the history
path is shared with replication apply, which must delete on passive
clusters — that path, the
delete-namespace worker, and admin force-delete are untouched.
## Why?
An unversioned delete task generated before a failover kept being
applied afterwards, deleting a run
the new active cluster owns.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
```
temporal --address :7233 workflow -n global-ns delete -w pay-invoice-0 -r 019fcf5b-996e-7416-9a2d-4ffb0ae7fc13 --grpc-meta xdc-redirection=false
WARNING: Deleting Workflow Executions in a global Namespace removes them from all replicas. Requests sent to a passive cluster are forwarded to the active cluster by default; to target the passive cluster directly, specify `--grpc-meta xdc-redirection=false`.
Delete Workflow "pay-invoice-0" with Run ID "019fcf5b-996e-7416-9a2d-4ffb0ae7fc13"? y/N y
Error: failed to delete workflow: Namespace: global-ns is active in cluster: cluster-b, while current cluster cluster-a is a standby cluster.
```
## Potential risks
- A deletion issued just before a failover is dropped on targets;
cleanup waits for the new active
cluster's retention timer. Delayed, not leaked — intended trade-off.
- API behavior change: deleting against a passive cluster now fails,
including batch delete
(`batcher/activities.go:704`). No killswitch; admin force-delete is the
escape hatch.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
- Keep attaching completion callbacks to every CHASM scheduler workflow
start, including actions whose resolved overlap policy is `ALLOW_ALL`,
so start requests remain safe across rolling upgrades.
- Exclude scheduler-wide last-completion result/failure input from new
`ALLOW_ALL` actions.
- After `StartWorkflowExecution` succeeds, remove the new `ALLOW_ALL`
`BufferedStart` in the same transaction. Separately, copy its start data
to `ScheduleInfo.RecentActions` as a start-only `RUNNING` history
record.
- Resolve an unspecified pending-start policy while building the
V1-to-V2 migration request, so every V2 dispatch and retry uses the
policy persisted on the buffered start.
- For a pre-existing callback, use the policy stamped on that start
rather than resolving against the schedule's current policy; an
unstamped V1-migrated running workflow remains tracked, as it was in V1.
- Preserve compatibility for previously persisted `ALLOW_ALL` callbacks
without updating last-completion state or `PauseOnFailure`.
- Carry start-only history through CHASM-to-V1 migration.
- Merge start-only and completion-tracked history by actual start time,
retaining the newest ten actions across both sources.
- Re-run generation after recording a start-only action, so a finite or
manual-only schedule rearms its idle timer from that start.
## Behavior
This makes CHASM match modern V1:
- A new `ALLOW_ALL` workflow starts normally, but is absent from
`DescribeSchedule.Info.RunningWorkflows` and does not consume
active-buffer capacity.
- It appears in `DescribeSchedule.Info.RecentActions` (and List's recent
actions) with its start time, workflow execution, and `RUNNING` status.
- Its completion callback remains attached for rolling-upgrade
compatibility. Because the start has already moved out of active
buffered state, the callback is ignored and success or failure cannot
change shared completion input or `PauseOnFailure`; its recent status
therefore remains `RUNNING`.
- For a final `ALLOW_ALL` action, an idle task armed before the start is
invalidated by the newer start time; generation immediately arms its
replacement, so the schedule still closes after `IdleTime`.
- If an older handler retained an `ALLOW_ALL` start, its callback
remains a compatibility path: the terminal action record is retained,
but its completion cannot update shared completion state or pause the
schedule once handled by this version.
- `DescribeSchedule.Info.RecentActions` is ordered by actual start time
and bounded to the newest ten actions across start-only and
completion-tracked history. Newer retained actions evict the oldest;
completion does not remove a start-only action.
- Non-`ALLOW_ALL` actions remain active and appear in `RunningWorkflows`
until their completion is handled.
- A V1 pending start with an unspecified override snapshots the
schedule's effective policy when its V2 migration request is built.
- A V1-migrated `RunningWorkflows` entry has an unspecified policy but
is nevertheless tracked: its completion updates sequential state and may
pause the schedule, regardless of the schedule's current `ALLOW_ALL`
default.
## Why?
`ALLOW_ALL` actions are independent executions. Tracking their
completions made scheduler-wide last result/failure and `PauseOnFailure`
depend on callback arrival order, and retaining them as active could
affect overlap and capacity behavior.
Callbacks remain attached because `StartWorkflowExecution` deduplicates
by request ID without reconciling callback differences. Keeping the
request callback-compatible prevents a mixed-version retry from
retaining a start that waits for a callback the workflow never received.
## How did you test it?
- [x] added unit coverage
- [x] added shared V1/CHASM functional coverage
Commands:
- `go test -tags test_dep ./chasm/lib/scheduler/... -count=1`
- `go test -tags test_dep ./tests -run
'^TestSchedule(CHASM|V1)/TestAllowAllDescribeContract$' -count=1`
- `make fmt-imports`
- `git diff --check`
- `env GOCACHE=/tmp/sch-038-gocache go vet -tags
disable_grpc_modules,,test_dep -vettool=.bin/errortype
-style-check=false ./chasm/lib/scheduler/...`
The functional test uses workflow signals to control completion. It
asserts counters, buffer size, recent-action status/timestamps, active
workflows, and failure/pause isolation across both backends. A CHASM
functional idle-close case verifies that a final `ALLOW_ALL` action
still closes after `IdleTime`. The migration regression and
callback-reason metric test drive real component transactions through
the CHASM test engine.
`make lint-code` currently exits before analysis with `no go files to
analyze` from its `--new-from-rev` filter, despite the Go diff; package
`go vet` is clean.
## Additional observability
Ignored callbacks are tagged as either `unrecognized_request_id` or
`already_completed`. A newly started `ALLOW_ALL` callback is expected to
be unrecognized after its buffered start moves to start-only history. An
already-completed callback is a valid redelivery (for example, after a
workflow reset). ~~Both preserve scheduler state while emitting a
warning and counter increment.~~ Missing request IDs are now metric-only
because they include expected `ALLOW_ALL` callbacks; known
`already_completed` redeliveries still emit the warning, event, and
counter.
## Potential risks
Keeping callbacks attached avoids permanently orphaning buffered starts
when old and new binaries race on the same request ID. During a rolling
upgrade, however, old and new requests still differ in whether
scheduler-wide last-completion result/failure input is included for
`ALLOW_ALL`; the request that wins deduplication determines whether that
workflow receives the legacy input. An older callback handler can also
temporarily apply the legacy completion and pause semantics. These
mixed-version differences end once the rollout completes, and the
callback ensures an older handler cannot wait indefinitely.
New `ALLOW_ALL` terminal status is intentionally not reflected in
schedule Describe/List results, as in modern V1. ~~Its expected late
callback is recorded as `unrecognized_request_id`, which adds callback
delivery plus warning/metric volume compared with omitting callbacks.~~
Its expected late callback remains recorded as
`unrecognized_request_id`, but only as a metric; warning and event
logging are suppressed until these callbacks can be safely omitted.
Migration resolves an unspecified pending-start policy at the V1-to-V2
boundary. If the schedule policy changes while that start remains
pending and the schedule then rolls back to V1, the explicit migrated
policy is preserved instead of inheriting the newer schedule policy.
This is a narrow semantic difference that keeps V2 dispatch and retry
behavior stable.
---------
Co-authored-by: David Porter <david.porter@temporal.io>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## What changed?
Preserved the caller-provided visibility scope when starting type-based
batch activity unpause operations. The activity-type predicate is now
safely combined with the original query.
## Why?
The server previously replaced the caller’s visibility query with the
activity-type predicate. This could broaden the batch scope and unpause
activities in workflows the caller did not select.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Type-based unpause batches now strictly honor the caller’s visibility
scope, so they may process fewer workflows than before, but this is the
correct design
## What changed?
- Added a reusable parent-child XDC test harness for controlling
replication task application, delay, ordering, and omission around
namespace failover.
- Added five functional scenarios covering orphaned children, missing or
incomplete children, task discard, missing parent completion, and parent
resend recovery.
- Added unit tests for the replication gate and
legacy/transition-history task decoding.
Configured two history shards per cluster and placed parent and child
workflows on different shards.
- Updated XDC synchronization checks to support multiple history shards.
## Why?
Parent-child replication failures depend on rare cross-shard ordering
and failover timing, making them difficult to reproduce reliably.
These tests deterministically construct the relevant partial states
while still exercising real Temporal services, persistence, replication,
verification RPCs, task retry/discard behavior, and namespace failover.
The harness also makes future scenarios easier to add and review.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed
Adds `TestLogger.StartCapture` / `StopCapture` to records log calls and
make them queryable.
## Why
We want ability to verify certain logs where emitted to verify
observability. See #11689 for first use case.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Go 1.27 prerequisite that applies the `errorsastype` Go fixer and its
required error-interface updates.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
> **Part 3 of a 5-PR series** building to replication stream namespace
isolation (a restructuring of #10147): read buffer → reader group → lane
protocol → isolation manager → sender isolation.
> #11302 (reader group) has merged, so this PR's diff is now standalone
against `main`. · **Next in series: #11304** (isolation manager).
## What changed?
The wire-level building blocks for per-namespace lane isolation,
receiver side only:
- **Proto**: `SyncReplicationState` gains `throttle_high_namespace_ids`
(namespaces the receiver reports as overwhelming the HIGH lane — the
priority lives in the field name, so a future LOW extension adds its own
field rather than widening this one), `isolated_lane_states` (per-lane
applied watermarks keyed by namespace ID — with the documented caveat
that a missing key is ambiguous between "not tracked yet" and "retired
and drained", so a sender must never treat absence alone as drain
proof), and `supports_namespace_isolation` (capability advertisement, so
a sender never emits lane-tagged traffic to a receiver that would
misroute it). `WorkflowReplicationMessages` gains
`isolated_namespace_id` — when set, the batch belongs to that
namespace's dedicated lane — and `retire_isolated_lane`, marking a
lane's final message.
- **Receiver**: lane-tagged batches route to lazily-created
per-namespace task trackers. Each lane is its own monotonic stream for
the life of the connection — there is no rewind or rotation machinery,
because the sender-side design (later in the series) gives every lane a
single owner cursor that never goes backwards. Member-lane watermarks
fold into the overall ack minimum (cleanup safety) and are reported per
lane; member-lane backlogs count toward HIGH flow control. Lane
lifecycle is defensive about ordering: a batch's tasks are tracked
BEFORE its retire flag is applied (so the concurrent ack loop can never
delete a lane whose final batch is mid-track), a retiring lane is only
dropped once it is drained AND has tracked at least one batch, and
non-retire traffic arriving on a retiring lane revives it (the sender
re-isolated the namespace before the lane drained). Lane-tagged traffic
at any priority other than HIGH is a protocol violation and fails the
stream rather than silently mis-acking (isolation splits the HIGH lane
only). Lanes created concurrently with `Stop()` are pre-cancelled so no
tasks run after shutdown.
- **`NamespaceThrottler`** interface (default: noop, via fx) observes
per-namespace HIGH-priority task load and decides which namespaces to
report.
The sender does not tag lanes yet, so this is inert until the
sender-side isolation lands.
## Why?
Isolation needs a wire contract before the sender can use it: capability
advertisement, per-lane routing and progress reporting, and the
throttled-namespace feedback channel. Landing the receiver first makes
mixed-version clusters safe by construction. Compared to #10147, lanes
are per-namespace rather than shared per severity tier — which is what
eliminates that design's cursor rewinds and the
watermark-regression/tracker-rotation protocol this PR previously needed
to compensate for them.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s) — lane routing (priority routing when
unset, per-namespace tracker identity, non-HIGH rejection), retirement
lifecycle (drop once drained, never-tracked retiring lane survives the
ack snapshot, revive on re-isolation traffic, fresh lane after drop),
and post-Stop lane creation being pre-cancelled
- [x] added new functional test(s) — exercised end-to-end by the xdc
test in the final PR of the series
## Potential risks
Inert until a sender emits `isolated_namespace_id`, which is gated
behind both a config flag and the capability advertisement.
Receiver-side lane state is bounded by the sender's isolation cap (final
PR).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches replication ack watermarks and stream failure paths on the
passive cluster; lane mis-handling could stall cleanup or mis-ack,
though lane-tagged traffic is not sent until follow-up sender work.
>
> **Overview**
> Adds the **wire contract and receiver behavior** for per-namespace
HIGH-lane isolation before the sender starts tagging traffic.
>
> **Proto:** `SyncReplicationState` now carries
`throttle_high_namespace_ids`, per-namespace `isolated_lane_states`, and
`supports_namespace_isolation`. `WorkflowReplicationMessages` adds
`isolated_namespace_id` and `retire_isolated_lane` so batches can be
routed and retired on dedicated lanes.
>
> **Receiver:** Lane-tagged HIGH batches use lazily created
per-namespace task trackers (monotonic per connection). Member-lane
watermarks fold into the overall ack minimum and are reported per
namespace; member backlog counts toward HIGH flow control. Acks include
shard-scoped throttled namespace IDs via a new **`NamespaceThrottler`**
(noop by default). Lane lifecycle handles retire/drain/revive and
rejects non-HIGH lane traffic.
>
> Sender emission of lane tags is not in this PR, so behavior stays
inert until a later change gates on capability advertisement.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
688a5173f0. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
## What changed?
Back `testSequentialTaskQueue` with an unbounded slice instead of a
3000-capacity channel, so `Add` no longer blocks.
## Why?
`common/tasks` deadlocked and timed out after 15m on main ([run
32773080605](https://github.com/temporalio/temporal/actions/runs/32773080605)).
`SequentialScheduler.Submit` calls `Add` from inside `PutOrDo`'s
callback, which runs under the shard write lock. Once the test queue's
channel filled, the submitter blocked in `Add` while holding that lock,
and the only worker was blocked on the same lock in `RemoveIf`. Holding
the lock across the add is deliberate — it keeps the add atomic against
`RemoveIf`'s empty-check — so the invariant is that `Add` must not
block, and only the test queue violated it.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
Adds generic `await.Rcv` and `await.Snd` helpers that bound blocking
channel operations by the test context and report closure or
cancellation clearly.
Replaces both previous helpers and existing unsafe channel rcv/snd
across tests.
## What changed?
- Moves `activity.linkValidator` into `common/links`.
- Moves `callback.Validator` into `common/callbacks`
In addition, this PR performs some minor refactorings for consistency
and clarity.
- Moved some `links.Validator`-specific tests from
`chasm/lib/activity/validator_test.go` elsewhere.
- Introduced a `callbacks.ValidatorConfig` to bundle all of the specific
settings. (Since we'll need to wire 3+ more parameters when updating the
`callbacks.Validator` to support worker callbacks.)
> The singular package names `common/link` or `common/callback` would be
more consistent. But `common/links` already existed, there are other
pluralized ones like `common/enums` or `common/headers`. And IMHO, the
plural seems a little more applicable since the validations are only on
groupings of links or callbacks.
## Why?
The `activity.linkValidator` and `callback.Validator` types are great,
but they aren't able to be used as across other CHASM components as
easily. Moreover, `callback.Validator` uses types that are exposed from
the CHASM `callback` package, it will lead to circular dependencies in
the future. (I'm hitting this now in PRs for landing worker callbacks.)
Moving the `commonpb` protobuf validation into `common/` means we can
better separate the the distinction between validation logic and the
CHASM executions that rely on it.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
This should just be a standard refactoring. There should not be any new
validation checks enabled on codepaths where they weren't already
present. (Or in test cases, we initialize fields of
`callback.ValidatorConfig` that weren't used before.)
## What changed?
When `dispatchForExistingWorkflow` finds `currentRunID == ""`, it treats
the missing current execution record as a deletion issued by the user,
and handles the target run accordingly:
- **Running target:** return an internal invariant error. Replication
tasks are ordered, so a deletion cannot overtake the close event — a
running run with no current record is conceptually impossible. The task
retries and is eventually sent to the DLQ.
- **Closed target:** apply the target as a zombie with
`UpdateWorkflowModeBypassCurrent`, leaving the current record missing.
The current record is never re-established here, because that would
resurrect a user-deleted workflow.
If the task carries a new run (continue-as-new / cron / retry
successor), it is **not dropped**: when it is not already present
locally it is persisted as a zombie via bypass-current, so **no history
is lost** — but it never becomes the current run. A zombie is not a dead
end: when the successor's own close event later replicates, `ZOMBIE ->
COMPLETED` is a valid transition and it converges like any other run
(intermediate events keep it a zombie, so they never hit the
running-invariant error). While open, a zombie is invisible to
visibility, so it never shows up as a stray running workflow.
For example, given `r1 -> r2 -> r3` (continued-as-new chain) followed by
deletion of the current run `r3`, a later replication update of `r1` may
carry `r2` or `r3` as its new run. Either way the successor is persisted
as a zombie, never as current, so the user's deletion intent is
preserved while its history is retained. The current record is
(re)established only by a separately replicated new/reset run through
its own new-workflow path.
## Why?
Cross-cluster deletion removes the passive cluster's current execution
record while closed run rows can survive until retention.
Re-establishing a current record from a later replication task would
resurrect a deliberately deleted workflow, so the passive cluster
preserves the deletion intent: closed historical runs converge as
zombies, and only a
separately replicated new/reset run may establish a new current record.
Carried successors are still persisted (as zombies) so their history is
never lost, and they close normally once their own close events
replicate.
## How did you test it?
- [x] built
- [x] run locally and tested manually — reproduced the original
duplicate-run failure with
`r1 -> r2 -> r3`, deleted `r3` on both clusters, then reset `r1`
- [x] covered by existing tests
- [x] added unit tests — missing-current dispatch for the closed,
rebuilt, and impossible
running cases, plus a carried new run in both the persist-if-absent and
skip-if-already-present paths
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What changed?
Fixes a build error, seemingly introduced when multiple changes were
merged automatically after approval.
## Why?
Because a broken build stops the flow of spice. And the spice must flow.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None. But there might be other issues, I'm curious to see if there are
any other issues flagged by CI/CD.
## What changed
The HSM Nexus executor logged outbound call failure logs were missing
tags.
## Why
`chasm/lib/nexusoperation` already logs exactly these fields via
`invocationTraceContext.tags()`,
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed
All three warnings in `ConvertNexusLinksToProtoLinks` interpolated the
link type into the message, and two also embedded the link URL. Moved
both into tags and kept the message static.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
The `commonpb.Callback`-variant of `commonpb.Callback_Internal` is
unused and should be removed entirely. This PR removes the remaining
unnecessary instances of that type.
(My actual motivation is really to avoid a larger diff later, since
introducing Worker-variant callbacks will start returning errors when
you try to attach an `Internal`-variant callback.)
However, changing `Callback_Internal` to `Callback_Nexus` changed the
behavior of `TestDedupLinksFromCallbacks`. After scratching my head for
a while, I add a doc comment to clarify exactly what the function does,
and then updated the tests to be easier to read and understand.
## Why?
The call to `dedupLinksFromCallbacks(...)` in the testcase did _not_
dedupe the links attached to `callbacks[0]` because it was the
`commonpb.Callback_Internal` variant. (Relying on a quirk of the
function only filtering callbacks from Nexus-variant callbacks.)
I kept that behavior in, but added a couple more test scenarios to
provide better coverage and clarify the expected behavior.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None.
## What changed?
Pass one test-owned context through namespace creation, namespace cache
polling, and search attribute registration during functional test setup.
## Why?
Reusing the test context avoids creating independent timeout contexts
for each setup RPC and ties their resources to the test lifecycle.
## What changed?
- Reorganize `chasm/lib/activity`
## Why?
- Improve navigability and codebase comprehensibility
## How did you test it?
- [x] covered by existing tests
## What changed?
ForkHistoryBranchResponse now carries BaseBranchToken, and the two reset
paths rebuild through it when the store set it. A store that doesn't
(Cassandra, SQL) leaves it nil and both call sites fall back to the
token the caller already had, so behavior is unchanged everywhere else.
## Why?
ForkHistoryBranch can modify the base branch token in ways the caller
may not have visibility into. This PR fixes it so that the changed token
is returned to the caller.
## How did you test it?
- [ ] built
- [ ] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
`GetWorkflowExecutionHistory` and `GetWorkflowExecutionHistoryReverse`
now check `branch_token` in the page token against the token in mutable
state.
## Why?
To confirm if it is still the correct branch after conflict resolution.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Adds metrics for how often and by how much queue slices
fail to narrow their predicate, and how large persisted queue state
actually is.
- `queue_slice_pending_keys` — histogram, recorded on every narrowing
attempt (declined or
succeeded). This is the distribution
`queueShrinkPredicateMaxPendingKeys` should be sized against.
- `shard_info_size` / `queue_state_size` — histograms recorded when a
shard record is actually
written, giving whole-record and per-category size.
- `queue_state_size_total` / `queue_slice_count_total` — counters paired
with the histograms above
(and with the existing `queue_slice_count`), so an exact bytes-per-slice
ratio is possible.
- `queue_slice_count` gains a `task_category` tag (previously untagged
beyond `operation`).
These are only metrics changes - no behavior changes.
## Why?
A slice only narrows its predicate below
`queueShrinkPredicateMaxPendingKeys` (10) pending
namespaces; above that it stays universal and re-reads the whole range
every time. Raising that
threshold safely requires knowing the pending-key distribution and the
persisted size.
This PR is the baseline for evaluating a follow-on encoding.
There are two counters because this server's tally-backed Prometheus
reporter doesn't preserve the
true recorded value when a histogram flushes — it replays each sample as
its bucket's upper bound,
so a histogram's `_sum` has no more precision than its buckets.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] run locally and tested manually (`queue_predicate_resolution_loss`
confirmed live against a local server under forced
narrowing-decline conditions)
## Potential risks
## What changed?
- emit remote cluster and namespace replication lifecycle records under
`namespace_lifecycle`
- retain compatibility aliases for specialized event-name constants with
TODO cleanup
- update tests and shared envelope documentation
## Why?
Update schema event names to match namespace lifecycle schema which
currently exist so they are properly interpreted.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Refactor NamespaceRateLimitInterceptor with functions to consume N
tokens:
- removed `tokens` overwrite argument as it's never used
- added functions to consume N tokens
The changes itself in this PR is no-op since it's introducing new
functions to the interface.
## Why?
Added flexibility to wrap `NamespaceRateLimitInterceptor`.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed
The completion (callback) handler's request-scoped logger carried only
the namespace, even though a richer one was built just above it.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Go 1.27 prerequisite that applies the `slicesbackward` Go fixer for
reverse iteration.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What
Treat `NotFound` from the Version workflow delete update as success, so
the Deployment workflow can clean up its stale version reference.
## Why
When a Version workflow is already closed or its history is gone, the
delete update returns `NotFound`, blocking the Deployment workflow from
removing the reference. Fixes#11539.
## How did you test it?
Unit test covering the `NotFound` → success path and verifying other
History errors still propagate.
## What changed?
Added Temporal Nexus attributes to spans.
## Why?
Domain attributes make Nexus traces more useful.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Every task produced by `GenerateMigrationTasks` is now low priority end
to end:
- Added a `Priority` field to `HistoryReplicationTask`,
`SyncActivityTask` and `SyncHSMTask`.
- Round-tripped `Priority` through the replication task serializer for
those three, plus `SyncVersionedTransitionTask` (its `Priority` field
existed but was never persisted). The proto
`ReplicationTaskInfo.Priority` field already existed, so no proto
change.
- `StreamSenderImpl.getTaskPriority` now honors `Priority` on all five
replication task types, defaulting to high when unspecified.
- `GenerateMigrationTasks` stamps `TASK_PRIORITY_LOW` on the
`HistoryReplicationTask`, the sync activity tasks and the `SyncHSMTask`
it returns (`SyncWorkflowStateTask` and `SyncVersionedTransitionTask`
already set it).
## Why?
`GenerateMigrationTasks` is only reachable through force replication
(`GenerateLastHistoryReplicationTasks`, called by the
migration/force-replication workflow and by tdbg). That traffic is bulk
backfill and should not compete with live replication. Only
`SyncWorkflowStateTask` was actually being treated as low priority;
everything else fell through to `TASK_PRIORITY_HIGH` in the stream
sender, and `SyncVersionedTransitionTask`'s low priority was silently
dropped on write.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
`TestTaskGeneratorImpl_GenerateMigrationTasks` now asserts
`TASK_PRIORITY_LOW` on every returned task and every task equivalent.
`service/history/replication`, `common/persistence/serialization` and
`service/history/tasks` pass.
Note: `service/history/workflow` has a pre-existing failure in
`TestTaskRefresherSuite/TestRefreshSubStateMachineTasks` that reproduces
on unmodified `main` at 39fc2c45e and is unrelated to this change.
## Potential risks
- `SyncVersionedTransitionTask.Priority` is now persisted where it was
previously dropped. Tasks written before this change still deserialize
with `TASK_PRIORITY_UNSPECIFIED`.
- `getTaskPriority` defaults to high on `UNSPECIFIED` for the
newly-handled types, so normal (non-force) replication keeps its current
priority. Only tasks explicitly stamped low move to the low priority
stream.
- Force replication tasks now share the low priority stream and its rate
limiting with sync-state traffic, so a large force replication may
progress more slowly than before — which is the intent.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
Adds server support for WorkflowQuery-backed Nexus Operations
## Why?
Part of effort to expose all Temporal primitives as Nexus Operations
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## TODO:
- [x] https://github.com/temporalio/api/pull/842
## Summary
- capture post-transaction mutable state for fresh `NotFound` snapshots
and `IsFirstSync` creation
- report successful verify history repairs as `outcome=backfilled`
- include the repaired event range and `new_run_id` in the verify
applied event
- add regression coverage for fresh zombie applies and
non-current-branch backfills
## Testing
- `go test ./service/history/ndc ./service/history/replication
./common/wideevents -count=1`
## What changed?
Adds `testcontext.EnsureRemaining` and has `await` use it so long await
calls can request additional test-scoped context time while still
respecting the test context cap.
## Why?
Await calls can need more time than the default test context has left
(esp after the environment setup). Extending the test timeout in this
way allows for (1) stuck tests to fail earlier than the default test
timeout and (2) legitimately longer running tests to pass without
manually tweaking the test timeout.
---------
Co-authored-by: Sean Kane <sean.kane@temporal.io>
## What changed?
- Clear an activity’s timer-task status when it is unpaused so timeout
tasks are regenerated.
- Make ResetActivity with keepPaused=false fully unpause both scheduled
and running activities, including clearing pause metadata.
- Add and strengthen unit and functional coverage for unpause,
reset-unpause, timer regeneration, and keepPaused=true.
## Why?
Timeout tasks can fire while an activity is paused and be discarded.
Previously, the activity’s timer-task status still indicated that those
tasks existed, preventing them from being recreated after unpause and
potentially making the timeout ineffective.
ResetActivity also bypassed normal unpause handling, and running
activities returned early without clearing their paused state.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Unpausing now invalidates existing timeout tasks and recreates the next
applicable timer during transaction close. This is correct, but a
behavioral change. Stale queued tasks may still be processed and
discarded through the existing stamp validation.
## What changed?
Adds `namespace_lifecycle` start and finish events for the namespace
handover, force replication, and catchup system workflows.
The events carry workflow identity and the core operation inputs.
Finished events classify the result as succeeded, canceled, or failed.
Force replication reports its cumulative verified workflow count and
emits only one start and one finish across a continue-as-new chain.
Emission uses one shared activity, the existing
`system.emitNamespaceLifecycleEvents` gate, disconnected cleanup for
cancellation, and workflow versioning for replay compatibility. Existing
shard handover events are unchanged.
## Why?
These system workflows currently have no consistent operation-level
event pair, which makes it difficult to correlate a namespace migration
request with its final outcome.
## How did you test it?
- [x] covered by existing tests
- [x] added new unit test(s)
`go test -tags test_dep ./common/wideevents ./service/worker/migration`
`make fmt-imports`
`make lint-code` reports no issues introduced by this change; the
repository-wide target still reports existing findings on current
`main`.
## Potential risks
The terminal event is best effort and cannot run after server-side
workflow termination or workflow run timeout because those outcomes do
not execute workflow cleanup.
## What changed?
Makes Standalone Activity conflict updates idempotent by recording the
`requestID` when attaching callbacks or links and recognizing duplicate
request IDs. I needed to add a dedicated CHASM error for
## Why?
This prevents a successful attachment whose response was lost from
failing on retry or duplicating/replacing callbacks and links.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
