## Summary
- Removed the skip on `testBackfillWithBufferOneOverlap` - verified
passing on both V1 and CHASM: the deferred backfill start under
`BUFFER_ONE` now correctly re-enables once the running workflow
completes.
- Removed `testUpdateScheduleMemoOnly` and its registration -
`UpdateSchedule` has no partial-update path for the `schedule` field;
every update (including memo/search-attribute-only ones) requires
resending the full schedule, so the test's premise no longer applies.
Other previously-skipped scheduler tests (`TestFailedStart`,
`TestPauseUnpauseBetweenNominalAndJittered`) were checked and still
reproduce their underlying issues, so they remain skipped.
## Test plan
- [x] `go test -tags=test_dep ./tests/ -run
'TestScheduleCHASM/Backfill|TestScheduleV1/Backfill' -count=1 -v`
- [x] `go build -tags=test_dep ./tests/...`
## What changed?
Capture panic in `ServiceErrorInterceptor`
## Why?
`ServiceErrorInterceptor` is a top-level interceptor for frontend,
history and matching. Capturing panics to make sure unhandled panics
won't crash the service.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed
Kept dynamic data out of Nexus log messages and moved it to tags. Added
a review guideline requiring static logger messages and structured tags
for all dynamic content.
## Why
Ensures that Nexus logs are aggregatable.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
Added wide events for the child workflow resend lifecycle, including:
- scheduled
- started
- succeeded
- source not found
- failed
- deduplicated
- limited by the host-level concurrency cap
## Why?
Child workflow resends run asynchronously, making missing-child
replication issues difficult to diagnose from request logs alone.
These events make it possible to trace scheduler admission, remote state
synchronization, verification, and terminal outcomes while keeping
parent and child resend events consistent.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
## What changed?
- Added an opt-in recovery path for a child workflow missing from a
standby cluster.
- After the existing resend delay, VerifyFirstWorkflowTaskScheduled
asynchronously fetches the child state from the active cluster, applies
it locally, and verifies it again.
- Added deduplication, per-shard concurrency limits, metrics, namespace
checks, and transition-history gating.
Corrected the discard-time source check to verify the child workflow
rather than the parent.
- Moved the reusable in-flight resend tracker into the shared
workflowresend package.
- Updated the existing XDC parent-child test to assert that the missing
child and its first workflow task are restored.
## Why?
Cross-shard replication may deliver the parent’s
ChildWorkflowExecutionStarted event before the child workflow reaches
the standby cluster. Previously, verification repeatedly returned
NotFound and eventually discarded the standby task, leaving the child
missing.
This adds the child-side symmetric recovery behavior to the parent
resend implemented in #11424 .
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
### Rollout
`history.enableChildWorkflowResend` default to `false`.
### Known
- Enabling this feature introduces additional cross-cluster state-sync
traffic. It is disabled by default and protected by deduplication and a
per-shard concurrency limit.
- Regular replication may race with state sync; duplicate application is
treated as success.
- The resend delay and replication timeout should remain below the
standby task discard delay so recovery has time to complete.
## What changed?
- Atomically return and hold the next parent workflow task after
starting child workflows.
- Release pending intercepted replication tasks during test cleanup.
- Ensure each blocked replication task executes at most once.
## Why?
This is a follow-up to
[#11789](https://github.com/temporalio/temporal/pull/11789).
The test separately scheduled and polled the next workflow task, leaving
a race where a duplicate child outcome could be recorded before the
parent workflow task started. When the assertion failed, blocked
replication tasks were not released, potentially stalling subsequent XDC
tests until CI timed out.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
Ran the affected test with the race detector 10 times:
```bash
go test -tags test_dep -race -count=10 ./tests/xdc \
-run '^TestFuncClustersTestSuite$/^EnableTransitionHistory$/^TestNaturallyBufferedChildWorkflowOutcomesFlushedToLosingBranch$' \
-args -persistenceType=sql -persistenceDriver=sqlite
## What changed?
Stop installing Temporal-specific process-global OTEL error handlers.
OTEL errors now use the default ie stderr.
## Why?
The process-global OTEL handler retained the Temporal server logger
after shutdown; ie it leaked.
## What changed?
`replicator_errors` now records the existing `replication_task_type`
metric dimension when a namespace replication processor exhausts its
task-application retries. This distinguishes values such as
`NamespaceTask` and `TaskQueueUserData` without introducing a new
metric.
A unit test verifies that the failure counter carries the task type from
the failed replication task.
## Why?
This is a prerequisite for splitting task queue user data
(`TaskQueueUserData`) failures from namespace metadata replication
(`NamespaceTask`) alerts. Today both task kinds feed `replicator_errors`
under `NamespaceReplicationTask` and are indistinguishable, so TQUD
failures can be reported as namespace replication poison pills.
This PR only adds the server metric label. Alert definitions will be
updated separately after the label is available in deployed server
versions.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added unit tests
- [ ] added new functional test(s)
Manual two-cluster XDC E2E using `development-cluster-a.yaml` and
`development-cluster-b.yaml`:
- A namespace replication conflict exported
`replicationTaskType="NamespaceTask"`.
- A TQUD apply failure exported
`replicationTaskType="TaskQueueUserData"`.
## What changed?
Capture any panics in Visibility query converter.
## Why?
Visibility query converter is complex, and at times might make
assumptions that might not hold (due to bugs in the store query
converter implementation for example). Capturing at top level, and
returning an error instead.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed
- Add `worker_deployment_name` and `worker_build_id` labels to Workflow
Task completion and failure metrics.
- Add the split labels to Activity success, failure, cancellation,
timeout, and completion-latency metrics.
- Add the split labels to Workflow Task and Activity schedule-to-start
latency and poll-time task-dispatch latency.
- Continue emitting the existing combined `worker_version` label on
task-dispatch latency for compatibility.
- Reuse `metrics.breakdownByBuildID` as the task-queue-scoped
cardinality gate for the new labels.
## Which metrics changed
Every metric below now emits both `worker_deployment_name` and
`worker_build_id` when `metrics.breakdownByBuildID` is enabled and
deployment attribution is available. When the gate is disabled or
attribution is unavailable—including unversioned tasks and timeouts
before a worker starts—both labels remain present with empty values.
### Workflow Task outcomes
- `workflow_tasks_completed`
- `failed_workflow_tasks`
### Activity outcomes
- `activity_success`
- `activity_fail`
- `activity_task_fail`
- `activity_cancel`
- `activity_timeout`
- `activity_task_timeout`
### Activity latency
- `activity_end_to_end_latency` (deprecated; use
`activity_start_to_close_latency` instead)
- `activity_start_to_close_latency`
- `activity_schedule_to_close_latency`
### Task-routing latency
- `task_schedule_to_start_latency`
- `task_dispatch_latency` (continues to emit the existing combined
`worker_version` label as well)
## Why
- To improve user-experience for worker-versioning by having more
insights
- Worker Deployment name and build ID need to be independently
filterable. Emitting them separately avoids requiring consumers to parse
the combined `worker_version` value. This PR preserves `worker_version`
on `task_dispatch_latency` for compatibility and does not remove or
deprecate it.
## How each metric was tested
The functional tests run real versioned Workflow and Activity workers
with `metrics.breakdownByBuildID` enabled. They assert that emitted
server metrics contain the actual Worker Deployment name and build
ID—not merely that the label keys exist.
| Functional scenario | Metrics verified | What is asserted |
|---|---|---|
| Workflow and Activity task dispatch |
`task_schedule_to_start_latency`, `task_dispatch_latency` | Both
Workflow and Activity task series contain real deployment/build values
when enabled. Disabled and unversioned cases emit empty values. |
| Successful Workflow and Activity | `workflow_tasks_completed`,
`activity_success`, `activity_end_to_end_latency`,
`activity_start_to_close_latency`, `activity_schedule_to_close_latency`
| Successful completion series contain real deployment/build values. |
| Terminal Activity failure | `activity_task_fail`, `activity_fail`,
`activity_end_to_end_latency`, `activity_start_to_close_latency`,
`activity_schedule_to_close_latency` | Both the failed attempt and
terminal-failure series retain real worker attribution. |
| Activity cancellation | `activity_cancel` | The cancellation series is
attributed to the worker that started the Activity. |
| Terminal Activity timeout | `activity_task_timeout`,
`activity_timeout` | Both attempt-level and terminal timeout series
retain the started worker’s deployment/build values. |
| Failed Workflow Task | `failed_workflow_tasks` | A real versioned
Workflow Task is polled and failed; the resulting series contains the
poller’s deployment/build values. |
### Local end-to-end validation
The PR server was also run locally with the Worker Deployment versioning
canary and a `bench-go` workload.
Prometheus was scraped directly to verify that:
- real `worker_deployment_name` and `worker_build_id` values were
emitted;
- versioned and unversioned series were both accepted without
inconsistent-label errors;
- `activity_success`, `activity_task_fail`,
`activity_end_to_end_latency`, `activity_start_to_close_latency`, and
`activity_schedule_to_close_latency` carried the expected real
deployment/build values.
The remaining failure, cancellation, timeout, and failed-Workflow-Task
paths are covered by the functional tests above.
NEW: Also tested each of these 13 metric families in a cloud test cell
with sample metrics pasted here:
https://grafana.tmprl-internal.cloud/d/shdb6gf/new-dashboard?orgId=1&from=2026-08-27T00:00:00.000Z&to=2026-08-27T23:59:59.000Z&timezone=utc
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Broad metrics surface area across History and Matching with
cardinality gated by config; deployment attribution on timeouts and
eager starts changes which tag values appear on existing metric names.
>
> **Overview**
> Adds **`worker_deployment_name`** and **`worker_build_id`** to History
and Matching metrics so worker versioning can be filtered without
parsing the combined **`worker_version`** label.
**`metrics.breakdownByBuildID`** (task-queue scoped) controls whether
those tags carry real values or empty strings;
**`task_dispatch_latency`** still emits **`worker_version`** for
compatibility.
>
> **Workflow tasks:** Completion and failure counters
(`workflow_tasks_completed`, `failed_workflow_tasks`) now go through
shared helpers that attach versioning behavior plus deployment tags from
the poller’s **`DeploymentOptions`**.
>
> **Workflow activities:** Success, failure, cancel, timeout, and
related latency metrics get the same split labels via
**`VersioningMetricContext`** on respond paths (from request deployment
options), timer-driven timeouts (from **`LastDeploymentVersion`** when
an attempt had started), and schedule-to-start latency on
**`RecordActivityTaskStarted`**. Eager activities started during WFT
completion now record the completing worker’s deployment on the started
event.
>
> **Standalone activities (CHASM):** Completion/timeout/cancel paths use
**`completionMetricsHandler`**, which always includes the deployment
label keys with empty values until standalone versioning exists—keeping
Prometheus label-set parity with workflow-embedded activities.
>
> **Other:** **`AddActivityTaskStartedEvent`** clears stored deployment
when the poller is unversioned; Matching **`task_dispatch_latency`**
adds the new tags alongside **`worker_version`**.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
1c60b05f06. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What
This fixes the Schedule invariant scanner's false-positives coming from
replication. Due to carelessness it was firing on the passive side
because I forgot to filter this out, and for a while during
post-replication disconnection, the task processing will cease. Also
adds a small check for Described schedules to filter out visibility
drift.
## How
- Adds a guard for only checking active NS
- Adds a describe check for the next fire time, so that
visibility-delayed schedules are excluded
## Risks:
- That I make a mistake and break the scanner
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- add transition-history XDC E2E coverage for every buffered event that
current production code can emit
- create buffered events naturally through workflow/activity APIs,
workflow commands, transfer and timer tasks, child callbacks, Nexus
callbacks, and conflict reapplication
- hold a workflow task, fail over, let its timeout flush the old active
cluster's buffer, and then release replication to exercise conflict
resolution
- prove every flushed event is persisted with a real event ID on the
non-current losing history branch
- verify cherry-pickable inputs are reapplied to the winning branch and
branch-dependent outcomes are skipped
- reproduce the Nexus #10986 conflict shape: reapply a shared-operation
completion while skipping an operation that exists only on the losing
branch
- keep all test infrastructure in
`buffered_events_replication_helpers_test.go`; no production or testcore
code is changed
## Coverage
The scenarios naturally cover all 30 production-reachable buffered event
types:
- activity started, completed, failed, timed out, and canceled
- timer fired
- workflow cancel requested and signaled
- workflow options updated, paused, and unpaused
- external signal/cancel success and failure callbacks
- child start failure, child started, and all five child terminal
outcomes
- update admitted through real conflict reapplication
- Nexus started, completed, failed, canceled, timed out, and both
cancel-request outcomes
Four values in the buffered-event set cannot be naturally buffered on
current `main` and are intentionally not fabricated:
- `WORKFLOW_EXECUTION_UPDATE_REJECTED`,
`WORKFLOW_PROPERTIES_MODIFIED_EXTERNALLY`, and
`ACTIVITY_PROPERTIES_MODIFIED_EXTERNALLY` have no production emitter
- `WORKFLOW_EXECUTION_TIME_SKIPPING_TRANSITIONED` has an emitter, but
its production precondition rejects workflows with a pending workflow
task
## What the tests prove
- expected events first exist in mutable-state buffering with
`BufferedEventID`
- failover creates a winning branch before the old active cluster's
replication is released
- workflow-task timeout flushes the old active cluster's buffer
- every expected event is found on the non-current losing branch with a
positive, non-buffered event ID
- a naturally buffered marker signal proves the losing batch passed
through conflict reapplication
- signals and updates that are eligible for reapplication reach the
winner
- activity, timer, child, external-command-result, Nexus
cancel-request-result, and losing-only Nexus events remain on the losing
branch when the winner lacks the state needed to apply them
- shared Nexus operation outcomes are reapplied by scheduled event ID,
while losing-only operations are skipped
- current histories converge after the relevant replication tasks are
released
## Determinism
- replication is intercepted and blocked only for the workflow under
test
- the next workflow task is explicitly created and polled before
callbacks are released
- Nexus handlers use response barriers released only after the held
workflow task is confirmed
- losing branches are read directly from persistence and identified by
their expected events
- assertions poll observable conditions instead of relying on fixed
sleeps
- the pause feature flag is overridden only for the test that exercises
pause/unpause
- scenarios skip when transition history is disabled; Nexus conflict
scenarios also skip for the CHASM implementation
## Validation
- repository `gci` formatting and `git diff --check`
- `GOLANGCI_LINT_FIX=false GOLANGCI_LINT_BASE_REV=HEAD~ make lint-code`
- XDC package compilation with `test_dep`
- focused transition-history E2E runs for mixed inputs/update
reapplication, activity outcomes, child outcomes, external workflow
outcomes, and Nexus outcomes
- CI PostgreSQL XDC, unit, integration, mixed-brain, formatting, and all
linter checks pass
## What changed?
- Add the Temporal failure response capability header to Nexus
cancellation requests in both HSM and CHASM executors.
- Respect the `nexusoperation.useNewFailureWireFormat` dynamic config
when adding the header to cancellation requests.
## Why?
Nexus start requests explicitly advertise support for Temporal failure
responses, but cancellation requests relied on the request header map
being mutated elsewhere.
[#10720](https://github.com/temporalio/temporal/pull/10720) and
[#10746](https://github.com/temporalio/temporal/pull/10746) introduced
defensive copies of Nexus request headers. That exposed the missing
capability header on cancellation requests, causing SDK workers to use
the legacy failure wire format for cancel handler failures.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Cancellation requests now advertise support for Temporal failure
responses by default. The existing dynamic config can disable the
behavior if needed.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed
- Add Worker Deployment name and build ID labels to
`worker_deployment_version_not_found_during_delete`.
- Assert both label values in the existing activity test.
## Why
The counter diagnoses stale Deployment workflow references, so it needs
to identify the exact missing Worker Deployment Version.
## Testing
- `GOWORK=off go test -tags test_dep ./service/worker/workerdeployment
-run '^TestDeleteWorkerDeploymentVersion$' -count=1`\n- `GOWORK=off go
vet -tags disable_grpc_modules,test_dep
./service/worker/workerdeployment`
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Observability-only changes to metric tags and log fields on an
already-handled NotFound path; no change to deletion behavior.
>
> **Overview**
> When deleting a worker deployment version hits a **NotFound** from
history (stale version workflow), the
**`worker_deployment_version_not_found_during_delete`** counter now
records **worker deployment name** and **build ID** tags, in addition to
namespace, so dashboards can pinpoint which version was missing.
>
> The same path updates the warning log to use **`versionObj`** for
deployment name and build ID, **`namespace.Info().GetName()`** for
namespace, and **`args.GetRequestId()`** for the request ID. The
activity test now asserts the new metric tag values.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
8776d7fd78. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## Summary
- mock the normal StartWorkflow and WatchWorkflow activities exercised
by the disabled-migration test
- mock migration safely and assert that it is never invoked
- preserve the active schedule and continue-as-new behavior under test
This prevents the test environment from invoking activities through a
nil receiver and logging repeated recovered panics.
## Testing
- `go test -tags test_dep -count=1 -run
"^TestWorkflow$/^TestMigrateDynamicConfigDisabledNoMigration$" -v
./service/worker/scheduler`
- `go test -tags test_dep -count=1 ./service/worker/scheduler/...`
- `make GOLANGCI_LINT_BASE_REV=origin/main GOLANGCI_LINT_FIX=false
LINT_CODE_TARGETS="./service/worker/scheduler" lint-code`
## Context:
We've seen several problems in the SchedulesV2 where we have failed to
catch genuine problems where we should re-arm the next timer and
therefore the schedule gets 'stuck' (in the sense that it's performed an
invalid state transition and now cannot continue.
We've done a lot of fancy exploration to try and solve for this and I
hope we can land some of these. But in the mean-time I was pretty
frustrated that it's still very easy to make these mistakes (there's a
lot here I want to explore in testing, but it's very much a source of
sharp edges).
Concretely, I just asked the LLM to go try and find a low-effort way of
catching these and I'm pretty sure this is a success in that it's very
dumb: It just checks the state for 'stuck'ness on close. It's very
basic, but hopefully a starting-point to address the state-change
sharp-edges problem.
### LLM summary:
## How did you test it?
- [x] added unit coverage
- [ ] functional suite not yet run
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
This adds a lastEventTime tracking value which keeps track of the last
value even if they fall out of the buffer. It's a bugfix for the current
state which can presently cause multiple idle-tasks to get assigned as
the calculation of when to set the idle-task can jump around as actions
are removed from the buffer. By tracking the last seen value, it can be
forced to be a monotonic value.
### LLM summary:
`getLastEventTime` maxed over `Invoker.recentActions()`, which reads
`BufferedStarts` — truncated by `applyCompletedRetention` to the last
`recentActionCount` (10) completed entries, **ordered by `CloseTime`**.
Whenever completion order differs from start order, evicting the start
that held the largest `StartTime` drags the value, and therefore the
idle deadline, backwards.
**Unbounded growth of the mutable state blob and the timer queue.**
Idle tasks are reaped only via `Validate`, whose predicate
(`idleExpiration.After(scheduledTime)`) drops a task only when the
deadline moved *later*. A flat or regressed deadline invalidates
nothing, so every Generator tick leaves another task behind — one per
action between idle ticks.
### Observed in production
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
New file `chasm/lib/scheduler/scheduler_idle_deadline_test.go`, 10
tests. Verified they fail without the fix by reverting both production
changes — 9 of 10 fail, reproducing the production numbers:
```
TestGeneratorTask_IdleTask_NotReArmedWhenDeadlineUnchanged
expected: 1 actual: 21 # 20 no-op ticks -> 21 tasks, one per tick
TestGeneratorTask_IdleTask_DoesNotAccumulateAcrossActionBurst
idle deadline regressed at action 10:
2026-08-28 04:43:25.199231 -> 2026-08-28 04:43:25.197731
```
A 1.5ms regression at action 10 — landing exactly where
`recentActionCount = 10` first evicts. The tenth test
(`NotArmedWhileSpecHasWork`) passes before and after by design; it
guards against the skip logic swallowing the non-idle branch.
Coverage: the root cause independent of the idle machinery (recomputed
value regresses on eviction, floored read does not, mark never moves
backwards, nil mark degrades to old behaviour); accumulation (repeated
no-op ticks, and 43 actions completing one at a time through the
retention window); stuck-open safety for the skip guard (deadline
advance re-arms and reaps, pause/unpause re-arms, live spec arms
nothing); `Validate` no longer entering the regressed branch after an
eviction; and a sanity test that the snapshot helper sees armed tasks so
the count assertions can't pass vacuously.
Whole package green, stable under `-count=3 -test.shuffle=on`, `make
lint-code` 0 issues, `go build ./...` and `go test ./chasm/...` clean.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
1. Object leak reports now includes sorted heap addresses for unexpected
retained objects.
2. The leak test can optionally capture a raw heap dump with
`LEAK_HEAP_DUMP=1`.
## Why?
Local debugging; it helps identify the runtime root retaining a reported
object.
## Potential risks
Raw heap diagnostics can be large and may contain sensitive process
data. They are only captured after an object leak failure when
`LEAK_HEAP_DUMP=1` is explicitly set.
## What changed?
Add data race summary to CI report
## Why?
Notify when CI detects data race issues in main.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
When using `git worktree` for agentic workflows during development, the
golangci-lint cache is always cold.
That means the first invocation scans the entire codebase (consuming all
CPU cores) unnecessarily, since almost all lints can be found by just
scanning the changed packages. The exceptions are type-based checks, but
they are rare.
This PR speeds up the agentic development cycle greatly for local
development; and anything that slips through is still caught in CI where
the full codebase is checked.
Example benchmark for cold cache with 1 changed file with 1 finding:
- `lint-code-fast`: 8s
- `lint-code`: 14min
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Dan Davison <dan.davison@temporal.io>
**What**
Handle terminal errors from asynchronous task-queue delete propagation
without marking propagation complete, so the Version workflow stays open
until cleanup succeeds.
**Why**
Retryable activity failures retry indefinitely, but a terminal error can
escape. The current code ignores that error and allows the Version
workflow to complete without confirming task-queue cleanup.
Note: We don't know how this non retryable case can happen. We observed
1 setup in this state; so this PR exports a metric for us to observe
when it happens.
**How did you test it?**
Workflow test covering a non-retryable cleanup failure and verifying the
Version workflow remains open.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
**1. Delete execution replication tasks carry a failover version.**
Stamped with the source cluster's
failover version at generation (`shard/context_impl.go:1013`), carried
in the existing
`ReplicationTaskInfo.version` (no proto change), and skipped on apply
when older than the target's
namespace failover version (`executable_delete_execution_task.go:130`).
Still applied: unversioned
tasks (queued pre-upgrade), tasks newer than the target's namespace
entry, and deletions synthesized
by versioned-transition tasks.
**2. `DeleteWorkflowExecution` is rejected on a cluster passive for the
workflow**
(`workflow_handler.go:2513`), with the usual `NamespaceNotActive`. In
the frontend, because the history
path is shared with replication apply, which must delete on passive
clusters — that path, the
delete-namespace worker, and admin force-delete are untouched.
## Why?
An unversioned delete task generated before a failover kept being
applied afterwards, deleting a run
the new active cluster owns.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
```
temporal --address :7233 workflow -n global-ns delete -w pay-invoice-0 -r 019fcf5b-996e-7416-9a2d-4ffb0ae7fc13 --grpc-meta xdc-redirection=false
WARNING: Deleting Workflow Executions in a global Namespace removes them from all replicas. Requests sent to a passive cluster are forwarded to the active cluster by default; to target the passive cluster directly, specify `--grpc-meta xdc-redirection=false`.
Delete Workflow "pay-invoice-0" with Run ID "019fcf5b-996e-7416-9a2d-4ffb0ae7fc13"? y/N y
Error: failed to delete workflow: Namespace: global-ns is active in cluster: cluster-b, while current cluster cluster-a is a standby cluster.
```
## Potential risks
- A deletion issued just before a failover is dropped on targets;
cleanup waits for the new active
cluster's retention timer. Delayed, not leaked — intended trade-off.
- API behavior change: deleting against a passive cluster now fails,
including batch delete
(`batcher/activities.go:704`). No killswitch; admin force-delete is the
escape hatch.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
- Keep attaching completion callbacks to every CHASM scheduler workflow
start, including actions whose resolved overlap policy is `ALLOW_ALL`,
so start requests remain safe across rolling upgrades.
- Exclude scheduler-wide last-completion result/failure input from new
`ALLOW_ALL` actions.
- After `StartWorkflowExecution` succeeds, remove the new `ALLOW_ALL`
`BufferedStart` in the same transaction. Separately, copy its start data
to `ScheduleInfo.RecentActions` as a start-only `RUNNING` history
record.
- Resolve an unspecified pending-start policy while building the
V1-to-V2 migration request, so every V2 dispatch and retry uses the
policy persisted on the buffered start.
- For a pre-existing callback, use the policy stamped on that start
rather than resolving against the schedule's current policy; an
unstamped V1-migrated running workflow remains tracked, as it was in V1.
- Preserve compatibility for previously persisted `ALLOW_ALL` callbacks
without updating last-completion state or `PauseOnFailure`.
- Carry start-only history through CHASM-to-V1 migration.
- Merge start-only and completion-tracked history by actual start time,
retaining the newest ten actions across both sources.
- Re-run generation after recording a start-only action, so a finite or
manual-only schedule rearms its idle timer from that start.
## Behavior
This makes CHASM match modern V1:
- A new `ALLOW_ALL` workflow starts normally, but is absent from
`DescribeSchedule.Info.RunningWorkflows` and does not consume
active-buffer capacity.
- It appears in `DescribeSchedule.Info.RecentActions` (and List's recent
actions) with its start time, workflow execution, and `RUNNING` status.
- Its completion callback remains attached for rolling-upgrade
compatibility. Because the start has already moved out of active
buffered state, the callback is ignored and success or failure cannot
change shared completion input or `PauseOnFailure`; its recent status
therefore remains `RUNNING`.
- For a final `ALLOW_ALL` action, an idle task armed before the start is
invalidated by the newer start time; generation immediately arms its
replacement, so the schedule still closes after `IdleTime`.
- If an older handler retained an `ALLOW_ALL` start, its callback
remains a compatibility path: the terminal action record is retained,
but its completion cannot update shared completion state or pause the
schedule once handled by this version.
- `DescribeSchedule.Info.RecentActions` is ordered by actual start time
and bounded to the newest ten actions across start-only and
completion-tracked history. Newer retained actions evict the oldest;
completion does not remove a start-only action.
- Non-`ALLOW_ALL` actions remain active and appear in `RunningWorkflows`
until their completion is handled.
- A V1 pending start with an unspecified override snapshots the
schedule's effective policy when its V2 migration request is built.
- A V1-migrated `RunningWorkflows` entry has an unspecified policy but
is nevertheless tracked: its completion updates sequential state and may
pause the schedule, regardless of the schedule's current `ALLOW_ALL`
default.
## Why?
`ALLOW_ALL` actions are independent executions. Tracking their
completions made scheduler-wide last result/failure and `PauseOnFailure`
depend on callback arrival order, and retaining them as active could
affect overlap and capacity behavior.
Callbacks remain attached because `StartWorkflowExecution` deduplicates
by request ID without reconciling callback differences. Keeping the
request callback-compatible prevents a mixed-version retry from
retaining a start that waits for a callback the workflow never received.
## How did you test it?
- [x] added unit coverage
- [x] added shared V1/CHASM functional coverage
Commands:
- `go test -tags test_dep ./chasm/lib/scheduler/... -count=1`
- `go test -tags test_dep ./tests -run
'^TestSchedule(CHASM|V1)/TestAllowAllDescribeContract$' -count=1`
- `make fmt-imports`
- `git diff --check`
- `env GOCACHE=/tmp/sch-038-gocache go vet -tags
disable_grpc_modules,,test_dep -vettool=.bin/errortype
-style-check=false ./chasm/lib/scheduler/...`
The functional test uses workflow signals to control completion. It
asserts counters, buffer size, recent-action status/timestamps, active
workflows, and failure/pause isolation across both backends. A CHASM
functional idle-close case verifies that a final `ALLOW_ALL` action
still closes after `IdleTime`. The migration regression and
callback-reason metric test drive real component transactions through
the CHASM test engine.
`make lint-code` currently exits before analysis with `no go files to
analyze` from its `--new-from-rev` filter, despite the Go diff; package
`go vet` is clean.
## Additional observability
Ignored callbacks are tagged as either `unrecognized_request_id` or
`already_completed`. A newly started `ALLOW_ALL` callback is expected to
be unrecognized after its buffered start moves to start-only history. An
already-completed callback is a valid redelivery (for example, after a
workflow reset). ~~Both preserve scheduler state while emitting a
warning and counter increment.~~ Missing request IDs are now metric-only
because they include expected `ALLOW_ALL` callbacks; known
`already_completed` redeliveries still emit the warning, event, and
counter.
## Potential risks
Keeping callbacks attached avoids permanently orphaning buffered starts
when old and new binaries race on the same request ID. During a rolling
upgrade, however, old and new requests still differ in whether
scheduler-wide last-completion result/failure input is included for
`ALLOW_ALL`; the request that wins deduplication determines whether that
workflow receives the legacy input. An older callback handler can also
temporarily apply the legacy completion and pause semantics. These
mixed-version differences end once the rollout completes, and the
callback ensures an older handler cannot wait indefinitely.
New `ALLOW_ALL` terminal status is intentionally not reflected in
schedule Describe/List results, as in modern V1. ~~Its expected late
callback is recorded as `unrecognized_request_id`, which adds callback
delivery plus warning/metric volume compared with omitting callbacks.~~
Its expected late callback remains recorded as
`unrecognized_request_id`, but only as a metric; warning and event
logging are suppressed until these callbacks can be safely omitted.
Migration resolves an unspecified pending-start policy at the V1-to-V2
boundary. If the schedule policy changes while that start remains
pending and the schedule then rolls back to V1, the explicit migrated
policy is preserved instead of inheriting the newer schedule policy.
This is a narrow semantic difference that keeps V2 dispatch and retry
behavior stable.
---------
Co-authored-by: David Porter <david.porter@temporal.io>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## What changed?
Preserved the caller-provided visibility scope when starting type-based
batch activity unpause operations. The activity-type predicate is now
safely combined with the original query.
## Why?
The server previously replaced the caller’s visibility query with the
activity-type predicate. This could broaden the batch scope and unpause
activities in workflows the caller did not select.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Type-based unpause batches now strictly honor the caller’s visibility
scope, so they may process fewer workflows than before, but this is the
correct design
## What changed?
- Added a reusable parent-child XDC test harness for controlling
replication task application, delay, ordering, and omission around
namespace failover.
- Added five functional scenarios covering orphaned children, missing or
incomplete children, task discard, missing parent completion, and parent
resend recovery.
- Added unit tests for the replication gate and
legacy/transition-history task decoding.
Configured two history shards per cluster and placed parent and child
workflows on different shards.
- Updated XDC synchronization checks to support multiple history shards.
## Why?
Parent-child replication failures depend on rare cross-shard ordering
and failover timing, making them difficult to reproduce reliably.
These tests deterministically construct the relevant partial states
while still exercising real Temporal services, persistence, replication,
verification RPCs, task retry/discard behavior, and namespace failover.
The harness also makes future scenarios easier to add and review.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed
Adds `TestLogger.StartCapture` / `StopCapture` to records log calls and
make them queryable.
## Why
We want ability to verify certain logs where emitted to verify
observability. See #11689 for first use case.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Go 1.27 prerequisite that applies the `errorsastype` Go fixer and its
required error-interface updates.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
> **Part 3 of a 5-PR series** building to replication stream namespace
isolation (a restructuring of #10147): read buffer → reader group → lane
protocol → isolation manager → sender isolation.
> #11302 (reader group) has merged, so this PR's diff is now standalone
against `main`. · **Next in series: #11304** (isolation manager).
## What changed?
The wire-level building blocks for per-namespace lane isolation,
receiver side only:
- **Proto**: `SyncReplicationState` gains `throttle_high_namespace_ids`
(namespaces the receiver reports as overwhelming the HIGH lane — the
priority lives in the field name, so a future LOW extension adds its own
field rather than widening this one), `isolated_lane_states` (per-lane
applied watermarks keyed by namespace ID — with the documented caveat
that a missing key is ambiguous between "not tracked yet" and "retired
and drained", so a sender must never treat absence alone as drain
proof), and `supports_namespace_isolation` (capability advertisement, so
a sender never emits lane-tagged traffic to a receiver that would
misroute it). `WorkflowReplicationMessages` gains
`isolated_namespace_id` — when set, the batch belongs to that
namespace's dedicated lane — and `retire_isolated_lane`, marking a
lane's final message.
- **Receiver**: lane-tagged batches route to lazily-created
per-namespace task trackers. Each lane is its own monotonic stream for
the life of the connection — there is no rewind or rotation machinery,
because the sender-side design (later in the series) gives every lane a
single owner cursor that never goes backwards. Member-lane watermarks
fold into the overall ack minimum (cleanup safety) and are reported per
lane; member-lane backlogs count toward HIGH flow control. Lane
lifecycle is defensive about ordering: a batch's tasks are tracked
BEFORE its retire flag is applied (so the concurrent ack loop can never
delete a lane whose final batch is mid-track), a retiring lane is only
dropped once it is drained AND has tracked at least one batch, and
non-retire traffic arriving on a retiring lane revives it (the sender
re-isolated the namespace before the lane drained). Lane-tagged traffic
at any priority other than HIGH is a protocol violation and fails the
stream rather than silently mis-acking (isolation splits the HIGH lane
only). Lanes created concurrently with `Stop()` are pre-cancelled so no
tasks run after shutdown.
- **`NamespaceThrottler`** interface (default: noop, via fx) observes
per-namespace HIGH-priority task load and decides which namespaces to
report.
The sender does not tag lanes yet, so this is inert until the
sender-side isolation lands.
## Why?
Isolation needs a wire contract before the sender can use it: capability
advertisement, per-lane routing and progress reporting, and the
throttled-namespace feedback channel. Landing the receiver first makes
mixed-version clusters safe by construction. Compared to #10147, lanes
are per-namespace rather than shared per severity tier — which is what
eliminates that design's cursor rewinds and the
watermark-regression/tracker-rotation protocol this PR previously needed
to compensate for them.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s) — lane routing (priority routing when
unset, per-namespace tracker identity, non-HIGH rejection), retirement
lifecycle (drop once drained, never-tracked retiring lane survives the
ack snapshot, revive on re-isolation traffic, fresh lane after drop),
and post-Stop lane creation being pre-cancelled
- [x] added new functional test(s) — exercised end-to-end by the xdc
test in the final PR of the series
## Potential risks
Inert until a sender emits `isolated_namespace_id`, which is gated
behind both a config flag and the capability advertisement.
Receiver-side lane state is bounded by the sender's isolation cap (final
PR).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches replication ack watermarks and stream failure paths on the
passive cluster; lane mis-handling could stall cleanup or mis-ack,
though lane-tagged traffic is not sent until follow-up sender work.
>
> **Overview**
> Adds the **wire contract and receiver behavior** for per-namespace
HIGH-lane isolation before the sender starts tagging traffic.
>
> **Proto:** `SyncReplicationState` now carries
`throttle_high_namespace_ids`, per-namespace `isolated_lane_states`, and
`supports_namespace_isolation`. `WorkflowReplicationMessages` adds
`isolated_namespace_id` and `retire_isolated_lane` so batches can be
routed and retired on dedicated lanes.
>
> **Receiver:** Lane-tagged HIGH batches use lazily created
per-namespace task trackers (monotonic per connection). Member-lane
watermarks fold into the overall ack minimum and are reported per
namespace; member backlog counts toward HIGH flow control. Acks include
shard-scoped throttled namespace IDs via a new **`NamespaceThrottler`**
(noop by default). Lane lifecycle handles retire/drain/revive and
rejects non-HIGH lane traffic.
>
> Sender emission of lane tags is not in this PR, so behavior stays
inert until a later change gates on capability advertisement.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
688a5173f0. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
## What changed?
Back `testSequentialTaskQueue` with an unbounded slice instead of a
3000-capacity channel, so `Add` no longer blocks.
## Why?
`common/tasks` deadlocked and timed out after 15m on main ([run
32773080605](https://github.com/temporalio/temporal/actions/runs/32773080605)).
`SequentialScheduler.Submit` calls `Add` from inside `PutOrDo`'s
callback, which runs under the shard write lock. Once the test queue's
channel filled, the submitter blocked in `Add` while holding that lock,
and the only worker was blocked on the same lock in `RemoveIf`. Holding
the lock across the add is deliberate — it keeps the add atomic against
`RemoveIf`'s empty-check — so the invariant is that `Add` must not
block, and only the test queue violated it.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
Adds generic `await.Rcv` and `await.Snd` helpers that bound blocking
channel operations by the test context and report closure or
cancellation clearly.
Replaces both previous helpers and existing unsafe channel rcv/snd
across tests.
## What changed?
- Moves `activity.linkValidator` into `common/links`.
- Moves `callback.Validator` into `common/callbacks`
In addition, this PR performs some minor refactorings for consistency
and clarity.
- Moved some `links.Validator`-specific tests from
`chasm/lib/activity/validator_test.go` elsewhere.
- Introduced a `callbacks.ValidatorConfig` to bundle all of the specific
settings. (Since we'll need to wire 3+ more parameters when updating the
`callbacks.Validator` to support worker callbacks.)
> The singular package names `common/link` or `common/callback` would be
more consistent. But `common/links` already existed, there are other
pluralized ones like `common/enums` or `common/headers`. And IMHO, the
plural seems a little more applicable since the validations are only on
groupings of links or callbacks.
## Why?
The `activity.linkValidator` and `callback.Validator` types are great,
but they aren't able to be used as across other CHASM components as
easily. Moreover, `callback.Validator` uses types that are exposed from
the CHASM `callback` package, it will lead to circular dependencies in
the future. (I'm hitting this now in PRs for landing worker callbacks.)
Moving the `commonpb` protobuf validation into `common/` means we can
better separate the the distinction between validation logic and the
CHASM executions that rely on it.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
This should just be a standard refactoring. There should not be any new
validation checks enabled on codepaths where they weren't already
present. (Or in test cases, we initialize fields of
`callback.ValidatorConfig` that weren't used before.)
## What changed?
When `dispatchForExistingWorkflow` finds `currentRunID == ""`, it treats
the missing current execution record as a deletion issued by the user,
and handles the target run accordingly:
- **Running target:** return an internal invariant error. Replication
tasks are ordered, so a deletion cannot overtake the close event — a
running run with no current record is conceptually impossible. The task
retries and is eventually sent to the DLQ.
- **Closed target:** apply the target as a zombie with
`UpdateWorkflowModeBypassCurrent`, leaving the current record missing.
The current record is never re-established here, because that would
resurrect a user-deleted workflow.
If the task carries a new run (continue-as-new / cron / retry
successor), it is **not dropped**: when it is not already present
locally it is persisted as a zombie via bypass-current, so **no history
is lost** — but it never becomes the current run. A zombie is not a dead
end: when the successor's own close event later replicates, `ZOMBIE ->
COMPLETED` is a valid transition and it converges like any other run
(intermediate events keep it a zombie, so they never hit the
running-invariant error). While open, a zombie is invisible to
visibility, so it never shows up as a stray running workflow.
For example, given `r1 -> r2 -> r3` (continued-as-new chain) followed by
deletion of the current run `r3`, a later replication update of `r1` may
carry `r2` or `r3` as its new run. Either way the successor is persisted
as a zombie, never as current, so the user's deletion intent is
preserved while its history is retained. The current record is
(re)established only by a separately replicated new/reset run through
its own new-workflow path.
## Why?
Cross-cluster deletion removes the passive cluster's current execution
record while closed run rows can survive until retention.
Re-establishing a current record from a later replication task would
resurrect a deliberately deleted workflow, so the passive cluster
preserves the deletion intent: closed historical runs converge as
zombies, and only a
separately replicated new/reset run may establish a new current record.
Carried successors are still persisted (as zombies) so their history is
never lost, and they close normally once their own close events
replicate.
## How did you test it?
- [x] built
- [x] run locally and tested manually — reproduced the original
duplicate-run failure with
`r1 -> r2 -> r3`, deleted `r3` on both clusters, then reset `r1`
- [x] covered by existing tests
- [x] added unit tests — missing-current dispatch for the closed,
rebuilt, and impossible
running cases, plus a carried new run in both the persist-if-absent and
skip-if-already-present paths
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What changed?
Fixes a build error, seemingly introduced when multiple changes were
merged automatically after approval.
## Why?
Because a broken build stops the flow of spice. And the spice must flow.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None. But there might be other issues, I'm curious to see if there are
any other issues flagged by CI/CD.
## What changed
The HSM Nexus executor logged outbound call failure logs were missing
tags.
## Why
`chasm/lib/nexusoperation` already logs exactly these fields via
`invocationTraceContext.tags()`,
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed
All three warnings in `ConvertNexusLinksToProtoLinks` interpolated the
link type into the message, and two also embedded the link URL. Moved
both into tags and kept the message static.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
The `commonpb.Callback`-variant of `commonpb.Callback_Internal` is
unused and should be removed entirely. This PR removes the remaining
unnecessary instances of that type.
(My actual motivation is really to avoid a larger diff later, since
introducing Worker-variant callbacks will start returning errors when
you try to attach an `Internal`-variant callback.)
However, changing `Callback_Internal` to `Callback_Nexus` changed the
behavior of `TestDedupLinksFromCallbacks`. After scratching my head for
a while, I add a doc comment to clarify exactly what the function does,
and then updated the tests to be easier to read and understand.
## Why?
The call to `dedupLinksFromCallbacks(...)` in the testcase did _not_
dedupe the links attached to `callbacks[0]` because it was the
`commonpb.Callback_Internal` variant. (Relying on a quirk of the
function only filtering callbacks from Nexus-variant callbacks.)
I kept that behavior in, but added a couple more test scenarios to
provide better coverage and clarify the expected behavior.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None.
## What changed?
Pass one test-owned context through namespace creation, namespace cache
polling, and search attribute registration during functional test setup.
## Why?
Reusing the test context avoids creating independent timeout contexts
for each setup RPC and ties their resources to the test lifecycle.
## What changed?
- Reorganize `chasm/lib/activity`
## Why?
- Improve navigability and codebase comprehensibility
## How did you test it?
- [x] covered by existing tests
## What changed?
ForkHistoryBranchResponse now carries BaseBranchToken, and the two reset
paths rebuild through it when the store set it. A store that doesn't
(Cassandra, SQL) leaves it nil and both call sites fall back to the
token the caller already had, so behavior is unchanged everywhere else.
## Why?
ForkHistoryBranch can modify the base branch token in ways the caller
may not have visibility into. This PR fixes it so that the changed token
is returned to the caller.
## How did you test it?
- [ ] built
- [ ] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
`GetWorkflowExecutionHistory` and `GetWorkflowExecutionHistoryReverse`
now check `branch_token` in the page token against the token in mutable
state.
## Why?
To confirm if it is still the correct branch after conflict resolution.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Adds metrics for how often and by how much queue slices
fail to narrow their predicate, and how large persisted queue state
actually is.
- `queue_slice_pending_keys` — histogram, recorded on every narrowing
attempt (declined or
succeeded). This is the distribution
`queueShrinkPredicateMaxPendingKeys` should be sized against.
- `shard_info_size` / `queue_state_size` — histograms recorded when a
shard record is actually
written, giving whole-record and per-category size.
- `queue_state_size_total` / `queue_slice_count_total` — counters paired
with the histograms above
(and with the existing `queue_slice_count`), so an exact bytes-per-slice
ratio is possible.
- `queue_slice_count` gains a `task_category` tag (previously untagged
beyond `operation`).
These are only metrics changes - no behavior changes.
## Why?
A slice only narrows its predicate below
`queueShrinkPredicateMaxPendingKeys` (10) pending
namespaces; above that it stays universal and re-reads the whole range
every time. Raising that
threshold safely requires knowing the pending-key distribution and the
persisted size.
This PR is the baseline for evaluating a follow-on encoding.
There are two counters because this server's tally-backed Prometheus
reporter doesn't preserve the
true recorded value when a histogram flushes — it replays each sample as
its bucket's upper bound,
so a histogram's `_sum` has no more precision than its buckets.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] run locally and tested manually (`queue_predicate_resolution_loss`
confirmed live against a local server under forced
narrowing-decline conditions)
## Potential risks
## What changed?
- emit remote cluster and namespace replication lifecycle records under
`namespace_lifecycle`
- retain compatibility aliases for specialized event-name constants with
TODO cleanup
- update tests and shared envelope documentation
## Why?
Update schema event names to match namespace lifecycle schema which
currently exist so they are properly interpreted.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Refactor NamespaceRateLimitInterceptor with functions to consume N
tokens:
- removed `tokens` overwrite argument as it's never used
- added functions to consume N tokens
The changes itself in this PR is no-op since it's introducing new
functions to the interface.
## Why?
Added flexibility to wrap `NamespaceRateLimitInterceptor`.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
