mirror of
https://github.com/temporalio/temporal.git
synced 2026-08-30 18:41:49 -07:00
main
1114 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Qian Chen
|
1863fbe665 |
Tag replicator errors by replication task type (#11760)
## What changed? `replicator_errors` now records the existing `replication_task_type` metric dimension when a namespace replication processor exhausts its task-application retries. This distinguishes values such as `NamespaceTask` and `TaskQueueUserData` without introducing a new metric. A unit test verifies that the failure counter carries the task type from the failed replication task. ## Why? This is a prerequisite for splitting task queue user data (`TaskQueueUserData`) failures from namespace metadata replication (`NamespaceTask`) alerts. Today both task kinds feed `replicator_errors` under `NamespaceReplicationTask` and are indistinguishable, so TQUD failures can be reported as namespace replication poison pills. This PR only adds the server metric label. Alert definitions will be updated separately after the label is available in deployed server versions. ## How did you test it? - [x] built - [x] covered by existing tests - [x] added unit tests - [ ] added new functional test(s) Manual two-cluster XDC E2E using `development-cluster-a.yaml` and `development-cluster-b.yaml`: - A namespace replication conflict exported `replicationTaskType="NamespaceTask"`. - A TQUD apply failure exported `replicationTaskType="TaskQueueUserData"`. |
||
|
Rodrigo Zhou
|
34c2032e01 |
Capture panics in Visibility query converter (#11800)
## What changed? Capture any panics in Visibility query converter. ## Why? Visibility query converter is complex, and at times might make assumptions that might not hold (due to bugs in the store query converter implementation for example). Capturing at top level, and returning an error instead. ## How did you test it? - [x] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) ## Potential risks |
||
|
David Porter
|
c9fd978d45 |
Fix: Schedule scanner false-positives for inactive NS (#11703)
## What This fixes the Schedule invariant scanner's false-positives coming from replication. Due to carelessness it was firing on the passive side because I forgot to filter this out, and for a while during post-replication disconnection, the task processing will cease. Also adds a small check for Described schedules to filter out visibility drift. ## How - Adds a guard for only checking active NS - Adds a describe check for the next fire time, so that visibility-delayed schedules are excluded ## Risks: - That I make a mistake and break the scanner --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Shivam
|
a4a452c003 |
Add version labels to missing deployment metric (#11799)
## What changed
- Add Worker Deployment name and build ID labels to
`worker_deployment_version_not_found_during_delete`.
- Assert both label values in the existing activity test.
## Why
The counter diagnoses stale Deployment workflow references, so it needs
to identify the exact missing Worker Deployment Version.
## Testing
- `GOWORK=off go test -tags test_dep ./service/worker/workerdeployment
-run '^TestDeleteWorkerDeploymentVersion$' -count=1`\n- `GOWORK=off go
vet -tags disable_grpc_modules,test_dep
./service/worker/workerdeployment`
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Observability-only changes to metric tags and log fields on an
already-handled NotFound path; no change to deletion behavior.
>
> **Overview**
> When deleting a worker deployment version hits a **NotFound** from
history (stale version workflow), the
**`worker_deployment_version_not_found_during_delete`** counter now
records **worker deployment name** and **build ID** tags, in addition to
namespace, so dashboards can pinpoint which version was missing.
>
> The same path updates the warning log to use **`versionObj`** for
deployment name and build ID, **`namespace.Info().GetName()`** for
namespace, and **`args.GetRequestId()`** for the request ID. The
activity test now asserts the new metric tag values.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
David Porter
|
539790267f |
Fix scheduler migration test activity panic (#11803)
## Summary - mock the normal StartWorkflow and WatchWorkflow activities exercised by the disabled-migration test - mock migration safely and assert that it is never invoked - preserve the active schedule and continue-as-new behavior under test This prevents the test environment from invoking activities through a nil receiver and logging repeated recovered panics. ## Testing - `go test -tags test_dep -count=1 -run "^TestWorkflow$/^TestMigrateDynamicConfigDisabledNoMigration$" -v ./service/worker/scheduler` - `go test -tags test_dep -count=1 ./service/worker/scheduler/...` - `make GOLANGCI_LINT_BASE_REV=origin/main GOLANGCI_LINT_FIX=false LINT_CODE_TARGETS="./service/worker/scheduler" lint-code` |
||
|
Kannan
|
30e0884ac0 |
Keep Version workflow open if delete propagation Activity fails for unknown reasons (#11698)
**What** Handle terminal errors from asynchronous task-queue delete propagation without marking propagation complete, so the Version workflow stays open until cleanup succeeds. **Why** Retryable activity failures retry indefinitely, but a terminal error can escape. The current code ignores that error and allows the Version workflow to complete without confirming task-queue cleanup. Note: We don't know how this non retryable case can happen. We observed 1 setup in this state; so this PR exports a metric for us to observe when it happens. **How did you test it?** Workflow test covering a non-retryable cleanup failure and verifying the Version workflow remains open. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Stephan Behnke
|
5c210f4c73 |
Replace errors.As with errors.AsType (#11674)
Go 1.27 prerequisite that applies the `errorsastype` Go fixer and its required error-interface updates. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Kannan
|
c9b99295b1 |
Treat missing Worker Deployment Version as deleted (#11696)
## What Treat `NotFound` from the Version workflow delete update as success, so the Deployment workflow can clean up its stale version reference. ## Why When a Version workflow is already closed or its history is gone, the delete update returns `NotFound`, blocking the Deployment workflow from removing the reference. Fixes #11539. ## How did you test it? Unit test covering the `NotFound` → success path and verifying other History errors still propagate. |
||
|
michaely520
|
8cadb77011 |
Emit namespace migration workflow lifecycle events (#11658)
## What changed? Adds `namespace_lifecycle` start and finish events for the namespace handover, force replication, and catchup system workflows. The events carry workflow identity and the core operation inputs. Finished events classify the result as succeeded, canceled, or failed. Force replication reports its cumulative verified workflow count and emits only one start and one finish across a continue-as-new chain. Emission uses one shared activity, the existing `system.emitNamespaceLifecycleEvents` gate, disconnected cleanup for cancellation, and workflow versioning for replay compatibility. Existing shard handover events are unchanged. ## Why? These system workflows currently have no consistent operation-level event pair, which makes it difficult to correlate a namespace migration request with its final outcome. ## How did you test it? - [x] covered by existing tests - [x] added new unit test(s) `go test -tags test_dep ./common/wideevents ./service/worker/migration` `make fmt-imports` `make lint-code` reports no issues introduced by this change; the repository-wide target still reports existing findings on current `main`. ## Potential risks The terminal event is best effort and cannot run after server-side workflow termination or workflow run timeout because those outcomes do not execute workflow cleanup. |
||
|
Stephan Behnke
|
430968b08a |
Use typed atomic values (#11675)
Go 1.27 prerequisite that applies the `atomictypes` Go fixer to use typed atomic values. |
||
|
Qian Chen
|
ba919854a1 |
Emit namespace replication lifecycle wide events (#11632)
## What changed? - Added a separate `namespace_replication_lifecycle` wide event with `created`, `received`, `processed`, and `dlqed` phases. - Included namespace/task identity, source and target clusters, source task ID, retry attempt count, deterministic task fingerprint, and the serialized namespace replication task. - Included the successful `CreateNamespaceRequest` or resolved `UpdateNamespaceRequest` as `persistence_request` on `processed`; duplicate, stale, and skipped tasks omit it. - Passed receiver-side diagnostic metadata through a typed context so the existing `TaskExecutor.Execute` and create/update handler signatures remain unchanged. - Added the dedicated, default-off `system.emitNamespaceReplicationLifecycleEvents` dynamic-config gate, checked explicitly at both the processor and processed-event emitter. - Preserved the namespace replication queue message ID as `source_task_id` when reading tasks. ## Why? Namespace CRUD events describe user-visible namespace mutations, but do not show whether the resulting namespace replication task was queued, received, applied, retried, or sent to the DLQ. These events provide that transport and processing audit trail without additional persistence reads. ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) Commands: ```text go test -tags test_dep ./common/wideevents ./common/namespace/nsreplication ./service/worker/replicator ./service/frontend make GOLANGCI_LINT_FIX=false GOLANGCI_LINT_BASE_REV=origin/main lint-code ``` Local two-cluster testing covered create, update, and failover after rebasing onto `origin/main`. Each task produced linked `created -> received -> processed` events, and `processed` contained the expected persistence request. With the dynamic-config flag off, namespace replication still completed and neither cluster emitted a matching lifecycle event. |
||
|
Fred Tzeng
|
bfb8142d21 |
Fix batch operations targeting paused executions (#11642)
## What changed? Batch operations now target Paused executions in addition to Running ones. The filter auto-appended by adjustQueryBatchTypeEnum changed from ExecutionStatus='Running' to ExecutionStatus='Running' OR ExecutionStatus='Paused', affecting all workflow batch types (terminate/signal/cancel/update-options) and activity batch types (unpause/update-options/reset/terminate/cancel). ## Why? A paused execution is still non-terminal. Users issuing a batch terminate/signal/cancel reasonably expect it to apply to paused targets, but the old Running-only filter silently skipped them. ## How did you test it? - [X] built - [ ] run locally and tested manually - [X] covered by existing tests - [X] added new unit test(s) - [X] added new functional test(s) ## Potential risks - The filter now references ExecutionStatus='Paused' on every batch operation. This is ok as it's an additive clause to the visibilty query - This would be changing behavior for batch callers as paused activities are now affected. |
||
|
Qian Chen
|
d83b12e87e |
Emit namespace CRUD lifecycle wide events (register / update / failover / delete) (#11563)
## What changed?
Adds structured `namespace_lifecycle` wide events for namespace
mutations.
### Namespace mutations
- `namespace_registered` is emitted after successful registration.
- `namespace_updated` is emitted after successful namespace updates and
includes full `before` / `after` snapshots, requested values, and
`requested_fields` identifying what the caller explicitly set.
- Local-to-global promotion is distinguished by `is_promotion` and
`promote_namespace_requested`.
- Active-cluster failover is distinguished by `is_failover` and the
active-cluster transition in `before` / `after`.
- Deprecation is represented by the `Registered` to `Deprecated` state
transition.
- Workflow-rule creation and deletion include the affected rule ID and
detail, plus force-scan and request-ID data when supplied.
- `namespace_renamed` is emitted by the delete-namespace worker when the
namespace is renamed to its tombstone name. This is the observable
deletion point because namespace deletion is local and is not replicated
as a namespace operation.
The snapshots cover namespace info, configuration, archival settings,
replication topology/state, failover versions/history, custom
search-attribute aliases, bad binaries, and workflow-rule IDs. Request
security tokens are not captured.
### Dynamic configuration
All namespace lifecycle emission is gated by the new global
dynamic-config setting:
```yaml
system.emitNamespaceLifecycleEvents:
- value: true
```
The setting defaults to `false` and is evaluated dynamically by
frontend, history, and worker producers. In addition to the new
namespace mutation events, the gate covers the existing handover-related
namespace lifecycle events:
- `shard_handover_watermark_set`
- `shard_handover_watermark_removed`
- `shard_handover_incomplete`
This PR does not introduce or change those handover event payloads; it
only makes their emission follow the same namespace lifecycle flag.
## Why?
Existing RPC metrics and logs do not provide a structured, field-level
record of namespace control-plane changes. These events provide an
attributable and queryable view of what changed, including promotion
versus failover, requested versus persisted values, rule mutations, and
delete-pipeline renames. The shared gate lets operators enable the
complete namespace lifecycle signal consistently across services.
## How did you test?
- [x] Unit tests with `-tags test_dep` for frontend emission and
disabled gating, history handover gating, delete-namespace rename
emission, migration incomplete-handover gating, dynamic config, and
common wide-event payloads.
- [x] `make lint-code` (`0 issues`).
- [x] Full local two-cluster E2E using
`config/development-cluster-a.yaml` and
`config/development-cluster-b.yaml` with a JSON event logger.
- With the flag enabled, validated register, ordinary update,
workflow-rule create/delete, deprecate, delete rename, promotion,
cluster-list update, and failover from cluster A to B.
- Also verified that the existing handover producers remain functional
when enabled: all 16 shard watermark additions and removals on cluster B
and a forced 32-shard incomplete-handover event on cluster A.
- With the flag disabled, repeated all producer paths and confirmed zero
emitted bytes. The failure-only incomplete-handover path was rerun after
disabling the flag and left both event-log counts unchanged.
- Confirmed live dynamic-config enable/disable behavior without
restarting either cluster.
### Abridged failover event
```json
{
"event_name": "namespace_lifecycle",
"phase": "namespace_updated",
"details": {
"before": {
"active_cluster": "cluster-a",
"failover_version": 1
},
"after": {
"active_cluster": "cluster-b",
"failover_version": 2
},
"requested": {
"active_cluster": "cluster-b"
},
"requested_fields": ["active_cluster"],
"is_failover": true,
"is_promotion": false
}
}
```
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
David Porter
|
471f58d2d2 |
Fix: log for scanner hitting cap (#11606)
The "overdue scan hit per-namespace check cap" warning previously only logged the namespace and the cap value, giving no way to tell which schedule the scan stopped at or how much progress it had made before hitting the cap. Add the schedule ID it stopped on, how many schedules were checked this pass, and how many anomalies had already been found, so the log line is actionable on its own. ## What changed? adds some logs ## Why? Trying to debug some odd results ## How did you test it? - [ ] built - [ ] run locally and tested manually - [X] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks very low, feature is turned off by default, log line only |
||
|
michaely520
|
232fb7bf51 |
Emit handover watermark and shard readiness wide events (#11401)
## What changed?
Adds `NamespaceLifecycle` wide events on the two paths that decide when
a namespace handover can complete. Also threads `ShardID` and an
`EventLogger` into `HandoverTrackerParams` so the tracker can attribute
an event to its shard.
**Shard handover tracker** (`service/history/shard/`) — each shard holds
its own replication watermark for a namespace in handover, and the
handover cannot complete until every shard's watermark has been acked by
the target:
| Phase | When |
|---|---|
| `shard_handover_watermark_set` | the shard takes or advances a
watermark — `reason` is `added` or `updated` |
| `shard_handover_watermark_removed` | the shard drops it —
`deleted_from_db` separates a namespace deletion from a normal exit out
of handover |
Both fire only on an actual mutation of the tracker map. Notably, the
non-handover branch of `UpdateHandoverState` runs for *every* namespace
notification on *every* shard; it now guards on presence before
deleting, so it emits only on a real removal rather than on every
notification.
**`WaitHandover`** (`service/worker/migration/`) — one event, emitted on
the way out:
| Phase | When |
|---|---|
| `shard_handover_incomplete` | the wait ended with shards still behind
|
Carries `not_ready_count`, `total_shards`,
`missing_handover_info_count`, `max_lagging_tasks{,_shard_id}`,
`elapsed_seconds`, `exit_reason`, and a `lagging_shards` list of
`{shard_id, lagging_tasks}` capped at 64 (`not_ready_count` is always
the true count).
A handover that completes leaves no laggards and emits nothing, so the
happy path — nearly all of them — is silent. The wait is capped at
`maximumHandoverTimeoutSeconds` (30) with `MaximumAttempts: 1`, so this
is at most one event per handover.
`exit_reason` separates the wait being killed while shards were still
behind from a failed `GetReplicationStatus`. The activity never returns
on its own in the former case: the SDK cancels the activity context off
the heartbeat, the next `GetReplicationStatus` fails, `WaitHandover`
returns, and the deferred summary runs.
A not-ready shard with `lagging_tasks == 0` is the missing-handover-info
case (that shard's namespace cache hasn't picked up the handover yet);
any other not-ready shard is behind by `lagging_tasks`.
## Why?
When a handover stalls, the only signal today is `Wait handover not
ready`, which reports counts plus the single worst shard, once a second
for the life of the wait. That is not enough to name the shards actually
holding it up, or to tell a shard that never took a watermark from one
whose watermark is not being acked. These events make both attributable,
and the summary form costs one row per failed handover instead of one
log line per second.
Emission is a no-op unless a `LoggerProvider` is configured, so there is
no cost for deployments that have not opted in.
## How did you test it?
- [x] covered by new unit tests
- [x] built
New tests pin the behavior that keeps these off the hot path:
- `TestHandoverWatermarkRemovedOnlyWhenTracked` — the per-notification
branch emits nothing when there is no watermark to remove.
- `TestHandoverWatermarkEvents` — set/update/remove, and that a repeat
notification at the same version emits nothing.
- `TestHandoverWatermarkPendingThenResolved` — the unacquired-shard
sentinel is reported as `pending`, and resolving it on acquire emits
nothing.
- `TestEmitHandoverLagSummarySilentWhenReady` — a completed handover
emits nothing.
- `TestCheckHandoverOnceSnapshotIsLastPollOnly` — the snapshot is
overwritten per poll, so laggards from an earlier poll cannot leak into
the summary.
- `TestEmitHandoverLagSummaryNamesLaggingShards` / `ExitReason` /
`Truncates` — payload contents, exit-reason classification, and the
per-shard cap.
`go test ./service/history/shard/... ./service/worker/migration/...`
passes; `go vet` and `gofmt` clean.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Sean Kane
|
0d60f98d06 |
fix(batcher): scope deterministic request IDs to the batch job ID (#11546)
## What changed? Use the `jobID` parameter in the `deterministicRequestID` function to allow multiple batch operations to signal the same workflow. ## Why? Two batch operations that send the same signal to the same workflow/run ID will only send one signal, the second will be de-duped in the signal logic because the request ID was hashed from workflow id / run id / signal name. Including the batch job ID gives proper hashing to prevent these separate signals from being de-duped ## How did you test it? - [x] added new unit test(s) - [x] added new functional test(s) ## Potential risks NA, this is a bug fix. |
||
|
Prathyush PV
|
264a483cd5 |
Stop the SDK workers and clients the worker service starts (#11436)
## What changed? `parentclosepolicy.Processor` and `scanner.Scanner` keep a reference to the SDK workers they start and stop them on shutdown, and the batch activity closes the SDK client it creates per execution. The SDK worker goleak ignores go with them. ## Why? Neither component kept a reference before, so nothing could stop the workers and their pollers ran until the process exited. The batcher leaked a client reference per activity execution, so that one accumulated per server process rather than per cluster. Part of removing the TODO ignores in `tests/leakcheck/leak_test.go` (see #11322). ## How did you test it? - [x] built - [x] covered by existing tests |
||
|
Shivam
|
f30cf705bd |
Gate worker deployment version demotion signals (#11426)
## What changed
- Add a global `matching.enableWorkerDeploymentVersionDemotionSignal`
dynamic config, defaulting to `false` for OSS.
- Preserve the existing signal-based demotion path when the config is
`true`.
- Restore the legacy `SyncWorkerDeploymentVersion` update path when the
config is `false`.
- Record the dynamic-config decision deterministically with workflow
versioning and `MutableSideEffect`.
- Cover SetCurrent and SetRamping demotions under both modes.
- Replay the complete Worker Deployment history corpus with the config
both disabled and enabled.
Temporal Cloud configures this value to `true`. The OSS default is
intended to change to `true` in v1.33, after existing Version workflows
have had time to Continue-As-New onto code that registers the demotion
signal handler.
## Why
#9973 changed version demotion from a synchronous update to a
fire-and-forget signal. Long-running Worker Deployment Version workflows
that had started before the signal handler was available could ignore
that signal, leaving versions stuck in `Draining`.
Cloud has already repaired affected Version workflows through forced
Continue-As-New and should keep the signal path enabled. OSS v1.31 does
not contain #9973, so v1.32 should default to the legacy update path and
avoid introducing this failure mode during upgrade. The main idea, with
OSS, would be to keep this feature disabled for now (default value of
the new dynamic config value introduced is false) and then turn the knob
on to true in the next OSS release.
Existing histories remain deterministic:
- OSS histories without `commit-routing-first` continue replaying the
update path.
- Cloud histories that already contain the `demote-version` signal but
lack `version-demotion-signal-dynamic-config` continue replaying the
signal path, even if the current config is `false`. (this should never
happen since I shall be doing a global rollout of this dynamic config
for cloud very soon)
- Workflows with no recorded demotion decision use and record the
current dynamic-config value when their first demotion occurs.
## User impact
OSS users upgrading to v1.32 retain the pre-#9973 demotion behavior by
default. Cloud retains the currently deployed signal behavior. Existing
OSS and Cloud workflow histories replay without nondeterminism under
either current config value.
## Validation
- `go test -tags test_dep ./service/worker/workerdeployment`
- `go test -tags test_dep ./service/worker/workerdeployment/replaytester
-run '^TestReplays$' -count=1`
#11421 adds the freshly generated histories that prove the signal-based
demotion command path.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **High Risk**
> Changes worker deployment routing/demotion behavior and workflow
determinism rules; mis-toggling the config or replay mismatches could
leave deployment versions stuck in Draining or cause nondeterminism on
long-running workflows.
>
> **Overview**
> Adds global dynamic config
**`matching.enableWorkerDeploymentVersionDemotionSignal`** (default
**`false`** for OSS) so Worker Deployment workflows can choose between
**fire-and-forget demotion signals** and the legacy
**`SyncWorkerDeploymentVersion`** activity path when changing current or
ramping versions.
>
> The deployment workflow records the choice deterministically via
**`workflow.GetVersion`** checkpoints (`commit-routing-first`,
`version-demotion-signal-dynamic-config`) and **`MutableSideEffect`** on
the config value. Histories that already recorded signal-based demotion
keep the signal path on replay even if the config is off; OSS histories
without those markers stay on the update path.
>
> **`fx.go`** wires the config getter into **`Workflow`**. Replay tests
run the full history corpus with the flag both enabled and disabled.
Workflow tests cover SetCurrent and SetRamping demotion for each mode.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Shivam
|
df8a7f4184 |
Add demote-version replay histories (#11421)
## What changed Adds a freshly generated Worker Deployment replay corpus containing 20 Deployment workflow histories and 11 Version workflow histories. Two Deployment workflow histories exercise the signal-based version-demotion path: they record the `commit-routing-first` version marker at version `0`, initiate the `demote-version` external workflow signal, and record successful external signaling. The corresponding Version workflow histories record receipt of those signals. ## Why The existing replay corpus did not contain Deployment workflow histories that emitted `demote-version`. These fixtures give us real signal-path histories to use when validating workflow compatibility changes around OSS upgrades and the demotion gate. There are no production code changes in this PR. ## Historical context PR #9973 initially added `run_1776382947`, containing 21 Deployment workflow histories and 11 Version workflow histories. That corpus included two Deployment histories sending `demote-version` and two Version histories receiving it, together with the new workflow version markers. Later in the same PR's branch history, commit [`4e41ba1`]( |
||
|
Sean Kane
|
bf04d5a7d0 |
fix(batcher): stop retrying terminal-state errors and fix flaking activity batch tests (#11423)
## What changed?
1. Test fix `tests/activity_api_batch_{cancel,terminate}_test.go`: both
`*_ExcludesNonRunning` tests waited only for two activities to match
`ActivityType` before starting the batch, but the just-completed one may
still be indexed as `ExecutionStatus='Running'`. New shared helper
`waitForRunningFilterToSettle` also waits on the `Running` count,
mirroring the query the batcher counts with in
`adjustQueryBatchTypeEnum`.
2. `isNonRetryableError` treats `serviceerror.FailedPrecondition` as
non-retryable for
`TERMINATE_ACTIVITY`/`CANCEL_ACTIVITY`/`DELETE_ACTIVITY`. Every
reachable FailedPrecondition on those paths is permanent — a terminal
status, or an already-recorded terminate/cancel request with a different
request ID — and processTaskWithRetries retries in place with no
backoff.
## Why?
TestActivityBatchCancel_ExcludesNonRunning was a top CI breaker in the
2026-08-04 Flaky Tests Report.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s) — the two existing ones are the
subject of the fix
## Potential risks
- Activity batch targets in a terminal state now fail immediately rather
than after N attempts. They are still counted as failures, so
`DescribeBatchOperation` totals are unchanged.
|
||
|
Muneeb Ahmad
|
27b67bd9c8 |
Propagate ComputeStatus to deployment workflow (#11273)
## What changed? Added a missing `d.syncSummary()` call in `syncVersionDataToComputeStatus`, so it now notifies the parent Deployment workflow after pulling a compute status from WCI. ## Why? Without this, the pull only updates the Version workflow's own state. The Deployment workflow (which `ListWorkerDeployments`/`DescribeWorkerDeployment` actually read from) is not updated, so `computeStatus` can stay permanently missing from the API even when the data is available. ## How did you test it? - [x] built - [x] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Carly de Frondeville
|
91fb96ad3c |
Revert flaky MaxDeployments functional test (#11387)
### Motivation
- The change that updated MaxDeployments behavior and error text caused
flakiness in `main` by making `TestNamespaceDeploymentsLimit` unstable
under shared test namespaces.
- Revert and re-disable the affected functional test to restore stable
test behavior while the underlying flakiness is investigated.
### Description
- Restore the previous worker-deployment limit error wording in
`service/worker/workerdeployment/client.go` (reverting the wording
change that exposed "worker deployments" in the message).
- Re-disable the unstable functional test by adding `s.T().Skip()` in
`tests/worker_deployment_test.go` for `TestNamespaceDeploymentsLimit`
and restore its prior flow/assertions that expect the original error
message.
- Adjust test assertions in
`TestCreateWorkerDeployment_MaxDeploymentsLimit` to match the restored
error text (`"reached maximum deployments in namespace"`).
- Modified files: `service/worker/workerdeployment/client.go` and
`tests/worker_deployment_test.go`.
### Testing
- Ran the targeted functional test with `go test -tags test_dep ./tests
-run 'TestWorkerDeploymentSuite/TestNamespaceDeploymentsLimit$'
-count=1`, which completed successfully.
- Ran `make lint-code`, which failed due to a network error downloading
`golangci-lint` (HTTP 403 from `proxy.golang.org`).
------
[Codex
Task](https://chatgpt.com/codex/cloud/tasks/task_b_6a6cedda33c08324a2863a637b5c00ca)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> User-visible error string reversion and test skip/assertion alignment
only; no limit logic changes.
>
> **Overview**
> Reverts namespace **max deployment** limit errors to say **"reached
maximum deployments in namespace"** instead of **"worker deployments"**,
in both `CreateWorkerDeployment` and auto-create via poll paths in
`client.go`.
>
> **`TestNamespaceDeploymentsLimit`** is skipped again (shared namespace
/ visibility flake) with a TODO on poller error messaging; the in-test
flow is simplified to shared helpers when re-enabled.
**`TestCreateWorkerDeployment_MaxDeploymentsLimit`** now expects the
restored error substring.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Carly de Frondeville
|
53f26583ce |
test MaxDeployments error is exposed correctly (#10469)
## What changed?
revive MaxDeployments error test by giving the test its own namespace to
hit the max in
## Why?
test that MaxDeployments error is exposed correctly
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> User-visible error string tweak plus test-only namespace isolation; no
change to limit enforcement logic.
>
> **Overview**
> Clarifies namespace **worker deployment** limit errors and re-enables
coverage for pollers hitting that cap.
>
> **Error text:** `ResourceExhausted` messages in
`CreateWorkerDeployment` and auto-create
(`updateWithStartWorkerDeployment`) now say *reached maximum worker
deployments in namespace* instead of *maximum deployments*, so clients
can tell this limit apart from other deployment concepts.
>
> **Tests:** `TestNamespaceDeploymentsLimit` is no longer skipped. It
registers a dedicated namespace with `MatchingMaxDeployments` set to 1,
creates one deployment via poll, then asserts a second deployment’s poll
fails with the exact new message.
`TestCreateWorkerDeployment_MaxDeploymentsLimit` expects the updated
substring in the API error.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Shivam
|
39905ab1a4 |
Deduplicate Worker Deployments within a "single" Page (#11110)
## Summary
- deduplicate Worker Deployment summaries by deployment name within each
fetched Visibility page
- retain the record with the newest workflow start time while preserving
the page's first-occurrence ordering
- keep Visibility pagination state and next-page tokens unchanged
- cover Continue-As-New, missing-memo, delete-and-recreate,
unrelated-deployment, and page-boundary behavior
## Root cause
Worker Deployment creation can immediately Continue-As-New, and
delete-and-recreate starts another run with the same deployment
identity. Because Visibility updates are eventually consistent, the
successor start can be visible before the predecessor close update.
`ListWorkerDeployments` previously converted every returned execution
into a summary, so overlapping running records in the same Visibility
page appeared as duplicate deployments.
This change post-processes only the fetched page. It does not add
cross-page state or alter Worker Deployment workflow behavior.
## Testing
- added unit tests that mock visibility to return duplicate data which
verifies if the added logic works or not
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Read-path-only listing change with bounded per-page dedup; pagination
contract unchanged and covered by new unit tests.
>
> **Overview**
> **Fixes duplicate worker deployments in `ListWorkerDeployments`** when
Visibility returns multiple RUNNING records for the same deployment on
one page (e.g. Continue-As-New or delete/recreate while visibility is
eventually consistent).
>
> `ListWorkerDeployments` now post-processes each Visibility page via
**`collapseDuplicateDeploymentSummaries`**: summaries are keyed by
deployment name, the run with the **latest workflow start time** wins,
and results stay in the page’s **first-seen slot order**. **Next-page
tokens are unchanged**; dedup does not span pages.
>
> Unit tests mock Visibility for same-page collapse (including
memo-missing newest run), order preservation, and no cross-page
collapse.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Sean Kane
|
a1b0e621c1 |
Implement terminate/cancel/delete batch operation for standalone activities (#10803)
## What changed? - terminate, cancel, delete standalone activity batch operations - Add OperationType to ListBatchOperations - Show query/executions and operation type on DescribeBatchOperation ## Why? Terminate, cancel, delete are available batch operations for workflows, providing them for standalone activities brings SAA up to parity. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [X] covered by existing tests - [X] added new unit test(s) - [X] added new functional test(s) ## Potential risks Changes to batch operations introduces the risk of non-backwards compatible code. This code has been manually (human-read) audited for backwards compatability. --------- Co-authored-by: ks-temporal <281732484+ks-temporal@users.noreply.github.com> |
||
|
stuart-wells
|
07478bca3b |
Treat no watermark as a 0-lag but notReady shard to avoid garbage lag values (#10939)
## What changed? Zero out metrics for non-acked/no watermark shards instead of logging them as extremely far behind. Adds additional diagnostics around this including histograms for timeLag. ## Why? It's possible for a newly connected shard to be connected and caught up, but not have a watermark. This PR also prevents metrics from being garbled by incorrect lag times from unacked shards. Hopefully the additional metrics allow for a more targeted fix of issues that cause graceful failovers to become forceful. ## How did you test it? - [x] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) ## Potential risks This could adjust metrics thresholds that alarms rely on. Very low risk of this since the data being zeroed was unreliable anyways. |
||
|
David Porter
|
c10c730e53 |
List matching perf (#11014)
## What changed? This addresses a performance concern in listing some schedules with specs which are excessive in what they exclude by adding a bound (credit to @lina-temporal ) . ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [X] added new unit test(s) - [ ] added new functional test(s) ## Potential risks Relatively high risk change, risks breaking V1 schedules extensively if we mess up nondeterministism, will have to roll out with care. There's also the (unlikely) risk that some customers may actually hit the cap. For use-cases where this is happening, the operator should raise the limit and rely on warnings first. --------- Co-authored-by: Lina Jodoin <lina.jodoin@temporal.io> |
||
|
Stefan Richter
|
1e125d7520 |
Sync WCI compute status to worker deployment version state (#10908)
## What changed? Add DescribeWorkerControllerInstanceStatus activity to fetch the WCI validation status and map it to a ComputeStatus. During handleSyncState, invoke syncVersionDataToComputeStatus to update the version state's ComputeStatus with the provider validation result. The sync is gated behind the "sync-compute-validation-status" workflow version for backward compatibility. ## Why? During creation of a new version there is a risk of the compute provider validation finishing before the version is created, and thereby the sync signal for the compute status being missed. This highlighted that there is no backstop for the status sync in this and other cases of missing the signal so adding one. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Sean Kane
|
193c5995d0 |
Fix Schedules V1 behavior with Pause (#10807)
## What changed? * Treat PAUSED status the same as RUNNING in V1 scheduler, which allows the scheduler to terminate those workflows if the overlap policy requests that. * Add tests to validate the `OverlapPolicy` behavior with paused workflows. ## Why? In preparation for a workflow pause release, encoding the overlap policy and workflow pause interactions operate the same for CHASM and V1 schedules. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [X] added new functional test(s) ## Potential risks Test changes only |
||
|
Stefan Richter
|
fb9f59cc23 |
Pass Nexus endpoint through to WCI scaling group compute spec (#10938)
## What changed? While it had been in the API spec from early on, it was missing on the WCI side and has now been added. ## Why? Eventually we want to use Nexus as an extension point to allow custom compute providers for Serverless, which will need the endpoint to work correctly. ## How did you test it? - [x] built - [x] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Muneeb Ahmad
|
d4cab6b2b2 |
Include WCI connectivity health in worker deployment API (#10778)
## What changed? `DescribeWorkerDeployment` now returns compute status per version, checking whether Temporal can successfully interact with the version's compute resource. `ListWorkerDeployments` also returns compute status on the current, ramping, and latest version summaries, fetched in parallel. When connectivity changes (e.g. Lambda becomes unreachable or is restored), WCI signals the version workflow, which propagates the update to the deployment workflow memo. The list view reads from the memo — versions that have been validated since deployment will show their status immediately; others will appear once the first validation runs. ## Why? Allows customers to see whether Temporal can successfully interact with their compute resource directly from the Worker Deployments list and detail views, without navigating into each individual version. ## How did you test it? - [x] built - [x] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) |
||
|
Stephan Behnke
|
5e5b6adc03 |
Run all gofix analyzers by default (#10828)
## What changed? Set `GOFIX_FLAGS` to empty so `make fmt` runs all `go fix` analyzers. |
||
|
Stephan Behnke
|
3bc2ffb276 |
Apply lint fixes for oss-foundations (#10796)
## What changed? - Applied testifylint, staticcheck, and gofix auto-fixes. - Exact commands that were run: ```sh .bin/golangci-lint-v2.9.0 run --allow-parallel-runners --concurrency 4 --fix --enable-only testifylint --build-tags disable_grpc_modules,test_dep --timeout 20m --config=.github/.golangci.yml .bin/golangci-lint-v2.9.0 run --allow-parallel-runners --concurrency 4 --fix --enable-only staticcheck --build-tags disable_grpc_modules,test_dep --timeout 20m --config=.github/.golangci.yml make fmt-gofix make goimports make fmt git diff --check ``` - No manual or AI changes were made; except where commented on. - Some fixes caused lint errors; those were reverted again. - Changes were all reviewed by me. |
||
|
Stephan Behnke
|
8868e726a7 |
Apply lint fixes for CGS (#10795)
## What changed? - Applied testifylint, staticcheck, and gofix auto-fixes. - Exact commands that were run: ```sh .bin/golangci-lint-v2.9.0 run --allow-parallel-runners --concurrency 4 --fix --enable-only testifylint --build-tags disable_grpc_modules,test_dep --timeout 20m --config=.github/.golangci.yml .bin/golangci-lint-v2.9.0 run --allow-parallel-runners --concurrency 4 --fix --enable-only staticcheck --build-tags disable_grpc_modules,test_dep --timeout 20m --config=.github/.golangci.yml make fmt-gofix make goimports make fmt git diff --check ``` - No manual or AI changes were made. - Some fixes caused lint errors; those were reverted again. - Changes were all reviewed by me. |
||
|
Shivam
|
40eeb483e9 |
Fix per-namespace worker shutdown go-routine leak! (#10802)
## What changed?
- WISOTT
## Why?
- Not impacting prod, but @stephanos 🐐 noticed that there were some OOM
kills that were happening in our test clusters.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- Not sure honestly, this should make life easier not tougher.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Shutdown-path change in worker service only; behavior of
refresh/membership handling is unchanged aside from proper goroutine
teardown.
>
> **Overview**
> Fixes a **goroutine leak** in `PerNamespaceWorkerManager` where the
membership listener and periodic refresh loops never exited on shutdown.
>
> **Lifecycle:** Background work now runs under `goro.Group` instead of
bare `go` calls. `Stop()` **cancels** that group and **waits** for both
loops to finish, replacing `close(membershipChangedCh)` and removing
status polling from the periodic ticker loop (loops exit on
`ctx.Done()`).
>
> **Tests:** Adds `TestPerNsWorkerManagerStopStopsBackgroundLoops` and a
`stopManager` helper with a timeout so suite teardown cannot hang;
functional leak tests drop the `periodicRefresh` goleak ignore.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Stephan Behnke
|
4c33955f06 |
Apply staticcheck fixes (#10793)
## What changed? - Applied the server-owned subset of staticcheck auto-fixes. - Exact commands that were run: ```sh .bin/golangci-lint-v2.9.0 run --allow-parallel-runners --concurrency 4 --fix --enable-only staticcheck --build-tags disable_grpc_modules,test_dep --timeout 20m --config=.github/.golangci.yml make goimports git diff --check ``` - No manual or AI changes were made. - Some fixes caused lint errors; those were reverted again. - Changes were all reviewed by me. |
||
|
Stephan Behnke
|
f0607b84ab |
Apply lint fixes for oss-matching (#10798)
## What changed? - Applied testifylint, staticcheck, and gofix auto-fixes. - Exact commands that were run: ```sh .bin/golangci-lint-v2.9.0 run --allow-parallel-runners --concurrency 4 --fix --enable-only testifylint --build-tags disable_grpc_modules,test_dep --timeout 20m --config=.github/.golangci.yml .bin/golangci-lint-v2.9.0 run --allow-parallel-runners --concurrency 4 --fix --enable-only staticcheck --build-tags disable_grpc_modules,test_dep --timeout 20m --config=.github/.golangci.yml make fmt-gofix make fmt-gofix GOFIX_FLAGS="" make goimports git diff --check ``` - No manual or AI changes were made. - Some fixes caused lint errors; those were reverted again. - Changes were all reviewed by me. |
||
|
Stephan Behnke
|
8ab1cc0155 |
Apply lint fixes for ACT (#10797)
## What changed? - Applied testifylint, staticcheck, and gofix auto-fixes. - Exact commands that were run: ```sh .bin/golangci-lint-v2.9.0 run --allow-parallel-runners --concurrency 4 --fix --enable-only testifylint --build-tags disable_grpc_modules,test_dep --timeout 20m --config=.github/.golangci.yml .bin/golangci-lint-v2.9.0 run --allow-parallel-runners --concurrency 4 --fix --enable-only staticcheck --build-tags disable_grpc_modules,test_dep --timeout 20m --config=.github/.golangci.yml make fmt-gofix make goimports make fmt git diff --check ``` - No manual or AI changes were made. - Some fixes caused lint errors; those were reverted again. - Changes were all reviewed by me. |
||
|
Alan Wu
|
7ba4645d31 |
Add nexus operation archetype ID to force migration translation (#10769)
## What changed? Add nexus operation hardcoded archetypeID for force migration handler. Worker service does not register Nexus operations in the chasm Registry, so we need a manual translation. Will fast follow up to use archetypeID directly in the future to avoid more hardcoding. ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [X] added new unit test(s) - [ ] added new functional test(s) |
||
|
Rodrigo Zhou
|
18c8ab7f77 |
Replace payload.Encode with sadefs.MustEncodeValue (#10432)
## What changed? Replace `payload.Encode` with `sadefs.MustEncodeValue` for encoding predefined search attributes used internally. ## Why? `sadefs.MustEncodeValue` adds metadata type to the payload. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks |
||
|
Rodrigo Zhou
|
5fb11aef2b |
Add StartTime and IsRetentionDelete to delete visibility task (#10614)
## What changed? Add StartTime and IsRetentionDelete to delete visibility task ## Why? Additional info for delete visibility task that might be useful for implementing VisibilityStore. ## How did you test it? - [x] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks |
||
|
Prathyush PV
|
4d6f69d9b9 |
fix: prevent batcher activity deadlock when retrying failed tasks (#10630)
## What changed? Batch worker retries now run in place on the worker goroutine instead of being re-queued onto the task channel, and the heartbeat resume point now tracks the oldest not-yet-done page. Also clamps the worker count to at least 1 and stops offering empty tasks once a page is fully submitted. ## Why? Re-queuing retries onto the worker-only task channel could deadlock the worker pool under a burst of retryable errors, leaving the activity heartbeating with no progress. Advancing the resume token while fetching ahead could also skip still-in-flight pages on restart. ## How did you test it? - [x] built - [x] covered by existing tests - [x] added new unit test(s) ## Potential risks Retries hold a worker slot while retrying, bounded by AttemptsOnRetryableError and the per-task timeout. |
||
|
Jiechen Zhong
|
2e4de2b3e6 |
Usage of ActiveInCluster should be justified (#10485)
## What changed? 1. Add linter rule to guard`ActiveInCluster` usage. 2. Fixed the callsites for `ActiveInCluster` where should use `ActiveClusterName` instead. 3. Add explicit `//nolint:forbidigo justifications` for genuine namespace-level checks. ## Why? ActiveInCluster is namespace-level and not businessID-aware. For workflow-scoped decisions, the active cluster must be resolved with `ActiveClusterName(routingKey)` so future businessID-aware routing does not incorrectly treat an entire namespace as active or standby. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Sean Kane
|
84ec388531 |
batch: set context timeout for a single batch task to 30s (#10627)
## What changed? Set the minimum timeout to 30s for a single batch activity task (activity pause,reset,etc or admin refresh tasks). ## Why? The timeout for an activity is 20years, so these activities could hang forever. Adding this timeout only for the processing of a single task will prevent a single task from stopping a batch operation from completing. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [X] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks Minimal, this is not a breaking change. |
||
|
David Porter
|
9389c7ff12 |
fix: Scanner for invariants - followup (#10596)
## What changed? I let https://github.com/temporalio/temporal/pull/10406 autoland and didn't respond to all feedback, addressing the remaining here. - Pulls all the dynamic config into a single struct - Switches Namespace list to used the local cached version - Converts some minor functions to be iterators ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks The same as the previous scanner PR |
||
|
David Porter
|
4deaeeefde |
Fix: search-attribute nextActionTime is queryable and fixes test (#10606)
## What changed? - Fixes a broken test on main for verifying ScheduleNextActionTime is in use correctly as a SA - Fixes SA validation such that users may query with ScheduleNextActionTime. - Splits out the functional test into a standalone test, just because they're timing out locally otherwise. ## Why: Because I had a mega-pr which I split out, I got muddled and created a functional test which was broken but timing out but (owing to a CI bug) still got landed. Part of my muddle was about making search attributes usable, both visible and queryable by users and by the scanner that I created. As it turns out, it's still not presently possible to list schedules with the `ScheduleNextActionTime` owing to it failing validation. In order to get the test working again, i've re-enabled the ability to query the frontend with CHASM fields and have them pass validation. ## How did you test it? - [X] built - [ ] run locally and tested manually - [X] covered by existing tests - [ ] added new unit test(s) - [X] added new functional test(s) ## Potential risks Not any that I can easily identify |
||
|
Qian Chen
|
aded9c823a |
Widen retry budget for namespace replication tasks (#10571)
## What changed? Increase retry attempts for namespace replication to up to 30. ## Why? We saw under heavy load of failing over namespaces the replication tasks gets DLQ'd after 5 retries . Increasing the retry policy in namespace replication to reduce the likelihood of DLQ'ing ## How did you test it? - [x] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Shivam
|
7113cc0b13 |
Commit routing config before demoting old version in handleSetCurrent (#9973)
## What changed?
- Fix non-atomic two-step sync in handleSetCurrent and setRamp where
step 2 (demoting old version) failure leaves routing config uncommitted,
causing orphaned CURRENT versions and burned revision numbers.
- New flow: commit routing config immediately after step 1 (promote new
version) succeeds, then fire-and-forget signal to old version instead of
blocking sync activity. Gated behind workflow.GetVersion for NDE safety.
- The only thing that I don't have in this PR are new tests to test this
out. Happy to hear ideas if someone has any, but the core idea was that
the current ones should be passing and testing the code paths.
## Why?
- Reliability IMO
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- I would appreciate a very careful review on this one!
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **High Risk**
> Changes core worker-deployment routing and task-queue propagation in
async mode; demotion is now signal-based and eventually consistent, with
revision tracking on signal failure.
>
> **Overview**
> Fixes a reliability bug in **async** `set current` / `set ramp` where
promoting the new version succeeded but **demoting** the previous
current or ramping version could fail, leaving deployment routing
uncommitted and inconsistent summaries.
>
> When `workflow.GetVersion("commit-routing-first")` is enabled in async
mode, the deployment workflow **commits** `pendingRoutingConfig` to
local state right after the promote step, then **signals** the old
version workflow via new **`demote-version`** (`DemoteVersionSignalArgs`
carrying full `RoutingConfig`) instead of a blocking `syncVersion`
activity. Version workflows handle the signal (gated by
`demote-version-signal`) by deriving status from routing config, syncing
task queues, and starting drainage when needed.
**`signalDemoteVersion`** tracks propagating revision numbers until
`PropagationComplete`; failed signal delivery untracks the revision.
**`setVersionSummaryDraining`** updates deployment-side version
summaries immediately in the new path. Sync mode and workflows without
the version gates keep the prior behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Alex Stanfield
|
71dd8ed694 |
feat: add scheduler observability metrics for missed actions (#10503)
## Summary
Add and improve scheduler metrics to detect missed actions caused by
system instability, slow callback delivery, and overlap policy behavior.
Changes apply to both v1 (workflow-based) and v2 (CHASM) schedulers.
## What is a missed action?
A scheduled action is "missed" when the scheduler decides not to execute
an action that was due according to the schedule spec. There are three
scenarios where this happens:
1. **Catchup window exceeded.** The scheduler was delayed in processing
and the action's scheduled execution time is now older than the
configured catchup window. The action is permanently dropped. This can
be caused by a previous action still running or by system instability
(worker downtime, shard movement, task queue backlog). Tracked by
`schedule_missed_catchup_window`.
2. **Overlap policy skip.** A previous action is still running and the
overlap policy is SKIP (or BUFFER_ONE with a full buffer). The scheduler
intentionally drops the action because the policy says not to start
concurrent runs. This is expected behavior, not a system issue -- but a
high skip rate may indicate action duration exceeds the schedule
interval. Tracked by `schedule_overlap_skipped`.
3. **Buffer overrun.** The internal action buffer is full (bounded by
`MaxBufferSize`). New actions are dropped regardless of overlap policy.
Tracked by `schedule_buffer_overruns`.
Scenarios 1 and 2 are the focus of this PR. Scenario 3 already had
metrics.
### Catchup window vs. overlap policy: v1 and v2 differences
An action is only ever counted in one of the above scenarios, but the
evaluation order and the possible outcomes differ between v1 and v2.
Both v1 and v2 have the same two-phase structure: a generation phase
that checks the catchup window and buffers actions, followed by a buffer
processing phase that evaluates overlap policy and executes actions. The
difference is in what happens to buffered actions that can't be executed
immediately.
**V1 (workflow-based):**
1. `processTimeRange` (generation) checks the catchup window. Actions
whose nominal time is older than the window are dropped
(`schedule_missed_catchup_window{reason="not_buffered"}`). Surviving
actions are added to the buffer.
2. `processBuffer` (invocation) refreshes running action state via
`PollMutableState`, then evaluates overlap policy on the entire buffer,
including actions deferred on previous ticks. Actions are either
started, skipped by overlap policy (`schedule_overlap_skipped`), or
deferred (BUFFER_ONE/BUFFER_ALL).
V1 has no `reason=buffer_expired` case because it never re-checks the
catchup window on buffered actions. Once an action passes the initial
catchup window check and enters the buffer, it will never be dropped for
being too old. If a BUFFER_ONE schedule has a long-running action, the
deferred action waits indefinitely across ticks until the running action
completes, at which point it is started regardless of how much time has
passed since its original schedule time.
**V2 (CHASM):**
1. The generator task checks the catchup window. Same as v1 -- actions
past the window are dropped
(`schedule_missed_catchup_window{reason="not_buffered"}`). Surviving
actions are buffered.
2. The invoker's `ProcessBuffer` task evaluates overlap policy. Actions
are either marked for execution, skipped by overlap
(`schedule_overlap_skipped`), or deferred (BUFFER_ONE with a running
action sets `Attempt = -1`).
3. Actions that survive overlap policy evaluation are checked against
the catchup window a second time. If the action was buffered in time but
has now expired -- due to the gap between CHASM tasks, overlap policy
deferral, or start retries -- it is dropped
(`schedule_missed_catchup_window{reason="buffer_expired"}`).
This second deadline check is unique to v2. It means a BUFFER_ONE action
that waits too long for a slot can be dropped, unlike v1 where it would
wait indefinitely. The `action_running` tag on `reason=buffer_expired`
distinguishes whether the expiration was caused by a long-running action
blocking the slot (`action_running=true`) or by system delay in CHASM
task processing (`action_running=false`). For starts deferred by
BUFFER_ONE/BUFFER_ALL, `action_running=true` is preserved when the start
expires because it waited behind a running action, even if that action
has completed by the time the metric is emitted.
## New metrics
### `schedule_overlap_skipped` (counter)
**Tags:** `namespace`, `scheduler_backend`, `schedule_overlap_policy`
Actions are skipped when the overlap policy prevents starting a new
action while a previous one is still running. Today this is only tracked
as a running total in `Info.OverlapSkipped` (visible via
DescribeSchedule) with no time-series visibility.
This counter is emitted for each action dropped by `ProcessBuffer` due
to overlap policy. The `schedule_overlap_policy` tag carries the overlap
policy that was actually applied when that action was evaluated (e.g.,
`SCHEDULE_OVERLAP_POLICY_SKIP` or `SCHEDULE_OVERLAP_POLICY_BUFFER_ONE`).
One subtlety is that buffered actions do not all resolve overlap policy
the same way:
- **v1:** normal scheduled actions are buffered with `UNSPECIFIED` and
resolved later in `ProcessBuffer` against the schedule's current policy.
If `UpdateSchedule` changes the overlap policy while actions are already
buffered, those existing buffered scheduled actions will be evaluated
under the new policy.
- **v2:** the generator resolves overlap policy before buffering and
stores the resolved policy on each `BufferedStart`. If `UpdateSchedule`
changes the overlap policy later, already-buffered scheduled actions
keep the policy they were generated with.
This distinction is why the metric is emitted per skipped action using
the policy that was actually applied at evaluation time, rather than
once per `ProcessBuffer` pass with a single schedule-level tag.
**How to use:**
- `rate(schedule_overlap_skipped[5m])` shows how often actions are being
dropped due to overlap.
- Group by `schedule_overlap_policy` to see which policies are causing
the most skips.
- Compare with `schedule_action_e2e_delay` to see whether skipped
actions correlate with actions whose duration exceeds the schedule
interval, pushing the total schedule-to-start delay higher.
Emitted in v1 (`processBuffer`) and v2
(`InvokerProcessBufferTaskHandler.Execute`).
### `schedule_callback_latency` (timer, v2 only)
**Tags:** `namespace`, `scheduler_backend`
The v2 scheduler learns about action completions via Nexus completion
callbacks. This timer records `time.Since(closeTime)` when
`HandleNexusCompletion` processes a callback, measuring the end-to-end
delay from action completion to the scheduler receiving the result.
This latency includes: callback task creation, callback task queue
processing, cross-shard RPC delivery, and handler execution.
**Why v2 only:** The v1 and v2 schedulers use fundamentally different
mechanisms to discover action completions, and only v2's is worth
measuring independently.
In v1, completion discovery is **tick-driven**. The scheduler wakes on
each scheduled time and does a synchronous `PollMutableState` refresh of
all running actions before making overlap decisions. For a SKIP schedule
with a 1s interval, the scheduler discovers completions at most 1s after
they happen (on the next tick). The long-poll watcher activity exists
but is only useful for BUFFER_ONE/BUFFER_ALL where starts sit in the
buffer waiting for a completion -- for SKIP, there are no buffered
starts and the watcher is irrelevant. The completion discovery latency
in v1 is inherently bounded by the schedule interval itself, so there's
nothing additional to measure.
In v2, completion discovery is **event-driven** via Nexus callbacks. The
scheduler has no periodic refresh; it only learns about completions when
the callback is delivered. This means callback delivery latency directly
gates when the next action can start and when overlap policies are
re-evaluated. A slow callback pipeline silently degrades scheduler
throughput in a way that has no v1 equivalent.
**How to use:**
- High values indicate the callback delivery pipeline is slow, which
directly delays the next action for schedules using BUFFER_ONE or
BUFFER_ALL policies.
- Compare with `schedule_action_delay` to understand how much of the
total action delay is attributable to callback delivery vs. other
factors.
- Useful for diagnosing scenarios where v2 schedules appear to "stall"
between actions.
Emitted in `Scheduler.HandleNexusCompletion`.
## Modified metrics
### `schedule_generate_latency` (timer) -- added to v1, fixed in both
Previously v2-only. This timer measures how late the scheduler is in
processing a due action: `now - scheduledExecutionTime` (the action's
`ActualTime`, after jitter).
**Changes:**
1. **Added to v1 scheduler.** The v1 `processTimeRange` now emits this
metric, matching the existing v2 behavior.
2. **Emit only for the first action per batch.** When the scheduler
catches up over a large time range (e.g., after being down), it
processes many due actions in a single pass. Previously every action got
its own latency sample against the same `now`. Now only the first action
is recorded, representing the actual system delay.
3. **Moved before the catchup window check.** Previously (v2) the metric
was emitted after the catchup window check, meaning the worst latencies
-- the ones that actually miss the window -- were never recorded. Now
it's emitted before, so all latencies including missed actions are
captured.
**How to use:**
- `histogram_quantile(0.99, schedule_generate_latency)` shows worst-case
scheduling delay.
- A rising trend is an early warning of system instability before
actions start being dropped.
### `schedule_missed_catchup_window` (counter) -- added `reason` and
`action_running` tags
Actions can miss the catchup window at two points in the v2 scheduler:
1. **Generator**: the action's scheduled execution time is older than
the catchup window when the generator processes the time range. The
action is never buffered.
2. **Invoker (process buffer)**: the action was buffered in time but
then sat too long waiting for execution (e.g., due to overlap policy
deferral, start retries, or system delay between the gen task and
process buffer task).
Previously both cases emitted the same untagged counter, making it
impossible to distinguish root cause. Now the counter carries two new
tags:
**`reason` tag:**
- `reason=not_buffered` -- the action's scheduled execution time was
already past the catchup window when the generator processed the time
range. It was never buffered for execution. Present in both v1 and v2.
- `reason=buffer_expired` -- the action was buffered in time but expired
before execution (e.g., due to overlap deferral, retries, or system
delay between CHASM tasks). V2 only.
V1 only has the `not_buffered` case because generation and buffer
processing happen synchronously in the same workflow task -- there is no
gap where a buffered start can sit and expire.
**`action_running` tag:**
- `action_running=true` -- a previous action was running when the
buffered start was deferred, or was still running when the start was
dropped. This indicates overlap wait contributed to the missed action.
- `action_running=false` -- no action was blocking the start when it
expired, so the miss was due to other factors (system delay, scheduler
processing time).
Present only on `reason=buffer_expired` (v2). Not added to
`reason=not_buffered` because: in v2, the generator doesn't have access
to the invoker's running action state; in v1, running action state is
stale at the point where the catchup window check runs (the refresh
happens later in `processBuffer`).
**How to use:**
- `schedule_missed_catchup_window{reason="not_buffered"}` means the
scheduler was too late to process the action. This check is independent
of whether a previous action is running.
- `schedule_missed_catchup_window{reason="buffer_expired"}` indicates
actions were buffered on time but the invoker was delayed in executing
them. V2 only.
- `schedule_missed_catchup_window{reason="buffer_expired",
action_running="true"}` narrows the cause to a long-running previous
action blocking execution of buffered starts, including starts that
expire after the previous action completes but before they can be
executed. For v2 it's possible that completion callbacks are not being
delivered in time.
- `schedule_missed_catchup_window{reason="buffer_expired",
action_running="false"}` means no previous action was blocking execution
-- the delay was in CHASM task processing between the generator and
invoker.
- Comparing the two rates shows whether the bottleneck is in waking the
scheduler vs. executing buffered actions.
## Existing metrics (not modified, included for context)
### `schedule_action_delay` (timer)
**Tags:** `namespace`, `scheduler_backend`, `schedule_action`
Measures the time between when an action became eligible for execution
and when it was actually started: `actualStartTime - desiredTime`.
`desiredTime` is either:
- The action's scheduled execution time (`ActualTime`), for the first
action after the scheduler wakes up.
- The `CloseTime` of the previous action, for subsequent buffered starts
(set in `processWatcherResult` in v1, `recordCompletedAction` in v2).
This means that for BUFFER_ONE/BUFFER_ALL, the delay measures only the
gap between the previous action completing and the next one starting,
not the total wait from the original schedule time.
This metric captures the delay from "action eligible" to "action
started," which includes buffer processing, rate limiting, and the
actual start RPC. However, it does **not** capture time spent waiting
for a previous action to complete under BUFFER_ONE/BUFFER_ALL overlap
policies, because `DesiredTime` is reset to the previous action's
`CloseTime` when it completes (in `processWatcherResult` for v1,
`recordCompletedAction` for v2). This means for buffered starts,
`schedule_action_delay` measures only the gap between the slot opening
and the next action starting -- not the total wall-clock wait from the
original schedule time.
For the full wall-clock delay including overlap policy wait, use
`schedule_action_e2e_delay`.
**Relationship to other metrics:**
- `schedule_generate_latency` measures a subset: how late the scheduler
was in waking up to process the action. This is always <=
`schedule_action_delay`.
- `schedule_callback_latency` (v2) measures a different leg: how long it
took to learn about the previous action's completion. For BUFFER_ONE,
high callback latency contributes to high action delay on the next
start.
- `schedule_action_delay` is the metric most directly visible to users
as "how fast did we start after the slot opened."
- `schedule_action_e2e_delay` is the metric most directly visible to
users as "how late was my scheduled action" -- it includes overlap
policy wait time that `schedule_action_delay` excludes.
### `schedule_action_e2e_delay` (timer) -- NEW
**Tags:** `namespace`, `scheduler_backend`, `schedule_action`
Measures the total wall-clock delay from the action's original schedule
time to when it was actually started: `actualStartTime - ActualTime`.
Unlike `schedule_action_delay`, this metric always uses the original
schedule time as the baseline and is never reset when a previous action
completes.
This means for BUFFER_ONE/BUFFER_ALL schedules where an action was
waiting in the buffer for a previous one to finish, this metric includes
the full overlap policy wait time. For the first action after the
scheduler wakes (where `DesiredTime` has not been reset), this metric
will equal `schedule_action_delay`.
**How to use:**
- Compare with `schedule_action_delay` to isolate how much delay is
caused by overlap policy wait vs. other factors. If
`schedule_action_e2e_delay` is high but `schedule_action_delay` is low,
the bottleneck is the previous action's duration.
- `histogram_quantile(0.99, schedule_action_e2e_delay)` shows worst-case
total delay from schedule time to actual start, as experienced by users.
Emitted in v1 (`startWorkflow`) and v2
(`InvokerExecuteTaskHandler.startWorkflow`).
## Detecting missed/delayed actions due to system instability
These metrics form a pipeline from early warning to confirmed impact.
The available signals differ between v1 and v2 due to their
architectural differences.
### V1 (workflow-based scheduler)
The v1 scheduler runs as a single workflow. Generation and buffer
processing happen synchronously in the same workflow task. The scheduler
discovers action completions via a synchronous `PollMutableState`
refresh on each tick.
**Available metrics:** `schedule_generate_latency`,
`schedule_action_delay`, `schedule_action_e2e_delay`,
`schedule_missed_catchup_window{reason="not_buffered"}`,
`schedule_overlap_skipped`
**Diagnosis flow:**
1. **Early warning: `schedule_generate_latency`**
The first metric to move. Measures how late the scheduler is in
processing due actions. Under normal conditions this should be
near-zero. A rising trend means the scheduler is delayed in processing
-- the worker is under load, matching is slow, or the task queue is
backed up.
```promql
histogram_quantile(0.99,
rate(schedule_generate_latency_bucket{scheduler_backend="legacy"}[5m]))
```
2. **Confirmed drops:
`schedule_missed_catchup_window{reason="not_buffered"}`**
Once generate latency exceeds the catchup window, actions are
permanently lost. In v1 this is always `reason=not_buffered` --
generation and buffer processing are synchronous, so there is no gap
where a buffered start can expire independently.
```promql
rate(schedule_missed_catchup_window{scheduler_backend="legacy",
reason="not_buffered"}[5m])
```
3. **User-facing impact: `schedule_action_delay` and
`schedule_action_e2e_delay`**
`schedule_action_delay` measures the delay from when the action became
eligible (after any overlap wait) to when it started. Absent high
`schedule_generate_latency`, a high `schedule_action_delay` points to
buffer processing -- rate limiting or the start RPC itself.
`schedule_action_e2e_delay` measures total delay from the original
schedule time, including overlap policy wait. Absent high
`schedule_action_delay`, a high `schedule_action_e2e_delay` indicates
the bottleneck is previous action duration, not system instability.
4. **Overlap context: `schedule_overlap_skipped`**
Absent high `schedule_callback_latency` (v2), a high skip rate suggests
actions are taking longer than the schedule interval (application-level
issue). In v2, delayed callback delivery can also cause skips -- the
scheduler still sees the previous action as running even though it has
completed, because the completion callback hasn't arrived yet.
### V2 (CHASM scheduler)
The v2 scheduler splits work across independent CHASM tasks: a generator
task buffers actions, and an invoker process-buffer task resolves and
executes them. The scheduler learns about action completions via Nexus
callbacks, not polling. This creates additional failure modes between
the generator and invoker, and in the callback delivery pipeline.
**Available metrics:** `schedule_generate_latency`,
`schedule_action_delay`, `schedule_action_e2e_delay`,
`schedule_missed_catchup_window{reason="not_buffered"|"buffer_expired"}`,
`schedule_callback_latency`, `schedule_overlap_skipped`
**Diagnosis flow:**
1. **Early warning: `schedule_generate_latency`**
Same as v1 -- measures how late the generator task is in processing due
actions.
```promql
histogram_quantile(0.99,
rate(schedule_generate_latency_bucket{scheduler_backend="chasm"}[5m]))
```
2. **Confirmed drops: `schedule_missed_catchup_window`**
The `reason` tag identifies where the drop occurred:
- `reason=not_buffered`: The generator was too late to buffer the
action. Same as v1 -- the clearest signal that the system is to blame.
- `reason=buffer_expired`: The action was buffered on time but expired
waiting for execution. The `action_running` tag distinguishes cause:
- `action_running=true`: A previous action was still running, or had
deferred the buffered start long enough that it expired before
execution. Absent high `schedule_callback_latency`, this points to
action duration exceeding the catchup window (application-level issue).
If `schedule_callback_latency` is also high, delayed callback delivery
may be the actual cause -- the previous action completed but the
scheduler hasn't learned about it yet.
- `action_running=false`: No action was blocking it. The delay was in
CHASM task processing between the generator and invoker -- system
instability.
```promql
# system-caused drops at the generator (strongest signal)
rate(schedule_missed_catchup_window{scheduler_backend="chasm",
reason="not_buffered"}[5m])
# system-caused drops at the invoker (no action blocking)
rate(schedule_missed_catchup_window{scheduler_backend="chasm",
reason="buffer_expired", action_running="false"}[5m])
# drops caused by long-running actions
rate(schedule_missed_catchup_window{scheduler_backend="chasm",
reason="buffer_expired", action_running="true"}[5m])
```
3. **Callback pipeline health: `schedule_callback_latency`**
For schedules using BUFFER_ONE or BUFFER_ALL, the callback pipeline is
on the critical path between one action completing and the next
starting. Absent high `schedule_generate_latency`, a high
`schedule_callback_latency` indicates the callback delivery pipeline is
the bottleneck. A slow callback pipeline means the scheduler doesn't
learn about completions promptly, delaying the next buffered action and
potentially causing overlap skips or buffer expirations.
```promql
histogram_quantile(0.99, rate(schedule_callback_latency_bucket[5m]))
```
4. **User-facing impact: `schedule_action_delay` and
`schedule_action_e2e_delay`**
`schedule_action_delay` measures delay from when the action became
eligible to when it started. `schedule_action_e2e_delay` measures total
delay from the original schedule time. If the latter is high, work
backwards:
- Is `schedule_generate_latency` high? -> Generator task is delayed
(system instability).
- Is `schedule_callback_latency` high? -> Completion callbacks are slow
(system instability in the callback pipeline).
- Is `schedule_missed_catchup_window{reason="buffer_expired",
action_running="false"}` firing? -> Invoker task is delayed (system
instability between CHASM tasks).
- Is `schedule_action_e2e_delay` high but `schedule_action_delay` low?
-> Previous action duration is the bottleneck (application-level issue).
- Is `schedule_overlap_skipped` spiking? -> Overlap policy is dropping
actions because previous actions run too long (application-level issue).
5. **Overlap context: `schedule_overlap_skipped`**
Same as v1. Absent high `schedule_callback_latency`, a high skip rate
suggests action duration exceeds the schedule interval
(application-level issue). If `schedule_callback_latency` is also high,
the skips may be caused by delayed completion callbacks rather than
actual action duration.
## Detecting missed/delayed actions due to schedule configuration
Actions can be missed or delayed due to the schedule's own configuration
-- for example, when action duration exceeds the schedule interval.
These are not caused by system instability but may still require
attention.
**Symptoms:**
- `schedule_overlap_skipped` is elevated, absent high
`schedule_callback_latency` (v2). Actions are being dropped because the
previous action hasn't completed by the time the next one is due.
- `schedule_action_e2e_delay` is high but `schedule_action_delay` is
low. The total delay from schedule time to start is large, but the
scheduler started the action promptly once the slot opened. The gap is
overlap policy wait.
- `schedule_missed_catchup_window{reason="buffer_expired",
action_running="true"}` is firing (v2 only), absent high
`schedule_callback_latency`. Buffered actions are expiring because they
spent too long waiting behind a previous action.
**Diagnosis:**
1. Compare `schedule_action_e2e_delay` with `schedule_action_delay`. A
large gap between them indicates time spent waiting for overlap policy
to allow execution.
2. Check `schedule_overlap_skipped` grouped by
`schedule_overlap_policy`. A high rate on SKIP or BUFFER_ONE indicates
the schedule interval is shorter than the action duration.
3. For BUFFER_ONE/BUFFER_ALL schedules, check whether
`schedule_missed_catchup_window{reason="buffer_expired",
action_running="true"}` is firing. This means buffered actions are
waiting so long for the slot that they exceed the catchup window.
**Resolution:** These issues are addressed by changing the schedule
configuration -- increasing the interval, switching to a more permissive
overlap policy, or reducing action duration. They do not indicate a
problem with the scheduler or the platform.
### `schedule_buffer_overruns` (counter)
**Tags:** `namespace`, `scheduler_backend`
Counts actions dropped because the internal action buffer is full
(bounded by `MaxBufferSize`).
Emitted in v1 (`processTimeRange` and signal handling) and v2
(`GeneratorTaskHandler.Execute`).
## Backfills and manual triggers
Backfills and manual triggers (TriggerImmediately) generate actions with
the `Manual` flag set. Each backfill request can specify its own overlap
policy -- if unspecified, it resolves to the schedule's configured
policy. Backfill requests can use ALLOW_ALL to run all backfilled
actions concurrently.
**Not emitted for manual/backfill actions:**
- `schedule_generate_latency` -- backfill actions are generated against
a historical time range, so `now - ActualTime` is meaningless as a
system delay indicator.
- `schedule_missed_catchup_window` -- backfill actions skip the catchup
window check entirely. The time range is intentionally in the past.
- `schedule_action_delay` and `schedule_action_e2e_delay` -- the delay
between a historical schedule time and now is not meaningful for
backfills.
**Emitted for manual/backfill actions:**
- `schedule_overlap_skipped` -- backfill actions go through
`ProcessBuffer` and are subject to overlap policy. A backfill with SKIP
policy against a running action will increment this counter. The
`schedule_overlap_policy` tag reflects the backfill's own policy, not
the schedule's configured policy.
- `schedule_buffer_overruns` -- backfills can fill the action buffer,
especially large backfills over wide time ranges. In v1, backfills are
limited to half the buffer (`MaxBufferSize/2`) and processed
incrementally (`BackfillsPerIteration` per tick). In v2, each backfill
request creates a separate `Backfiller` component that generates actions
independently.
- `schedule_callback_latency` (v2) -- recorded when any action's
completion callback arrives, regardless of whether it was manually
triggered.
**Interaction with regular actions:** Whether backfills contend with
regular actions depends on the backfill's overlap policy:
- **ALLOW_ALL (typical for backfills):** Actions started with ALLOW_ALL
are not tracked as running. This means they do not trigger overlap
policy evaluation on regular actions -- a SKIP schedule will continue to
start regular actions normally even while ALLOW_ALL backfill actions are
running. Buffer contention is the only possible interference: in v1,
backfills self-limit to half the buffer (`MaxBufferSize/2`) and add at
most `BackfillsPerIteration` (default 10) per tick, with the generator
running first each tick. Buffer contention is unlikely unless
backfill-started actions from previous ticks accumulate in the buffer
faster than they can be executed.
- **Non-ALLOW_ALL (SKIP, BUFFER_ONE, etc.):** Actions started with these
policies are tracked as running. A backfill-started action that is still
running will cause regular actions to be skipped or deferred by overlap
policy. This can show up as elevated `schedule_overlap_skipped` or
increased `schedule_action_e2e_delay` on regular actions while the
backfill is in progress. This is the primary contention scenario -- the
backfill's running actions block the schedule's normal execution.
## Glossary
- **Scheduled action**: The operation the scheduler executes at each
scheduled time. Currently this is always starting a workflow, but the
scheduler is designed to support other primitives (e.g., activities) in
the future.
- **Nominal time** (`NominalTime`): The exact time that matches the
schedule spec, before jitter is applied. Used to generate the workflow
ID (ensuring deterministic naming) and the request ID. Corresponds to
`GetNextTimeResult.Nominal`.
- **Actual time** (`ActualTime`): The nominal time with jitter applied.
This is the time the action is actually scheduled to execute.
Corresponds to `GetNextTimeResult.Next`. Stored on
`BufferedStart.ActualTime`.
- **Desired time** (`DesiredTime`): The baseline time used to compute
`schedule_action_delay`. Initially nil. When a previous action
completes, `DesiredTime` on the next pending buffered start is set to
the previous action's `CloseTime`. If nil at metric emission time, falls
back to `ActualTime`. This reset means `schedule_action_delay` measures
the gap from "slot opened" to "action started" for buffered starts, not
the total wait from the original schedule time.
- **Catchup window**: A configurable duration (default: 365 days,
minimum: 10 seconds) that determines how old an action can be before it
is permanently dropped. If `now - ActualTime > catchupWindow`, the
action is considered too stale to execute. Checked in the generator
(both v1 and v2) and again in the v2 invoker for buffered actions.
- **Overlap policy**: Determines what happens when a new action is due
but a previous action is still running.
- **SKIP**: Drop the new action. No concurrent or queued actions. This
is the default policy.
- **BUFFER_ONE**: Queue at most one pending action. When the running
action completes, the queued action starts. If another action comes due
while one is already queued, the new one is dropped.
- **BUFFER_ALL**: Queue all pending actions. Every due action is queued
and will eventually execute in order after the running action completes.
- **CANCEL_OTHER**: Cancel the running action and start the new one
immediately.
- **TERMINATE_OTHER**: Terminate the running action and start the new
one immediately.
- **Generator**: The phase that iterates over the schedule spec's time
range, checks the catchup window, and buffers actions for execution. In
v1 this is `processTimeRange`; in v2 this is the `GeneratorTaskHandler`.
- **Invoker / buffer processing**: The phase that evaluates overlap
policy on the buffer and executes actions. In v1 this is
`processBuffer`; in v2 this is the `InvokerProcessBufferTaskHandler`
followed by `InvokerExecuteTaskHandler`.
- **Deferred action**: A buffered action that was evaluated by
`ProcessBuffer` but could not be executed because a previous action is
still running under BUFFER_ONE or BUFFER_ALL policy. The action stays in
the buffer and is re-evaluated on the next processing pass. In v2,
deferred actions are marked with `Attempt = -1` to distinguish them from
newly enqueued actions. In v1, deferred actions remain in the buffer
with their original state and are re-evaluated on each tick.
|
||
|
David Porter
|
824d2d061f |
feat: Adding scanner for invariants for schedules v2 (#10406)
## What changed? This introduces an optional scanner for looking through v2 schedules and to perform some invariant checks and metric checks. The theory is that: - We should generally not see schedules with the nextActionTime in the past, and for those that do, they should be paused or be waiting for the workflow to fire. - We should not see workflows that are closed that are still around for a long period of time - We should expect (eventually, after migration and rollout of schedules-v2) all workflows to fit into the category of closed or running. Some of these assertions are possibly a bit premature as my understanding of the system is still ramping up, but it's relatively low risk, all the scanner does is emit a log or metric. ## Why? The intent is to just get a better handle on the schedules feature as it rolls out, owing to its criticality. It's hard to test for all edge-cases ahead of time, so looking for invariants in production can be illuminating. ## How did you test it? - [ ] built - [X] run locally and tested manually - [ ] covered by existing tests - [X] added new unit test(s) - [ ] added new functional test(s) For testing, I intentionally added a bug in the scheduler to hang the schedule after firing and then let it run and added lots of loud log messages so I could see if the scanner was able to detect it, which it was. This does *not* cover the cases where permissions might differ across namespaces, I still need to investigate that a bit more. ## Potential risks Probably fairly low: - A serious bug could cause a hot-loop of continue-as-new workflows if the activities were both misconfigured and unable to start, but otherwise it's unlikely to do much. - A misconfiguration in the rate-limit might cause visibility to get somewhat hammered by the scanner, generating some visibility load. |