mirror of
https://github.com/temporalio/temporal.git
synced 2026-08-30 18:41:49 -07:00
8451b876f3921169898d560ac4f94a86e9f1140f
620 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Feiyang Xie
|
8451b876f3 |
TS: fast forward and propagation policy change (#10576)
## What changed and why 1) make server compatible to breaking api change https://github.com/temporalio/api/pull/786 2) change the propagation policy of time skipping to - for the same execution (a chain of runs) all time skipping state and config are shared using StatePropagation - for child executions virtual time is always propagated to have causal time but the fast forward action is never propagated configuration propagation is controlled by a separate flag 3) rename bound to fast-forward ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) ## Potential risks the sequence of this pr 1. review this pr first 2. merge api 3. update this pr with public api dependency and merge |
||
|
Kannan
|
c0d871bc9a |
Implement CountWorkers API (#9476)
## What Add `CountWorkers` RPC to count workers matching a query filter without retrieving full worker details. ## Why The UI needs to display a worker count in places where listing isn't necessary. A dedicated count API follows the existing pattern (`CountWorkflowExecutions`, `CountSchedules`). ## How did you test it? - [x] new unit tests: count all, count with query filter, count with no matches, invalid query error --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Rodrigo Zhou
|
5fb11aef2b |
Add StartTime and IsRetentionDelete to delete visibility task (#10614)
## What changed? Add StartTime and IsRetentionDelete to delete visibility task ## Why? Additional info for delete visibility task that might be useful for implementing VisibilityStore. ## How did you test it? - [x] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks |
||
|
Shivam
|
7113cc0b13 |
Commit routing config before demoting old version in handleSetCurrent (#9973)
## What changed?
- Fix non-atomic two-step sync in handleSetCurrent and setRamp where
step 2 (demoting old version) failure leaves routing config uncommitted,
causing orphaned CURRENT versions and burned revision numbers.
- New flow: commit routing config immediately after step 1 (promote new
version) succeeds, then fire-and-forget signal to old version instead of
blocking sync activity. Gated behind workflow.GetVersion for NDE safety.
- The only thing that I don't have in this PR are new tests to test this
out. Happy to hear ideas if someone has any, but the core idea was that
the current ones should be passing and testing the code paths.
## Why?
- Reliability IMO
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- I would appreciate a very careful review on this one!
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **High Risk**
> Changes core worker-deployment routing and task-queue propagation in
async mode; demotion is now signal-based and eventually consistent, with
revision tracking on signal failure.
>
> **Overview**
> Fixes a reliability bug in **async** `set current` / `set ramp` where
promoting the new version succeeded but **demoting** the previous
current or ramping version could fail, leaving deployment routing
uncommitted and inconsistent summaries.
>
> When `workflow.GetVersion("commit-routing-first")` is enabled in async
mode, the deployment workflow **commits** `pendingRoutingConfig` to
local state right after the promote step, then **signals** the old
version workflow via new **`demote-version`** (`DemoteVersionSignalArgs`
carrying full `RoutingConfig`) instead of a blocking `syncVersion`
activity. Version workflows handle the signal (gated by
`demote-version-signal`) by deriving status from routing config, syncing
task queues, and starting drainage when needed.
**`signalDemoteVersion`** tracks propagating revision numbers until
`PropagationComplete`; failed signal delivery untracks the revision.
**`setVersionSummaryDraining`** updates deployment-side version
summaries immediately in the new path. Sync mode and workflows without
the version gates keep the prior behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
David Reiss
|
d69aede7e5 |
Dynamic partitioning: scale manager (#10365)
## What changed? Add scale manager component. ## Why? Part of dynamic partitioning. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) - to come in future PRs ## Potential risks No behavior changes yet. |
||
|
Shivam
|
26d6f98f16 |
Fix transient target version notification leak (#10444)
## What changed
- Adds a focused repro for a failed transient WFT that observes
`targetWorkerDeploymentVersionChanged=true` but never durably commits a
started event. Thus, in the next workflow task, we don't set the
`targetWorkerDeploymentVersionChanged=true` which results in errors with
respect to trampolining
- Now, to fix this, we just have the LastNotifiedTargetVersion present
in the MS to be the source of truth when filling in this flag.
## Why
A transient/speculative WFT can expose the target-version-changed flag
to the SDK before its started event is durable. If that WFT fails or is
discarded, we still should be sending the
`targetWorkerDeploymentVersionChanged=true` to our next workflow task.
## Verification
- `go test -tags test_dep ./tests -run
'TestVersioning3FunctionalSuite/TestPinnedCaN_FailedTransientNotificationDoesNotBecomeDeclinedOnPlainCaN'
-count=1`\n- `go test -tags test_dep ./tests -run
'TestVersioning3FunctionalSuite/TestPinnedCaN_FailedTransientNotificationDoesNotBecomeDeclinedOnPlainCaN'
-count=3`\n\nNote: broader existing trampolining functional tests hit
local test-cluster worker-deployment/user-data setup failures even from
a clean worktree at `c2c4bc1928`; CI should be the source of truth for
those.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches pinned versioning signals, persistence schema (reserved
field), and continue-as-new metadata; behavior change is intentional but
affects worker SDK upgrade notifications.
>
> **Overview**
> Fixes a **pinned worker-deployment** bug where a
**transient/speculative** workflow task could tell the SDK
`target_worker_deployment_version_changed=true` and set
`LastNotifiedTargetVersion`, but a failed/discarded task left the next
poll without that flag because it was no longer stored on mutable state.
>
> **`target_worker_deployment_version_changed` on `WorkflowTaskStarted`
is now derived at emit time** from `LastNotifiedTargetVersion` (plus
config and **PINNED** behavior) via
`targetWorkerDeploymentVersionChangedForStartedEvent()`, instead of
persisting `workflow_task_target_worker_deployment_version_changed` on
`WorkflowExecutionInfo` / `WorkflowTaskInfo`. Proto field **112** is
**reserved**; apply/rebuild paths and mocks drop the old parameter.
>
> **Continue-as-new:** `computeDeclinedTargetVersionUpgrade` only
propagates declined-upgrade metadata when the new run **inherits a
pinned version**; otherwise it returns nil.
>
> Adds
**`TestPinnedCaN_FailedTransientNotificationRefiresDespiteStaleMatching`**
(failed transient notification, mutable state still has
`LastNotifiedTargetVersion`, next WFT re-fires the signal even if
matching rolls back).
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Quinn Klassen
|
6258c4aab8 |
chasm: persist links and user_metadata at framework level (#10409)
## What changed? Add per-request links (keyed by request_id) and user_metadata fields to ChasmComponentAttributes so any CHASM component can store them uniformly without each library defining its own proto field. This first PR just adds support, I will open up follow up PRs to update any existing components use these new fields. ## Why? Per the discussion [here](https://github.com/temporalio/temporal/pull/10368#discussion_r3312238761) we think links and user metadata will be common components across CHASM components so it makes sense for the framework to handle this for us vs each component implementing it themselves. ## How did you test it? - [x] built - [ ] run locally and tested manually - [] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) |
||
|
David Reiss
|
503c6fb3c3 |
Dynamic partitioning: server validation (#9997)
## What changed? - Matching server propagates partition counts in ephemeral data. - Matching server validates partition counts sent by client, and rejects if they are too far. ## Why? Next part of dynamic partitioning. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) - covered by future integration tests ## Potential risks No change in behavior at this point, partition scale info is always nil, so nothing will be rejected. The server will start sending grpc trailers though. |
||
|
Alan Wu
|
1cc2e396e9 |
Add DeleteExecution history service API (#10119)
## What changed? Add a dedicated DeleteChasmExecution RPC to the history service for deleting CHASM executions by namespace, execution key, and archetype ID. Uses the CHASM engine's DeleteExecution path (terminate-if-running + async DeleteExecutionTask), replacing the ForceDeleteWorkflowExecution workaround in the delete namespace activity which bypassed the engine. Also adds NewComponentRefByArchetypeID to chasm/ref.go to construct a ComponentRef from a runtime archetype ID without a compile-time type parameter. ## Why? Remove dependency on force deletion API. ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [X] added new unit test(s) - [X] added new functional test(s) |
||
|
nikki-dag
|
573c2f787b |
Use proto-serialized context metadata in gRPC trailers (#10269)
## What changed
- Replaced per-key trailer format with a single protobuf
`ContextMetadata` message serialized into `contextmetadata-bin` trailer
key
- gRPC automatically base64-encodes the `-bin` value, making arbitrary
bytes (including HTTP/2-unsafe control chars) transport-safe
- Writer emits both proto format and legacy per-key format for backward
compatibility during rolling deploys
- Reader prefers proto key, falls back to legacy per-key format for old
writers
- Wired `TrailerToContextMetadataInterceptor` in test server to match
production behavior
## Why
Workflow type names containing control characters (newlines, NUL, etc.)
cause the gRPC HTTP/2 framer to reject trailer values. A single proto
message in a `-bin` key is simpler than per-key `-bin` suffixes: one
trailer key, one serialization, no key naming constraints, cleaner
backward compat removal.
## How tested
- Unit tests for proto round-trip, dual-format emission, reader
preference, legacy fallback, HTTP/2 safety
- Integration test suite (TestWorkflowTypeEncodingSuite) with control
chars, UTF-8, long names, -bin suffix workflow types
- All existing tests pass
## Risks
- During rolling deploy, old writers emit only legacy keys. New readers
handle this via fallback path. No data loss.
- After full rollout, legacy key emission can be removed in a follow-up.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes how context metadata is encoded/decoded in gRPC trailers,
which can affect cross-version compatibility and observability of
propagated metadata. Backward-compatible legacy fallback and extensive
unit/integration tests reduce the rollout risk.
>
> **Overview**
> **Switches context-metadata propagation in gRPC trailers to a single
proto-encoded payload.** Server-side `ContextMetadataInterceptor` now
serializes all context metadata into a new `ContextMetadata` protobuf
and emits it under `contextmetadata-bin`, avoiding HTTP/2-unsafe control
characters in values.
>
> **Maintains rolling-deploy compatibility.** Writers still emit legacy
per-key trailers (skipping unsafe values), and the client-side
`TrailerToContextMetadataInterceptor` now *prefers* the proto trailer
and falls back to legacy keys (including unprefixed well-known keys)
when needed.
>
> Adds the new `contextpropagation/v1` proto + generated Go types, plus
unit tests around proto/legacy behavior and an integration suite
(`WorkflowTypeEncodingSuite`) covering control characters, UTF-8, long
names, and `-bin` suffix workflow types.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Long Tran
|
27115b584d |
Propagate backlinks on Signal and Signal-with-Start responses (#9897)
## What changed? ### **High level** With https://github.com/temporalio/api/pull/761 to add the linking on the Signal and Signal-with-Start responses, This PR adds logic from the server that: * Adds `requestID` from Signal and Signal-with-Start requests to the CHASM workflow tree under a new map field `IncomingSignals`, and event store, so these requestIDs stay in buffer * Return a backlink in the response that references the `requestID` * On buffer flush to the DB transaction, attach these `requestID` to a concrete `eventID`, which would allow users to later know which event correlated w/ this request. We will wire the concrete event ID to the signal request IDs stored in the workflow component CHASM tree (`IncomingSignals` map) > [!NOTE] > Feature is gated behind a new dynamicconfig `EnableCHASMSignalBacklinks`, which implicitly is only checked if `EnableChasm` is enabled. ## Why? This will enable the caller of the signal to have a backlink to the cross-namespace signal invoked, which will become more relevant for Nexus SDK ergonomics. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [x] added new functional test(s) In functional tests, I augmented existing tests for Signal and Signal-with-Start to: * Ensure that backlink is returned via the responses * Later use `DescribeWorkflow` to ensure that we get a concrete EventID (mapped when buffer flushed) * Multiple signals with the same `requestID` gets de-duped ``` $ go test ./tests/ -run TestLinksTestSuite ok go.temporal.io/server/tests 1.486s ``` ``` $ go test ./tests/ -run 'TestNexusWorkflowTestSuite' -count=1 ok go.temporal.io/server/tests 4.714s ``` ## Potential risks Need to test end-to-end to see that the link shows up correctly in the Web UI. Feature is gated behind dynamicconfig since it requires CHASM-based workflow to be enabled. |
||
|
Alan Wu
|
4958253c8c |
Add CHASM lifecycle paused state (#10046)
## What changed? Add CHASM lifecycle paused state. Invalidates all pending logical tasks. ## Why? There are duplicate implementations in Libraries where authors have implemented their own statuses to act as a PAUSE state, preventing all tasks from executing. ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [X] added new unit test(s) - [ ] added new functional test(s) |
||
|
feiyang
|
e004436d8b |
time-skipping runtime-2 (with bound and backoff timers) (#10092)
## What changed? (1) elapsed-duration / skipped-duration **bounds** that auto-disable time-skipping (2) a new `TimeSkippingTimerTask` that wakes the workflow when an elapsed bound is reached (3) time-skipping support for backoff timers (start-with-delay / cron / retry / CaN-with-backoff) plus an exclusion for child workflows that haven't scheduled their first WT ## Why? the completion of the feature ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) |
||
|
feiyang
|
bc0354e194 |
time-skipping propagation (#10013)
## What changed and Why? Paired with [api change](https://github.com/temporalio/api/pull/770/changes) Define the default time-skipping propagation behavior for current features: continue-as-new (CAN), child workflows, retry, and reset. **Group 1 — Inherit both from the current execution:** - Continue-as-new (CAN): inherits the current config and accumulated skipped duration; the configured bound is shared across both the inherited skipped duration and any duration skipped by the new run. - Child workflows: same behavior as CAN This design is because CAN is a technical reason to start a new run of current run, so logically they can be viewed as a same "run". For Child WFs they can be viewed as an extension of previous workflows or separate workflows, and in either case, they shall inherit the skipped duration so that the virtual time doesn't rewind back, and the default behavior is designed to treat them as an extension of the parent and share the config. We only consider adding new config to provide flexibility on demand. **Group 2 — Inherit from a specific point in history:** - Retry: inherits the config and skipped duration recorded in the StartWorkflowExecutionEvent of the current workflow, since retry is defined as restarting execution from that event - Cron: same with Retry **Group3 -- Both inherit from a specific point and catch up of config change to current** - Reset: retains the current time-skipping config, since reset is designed to replay all events up to the reset point and apply any UpdateWorkflowExecutionOptions changes made after that point — with no option to exclude them (covered by tests only) ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) |
||
|
Stephan Behnke
|
ff3ed0b16f |
Nexus CHASM async completion (2/2) (#9972)
## What changed? Support completion-before-start. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) |
||
|
Shivam
|
562d26fc83 |
Skip reactivation signals for current/ramping/draining versions (#9778)
## Summary
- Extends `CheckTaskQueueVersionMembership` response with two new
fields: `is_version_active_or_draining` (bool) and `revision_number`
(int64). Matching populates both from its deployment data.
- Reactivation signals are **skipped** when matching reports the target
version as CURRENT/RAMPING/DRAINING; otherwise they are sent with a
**deterministic UUID v5 RequestId** derived from `revision_number`.
- Replaces the old TTL-based `ReactivationSignalCache` with a per-pod
**revision-based dedup LRU** on the worker-deployment client: each entry
records the highest revision this pod has successfully signaled for a
given version, so older or equal signals are skipped.
## What changed on the wire (matching → history)
`CheckTaskQueueVersionMembershipResponse` now has two new flat fields
(no wrapper message):
```proto
bool is_version_active_or_draining = 2; // true when status is CURRENT/RAMPING/DRAINING
int64 revision_number = 3; // from WorkerDeploymentVersionData.revision_number; 0 if unknown / legacy
```
Matching's `CheckTaskQueueVersionMembership` fills both via the helper
`worker_versioning.IsVersionActiveOrDraining(deploymentData, dep, build)
(bool, int64)`.
Naming choice — we picked `is_version_active_or_draining` (negative
polarity) rather than something like `supports_reactivation` so the
proto zero value (`false`) maps to the safe default ("send the signal").
Old matching binaries and runtime "version not found" both produce the
zero value, and history correctly falls through.
## Where `revision_number` flows
- **Matching**: populates the response field from the version's tracked
revision.
- **History-side helper/caches**:
`ValidateVersioningOverrideAndGetReactivationEligibility` returns
`(isVersionActiveOrDraining bool, revisionNumber int64, err)`.
`VersionMembershipAndReactivationStatusCache` stores both.
- **History signaler plumbing**: `VersionReactivationSignalerFn`,
`ReactivateVersionWorkflowIfPinned`, and all five call sites
(`startworkflow`, `signalwithstartworkflow`, `updateworkflowoptions`,
`resetworkflow`, `multioperation`) carry `revisionNumber int64`.
`resetworkflow.validatePostResetOperationInputs` returns parallel slices
`([]bool, []int64, error)` for per-operation inputs.
- **Signal RequestId**: `ClientImpl.SignalVersionReactivation` composes
`requestID = uuid.NewSHA1(uuid.NameSpaceOID,
[]byte("reactivation-signal:" + revisionNumber)).String()` — a
deterministic UUID v5 derived from the revision alone. Cassandra's
`signal_requested set<uuid>` column requires UUID-formatted RequestIds.
## Why revision-based dedup
History is sharded on `(namespaceID, workflowID)`. N concurrent
`StartWorkflow` calls pinned to the same drained version fan out across
potentially every history pod in the fleet. Before this PR each pod
independently fired a reactivation signal at the version workflow,
producing up to N `WorkflowExecutionSignaled` events — directly at odds
with the version workflow's design (it intentionally keeps history
minimal and CaNs aggressively, see `version_workflow.go:68-74`).
Per-pod caches alone can't fix this because they don't coordinate. What
we need is a **cluster-wide-deterministic dedup key** so all pods
converge on the same value for the same reactivation cycle. The
version's `revision_number` — incremented in `syncTaskQueuesAsync` on
every status change — is exactly that signal. Every pod reads the same
revision from matching, every pod composes the same UUID RequestId, and
Temporal's built-in `mutableState.pendingSignalRequestedIDs` dedup (see
`service/history/api/signalworkflow/api.go:40`) collapses concurrent
signals into exactly one event on the version workflow.
The per-pod map is a local optimization on top of that: it prevents a
single pod from re-sending the same-or-older-revision signal once it has
successfully sent one, cutting RPC volume.
## How the new caches look
### 1. `VersionMembershipAndReactivationStatusCache` (read-side,
per-pod)
Caches matching's `CheckTaskQueueVersionMembership` response so repeated
pinned-override validations on the same task queue don't re-hit
matching.
- **Key**: `(namespaceID, taskQueue, taskQueueType, deploymentName,
buildID)`
- **Value**: `(isMember bool, isVersionActiveOrDraining bool,
revisionNumber int64)`
- **Eviction**: `VersionMembershipCacheTTL` (1s default; 5s in
functional tests).
### 2. `highestRevSignaledToVersionWf` (write-side dedup, per-pod)
A field on `ClientImpl` in `service/worker/workerdeployment/client.go`.
For each target version workflow, stores the highest revision this pod
has successfully signaled. Subsequent calls at the same-or-lower
revision skip the RPC.
- **Key**: `reactivationVersionKey{namespaceID, deploymentName,
buildID}`
- **Value**: `int64` (highest revision successfully signaled)
- **Eviction**: LRU, bounded by `VersionReactivationSignalCacheMaxSize`.
The previous TTL-based `ReactivationSignalCache` module (in
`common/worker_versioning/`) has been deleted along with its provider
and `VersionReactivationSignalCacheTTL` config.
## Backwards/forwards compatibility
- **Old matching → new history**: old binaries don't set
`is_version_active_or_draining` or `revision_number`; both default to
proto zero values. `false` on the active bool → history falls through →
signal fires (safe default). `revisionNumber = 0` flows through as-is.
- **New matching → old history**: new fields on the response are ignored
by old history → identical to pre-PR behavior.
- **New matching → new history**: signal fires only when the version is
not active/draining; cross-pod fires converge on one UUID RequestId and
fold into one `WorkflowExecutionSignaled` event.
## Test plan
- [x] Unit tests for `IsVersionActiveOrDraining` covering all status
cases (CURRENT, RAMPING, DRAINING, DRAINED, INACTIVE, UNSPECIFIED), new
vs. old format, deleted and not-found versions.
- [x] Unit tests for
`ValidateVersioningOverrideAndGetReactivationEligibility` (cache
hit/miss, RPC with/without eligibility, Unimplemented fallback).
- [x] Unit tests for the per-pod dedup on
`ClientImpl.SignalVersionReactivation`: same-rev dedups, newer-rev
fires, older-rev skipped, different version isolated, signal-failure
allows retry.
- [x] Unit test for RequestId format (UUID v5, deterministic across
calls with the same revision).
- [x] Functional tests (all pass on SQLite and cass-es):
- `TestStartWorkflowExecution_ReactivateVersionOnPinned`
-
`TestStartWorkflowExecution_ReactivateVersionOnPinned_WithConflictPolicy`
- `TestSignalWithStartWorkflowExecution_ReactivateVersionOnPinned`
- `TestUpdateWorkflowExecutionOptions_ReactivateVersionOnPinned`
- `TestResetWorkflowExecution_ReactivateVersionOnPinned`
(The four `TestReactivationSignalCache_Deduplication_*` functional tests
from an earlier iteration were deleted — their coverage moved to unit
tests.)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes matching↔history API and reactivation signaling semantics by
skipping signals for active/draining versions and introducing
revision-based dedup via deterministic RequestIds; issues could affect
version workflow state transitions or signal fan-out during upgrades.
>
> **Overview**
> Matching’s `CheckTaskQueueVersionMembershipResponse` is extended with
`should_skip_reactivation` and `revision_number`, and matching now
populates both from per-task-queue deployment data.
>
> History-side versioning validation is refactored to return and cache
reactivation eligibility + revision, and reactivation signaling paths
(`StartWorkflow`, `SignalWithStart`, `UpdateWorkflowExecutionOptions`,
`ResetWorkflow`, multi-op) now **skip signals** when matching reports
the version as *CURRENT/RAMPING/DRAINING*.
>
> The old TTL-based `ReactivationSignalCache` is removed
(configs/metrics/providers updated), and the worker-deployment client
now performs **revision-based per-pod dedup** plus receiver-side dedup
by sending signals with a deterministic UUIDv5-like `RequestId` derived
from `revision_number`. Tests are updated/added to cover status
evaluation, new plumbing, and dedup behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Shivam
|
089fba53e1 |
Fix pinned workflow trampolining via revision-based signal suppression (#9895)
## Summary
- Matching now sends the real `targetDeploymentRevisionNumber` for
pinned workflows instead of hardcoded 0
- Added `revision_number` field to `LastNotifiedTargetVersion`
(server-internal) and `DeclinedTargetVersionUpgrade` (public API)
- Case 4 in the target version change switch now uses
`targetRevisionNumber <= declined.RevisionNumber` instead of version
string comparison, preventing trampolining when stale matching
partitions send outdated target versions
## Problem
When a pinned workflow CaNs and declines a target version upgrade, a
stale matching partition can send an older target version. The old case
4 compared version strings (`declined.buildId == target.buildId`), which
didn't match the stale version — causing the workflow to re-signal, CaN
again, and trampoline indefinitely between stale and up-to-date
partitions.
## Test plan
- [x] `TestStalePartition_RevisionSuppressesTrampolining` — integration
test that simulates a stale partition via `rollbackTaskQueueToVersion`
and verifies:
- Stale partition (revision 0) is suppressed after declining at a higher
revision
- Genuinely new version (v4 at higher revision) correctly fires the
signal
- [x] Verified test **fails** when matching + history changes are
reverted (matching sends 0, old string comparison)
## API repo dependency
- api: `temporalio/api@trampolining-rev-number`
- api-go: `temporalio/api-go@trampolining-rev-number`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches pinned-workflow versioning logic across matching, history, and
persistence; a bug here could change when workflows are signaled to
continue-as-new or upgrade, affecting routing behavior.
>
> **Overview**
> Prevents pinned workflows from repeatedly continue-as-new
“trampolining” when task queue partitions have stale routing data by
**tracking and comparing target routing-config revision numbers**.
>
> Matching now propagates the real `targetDeploymentRevisionNumber` for
pinned workflow tasks, and history threads this through
`AddWorkflowTaskStartedEvent` to persist
`LastNotifiedTargetVersion.revision_number` and carry it into
`DeclinedTargetVersionUpgrade` on continue-as-new. The pinned
target-change decision in `workflow_task_state_machine` switches case-4
suppression from deployment version string equality to
`targetRevisionNumber <= declined.RevisionNumber`, and adds an
integration test (`TestStalePartition_RevisionSuppressesTrampolining`)
covering stale vs fresh partition behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Stephan Behnke
|
b78391b445 |
Nexus CHASM async completion (1/2) (#9951)
## What changed? Support Nexus async completion in CHASM. PS: completion-before-start will be handled in a follow-up PR. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) ## Potential risks Is behind feature flag. |
||
|
Kannan
|
005d1849c6 |
Implement new task queue kind to send worker commands (#9899)
## What Add support for `TASK_QUEUE_KIND_WORKER_COMMANDS` in the matching service. This is a new task queue kind for server-to-worker communication (e.g. activity cancellations). It is represented by a new `WorkerCommandsPartition` type in the `tqid` package — analogous to how `StickyPartition` represents sticky queues. Also adds a distinct `partition` metric tag for worker-commands queues so their traffic can be distinguished from user-facing normal queues in metrics. Depends on [#9900](https://github.com/temporalio/temporal/pull/9900). ## Why Worker_Commands queue kind has different properties compared to a normal user defined queue. So we created a separate kind. This allows us to distinguish task queue metrics based on whether they are internal or user created queue. ## How did you test it? Unit tests covering metric tag generation for normal, sticky, and worker-commands partitions. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: David Reiss <david@temporal.io> |
||
|
David Reiss
|
1b2f50bea1 |
Dynamic partitioning: client side (#9732)
## What changed? Add client-side of dynamic partition scaling: `partitionCache`, `PartitionCounts`, `StalePartitionCounts` error, `invokeWithPartitionCounts`. The intended behavior is: For partition-aware calls (add+poll tasks), the client caches the partition count for active task queues, and uses those partition counts for load balancing (no change to load balancing algorithm yet). The client sends its cached partition counts to the server in a grpc header, and receives updated partition counts with the response in a grpc trailer. If the updated counts are different, it updates its cache it for subsequent requests. If the server indicates that the client's view of partition count is stale, it returns a special `StalePartitionCounts` error and then client makes one immediate retry with the newly received counts. With just this PR by itself, the server will never send counts, the cache will always be empty, and the client will always fall back to dynamic config for partition counts, so there's no change in behavior yet. ## Why? Implement half of dynamic partition scaling. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) – tests are in future PRs |
||
|
feiyang
|
c5badc94b5 |
time-skipping runtime foundation (#9965)
## What changed? 1) `taskGenerator`: - add regenerate tasks for time skipping 2) `mutableState`: - new methods: add key methods to transition time - `IsStateDirty`: captures changes of time-skipping related fields - `closeTransaction`: add closeTransactionHandlerTimeSkipping 3) event: new time-skipping runtime event added 4) persistence: new runtime data added to execution info ## Why? add the foundational mechanism of how time skipping works in the runtime of a workflow execution **for reviewers:** this is a foundation of t-s runtime, so it works without any granular and extended features as bound, replication, external-transfer tasks, etc. **Over 50% of lines are tests/mocks/pb-gen. can focus on code first, then functional tests, and last ut.** ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) |
||
|
Kannan
|
856e276e58 |
Dispatch activity cancellation to worker using Nexus (#9233)
## What Dispatches worker commands (starting with activity cancellation) to workers via their Nexus control queue. When the outbound queue processes a `WorkerCommandsTask`, the dispatcher sends an `ExecuteCommands` Nexus operation to the worker's control queue via `DispatchNexusTask`. - Retries are capped at 3 attempts since these commands are best-effort (the activity will eventually time out anyway). Uses the retryable matching client so transient RPC errors (ShardOwnershipLost, Unavailable) are retried at the RPC layer without consuming queue-level attempts. - Stores the clock used to generate the task token in the ActivityInfo. This is needed to reconstruct the same task token when dispatching to the worker. Otherwise, it will not match the token expected by the sdk. Suggested review order: `worker_commands_task_dispatcher.go` → `nexus_dispatch_response.go` → `recordactivitytaskstarted/api.go`. ## Why To support activity cancellation without activity heartbeat. This is the dispatch leg of the flow: 1. [#9231] Store `worker_control_task_queue` in `ActivityInfo` at activity start. 2. [#9232] On `RequestCancelActivityTask`, batch commands by control queue into `WorkerCommandsTask` outbound tasks. 3. **[This PR]** Dispatch each task as a Nexus `ExecuteCommands` operation to the worker, with a 3-attempt retry cap. 4. [SDK] Worker receives the cancel command and cancels the running activity. Gated by dynamic config `EnableCancelActivityWorkerCommand` (default: off). ## How did you test it? - **Unit tests** cover all dispatch outcomes (success, RPC error, timeout, worker error, feature-flag-off, max-attempts-exceeded) and response-to-error conversion paths. - **Functional test** verifies end-to-end: cancel request → Nexus dispatch → correct payload arrives on the control queue, and asserts that the cancel command's task token matches the one from the original activity poll response. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: rkannan82 <5853897+rkannan82@users.noreply.github.com> |
||
|
Veeral Patel
|
94556cc8ce |
feat: add GetTaskQueueUserData RPC to admin service (#9934)
## What changed? This PR adds a new `GetTaskQueueUserData` RPC to the admin service. Given a namespace, task queue name, task queue type, and optional partition ID (default is 0, which is `root`), it returns the user data currently loaded by that partition. This PR wraps the existing `GetTaskQueueUserData` RPC in Matching Service. We will also create a `tdbg` command which calls this Admin Service RPC, in a separate PR. ## Why? Each task queue family has associated metadata, stored in [TaskQueueUserData](https://github.com/temporalio/temporal/blob/main/proto/internal/temporal/server/api/persistence/v1/task_queues.proto). Metadata related to worker versioning, queue rate limiting, fairness are all stored in `TaskQueueUserData`. TaskQueueUserData is replicated from the root partition to all other partitions. However, there is no admin-accessible way to read the user data loaded by a specific partition or compare versions across partitions to diagnose replication lag. ## Files changed | File | Change | |---|---| | `proto/internal/.../adminservice/v1/request_response.proto` | Added `GetTaskQueueUserDataRequest` and `GetTaskQueueUserDataResponse` messages | | `proto/internal/.../adminservice/v1/service.proto` | Added `GetTaskQueueUserData` RPC to `AdminService` | | `service/frontend/admin_handler.go` | Implemented `AdminHandler.GetTaskQueueUserData`: validates request, resolves namespace → ID, builds partition RPC name via `tqid`, calls matching service, returns per-type entry + version | | `service/frontend/admin_handler_test.go` | Added unit tests | ## How did you test it? - [x] built - [x] run locally and tested manually - [x] added new unit test(s) - [x] added new integration test(s) - not applicable, not touching persistence layer - [x] added new functional test(s) ### Unit tests 100% unit test coverage | Test case | Input | Expected | |---|---|---| | Nil request | `request == nil` | `errRequestNotSet` | | Empty namespace | `namespace == ""` | `errNamespaceNotSet` | | Namespace not found | Namespace registry returns not-found | Error propagated; matching never called | | Invalid task queue name | `task_queue` starts with `/_sys/` | `INVALID_ARGUMENT` from `tqid.NewTaskQueueFamily`; matching never called | | Root partition | `partition_id=0`, workflow type | Sends bare name `my-queue` to matching; returns correct `user_data` and `version` | | Non-root partition | `partition_id=1`, workflow type | Sends mangled name `/_sys/my-queue/1` to matching | | No per-type data | Matching returns response with empty `per_type` map | `user_data` is nil; `version` still populated | | Matching error | Matching client returns error | Error propagated to caller | ### Functional tests | Test | Setup | What it verifies | |------------------------------------------------|--------------------------------|----------------------------------------------------------------------------------| | TestAdminGetTaskQueueUserData_RootPartition | Write fairness weight config to a workflow task queue | Admin RPC resolves namespace by name, routes to root partition (partition_id=0), returns version > 0 and non-nil per-type data | | TestAdminGetTaskQueueUserData_NonRootPartition | Same write, then poll until non-root partition replicates | Admin RPC routes to a non-root partition (partition_id=1) via mangled name, returns the same version as root after replication | ### Manual tests <details> <summary>Setup</summary> 1. Build and start the server: `make temporal-server && make start-sqlite` 2. Create namespace: `temporal operator namespace create default` 3. Insert assignment rule: `temporal task-queue versioning insert-assignment-rule` </details> <details> <summary>Case 1 — Root partition, workflow type</summary> ``` grpcurl -plaintext \ -d '{"namespace":"default","task_queue":"my-queue","task_queue_type":"TASK_QUEUE_TYPE_WORKFLOW"}' \ localhost:7233 \ temporal.server.api.adminservice.v1.AdminService/GetTaskQueueUserData ``` ```json { "version": "1" } ``` </details> <details> <summary>Case 2 — Root partition, activity type</summary> ``` grpcurl -plaintext \ -d '{"namespace":"default","task_queue":"my-queue","task_queue_type":"TASK_QUEUE_TYPE_ACTIVITY"}' \ localhost:7233 \ temporal.server.api.adminservice.v1.AdminService/GetTaskQueueUserData ``` ```json { "version": "1" } ``` </details> <details> <summary>Case 3 — Non-root partition, workflow type</summary> ``` grpcurl -plaintext \ -d '{"namespace":"default","task_queue":"my-queue","task_queue_type":"TASK_QUEUE_TYPE_WORKFLOW","partition_id":1}' \ localhost:7233 \ temporal.server.api.adminservice.v1.AdminService/GetTaskQueueUserData ``` ```json { "version": "1" } ``` </details> <details> <summary>Case 4 — Non-root partition, activity type</summary> ``` grpcurl -plaintext \ -d '{"namespace":"default","task_queue":"my-queue","task_queue_type":"TASK_QUEUE_TYPE_ACTIVITY","partition_id":1}' \ localhost:7233 \ temporal.server.api.adminservice.v1.AdminService/GetTaskQueueUserData ``` ```json { "version": "1" } ``` </details> <details> <summary>Case 5 — Non-root partition, activity type, with user data</summary> Setup: Add rate limit config ``` grpcurl -plaintext \ -d '{ "namespace": "default", "task_queue": "my-queue", "task_queue_type": "TASK_QUEUE_TYPE_ACTIVITY", "update_queue_rate_limit": { "rate_limit": { "requests_per_second": 50.0 }, "reason": "manual test" } }' \ localhost:7233 \ temporal.api.workflowservice.v1.WorkflowService/UpdateTaskQueueConfig ``` ``` grpcurl -plaintext \ -d '{ "namespace": "default", "task_queue": "my-queue", "task_queue_type": "TASK_QUEUE_TYPE_ACTIVITY" }' \ localhost:7233 \ temporal.server.api.adminservice.v1.AdminService/GetTaskQueueUserData ``` ```json { "userData": { "config": { "queueRateLimit": { "rateLimit": { "requestsPerSecond": 50 }, "metadata": { "reason": "manual test", "updateTime": "2026-04-13T21:40:39.888Z" } } } }, "version": "2" } ``` </details> <details> <summary>Case 6 — Namespace not found</summary> ``` grpcurl -plaintext \ -d '{"namespace":"nonexistent","task_queue":"my-queue","task_queue_type":"TASK_QUEUE_TYPE_WORKFLOW"}' \ localhost:7233 \ temporal.server.api.adminservice.v1.AdminService/GetTaskQueueUserData ``` ``` ERROR: Code: NotFound Message: Namespace nonexistent is not found. ``` `NOT_FOUND` from namespace registry; matching never called. </details> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
feiyang
|
ef19f17c95 |
timeskipping: add TimeSkippingConfig support to workflow-start APIs (#9834)
## What changed?
Add time-skipping configuration support.
(1) Feature flag in dynamic config
- frontend.TimeSkippingEnabled (namespace bool, default false) gates the
feature
(2) Frontend validation (validateTimeSkippingConfig)
- StartWorkflowExecution
- SignalWithStartWorkflowExecution
- ExecuteMultiOperation (via the shared prepareStartWorkflowRequest
path)
(3) Persistence
- TimeSkippingInfo proto added to WorkflowExecutionInfo
- MutableState stores the config when a workflow starts or options are
updated
(4) Tests
- New timeskipping functional tests in tests/timeskipping_test.go
## Why?
first step of the time skipping project
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
---------
Co-authored-by: Yichao Yang <yycptt@gmail.com>
|
||
|
Kannan
|
f5246b3fdd |
Add WorkerCommandsTask outbound task to dispatch worker commands via Nexus (#9232)
## What changed? New outbound task type (`WorkerCommandsTask`) that carries worker commands to be dispatched to workers via Nexus. Uses the generic `WorkerCommand` proto (not cancel-activity-specific), so this task type can carry any future command types. Suggested review order: proto changes → `worker_commands_task.go` → `task_generator.go` → `workflow_task_completed_handler.go` Key pieces: - **Proto**: `TASK_TYPE_WORKER_COMMANDS` enum, `WorkerCommandsTask` in `OutboundTaskInfo` with `repeated WorkerCommand`. - **Task definition**: `worker_commands_task.go` — implements outbound `Task` and `HasDestination` interfaces. - **Task creation** (`workflow_task_completed_handler.go`, `task_generator.go`): When `RequestCancelActivityTask` is processed for a started activity whose worker has a control queue, collects a `CancelActivityCommand` with the activity's task token. Commands are batched by destination control queue and flushed as one `WorkerCommandsTask` per queue at the end of WFT processing. - **Serialization**: `task_serializers.go` for persistence round-tripping. Dispatch is a no-op here — handled in #9233. Gated by dynamic config `EnableCancelActivityWorkerCommand` (default: off). ## Why? To support proactive activity cancellation without waiting for heartbeat. This is the task creation leg of the flow. 1. [#9231] Store `worker_control_task_queue` in `ActivityInfo` at activity start. 2. **[This PR]** On `RequestCancelActivityTask`, batch commands by control queue into `WorkerCommandsTask` outbound tasks. 3. [#9233] Dispatch each task as a Nexus `ExecuteCommands` operation to the worker, with a 3-attempt retry cap. 4. [SDK] Worker receives the cancel command and cancels the running activity. Gated by dynamic config `EnableCancelActivityWorkerCommand` (default: off). ## How did you test it? **Unit tests** cover task generation, command batching (including multi-queue batching), task serialization round-tripping, and the feature-flag-off path. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Kannan
|
4997c93fa8 |
Store worker attributes needed by server to propagate nexus tasks to worker (#9231)
## What changed? As part of RecordActivityTaskStarted flow, store worker_control_task_queue for an activity in the mutable state (ActivityInfo). Main changes: - executions.proto: Added the new worker_control_task_queue field. - mutable_state_impl.go: Update mutable state. - matching/forwarder.go: Propagate worker_control_task_queue when polls get forwarded. Otherwise, RecordActivityTaskStarted request will not have it set when invoked from a forwarded poll. ## Why? To support activity cancellation without activity heartbeat. Overall flow: - [This PR] Store worker attributes in ActivityInfo as part of RecordActivityTaskStarted call. - [#9232] When user cancels a workflow, create 1 or more tasks. Group all activities belonging to a worker into the task (for efficiency). - [#9233] Lookup the Nexus task queue for each worker, and send a Nexus operation for each transfer task. - [SDK] Worker will receive this cancel task and cancel the running activities. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Shivam Ajmera
|
578dd4d0d1 |
change extension number (#9898)
## What changed? Update proto extension number 7234 to 50234 ## Why? it was colliding with a saas-control-plane dep (discussed [here](https://temporaltechnologies.slack.com/archives/C0ARSCEHU75/p1775773612977229?thread_ts=1775773490.781049&cid=C0ARSCEHU75)) ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Carly de Frondeville
|
ac3c439a1a |
Add CONTINUE_AS_NEW_VERSIONING_BEHAVIOR_USE_RAMPING_VERSION (#9894)
## What changed?
Add `CONTINUE_AS_NEW_VERSIONING_BEHAVIOR_USE_RAMPING_VERSION` as a
`ContinueAsNewInitialVersioningBehavior` in `InheritedAutoUpgradeInfo`
and `VersioningInfo`, which if set, forces the first task of an
AutoUpgrade workflow to go to the Ramping Version of its task queue
instead of basing the Current/Ramping version selection on ramp
percentage and workflow id. If there is no Ramping Version at the time
of workflow dispatch, the workflow will use the Current Version.
This initial behavior comes from the continue-as-new command, and when
history sends workflow tasks to matching, is converted into an
internally defined `UseRampingVersionInitialTask bool` such that _only_
the first workflow task of an AutoUpgrade workflow will change its
Target Version selection. Retries of this workflow will start with the
same behavior, but child workflows and future continue-as-new workflows
initiated by this workflow will not inherit the initial behavior.
## Why?
To enable more fine grained control of upgrade-on-continue-as-new
upgrade stages before doing percentage based ramp. To use this, users
should set their promotion candidate version to the ramping version with
a ramp percentage of zero, and manually signal a certain cohort of
workflows to upgrade-on-continue-as-new with this option.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes workflow-task routing for AutoUpgrade executions by adding a
new directive flag that can bypass ramp-percentage hashing for the
initial task. Moderate risk because it affects matching/history version
selection and ContinueAsNew/retry inheritance semantics.
>
> **Overview**
> Adds a new `use_ramping_version` field to `TaskVersionDirective` and
threads it through history → matching so an AutoUpgrade workflow can
*force its initial workflow task* to route to the task queue’s ramping
deployment version (falling back to current when no ramping version
exists), bypassing the normal ramp-percentage/workflow-id hash.
>
> History now persists and evaluates the ContinueAsNew-requested initial
behavior via `MutableState.GetShouldUseRampingVersion()`, ensures it
applies only to the first workflow task, propagates it across retries,
and explicitly does **not** propagate it to child workflows or
subsequent ContinueAsNew hops. Updates tests (unit +
`versioning_3_test`) and bumps `go.temporal.io/api` to pick up the new
proto/enum support.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Shahab Tajik
|
744d3f2efe |
Add compute config to WorkerDeploymentVersionSummary (#9838)
## What changed? - Add compute config summary for latest and all other version so it is exposed in `ListWorkerDeployments` and `DescribeWorkerDeployment` APIs. ## Why? Mainly for UI to show compute provider types. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks None |
||
|
Shahab Tajik
|
d8813fd133 |
Serverless Feature Integration (#9779)
This PR merges the serverless feature branch into main. Individual PRs included in this branch: - #9380 - #9651 - #9412 - #9746 - #9759 --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Stefan Richter <stefan.richter@temporal.io> |
||
|
jiechenz
|
72567b9ed1 |
Replicate workflow deleteion (#9717)
## What changed? Add a new replication task type `DeleteExecutionReplicationTask` that replicates workflow deletion from the active cluster to passive/standby clusters. Gated by feature flag `history.enableDeleteWorkflowExecutionReplication` (default: false). #### **Key changes across the replication pipeline:** 1. Proto enums: `TASK_TYPE_REPLICATION_DELETE_EXECUTION` (34), `REPLICATION_TASK_TYPE_DELETE_EXECUTION_TASK` (13) 2. Replication task is associated with a new stage in `ShardContext.DeleteWorkflowExecution`, bundled with delete visibility task. 3. ~Engine interface: added `ForceDeleteWorkflowExecution` so the task can invoke the `ForceDeleteWorkflowExecution`.~ ## Why? Today, when a user delete workflow execution in source cluster, this operation will not replicate to the standby/target clusters. When a namespace failover to a target cluster, those deleted workflow may resurrected. <details> <summary>Race condition analysis</summary> **Before this change:** 1. **Cross-cluster resurrection:** Active deletes workflow → standby untouched → failover → workflow reappears. 2. **Termination event silently dropped:** Deleting a running workflow terminates it first, generating a `HistoryReplicationTask`. But the async `CloseExecutionTask` may delete mutable state before the stream sender converts that task. The converter calls `getBranchToken()` → `NotFound` → task silently dropped. The standby never sees the termination or the deletion. **After this change:** Race 1 is fixed — `DeleteExecutionReplicationTask` explicitly tells the standby to delete. Race 2 is mitigated — even if the termination event's replication task is dropped, the delete replication task ensures the standby cleans up. - If the workflow is still running (termination not yet replicated), the `DeleteExecutionTask` reschedules itself until the workflow closes. - If the termination event arrives later, the workflow closes normally, then the delete proceeds. - If the workflow is already deleted (e.g., by retention), the task is a no-op (`NotFound` treated as success). </details> <details> <summary>Deletion paths</summary> | Path | Replication task? | |------|-------------------| | User deletes workflow (active, running or closed) | Yes | | User deletes on passive (DC forwarding ON) | Forwarded to active → yes | | User deletes on passive (no forwarding) | No — `ActiveInCluster` check skips | | Retention expiry (with or without archival) | No — stage pre-marked as processed | | Admin ForceDelete (tdbg) | No — bypasses `DeleteWorkflowExecution` | </details> ## How did you test it? - [x] built - [x] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) Before change: <img width="1507" height="163" alt="Screenshot 2026-03-26 at 11 59 51 PM" src="https://github.com/user-attachments/assets/118cc50e-b69d-468a-9e45-5f49e4e4b9d1" /> After change: <img width="1507" height="135" alt="Screenshot 2026-03-27 at 12 00 07 AM" src="https://github.com/user-attachments/assets/8ccb7a11-2cb4-48b5-af89-b4a60ddb6333" /> ## Potential risks n/a |
||
|
Stephan Behnke
|
1d58ed21cc |
buf format (#9663)
## What changed? Adds `buf format` as a Makefile target; and integrates it into `make fmt`. All `.proto` changes are from running `make fmt`. ## Why? Consistent protobuf file style. ## How did you test it? - [x] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Shahab Tajik
|
d344f08be8 |
Ensure WD Version properly revives if recreated after deletion (#9382)
## What changed? - Ensure the TQs receive and apply the right version data after revive. - Made delete propagation to always happen serial to other propagations. It ensures all other propagations are cancelled before starting delete propagation. - Deprecate the `deleted` flag in version data and the GC logic around it. Now we use the good old forgetVersion path which immediately removes the version data from TQ. - Ensure version state is reset after revive, in case the recreation happened before workflow close. - Also, now workflows CaN based on SDK suggestion if no pending Signal or Update is present. ## Why? The version could stuck at deleted state from TQ POV if revived before the (now deprecated) GC logic cleans it up. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) ## Potential risks None --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
David Reiss
|
d0764befb4 |
Add drained field to InternalTaskQueueStatus (#9453)
## What changed? Add drained field to InternalTaskQueueStatus. ## Why? This is useful to tell if a queue is empty. Will be used for automatic partition scaling. ## How did you test it? - [ ] built - [x] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
David Reiss
|
24cea9d4f3 |
Separate ephemeral data for different task queue types (#9450)
## What changed and why? Ephemeral data was meant to be separate per task queue type (instead of for all types together like task queue user data), but the implementation didn't match that exactly: activity/other queues did propagate ephemeral data from the workflow queue. This fixes that to not pass ephemeral data along that edge. Also reduce the `returning user data` log to debug level. ## How did you test it? - [ ] built - [x] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) |
||
|
Rodrigo Zhou
|
6b11e07e8f |
[Chasm] Introduce proto ChasmExecutionInfo for Visibility (#9652)
## What changed? Introduce proto types for Visibility: - `ChasmExecutionInfo` - `ListChasmExecutionsRequest` - `ListChasmExecutionsResponse` Changed VisibilityManager API to use these protos. ## Why? Make VisibilityManager API more accessible for potential external usage. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks |
||
|
Yichao Yang
|
0463b59dd9 |
CHASM: support archetypeID in admin handler (#9309)
## What changed? - Accept archetypeID in admin API requests ## Why? - With this we no longer need to register chasm components to worker service, which currently performs an archetypeID to name conversion before calling admin apis. - For backward compatibility, we can only stop registering chasm components to worker service starting from next cloud release. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Lina Jodoin
|
e535cb5d01 |
[Scheduled Actions] Attach Nexus callbacks to schedules migrated from V1/Legacy (#9560)
## What changed? - Nexus callbacks are attached to running workflows after migration from a V1 workflow-backed schedule. This is done through a new side-effect task. ## Why? - Drives workflow completion events. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [x] added new functional test(s) |
||
|
Sean Kane
|
ff2754a711 |
Add CallbackRequestID to workflow execution for use by CHASM Schedules (#9479)
## What changed?
When a workflow is reset, `ApplyWorkflowExecutionStartedEvent`
re-registers the start-event callbacks using the reset operation's
request ID. `HandleNexusCompletion` cannot find a matching
`BufferedStart` and discards the completion. The `resetRequestID` param
is removed from `WorkflowResetter.ResetWorkflow` and the original
request ID is used. `findStartRequestID` reads the original request ID
back from `WorkflowExecutionInfo.RequestIds` by finding the
`EVENT_TYPE_WORKFLOW_EXECUTION_STARTED` entry.
## Why?
CHASM scheduler relies on callback `request_id` to match WF completions
to originating `BufferedStart` entries. When it cannot be found the
scheduler is permanently stuck with the workflow marked as still
running.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
-
`TestScheduledWorkflowDoubleReset_SchedulerSeesCompletion_{HSM,CHASM}Callbacks`:
create a schedule, trigger immediately, reset the workflow twice, signal
the completion to complete, poll `ListSchedules` until scheduler shows
`COMPLETED`.
## Potential risks
CHASM scheduler has not been enabled in production yet, the blast radius
should be minimal.
|
||
|
Shahab Tajik
|
26f2023316 |
Improve task-dispatch-latency metric (#9395)
## What changed? Emit task dispatch latency metric in matching_engine with the following improvements: - latency includes history calls - latency is not reset in case of sync-match forwards - metric is not lost for backlogged task in new matcher - origin partition is preserved during forward and used at the partition tag in the metric (keeping existing behavior) - task and poll forwarding do not cause duplicate emits, keeping existing behavior Also, fixed the following unrelated bugs that surfaced while testing the metric: - Query priority is not lost when forwarded. ## Why? Fixes bugs. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [x] added new functional test(s) ## Potential risks None. |
||
|
Shivam
|
b6ae24e69e |
Trampolining Part 2: Avoid infinite loops for Pinned workflows (#9374)
## What changed?
- Consists of the change to prevent any Pinned workflows, that may have
forgotten to have the initial CAN Behaviour as AU, from CAN'ing
infinitely.
- Also allows trampolining of a Pinned workflow onto the Unversioned if
un-versioned is the current version of the worker deployment at that
point in time. Note, the effective behaviour of the workflow would later
then be unversioned.
## Why?
- Worker-Versioning correctness.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- Pre-existing workflows (started before this fix) have nil
TargetVersionOnStart. On the first WFT after deployment, "" !=
"build-v2" → spurious targetDeploymentVersionChanged=true. This is a
one-time false-positive regression for those Pinned workflows.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches generated protobuf API surface used by internal services;
while the change is additive, mismatched proto versions across services
could cause integration/compatibility issues.
>
> **Overview**
> Adds `declined_target_version_upgrade` to
`StartWorkflowExecutionRequest` (HistoryService API) so
continue-as-new/retry chains can carry forward the SDK-declined target
deployment version and avoid pinned-workflow trampolining loops.
>
> Regenerates protobuf Go bindings, updating import/type references
across `request_response.pb.go`, and adds missing
`Marshal`/`Unmarshal`/`Size`/`Equal` helpers for the persistence
`LastNotifiedTargetVersion` message.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
|
||
|
Alex Stanfield
|
fe35b108fe |
V1 to CHASM scheduler migration RPC (#9261)
## What changed Adds an admin RPC (`MigrateSchedule`) that migrates workflow-backed (V1) schedulers to CHASM (V2) schedulers. The flow: 1. Admin API signals the V1 scheduler workflow with `migrate-to-chasm` 2. V1 workflow runs a local activity that snapshots state and calls `CreateFromMigrationState` on the CHASM scheduler service 3. On success, the V1 workflow closes. On failure, a `PendingMigration` flag persists so the next run loop iteration retries automatically Key details: - `CreateSchedulerFromMigration` initializes the full CHASM scheduler tree (generator, invoker, backfillers, visibility) from V1 state - Running/completed workflows are converted to buffered starts; ongoing backfills are preserved - Migration is idempotent -- if the CHASM schedule already exists, it's treated as success - Metrics: `schedule_migration_started`, `schedule_migration_completed`, `schedule_migration_failed` (with direction tag) - Local activity uses 10m schedule-to-close timeout; retries happen at the workflow level via the persistent flag ## Why Needed for migrating existing V1 schedulers to CHASM without downtime or data loss. ## Follow-up items - V2 to V1 migration (rollback path) - Sentinel key handling when `EnableCHASMSchedulerCreation` was previously enabled - Attach completion callbacks to running workflows after migration |
||
|
Yu Xia
|
7eae6ac604 |
Introduce api category into history APIs (#9435)
## What changed? Introduce api category into history APIs ## Why? There are some APIs we want to exclude from health check. Using API category is a more protective way to manage this group. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Kannan Rajah
|
450b7f95d6 |
Populate WorkerListInfo when returning ListWorkersResponse (#9418)
## What changed? - Build `WorkerListInfo` in matching handler and pass through to frontend - Update `go.temporal.io/api` to include merged WorkerListInfo proto ## Why? `WorkerListInfo` contains only static worker attributes for efficient listing. Building it in matching (closer to the data source) enables eventually dropping the deprecated `WorkersInfo` field from the internal response. ## How did you test it? - [x] built - [x] added new unit test for field coverage - [x] covered by existing functional tests ## Potential risks None - backward compatible. Deprecated `WorkersInfo` field continues to be populated. |
||
|
Lanie Hei
|
88ea88ce1a |
Add per-check diagnostics to DeepHealthCheck API (#9350)
## Summary
Extends the `DeepHealthCheck` API to return **per-check diagnostic
details** alongside the existing `HealthState` enum. When fault
detection triggers a cell failover, operators can now see exactly which
health check failed and why — not just that the cell is unhealthy.
### What changed
- **New proto package `health/v1`** — `HealthCheck` message with
`check_type` (string), `state`, `value`, `threshold`, and human-readable
`message`. `HostHealthDetail` and `ServiceHealthDetail` aggregate
per-host and per-service results.
- **New enum value `HEALTH_STATE_INTERNAL_ERROR`** — for infrastructure
failures like membership resolver errors (previously returned
`UNSPECIFIED`).
- **History handler** now runs all 5 checks unconditionally (gRPC
health, RPC latency, RPC error ratio, persistence latency, persistence
error ratio) and returns each with actual values and thresholds.
Previously it early-returned on first failure.
- **Frontend health checker** collects per-host `HostHealthDetail`
(address, state, checks) and builds a `ServiceHealthDetail` with
diagnostic messages for all paths — including resolver errors and empty
membership.
- **AdminService** passes `ServiceHealthDetail` through to callers.
- **`check_type` uses string constants**
(`common/health/check_types.go`) instead of a proto enum for
extensibility — new check types can be added without proto changes, and
the `message` field provides human-readable context with actual values
(e.g. `"RPC latency 850.00ms exceeded 500.00ms threshold"`).
### How it works
The call chain is: `AdminService.DeepHealthCheck()` →
`HealthChecker.Check()` → fan-out to all history hosts in membership →
`HistoryHandler.DeepHealthCheck()` per host.
Each history host runs **5 independent checks** and returns all results:
1. `grpc_health` — is the gRPC health server serving?
2. `rpc_latency` — average RPC latency vs threshold
3. `rpc_error_ratio` — RPC error rate vs threshold
4. `persistence_latency` — DB latency vs threshold
5. `persistence_error_ratio` — DB error rate vs threshold
The frontend collects results from **all hosts in membership**,
aggregates them, and returns the full breakdown.
### Example responses
#### Healthy cluster (3 hosts, all serving)
```json
{
"state": "HEALTH_STATE_SERVING",
"services": [{
"service": "history",
"state": "HEALTH_STATE_SERVING",
"hosts": [
{
"address": "10.0.1.5:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 45.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.01, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 12.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.6:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 52.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 18.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.7:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 38.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 15.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
}
]
}]
}
```
#### Degraded cluster — 1 host with high RPC latency (3 hosts, 1
failing, under threshold)
The failing host clearly shows which check triggered and the actual vs
threshold values. Because only 1/3 hosts failed (33%) and the failure
threshold is 25% but we require at least 2 hosts to fail, the overall
state remains `SERVING`.
```json
{
"state": "HEALTH_STATE_SERVING",
"services": [{
"service": "history",
"state": "HEALTH_STATE_SERVING",
"hosts": [
{
"address": "10.0.1.5:7234",
"state": "HEALTH_STATE_NOT_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_NOT_SERVING", "value": 850.0, "threshold": 500.0, "message": "RPC latency 850.00ms exceeded 500.00ms threshold"},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 120.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.6:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 52.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 18.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.7:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 38.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 15.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
}
]
}]
}
```
#### Unhealthy cluster — hosts unreachable (6 hosts in membership, 3
unreachable via RPC)
When the frontend cannot reach a host via RPC, it creates a synthetic
`host_availability` check with the error. The host appears in the
response with `NOT_SERVING` and the RPC error message. With 3/6 hosts
failing (50% > 25% threshold), the overall state is `NOT_SERVING`.
```json
{
"state": "HEALTH_STATE_NOT_SERVING",
"services": [{
"service": "history",
"state": "HEALTH_STATE_NOT_SERVING",
"hosts": [
{
"address": "10.0.1.5:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 45.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.01, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 12.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.6:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 52.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 18.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.7:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 38.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 15.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.8:7234",
"state": "HEALTH_STATE_NOT_SERVING",
"checks": [
{"check_type": "host_availability", "state": "HEALTH_STATE_NOT_SERVING", "message": "failed to reach host for health check: rpc error: code = Unavailable desc = connection refused"}
]
},
{
"address": "10.0.1.9:7234",
"state": "HEALTH_STATE_NOT_SERVING",
"checks": [
{"check_type": "host_availability", "state": "HEALTH_STATE_NOT_SERVING", "message": "failed to reach host for health check: rpc error: code = Unavailable desc = connection refused"}
]
},
{
"address": "10.0.1.10:7234",
"state": "HEALTH_STATE_NOT_SERVING",
"checks": [
{"check_type": "host_availability", "state": "HEALTH_STATE_NOT_SERVING", "message": "failed to reach host for health check: context deadline exceeded"}
]
}
]
}]
}
```
#### Host voluntarily draining — gRPC health declined (DECLINED_SERVING)
When a host's gRPC health server reports not serving (e.g. during
graceful shutdown), the check returns `DECLINED_SERVING`. If enough
hosts are in this state (exceeding the declined serving proportion
threshold), the overall service state becomes `DECLINED_SERVING`.
```json
{
"state": "HEALTH_STATE_DECLINED_SERVING",
"services": [{
"service": "history",
"state": "HEALTH_STATE_DECLINED_SERVING",
"hosts": [
{
"address": "10.0.1.5:7234",
"state": "HEALTH_STATE_DECLINED_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_DECLINED_SERVING", "message": "gRPC health server not serving"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 45.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.01, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 12.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.6:7234",
"state": "HEALTH_STATE_DECLINED_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_DECLINED_SERVING", "message": "gRPC health server not serving"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 52.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 18.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.7:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 38.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 15.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
}
]
}]
}
```
#### No hosts in membership
When the membership resolver returns an empty host list, the response
includes a service-level message but no hosts.
```json
{
"state": "HEALTH_STATE_NOT_SERVING",
"services": [{
"service": "history",
"state": "HEALTH_STATE_NOT_SERVING",
"message": "no available hosts in membership"
}]
}
```
#### Membership resolver failure (INTERNAL_ERROR)
When the frontend can't resolve the membership ring at all
(infrastructure failure), the response includes `INTERNAL_ERROR` with
the resolver error.
```json
{
"state": "HEALTH_STATE_INTERNAL_ERROR",
"services": [{
"service": "history",
"state": "HEALTH_STATE_INTERNAL_ERROR",
"message": "failed to get membership resolver: membership monitor not started"
}]
}
```
### Backward compatibility
- `DeepHealthCheckResponse.state` (field 1) unchanged in both history
and admin protos
- New fields (`checks`, `services`) are additive (field 2) — old clients
simply ignore them
- `GetState()` continues to work as before
### Related
- saas-control-plane PR #12203 — `HealthReport` + `CellHealthEvent`
(consumer side, ready to use these fields)
- Runbooks PR #1231 — end-to-end flow documentation
## Test plan
- [x] All existing `TestHealthCheckerSuite` tests pass (19 tests)
- [x] New tests: `Test_Check_ServiceDetail_Populated`,
`Test_Check_HostChecks_Propagated`, `Test_Check_GetResolver_Error`
(INTERNAL_ERROR + message), `Test_Check_No_Available_Hosts` (message)
- [x] Full `go build ./...` passes
- [ ] Verify saas-control-plane can import
`go.temporal.io/server/common/health` constants
- [ ] Integration test with actual history service
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
|
||
|
David Reiss
|
83174dfb02 |
Fairness counter: persist top K keys (#9188)
## What changed? The counts of some top keys are persisted in task queue metadata so they're preserved on queue movement or reloads. ## Why? More accurate fairness ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Sean Kane
|
0fbc386c55 |
Include transient and speculative WFT events in GetWorkflowExecutionHistoryResponse (#9325)
## What changed? Re-does #9138 which was incidentally merged. Include transient and speculative WFT events in `GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made request. * Adds `transient_or_speculative_events` back to `GetMutableStateResponse` * Reserve `transient_workflow_task` in `HisotryCOntinuation` token * Add validation helpers * Add query-compare-query for transient events at request start and end Re-implements #7732 ## Why? Fix "premature end of stream" errors when workers request history after cache eviction w/ transient/speculative workflow tasks present. This adds transient & speculative WFT events in `GetWorkflowExecutionHistory` (already in `PollWorkflowTask`). Worker cache eviction w/ speculative workflow tasks causes the expected and actual event counts to be different. #7732 passed transient events through continuation tokens, which could become stale during pagination. This PR implements mutable state querying at both start and end of pagination and compares transient event IDs to detect if WFT state changed during pagination and return a retryable error. ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks Same risks from #7732 |
||
|
Roey Berman
|
e0d9f48c84 |
System nexus endpoint (#9002)
## Overview
This feature introduces a special internal Nexus endpoint called
`__temporal_system` that enables adding functionality to workflows
**without requiring new workflow commands and events**. Operations on
this endpoint are routed internally within Temporal's history service
rather than via external HTTP calls.
## Key Components
### 1. System Endpoint Infrastructure
- **Endpoint Name**: `__temporal_system` (constant in
`common/nexus/constants.go:6`)
- **Callback URL**: `temporal://system` for internal routing
- **New History Service RPCs**
(`proto/internal/temporal/server/api/historyservice/v1/service.proto:433-437`):
- `StartNexusOperation` - Starts operations on the system endpoint
- `CancelNexusOperation` - Cancels operations on the system endpoint
### 2. Operation Processor Framework
A new processor pattern (`chasm/nexus_operation_processor.go`) that
allows CHASM libraries to:
- **Validate and transform input**: Processors can validate operation
inputs and set default values
- **Determine routing**: Each processor returns a routing key that
determines which history shard handles the operation
- **Re-serialize input**: Mutated inputs can be re-serialized to persist
default values
**Routing strategies**:
- `NexusOperationRoutingKeyExecution` - Routes to the shard owning a
specific workflow execution
- `NexusOperationRoutingKeyRandom` - Routes to a random shard
### 3. CHASM Library Integration
CHASM libraries can now provide (`chasm/library.go:16-19`):
- **`NexusServices()`**: Regular Nexus service handlers (implement the
actual operation logic)
- **`NexusServiceProcessors()`**: Input processors for validation and
routing
Example from test library (`chasm/lib/tests/nexus_service.go`):
```go
// Service handler - implements the actual operation
TestOperation = nexus.NewSyncOperation("TestOperation",
func(ctx context.Context, input string, options nexus.StartOperationOptions) (string, error) {
return "Hello, " + input, nil
})
// Processor - validates input and determines routing
func (o testOperationProcessor) ProcessInput(ctx chasm.NexusOperationProcessorContext, input string)
(*chasm.NexusOperationProcessorResult, error) {
return &chasm.NexusOperationProcessorResult{
RoutingKey: chasm.NexusOperationRoutingKeyExecution{
NamespaceID: ctx.Namespace.ID().String(),
BusinessID: input, // Route based on input
},
}, nil
}
```
### 4. Execution Flow
When a workflow schedules a Nexus operation on `__temporal_system`
(`components/nexusoperations/executors.go:233-238`):
1. **Input Processing**: The processor validates input and determines
routing
2. **Internal RPC**: Instead of HTTP, calls
`HistoryClient.StartNexusOperation` with the target shard ID
3. **Handler Execution**: The history service invokes the registered
Nexus handler (`service/history/handler.go:2707-2768`)
4. **Result Handling**: Supports both sync (immediate result) and async
(operation token) responses
5. **Workflow Completion**: Results flow back through the same
completion path as external Nexus operations
### 5. Benefits
✅ **No schema changes**: Add functionality without new commands/events
in workflow history
✅ **Consistent API**: Uses existing Nexus operation semantics
(sync/async, callbacks, links)
✅ **Proper routing**: Operations are intelligently routed to the correct
shard
✅ **Input validation**: Type-safe input validation and default value
handling
✅ **Future extensibility**: Foundation for direct client invocation (not
yet implemented)
### 6. Technical Details
- **Error handling** (`components/nexusoperations/executors.go:444`):
Non-retryable service errors are properly handled and fail operations
immediately
- **Metrics**: System operations are tracked separately with
`DestinationTag` set to the endpoint name
- **Link conversion**: Helper functions convert between Nexus SDK links
and protobuf links (`common/nexus/util.go:17-46`)
- **Operation token handling**: Moved link converters to common package
for reuse (`common/nexus/link_converter.go`)
### 7. Current Limitations
- Only accessible from workflows (via `ScheduleNexusOperation` command)
- Direct client invocation not yet implemented
- Headers not supported for system endpoint operations
## Test Coverage
New test (`tests/nexus_workflow_test.go:2763-2843`) demonstrates:
- Scheduling operation on `__temporal_system` endpoint
- Synchronous operation completion
- Result propagation back to workflow
## Architecture
This architecture provides a clean, extensible way to add internal
functionality while maintaining compatibility with Temporal's existing
workflow execution model. The system endpoint acts as a bridge between
workflows and internal CHASM components, enabling:
- **Extensibility**: New operations can be added by implementing CHASM
libraries
- **Type safety**: Input validation happens before operations are routed
- **Scalability**: Intelligent routing ensures operations land on the
correct shard
- **Consistency**: Same execution model as external Nexus operations
|
||
|
Sean Kane
|
bd19bc6059 |
Revert "Include transient and speculative WFT events in GetWorkflowExecutionHistoryResponse" (#9322)
Reverts temporalio/temporal#9138 |
||
|
Sean Kane
|
aad69c365b |
Include transient and speculative WFT events in GetWorkflowExecutionHistoryResponse (#9138)
## What changed? Include transient and speculative WFT events in `GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made request. * Adds `transient_or_speculative_events` back to `GetMutableStateResponse` * Reserve `transient_workflow_task` in `HisotryCOntinuation` token * Add validation helpers * Add query-compare-query for transient events at request start and end Re-implements #7732 ## Why? Fix "premature end of stream" errors when workers request history after cache eviction w/ transient/speculative workflow tasks present. This adds transient & speculative WFT events in `GetWorkflowExecutionHistory` (already in `PollWorkflowTask`). Worker cache eviction w/ speculative workflow tasks causes the expected and actual event counts to be different. #7732 passed transient events through continuation tokens, which could become stale during pagination. This PR implements mutable state querying at both start and end of pagination and compares transient event IDs to detect if WFT state changed during pagination and return a retryable error. ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks Same risks from #7732 |