mirror of
https://github.com/temporalio/temporal.git
synced 2026-08-30 18:41:49 -07:00
27115b584d2520ff303ab2c2043eb2bc94e7172b
610 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Long Tran
|
27115b584d |
Propagate backlinks on Signal and Signal-with-Start responses (#9897)
## What changed? ### **High level** With https://github.com/temporalio/api/pull/761 to add the linking on the Signal and Signal-with-Start responses, This PR adds logic from the server that: * Adds `requestID` from Signal and Signal-with-Start requests to the CHASM workflow tree under a new map field `IncomingSignals`, and event store, so these requestIDs stay in buffer * Return a backlink in the response that references the `requestID` * On buffer flush to the DB transaction, attach these `requestID` to a concrete `eventID`, which would allow users to later know which event correlated w/ this request. We will wire the concrete event ID to the signal request IDs stored in the workflow component CHASM tree (`IncomingSignals` map) > [!NOTE] > Feature is gated behind a new dynamicconfig `EnableCHASMSignalBacklinks`, which implicitly is only checked if `EnableChasm` is enabled. ## Why? This will enable the caller of the signal to have a backlink to the cross-namespace signal invoked, which will become more relevant for Nexus SDK ergonomics. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [x] added new functional test(s) In functional tests, I augmented existing tests for Signal and Signal-with-Start to: * Ensure that backlink is returned via the responses * Later use `DescribeWorkflow` to ensure that we get a concrete EventID (mapped when buffer flushed) * Multiple signals with the same `requestID` gets de-duped ``` $ go test ./tests/ -run TestLinksTestSuite ok go.temporal.io/server/tests 1.486s ``` ``` $ go test ./tests/ -run 'TestNexusWorkflowTestSuite' -count=1 ok go.temporal.io/server/tests 4.714s ``` ## Potential risks Need to test end-to-end to see that the link shows up correctly in the Web UI. Feature is gated behind dynamicconfig since it requires CHASM-based workflow to be enabled. |
||
|
Alan Wu
|
4958253c8c |
Add CHASM lifecycle paused state (#10046)
## What changed? Add CHASM lifecycle paused state. Invalidates all pending logical tasks. ## Why? There are duplicate implementations in Libraries where authors have implemented their own statuses to act as a PAUSE state, preventing all tasks from executing. ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [X] added new unit test(s) - [ ] added new functional test(s) |
||
|
feiyang
|
e004436d8b |
time-skipping runtime-2 (with bound and backoff timers) (#10092)
## What changed? (1) elapsed-duration / skipped-duration **bounds** that auto-disable time-skipping (2) a new `TimeSkippingTimerTask` that wakes the workflow when an elapsed bound is reached (3) time-skipping support for backoff timers (start-with-delay / cron / retry / CaN-with-backoff) plus an exclusion for child workflows that haven't scheduled their first WT ## Why? the completion of the feature ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) |
||
|
feiyang
|
bc0354e194 |
time-skipping propagation (#10013)
## What changed and Why? Paired with [api change](https://github.com/temporalio/api/pull/770/changes) Define the default time-skipping propagation behavior for current features: continue-as-new (CAN), child workflows, retry, and reset. **Group 1 — Inherit both from the current execution:** - Continue-as-new (CAN): inherits the current config and accumulated skipped duration; the configured bound is shared across both the inherited skipped duration and any duration skipped by the new run. - Child workflows: same behavior as CAN This design is because CAN is a technical reason to start a new run of current run, so logically they can be viewed as a same "run". For Child WFs they can be viewed as an extension of previous workflows or separate workflows, and in either case, they shall inherit the skipped duration so that the virtual time doesn't rewind back, and the default behavior is designed to treat them as an extension of the parent and share the config. We only consider adding new config to provide flexibility on demand. **Group 2 — Inherit from a specific point in history:** - Retry: inherits the config and skipped duration recorded in the StartWorkflowExecutionEvent of the current workflow, since retry is defined as restarting execution from that event - Cron: same with Retry **Group3 -- Both inherit from a specific point and catch up of config change to current** - Reset: retains the current time-skipping config, since reset is designed to replay all events up to the reset point and apply any UpdateWorkflowExecutionOptions changes made after that point — with no option to exclude them (covered by tests only) ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) |
||
|
Stephan Behnke
|
ff3ed0b16f |
Nexus CHASM async completion (2/2) (#9972)
## What changed? Support completion-before-start. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) |
||
|
Shivam
|
562d26fc83 |
Skip reactivation signals for current/ramping/draining versions (#9778)
## Summary
- Extends `CheckTaskQueueVersionMembership` response with two new
fields: `is_version_active_or_draining` (bool) and `revision_number`
(int64). Matching populates both from its deployment data.
- Reactivation signals are **skipped** when matching reports the target
version as CURRENT/RAMPING/DRAINING; otherwise they are sent with a
**deterministic UUID v5 RequestId** derived from `revision_number`.
- Replaces the old TTL-based `ReactivationSignalCache` with a per-pod
**revision-based dedup LRU** on the worker-deployment client: each entry
records the highest revision this pod has successfully signaled for a
given version, so older or equal signals are skipped.
## What changed on the wire (matching → history)
`CheckTaskQueueVersionMembershipResponse` now has two new flat fields
(no wrapper message):
```proto
bool is_version_active_or_draining = 2; // true when status is CURRENT/RAMPING/DRAINING
int64 revision_number = 3; // from WorkerDeploymentVersionData.revision_number; 0 if unknown / legacy
```
Matching's `CheckTaskQueueVersionMembership` fills both via the helper
`worker_versioning.IsVersionActiveOrDraining(deploymentData, dep, build)
(bool, int64)`.
Naming choice — we picked `is_version_active_or_draining` (negative
polarity) rather than something like `supports_reactivation` so the
proto zero value (`false`) maps to the safe default ("send the signal").
Old matching binaries and runtime "version not found" both produce the
zero value, and history correctly falls through.
## Where `revision_number` flows
- **Matching**: populates the response field from the version's tracked
revision.
- **History-side helper/caches**:
`ValidateVersioningOverrideAndGetReactivationEligibility` returns
`(isVersionActiveOrDraining bool, revisionNumber int64, err)`.
`VersionMembershipAndReactivationStatusCache` stores both.
- **History signaler plumbing**: `VersionReactivationSignalerFn`,
`ReactivateVersionWorkflowIfPinned`, and all five call sites
(`startworkflow`, `signalwithstartworkflow`, `updateworkflowoptions`,
`resetworkflow`, `multioperation`) carry `revisionNumber int64`.
`resetworkflow.validatePostResetOperationInputs` returns parallel slices
`([]bool, []int64, error)` for per-operation inputs.
- **Signal RequestId**: `ClientImpl.SignalVersionReactivation` composes
`requestID = uuid.NewSHA1(uuid.NameSpaceOID,
[]byte("reactivation-signal:" + revisionNumber)).String()` — a
deterministic UUID v5 derived from the revision alone. Cassandra's
`signal_requested set<uuid>` column requires UUID-formatted RequestIds.
## Why revision-based dedup
History is sharded on `(namespaceID, workflowID)`. N concurrent
`StartWorkflow` calls pinned to the same drained version fan out across
potentially every history pod in the fleet. Before this PR each pod
independently fired a reactivation signal at the version workflow,
producing up to N `WorkflowExecutionSignaled` events — directly at odds
with the version workflow's design (it intentionally keeps history
minimal and CaNs aggressively, see `version_workflow.go:68-74`).
Per-pod caches alone can't fix this because they don't coordinate. What
we need is a **cluster-wide-deterministic dedup key** so all pods
converge on the same value for the same reactivation cycle. The
version's `revision_number` — incremented in `syncTaskQueuesAsync` on
every status change — is exactly that signal. Every pod reads the same
revision from matching, every pod composes the same UUID RequestId, and
Temporal's built-in `mutableState.pendingSignalRequestedIDs` dedup (see
`service/history/api/signalworkflow/api.go:40`) collapses concurrent
signals into exactly one event on the version workflow.
The per-pod map is a local optimization on top of that: it prevents a
single pod from re-sending the same-or-older-revision signal once it has
successfully sent one, cutting RPC volume.
## How the new caches look
### 1. `VersionMembershipAndReactivationStatusCache` (read-side,
per-pod)
Caches matching's `CheckTaskQueueVersionMembership` response so repeated
pinned-override validations on the same task queue don't re-hit
matching.
- **Key**: `(namespaceID, taskQueue, taskQueueType, deploymentName,
buildID)`
- **Value**: `(isMember bool, isVersionActiveOrDraining bool,
revisionNumber int64)`
- **Eviction**: `VersionMembershipCacheTTL` (1s default; 5s in
functional tests).
### 2. `highestRevSignaledToVersionWf` (write-side dedup, per-pod)
A field on `ClientImpl` in `service/worker/workerdeployment/client.go`.
For each target version workflow, stores the highest revision this pod
has successfully signaled. Subsequent calls at the same-or-lower
revision skip the RPC.
- **Key**: `reactivationVersionKey{namespaceID, deploymentName,
buildID}`
- **Value**: `int64` (highest revision successfully signaled)
- **Eviction**: LRU, bounded by `VersionReactivationSignalCacheMaxSize`.
The previous TTL-based `ReactivationSignalCache` module (in
`common/worker_versioning/`) has been deleted along with its provider
and `VersionReactivationSignalCacheTTL` config.
## Backwards/forwards compatibility
- **Old matching → new history**: old binaries don't set
`is_version_active_or_draining` or `revision_number`; both default to
proto zero values. `false` on the active bool → history falls through →
signal fires (safe default). `revisionNumber = 0` flows through as-is.
- **New matching → old history**: new fields on the response are ignored
by old history → identical to pre-PR behavior.
- **New matching → new history**: signal fires only when the version is
not active/draining; cross-pod fires converge on one UUID RequestId and
fold into one `WorkflowExecutionSignaled` event.
## Test plan
- [x] Unit tests for `IsVersionActiveOrDraining` covering all status
cases (CURRENT, RAMPING, DRAINING, DRAINED, INACTIVE, UNSPECIFIED), new
vs. old format, deleted and not-found versions.
- [x] Unit tests for
`ValidateVersioningOverrideAndGetReactivationEligibility` (cache
hit/miss, RPC with/without eligibility, Unimplemented fallback).
- [x] Unit tests for the per-pod dedup on
`ClientImpl.SignalVersionReactivation`: same-rev dedups, newer-rev
fires, older-rev skipped, different version isolated, signal-failure
allows retry.
- [x] Unit test for RequestId format (UUID v5, deterministic across
calls with the same revision).
- [x] Functional tests (all pass on SQLite and cass-es):
- `TestStartWorkflowExecution_ReactivateVersionOnPinned`
-
`TestStartWorkflowExecution_ReactivateVersionOnPinned_WithConflictPolicy`
- `TestSignalWithStartWorkflowExecution_ReactivateVersionOnPinned`
- `TestUpdateWorkflowExecutionOptions_ReactivateVersionOnPinned`
- `TestResetWorkflowExecution_ReactivateVersionOnPinned`
(The four `TestReactivationSignalCache_Deduplication_*` functional tests
from an earlier iteration were deleted — their coverage moved to unit
tests.)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes matching↔history API and reactivation signaling semantics by
skipping signals for active/draining versions and introducing
revision-based dedup via deterministic RequestIds; issues could affect
version workflow state transitions or signal fan-out during upgrades.
>
> **Overview**
> Matching’s `CheckTaskQueueVersionMembershipResponse` is extended with
`should_skip_reactivation` and `revision_number`, and matching now
populates both from per-task-queue deployment data.
>
> History-side versioning validation is refactored to return and cache
reactivation eligibility + revision, and reactivation signaling paths
(`StartWorkflow`, `SignalWithStart`, `UpdateWorkflowExecutionOptions`,
`ResetWorkflow`, multi-op) now **skip signals** when matching reports
the version as *CURRENT/RAMPING/DRAINING*.
>
> The old TTL-based `ReactivationSignalCache` is removed
(configs/metrics/providers updated), and the worker-deployment client
now performs **revision-based per-pod dedup** plus receiver-side dedup
by sending signals with a deterministic UUIDv5-like `RequestId` derived
from `revision_number`. Tests are updated/added to cover status
evaluation, new plumbing, and dedup behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Shivam
|
089fba53e1 |
Fix pinned workflow trampolining via revision-based signal suppression (#9895)
## Summary
- Matching now sends the real `targetDeploymentRevisionNumber` for
pinned workflows instead of hardcoded 0
- Added `revision_number` field to `LastNotifiedTargetVersion`
(server-internal) and `DeclinedTargetVersionUpgrade` (public API)
- Case 4 in the target version change switch now uses
`targetRevisionNumber <= declined.RevisionNumber` instead of version
string comparison, preventing trampolining when stale matching
partitions send outdated target versions
## Problem
When a pinned workflow CaNs and declines a target version upgrade, a
stale matching partition can send an older target version. The old case
4 compared version strings (`declined.buildId == target.buildId`), which
didn't match the stale version — causing the workflow to re-signal, CaN
again, and trampoline indefinitely between stale and up-to-date
partitions.
## Test plan
- [x] `TestStalePartition_RevisionSuppressesTrampolining` — integration
test that simulates a stale partition via `rollbackTaskQueueToVersion`
and verifies:
- Stale partition (revision 0) is suppressed after declining at a higher
revision
- Genuinely new version (v4 at higher revision) correctly fires the
signal
- [x] Verified test **fails** when matching + history changes are
reverted (matching sends 0, old string comparison)
## API repo dependency
- api: `temporalio/api@trampolining-rev-number`
- api-go: `temporalio/api-go@trampolining-rev-number`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches pinned-workflow versioning logic across matching, history, and
persistence; a bug here could change when workflows are signaled to
continue-as-new or upgrade, affecting routing behavior.
>
> **Overview**
> Prevents pinned workflows from repeatedly continue-as-new
“trampolining” when task queue partitions have stale routing data by
**tracking and comparing target routing-config revision numbers**.
>
> Matching now propagates the real `targetDeploymentRevisionNumber` for
pinned workflow tasks, and history threads this through
`AddWorkflowTaskStartedEvent` to persist
`LastNotifiedTargetVersion.revision_number` and carry it into
`DeclinedTargetVersionUpgrade` on continue-as-new. The pinned
target-change decision in `workflow_task_state_machine` switches case-4
suppression from deployment version string equality to
`targetRevisionNumber <= declined.RevisionNumber`, and adds an
integration test (`TestStalePartition_RevisionSuppressesTrampolining`)
covering stale vs fresh partition behavior.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Stephan Behnke
|
b78391b445 |
Nexus CHASM async completion (1/2) (#9951)
## What changed? Support Nexus async completion in CHASM. PS: completion-before-start will be handled in a follow-up PR. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) ## Potential risks Is behind feature flag. |
||
|
Kannan
|
005d1849c6 |
Implement new task queue kind to send worker commands (#9899)
## What Add support for `TASK_QUEUE_KIND_WORKER_COMMANDS` in the matching service. This is a new task queue kind for server-to-worker communication (e.g. activity cancellations). It is represented by a new `WorkerCommandsPartition` type in the `tqid` package — analogous to how `StickyPartition` represents sticky queues. Also adds a distinct `partition` metric tag for worker-commands queues so their traffic can be distinguished from user-facing normal queues in metrics. Depends on [#9900](https://github.com/temporalio/temporal/pull/9900). ## Why Worker_Commands queue kind has different properties compared to a normal user defined queue. So we created a separate kind. This allows us to distinguish task queue metrics based on whether they are internal or user created queue. ## How did you test it? Unit tests covering metric tag generation for normal, sticky, and worker-commands partitions. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: David Reiss <david@temporal.io> |
||
|
David Reiss
|
1b2f50bea1 |
Dynamic partitioning: client side (#9732)
## What changed? Add client-side of dynamic partition scaling: `partitionCache`, `PartitionCounts`, `StalePartitionCounts` error, `invokeWithPartitionCounts`. The intended behavior is: For partition-aware calls (add+poll tasks), the client caches the partition count for active task queues, and uses those partition counts for load balancing (no change to load balancing algorithm yet). The client sends its cached partition counts to the server in a grpc header, and receives updated partition counts with the response in a grpc trailer. If the updated counts are different, it updates its cache it for subsequent requests. If the server indicates that the client's view of partition count is stale, it returns a special `StalePartitionCounts` error and then client makes one immediate retry with the newly received counts. With just this PR by itself, the server will never send counts, the cache will always be empty, and the client will always fall back to dynamic config for partition counts, so there's no change in behavior yet. ## Why? Implement half of dynamic partition scaling. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) – tests are in future PRs |
||
|
feiyang
|
c5badc94b5 |
time-skipping runtime foundation (#9965)
## What changed? 1) `taskGenerator`: - add regenerate tasks for time skipping 2) `mutableState`: - new methods: add key methods to transition time - `IsStateDirty`: captures changes of time-skipping related fields - `closeTransaction`: add closeTransactionHandlerTimeSkipping 3) event: new time-skipping runtime event added 4) persistence: new runtime data added to execution info ## Why? add the foundational mechanism of how time skipping works in the runtime of a workflow execution **for reviewers:** this is a foundation of t-s runtime, so it works without any granular and extended features as bound, replication, external-transfer tasks, etc. **Over 50% of lines are tests/mocks/pb-gen. can focus on code first, then functional tests, and last ut.** ## How did you test it? - [x] built - [x] run locally and tested manually - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) |
||
|
Kannan
|
856e276e58 |
Dispatch activity cancellation to worker using Nexus (#9233)
## What Dispatches worker commands (starting with activity cancellation) to workers via their Nexus control queue. When the outbound queue processes a `WorkerCommandsTask`, the dispatcher sends an `ExecuteCommands` Nexus operation to the worker's control queue via `DispatchNexusTask`. - Retries are capped at 3 attempts since these commands are best-effort (the activity will eventually time out anyway). Uses the retryable matching client so transient RPC errors (ShardOwnershipLost, Unavailable) are retried at the RPC layer without consuming queue-level attempts. - Stores the clock used to generate the task token in the ActivityInfo. This is needed to reconstruct the same task token when dispatching to the worker. Otherwise, it will not match the token expected by the sdk. Suggested review order: `worker_commands_task_dispatcher.go` → `nexus_dispatch_response.go` → `recordactivitytaskstarted/api.go`. ## Why To support activity cancellation without activity heartbeat. This is the dispatch leg of the flow: 1. [#9231] Store `worker_control_task_queue` in `ActivityInfo` at activity start. 2. [#9232] On `RequestCancelActivityTask`, batch commands by control queue into `WorkerCommandsTask` outbound tasks. 3. **[This PR]** Dispatch each task as a Nexus `ExecuteCommands` operation to the worker, with a 3-attempt retry cap. 4. [SDK] Worker receives the cancel command and cancels the running activity. Gated by dynamic config `EnableCancelActivityWorkerCommand` (default: off). ## How did you test it? - **Unit tests** cover all dispatch outcomes (success, RPC error, timeout, worker error, feature-flag-off, max-attempts-exceeded) and response-to-error conversion paths. - **Functional test** verifies end-to-end: cancel request → Nexus dispatch → correct payload arrives on the control queue, and asserts that the cancel command's task token matches the one from the original activity poll response. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: rkannan82 <5853897+rkannan82@users.noreply.github.com> |
||
|
Veeral Patel
|
94556cc8ce |
feat: add GetTaskQueueUserData RPC to admin service (#9934)
## What changed? This PR adds a new `GetTaskQueueUserData` RPC to the admin service. Given a namespace, task queue name, task queue type, and optional partition ID (default is 0, which is `root`), it returns the user data currently loaded by that partition. This PR wraps the existing `GetTaskQueueUserData` RPC in Matching Service. We will also create a `tdbg` command which calls this Admin Service RPC, in a separate PR. ## Why? Each task queue family has associated metadata, stored in [TaskQueueUserData](https://github.com/temporalio/temporal/blob/main/proto/internal/temporal/server/api/persistence/v1/task_queues.proto). Metadata related to worker versioning, queue rate limiting, fairness are all stored in `TaskQueueUserData`. TaskQueueUserData is replicated from the root partition to all other partitions. However, there is no admin-accessible way to read the user data loaded by a specific partition or compare versions across partitions to diagnose replication lag. ## Files changed | File | Change | |---|---| | `proto/internal/.../adminservice/v1/request_response.proto` | Added `GetTaskQueueUserDataRequest` and `GetTaskQueueUserDataResponse` messages | | `proto/internal/.../adminservice/v1/service.proto` | Added `GetTaskQueueUserData` RPC to `AdminService` | | `service/frontend/admin_handler.go` | Implemented `AdminHandler.GetTaskQueueUserData`: validates request, resolves namespace → ID, builds partition RPC name via `tqid`, calls matching service, returns per-type entry + version | | `service/frontend/admin_handler_test.go` | Added unit tests | ## How did you test it? - [x] built - [x] run locally and tested manually - [x] added new unit test(s) - [x] added new integration test(s) - not applicable, not touching persistence layer - [x] added new functional test(s) ### Unit tests 100% unit test coverage | Test case | Input | Expected | |---|---|---| | Nil request | `request == nil` | `errRequestNotSet` | | Empty namespace | `namespace == ""` | `errNamespaceNotSet` | | Namespace not found | Namespace registry returns not-found | Error propagated; matching never called | | Invalid task queue name | `task_queue` starts with `/_sys/` | `INVALID_ARGUMENT` from `tqid.NewTaskQueueFamily`; matching never called | | Root partition | `partition_id=0`, workflow type | Sends bare name `my-queue` to matching; returns correct `user_data` and `version` | | Non-root partition | `partition_id=1`, workflow type | Sends mangled name `/_sys/my-queue/1` to matching | | No per-type data | Matching returns response with empty `per_type` map | `user_data` is nil; `version` still populated | | Matching error | Matching client returns error | Error propagated to caller | ### Functional tests | Test | Setup | What it verifies | |------------------------------------------------|--------------------------------|----------------------------------------------------------------------------------| | TestAdminGetTaskQueueUserData_RootPartition | Write fairness weight config to a workflow task queue | Admin RPC resolves namespace by name, routes to root partition (partition_id=0), returns version > 0 and non-nil per-type data | | TestAdminGetTaskQueueUserData_NonRootPartition | Same write, then poll until non-root partition replicates | Admin RPC routes to a non-root partition (partition_id=1) via mangled name, returns the same version as root after replication | ### Manual tests <details> <summary>Setup</summary> 1. Build and start the server: `make temporal-server && make start-sqlite` 2. Create namespace: `temporal operator namespace create default` 3. Insert assignment rule: `temporal task-queue versioning insert-assignment-rule` </details> <details> <summary>Case 1 — Root partition, workflow type</summary> ``` grpcurl -plaintext \ -d '{"namespace":"default","task_queue":"my-queue","task_queue_type":"TASK_QUEUE_TYPE_WORKFLOW"}' \ localhost:7233 \ temporal.server.api.adminservice.v1.AdminService/GetTaskQueueUserData ``` ```json { "version": "1" } ``` </details> <details> <summary>Case 2 — Root partition, activity type</summary> ``` grpcurl -plaintext \ -d '{"namespace":"default","task_queue":"my-queue","task_queue_type":"TASK_QUEUE_TYPE_ACTIVITY"}' \ localhost:7233 \ temporal.server.api.adminservice.v1.AdminService/GetTaskQueueUserData ``` ```json { "version": "1" } ``` </details> <details> <summary>Case 3 — Non-root partition, workflow type</summary> ``` grpcurl -plaintext \ -d '{"namespace":"default","task_queue":"my-queue","task_queue_type":"TASK_QUEUE_TYPE_WORKFLOW","partition_id":1}' \ localhost:7233 \ temporal.server.api.adminservice.v1.AdminService/GetTaskQueueUserData ``` ```json { "version": "1" } ``` </details> <details> <summary>Case 4 — Non-root partition, activity type</summary> ``` grpcurl -plaintext \ -d '{"namespace":"default","task_queue":"my-queue","task_queue_type":"TASK_QUEUE_TYPE_ACTIVITY","partition_id":1}' \ localhost:7233 \ temporal.server.api.adminservice.v1.AdminService/GetTaskQueueUserData ``` ```json { "version": "1" } ``` </details> <details> <summary>Case 5 — Non-root partition, activity type, with user data</summary> Setup: Add rate limit config ``` grpcurl -plaintext \ -d '{ "namespace": "default", "task_queue": "my-queue", "task_queue_type": "TASK_QUEUE_TYPE_ACTIVITY", "update_queue_rate_limit": { "rate_limit": { "requests_per_second": 50.0 }, "reason": "manual test" } }' \ localhost:7233 \ temporal.api.workflowservice.v1.WorkflowService/UpdateTaskQueueConfig ``` ``` grpcurl -plaintext \ -d '{ "namespace": "default", "task_queue": "my-queue", "task_queue_type": "TASK_QUEUE_TYPE_ACTIVITY" }' \ localhost:7233 \ temporal.server.api.adminservice.v1.AdminService/GetTaskQueueUserData ``` ```json { "userData": { "config": { "queueRateLimit": { "rateLimit": { "requestsPerSecond": 50 }, "metadata": { "reason": "manual test", "updateTime": "2026-04-13T21:40:39.888Z" } } } }, "version": "2" } ``` </details> <details> <summary>Case 6 — Namespace not found</summary> ``` grpcurl -plaintext \ -d '{"namespace":"nonexistent","task_queue":"my-queue","task_queue_type":"TASK_QUEUE_TYPE_WORKFLOW"}' \ localhost:7233 \ temporal.server.api.adminservice.v1.AdminService/GetTaskQueueUserData ``` ``` ERROR: Code: NotFound Message: Namespace nonexistent is not found. ``` `NOT_FOUND` from namespace registry; matching never called. </details> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
feiyang
|
ef19f17c95 |
timeskipping: add TimeSkippingConfig support to workflow-start APIs (#9834)
## What changed?
Add time-skipping configuration support.
(1) Feature flag in dynamic config
- frontend.TimeSkippingEnabled (namespace bool, default false) gates the
feature
(2) Frontend validation (validateTimeSkippingConfig)
- StartWorkflowExecution
- SignalWithStartWorkflowExecution
- ExecuteMultiOperation (via the shared prepareStartWorkflowRequest
path)
(3) Persistence
- TimeSkippingInfo proto added to WorkflowExecutionInfo
- MutableState stores the config when a workflow starts or options are
updated
(4) Tests
- New timeskipping functional tests in tests/timeskipping_test.go
## Why?
first step of the time skipping project
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
---------
Co-authored-by: Yichao Yang <yycptt@gmail.com>
|
||
|
Kannan
|
f5246b3fdd |
Add WorkerCommandsTask outbound task to dispatch worker commands via Nexus (#9232)
## What changed? New outbound task type (`WorkerCommandsTask`) that carries worker commands to be dispatched to workers via Nexus. Uses the generic `WorkerCommand` proto (not cancel-activity-specific), so this task type can carry any future command types. Suggested review order: proto changes → `worker_commands_task.go` → `task_generator.go` → `workflow_task_completed_handler.go` Key pieces: - **Proto**: `TASK_TYPE_WORKER_COMMANDS` enum, `WorkerCommandsTask` in `OutboundTaskInfo` with `repeated WorkerCommand`. - **Task definition**: `worker_commands_task.go` — implements outbound `Task` and `HasDestination` interfaces. - **Task creation** (`workflow_task_completed_handler.go`, `task_generator.go`): When `RequestCancelActivityTask` is processed for a started activity whose worker has a control queue, collects a `CancelActivityCommand` with the activity's task token. Commands are batched by destination control queue and flushed as one `WorkerCommandsTask` per queue at the end of WFT processing. - **Serialization**: `task_serializers.go` for persistence round-tripping. Dispatch is a no-op here — handled in #9233. Gated by dynamic config `EnableCancelActivityWorkerCommand` (default: off). ## Why? To support proactive activity cancellation without waiting for heartbeat. This is the task creation leg of the flow. 1. [#9231] Store `worker_control_task_queue` in `ActivityInfo` at activity start. 2. **[This PR]** On `RequestCancelActivityTask`, batch commands by control queue into `WorkerCommandsTask` outbound tasks. 3. [#9233] Dispatch each task as a Nexus `ExecuteCommands` operation to the worker, with a 3-attempt retry cap. 4. [SDK] Worker receives the cancel command and cancels the running activity. Gated by dynamic config `EnableCancelActivityWorkerCommand` (default: off). ## How did you test it? **Unit tests** cover task generation, command batching (including multi-queue batching), task serialization round-tripping, and the feature-flag-off path. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Kannan
|
4997c93fa8 |
Store worker attributes needed by server to propagate nexus tasks to worker (#9231)
## What changed? As part of RecordActivityTaskStarted flow, store worker_control_task_queue for an activity in the mutable state (ActivityInfo). Main changes: - executions.proto: Added the new worker_control_task_queue field. - mutable_state_impl.go: Update mutable state. - matching/forwarder.go: Propagate worker_control_task_queue when polls get forwarded. Otherwise, RecordActivityTaskStarted request will not have it set when invoked from a forwarded poll. ## Why? To support activity cancellation without activity heartbeat. Overall flow: - [This PR] Store worker attributes in ActivityInfo as part of RecordActivityTaskStarted call. - [#9232] When user cancels a workflow, create 1 or more tasks. Group all activities belonging to a worker into the task (for efficiency). - [#9233] Lookup the Nexus task queue for each worker, and send a Nexus operation for each transfer task. - [SDK] Worker will receive this cancel task and cancel the running activities. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Shivam Ajmera
|
578dd4d0d1 |
change extension number (#9898)
## What changed? Update proto extension number 7234 to 50234 ## Why? it was colliding with a saas-control-plane dep (discussed [here](https://temporaltechnologies.slack.com/archives/C0ARSCEHU75/p1775773612977229?thread_ts=1775773490.781049&cid=C0ARSCEHU75)) ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Carly de Frondeville
|
ac3c439a1a |
Add CONTINUE_AS_NEW_VERSIONING_BEHAVIOR_USE_RAMPING_VERSION (#9894)
## What changed?
Add `CONTINUE_AS_NEW_VERSIONING_BEHAVIOR_USE_RAMPING_VERSION` as a
`ContinueAsNewInitialVersioningBehavior` in `InheritedAutoUpgradeInfo`
and `VersioningInfo`, which if set, forces the first task of an
AutoUpgrade workflow to go to the Ramping Version of its task queue
instead of basing the Current/Ramping version selection on ramp
percentage and workflow id. If there is no Ramping Version at the time
of workflow dispatch, the workflow will use the Current Version.
This initial behavior comes from the continue-as-new command, and when
history sends workflow tasks to matching, is converted into an
internally defined `UseRampingVersionInitialTask bool` such that _only_
the first workflow task of an AutoUpgrade workflow will change its
Target Version selection. Retries of this workflow will start with the
same behavior, but child workflows and future continue-as-new workflows
initiated by this workflow will not inherit the initial behavior.
## Why?
To enable more fine grained control of upgrade-on-continue-as-new
upgrade stages before doing percentage based ramp. To use this, users
should set their promotion candidate version to the ramping version with
a ramp percentage of zero, and manually signal a certain cohort of
workflows to upgrade-on-continue-as-new with this option.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes workflow-task routing for AutoUpgrade executions by adding a
new directive flag that can bypass ramp-percentage hashing for the
initial task. Moderate risk because it affects matching/history version
selection and ContinueAsNew/retry inheritance semantics.
>
> **Overview**
> Adds a new `use_ramping_version` field to `TaskVersionDirective` and
threads it through history → matching so an AutoUpgrade workflow can
*force its initial workflow task* to route to the task queue’s ramping
deployment version (falling back to current when no ramping version
exists), bypassing the normal ramp-percentage/workflow-id hash.
>
> History now persists and evaluates the ContinueAsNew-requested initial
behavior via `MutableState.GetShouldUseRampingVersion()`, ensures it
applies only to the first workflow task, propagates it across retries,
and explicitly does **not** propagate it to child workflows or
subsequent ContinueAsNew hops. Updates tests (unit +
`versioning_3_test`) and bumps `go.temporal.io/api` to pick up the new
proto/enum support.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
|
||
|
Shahab Tajik
|
744d3f2efe |
Add compute config to WorkerDeploymentVersionSummary (#9838)
## What changed? - Add compute config summary for latest and all other version so it is exposed in `ListWorkerDeployments` and `DescribeWorkerDeployment` APIs. ## Why? Mainly for UI to show compute provider types. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks None |
||
|
Shahab Tajik
|
d8813fd133 |
Serverless Feature Integration (#9779)
This PR merges the serverless feature branch into main. Individual PRs included in this branch: - #9380 - #9651 - #9412 - #9746 - #9759 --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Stefan Richter <stefan.richter@temporal.io> |
||
|
jiechenz
|
72567b9ed1 |
Replicate workflow deleteion (#9717)
## What changed? Add a new replication task type `DeleteExecutionReplicationTask` that replicates workflow deletion from the active cluster to passive/standby clusters. Gated by feature flag `history.enableDeleteWorkflowExecutionReplication` (default: false). #### **Key changes across the replication pipeline:** 1. Proto enums: `TASK_TYPE_REPLICATION_DELETE_EXECUTION` (34), `REPLICATION_TASK_TYPE_DELETE_EXECUTION_TASK` (13) 2. Replication task is associated with a new stage in `ShardContext.DeleteWorkflowExecution`, bundled with delete visibility task. 3. ~Engine interface: added `ForceDeleteWorkflowExecution` so the task can invoke the `ForceDeleteWorkflowExecution`.~ ## Why? Today, when a user delete workflow execution in source cluster, this operation will not replicate to the standby/target clusters. When a namespace failover to a target cluster, those deleted workflow may resurrected. <details> <summary>Race condition analysis</summary> **Before this change:** 1. **Cross-cluster resurrection:** Active deletes workflow → standby untouched → failover → workflow reappears. 2. **Termination event silently dropped:** Deleting a running workflow terminates it first, generating a `HistoryReplicationTask`. But the async `CloseExecutionTask` may delete mutable state before the stream sender converts that task. The converter calls `getBranchToken()` → `NotFound` → task silently dropped. The standby never sees the termination or the deletion. **After this change:** Race 1 is fixed — `DeleteExecutionReplicationTask` explicitly tells the standby to delete. Race 2 is mitigated — even if the termination event's replication task is dropped, the delete replication task ensures the standby cleans up. - If the workflow is still running (termination not yet replicated), the `DeleteExecutionTask` reschedules itself until the workflow closes. - If the termination event arrives later, the workflow closes normally, then the delete proceeds. - If the workflow is already deleted (e.g., by retention), the task is a no-op (`NotFound` treated as success). </details> <details> <summary>Deletion paths</summary> | Path | Replication task? | |------|-------------------| | User deletes workflow (active, running or closed) | Yes | | User deletes on passive (DC forwarding ON) | Forwarded to active → yes | | User deletes on passive (no forwarding) | No — `ActiveInCluster` check skips | | Retention expiry (with or without archival) | No — stage pre-marked as processed | | Admin ForceDelete (tdbg) | No — bypasses `DeleteWorkflowExecution` | </details> ## How did you test it? - [x] built - [x] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) Before change: <img width="1507" height="163" alt="Screenshot 2026-03-26 at 11 59 51 PM" src="https://github.com/user-attachments/assets/118cc50e-b69d-468a-9e45-5f49e4e4b9d1" /> After change: <img width="1507" height="135" alt="Screenshot 2026-03-27 at 12 00 07 AM" src="https://github.com/user-attachments/assets/8ccb7a11-2cb4-48b5-af89-b4a60ddb6333" /> ## Potential risks n/a |
||
|
Stephan Behnke
|
1d58ed21cc |
buf format (#9663)
## What changed? Adds `buf format` as a Makefile target; and integrates it into `make fmt`. All `.proto` changes are from running `make fmt`. ## Why? Consistent protobuf file style. ## How did you test it? - [x] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Shahab Tajik
|
d344f08be8 |
Ensure WD Version properly revives if recreated after deletion (#9382)
## What changed? - Ensure the TQs receive and apply the right version data after revive. - Made delete propagation to always happen serial to other propagations. It ensures all other propagations are cancelled before starting delete propagation. - Deprecate the `deleted` flag in version data and the GC logic around it. Now we use the good old forgetVersion path which immediately removes the version data from TQ. - Ensure version state is reset after revive, in case the recreation happened before workflow close. - Also, now workflows CaN based on SDK suggestion if no pending Signal or Update is present. ## Why? The version could stuck at deleted state from TQ POV if revived before the (now deprecated) GC logic cleans it up. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) ## Potential risks None --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
David Reiss
|
d0764befb4 |
Add drained field to InternalTaskQueueStatus (#9453)
## What changed? Add drained field to InternalTaskQueueStatus. ## Why? This is useful to tell if a queue is empty. Will be used for automatic partition scaling. ## How did you test it? - [ ] built - [x] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
David Reiss
|
24cea9d4f3 |
Separate ephemeral data for different task queue types (#9450)
## What changed and why? Ephemeral data was meant to be separate per task queue type (instead of for all types together like task queue user data), but the implementation didn't match that exactly: activity/other queues did propagate ephemeral data from the workflow queue. This fixes that to not pass ephemeral data along that edge. Also reduce the `returning user data` log to debug level. ## How did you test it? - [ ] built - [x] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) |
||
|
Rodrigo Zhou
|
6b11e07e8f |
[Chasm] Introduce proto ChasmExecutionInfo for Visibility (#9652)
## What changed? Introduce proto types for Visibility: - `ChasmExecutionInfo` - `ListChasmExecutionsRequest` - `ListChasmExecutionsResponse` Changed VisibilityManager API to use these protos. ## Why? Make VisibilityManager API more accessible for potential external usage. ## How did you test it? - [x] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks |
||
|
Yichao Yang
|
0463b59dd9 |
CHASM: support archetypeID in admin handler (#9309)
## What changed? - Accept archetypeID in admin API requests ## Why? - With this we no longer need to register chasm components to worker service, which currently performs an archetypeID to name conversion before calling admin apis. - For backward compatibility, we can only stop registering chasm components to worker service starting from next cloud release. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [x] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Lina Jodoin
|
e535cb5d01 |
[Scheduled Actions] Attach Nexus callbacks to schedules migrated from V1/Legacy (#9560)
## What changed? - Nexus callbacks are attached to running workflows after migration from a V1 workflow-backed schedule. This is done through a new side-effect task. ## Why? - Drives workflow completion events. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [x] added new functional test(s) |
||
|
Sean Kane
|
ff2754a711 |
Add CallbackRequestID to workflow execution for use by CHASM Schedules (#9479)
## What changed?
When a workflow is reset, `ApplyWorkflowExecutionStartedEvent`
re-registers the start-event callbacks using the reset operation's
request ID. `HandleNexusCompletion` cannot find a matching
`BufferedStart` and discards the completion. The `resetRequestID` param
is removed from `WorkflowResetter.ResetWorkflow` and the original
request ID is used. `findStartRequestID` reads the original request ID
back from `WorkflowExecutionInfo.RequestIds` by finding the
`EVENT_TYPE_WORKFLOW_EXECUTION_STARTED` entry.
## Why?
CHASM scheduler relies on callback `request_id` to match WF completions
to originating `BufferedStart` entries. When it cannot be found the
scheduler is permanently stuck with the workflow marked as still
running.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
-
`TestScheduledWorkflowDoubleReset_SchedulerSeesCompletion_{HSM,CHASM}Callbacks`:
create a schedule, trigger immediately, reset the workflow twice, signal
the completion to complete, poll `ListSchedules` until scheduler shows
`COMPLETED`.
## Potential risks
CHASM scheduler has not been enabled in production yet, the blast radius
should be minimal.
|
||
|
Shahab Tajik
|
26f2023316 |
Improve task-dispatch-latency metric (#9395)
## What changed? Emit task dispatch latency metric in matching_engine with the following improvements: - latency includes history calls - latency is not reset in case of sync-match forwards - metric is not lost for backlogged task in new matcher - origin partition is preserved during forward and used at the partition tag in the metric (keeping existing behavior) - task and poll forwarding do not cause duplicate emits, keeping existing behavior Also, fixed the following unrelated bugs that surfaced while testing the metric: - Query priority is not lost when forwarded. ## Why? Fixes bugs. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [x] added new functional test(s) ## Potential risks None. |
||
|
Shivam
|
b6ae24e69e |
Trampolining Part 2: Avoid infinite loops for Pinned workflows (#9374)
## What changed?
- Consists of the change to prevent any Pinned workflows, that may have
forgotten to have the initial CAN Behaviour as AU, from CAN'ing
infinitely.
- Also allows trampolining of a Pinned workflow onto the Unversioned if
un-versioned is the current version of the worker deployment at that
point in time. Note, the effective behaviour of the workflow would later
then be unversioned.
## Why?
- Worker-Versioning correctness.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- Pre-existing workflows (started before this fix) have nil
TargetVersionOnStart. On the first WFT after deployment, "" !=
"build-v2" → spurious targetDeploymentVersionChanged=true. This is a
one-time false-positive regression for those Pinned workflows.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches generated protobuf API surface used by internal services;
while the change is additive, mismatched proto versions across services
could cause integration/compatibility issues.
>
> **Overview**
> Adds `declined_target_version_upgrade` to
`StartWorkflowExecutionRequest` (HistoryService API) so
continue-as-new/retry chains can carry forward the SDK-declined target
deployment version and avoid pinned-workflow trampolining loops.
>
> Regenerates protobuf Go bindings, updating import/type references
across `request_response.pb.go`, and adds missing
`Marshal`/`Unmarshal`/`Size`/`Equal` helpers for the persistence
`LastNotifiedTargetVersion` message.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
|
||
|
Alex Stanfield
|
fe35b108fe |
V1 to CHASM scheduler migration RPC (#9261)
## What changed Adds an admin RPC (`MigrateSchedule`) that migrates workflow-backed (V1) schedulers to CHASM (V2) schedulers. The flow: 1. Admin API signals the V1 scheduler workflow with `migrate-to-chasm` 2. V1 workflow runs a local activity that snapshots state and calls `CreateFromMigrationState` on the CHASM scheduler service 3. On success, the V1 workflow closes. On failure, a `PendingMigration` flag persists so the next run loop iteration retries automatically Key details: - `CreateSchedulerFromMigration` initializes the full CHASM scheduler tree (generator, invoker, backfillers, visibility) from V1 state - Running/completed workflows are converted to buffered starts; ongoing backfills are preserved - Migration is idempotent -- if the CHASM schedule already exists, it's treated as success - Metrics: `schedule_migration_started`, `schedule_migration_completed`, `schedule_migration_failed` (with direction tag) - Local activity uses 10m schedule-to-close timeout; retries happen at the workflow level via the persistent flag ## Why Needed for migrating existing V1 schedulers to CHASM without downtime or data loss. ## Follow-up items - V2 to V1 migration (rollback path) - Sentinel key handling when `EnableCHASMSchedulerCreation` was previously enabled - Attach completion callbacks to running workflows after migration |
||
|
Yu Xia
|
7eae6ac604 |
Introduce api category into history APIs (#9435)
## What changed? Introduce api category into history APIs ## Why? There are some APIs we want to exclude from health check. Using API category is a more protective way to manage this group. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Kannan Rajah
|
450b7f95d6 |
Populate WorkerListInfo when returning ListWorkersResponse (#9418)
## What changed? - Build `WorkerListInfo` in matching handler and pass through to frontend - Update `go.temporal.io/api` to include merged WorkerListInfo proto ## Why? `WorkerListInfo` contains only static worker attributes for efficient listing. Building it in matching (closer to the data source) enables eventually dropping the deprecated `WorkersInfo` field from the internal response. ## How did you test it? - [x] built - [x] added new unit test for field coverage - [x] covered by existing functional tests ## Potential risks None - backward compatible. Deprecated `WorkersInfo` field continues to be populated. |
||
|
Lanie Hei
|
88ea88ce1a |
Add per-check diagnostics to DeepHealthCheck API (#9350)
## Summary
Extends the `DeepHealthCheck` API to return **per-check diagnostic
details** alongside the existing `HealthState` enum. When fault
detection triggers a cell failover, operators can now see exactly which
health check failed and why — not just that the cell is unhealthy.
### What changed
- **New proto package `health/v1`** — `HealthCheck` message with
`check_type` (string), `state`, `value`, `threshold`, and human-readable
`message`. `HostHealthDetail` and `ServiceHealthDetail` aggregate
per-host and per-service results.
- **New enum value `HEALTH_STATE_INTERNAL_ERROR`** — for infrastructure
failures like membership resolver errors (previously returned
`UNSPECIFIED`).
- **History handler** now runs all 5 checks unconditionally (gRPC
health, RPC latency, RPC error ratio, persistence latency, persistence
error ratio) and returns each with actual values and thresholds.
Previously it early-returned on first failure.
- **Frontend health checker** collects per-host `HostHealthDetail`
(address, state, checks) and builds a `ServiceHealthDetail` with
diagnostic messages for all paths — including resolver errors and empty
membership.
- **AdminService** passes `ServiceHealthDetail` through to callers.
- **`check_type` uses string constants**
(`common/health/check_types.go`) instead of a proto enum for
extensibility — new check types can be added without proto changes, and
the `message` field provides human-readable context with actual values
(e.g. `"RPC latency 850.00ms exceeded 500.00ms threshold"`).
### How it works
The call chain is: `AdminService.DeepHealthCheck()` →
`HealthChecker.Check()` → fan-out to all history hosts in membership →
`HistoryHandler.DeepHealthCheck()` per host.
Each history host runs **5 independent checks** and returns all results:
1. `grpc_health` — is the gRPC health server serving?
2. `rpc_latency` — average RPC latency vs threshold
3. `rpc_error_ratio` — RPC error rate vs threshold
4. `persistence_latency` — DB latency vs threshold
5. `persistence_error_ratio` — DB error rate vs threshold
The frontend collects results from **all hosts in membership**,
aggregates them, and returns the full breakdown.
### Example responses
#### Healthy cluster (3 hosts, all serving)
```json
{
"state": "HEALTH_STATE_SERVING",
"services": [{
"service": "history",
"state": "HEALTH_STATE_SERVING",
"hosts": [
{
"address": "10.0.1.5:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 45.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.01, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 12.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.6:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 52.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 18.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.7:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 38.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 15.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
}
]
}]
}
```
#### Degraded cluster — 1 host with high RPC latency (3 hosts, 1
failing, under threshold)
The failing host clearly shows which check triggered and the actual vs
threshold values. Because only 1/3 hosts failed (33%) and the failure
threshold is 25% but we require at least 2 hosts to fail, the overall
state remains `SERVING`.
```json
{
"state": "HEALTH_STATE_SERVING",
"services": [{
"service": "history",
"state": "HEALTH_STATE_SERVING",
"hosts": [
{
"address": "10.0.1.5:7234",
"state": "HEALTH_STATE_NOT_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_NOT_SERVING", "value": 850.0, "threshold": 500.0, "message": "RPC latency 850.00ms exceeded 500.00ms threshold"},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 120.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.6:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 52.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 18.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.7:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 38.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 15.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
}
]
}]
}
```
#### Unhealthy cluster — hosts unreachable (6 hosts in membership, 3
unreachable via RPC)
When the frontend cannot reach a host via RPC, it creates a synthetic
`host_availability` check with the error. The host appears in the
response with `NOT_SERVING` and the RPC error message. With 3/6 hosts
failing (50% > 25% threshold), the overall state is `NOT_SERVING`.
```json
{
"state": "HEALTH_STATE_NOT_SERVING",
"services": [{
"service": "history",
"state": "HEALTH_STATE_NOT_SERVING",
"hosts": [
{
"address": "10.0.1.5:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 45.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.01, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 12.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.6:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 52.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 18.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.7:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 38.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 15.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.8:7234",
"state": "HEALTH_STATE_NOT_SERVING",
"checks": [
{"check_type": "host_availability", "state": "HEALTH_STATE_NOT_SERVING", "message": "failed to reach host for health check: rpc error: code = Unavailable desc = connection refused"}
]
},
{
"address": "10.0.1.9:7234",
"state": "HEALTH_STATE_NOT_SERVING",
"checks": [
{"check_type": "host_availability", "state": "HEALTH_STATE_NOT_SERVING", "message": "failed to reach host for health check: rpc error: code = Unavailable desc = connection refused"}
]
},
{
"address": "10.0.1.10:7234",
"state": "HEALTH_STATE_NOT_SERVING",
"checks": [
{"check_type": "host_availability", "state": "HEALTH_STATE_NOT_SERVING", "message": "failed to reach host for health check: context deadline exceeded"}
]
}
]
}]
}
```
#### Host voluntarily draining — gRPC health declined (DECLINED_SERVING)
When a host's gRPC health server reports not serving (e.g. during
graceful shutdown), the check returns `DECLINED_SERVING`. If enough
hosts are in this state (exceeding the declined serving proportion
threshold), the overall service state becomes `DECLINED_SERVING`.
```json
{
"state": "HEALTH_STATE_DECLINED_SERVING",
"services": [{
"service": "history",
"state": "HEALTH_STATE_DECLINED_SERVING",
"hosts": [
{
"address": "10.0.1.5:7234",
"state": "HEALTH_STATE_DECLINED_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_DECLINED_SERVING", "message": "gRPC health server not serving"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 45.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.01, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 12.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.6:7234",
"state": "HEALTH_STATE_DECLINED_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_DECLINED_SERVING", "message": "gRPC health server not serving"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 52.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 18.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
},
{
"address": "10.0.1.7:7234",
"state": "HEALTH_STATE_SERVING",
"checks": [
{"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
{"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 38.0, "threshold": 500.0},
{"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
{"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 15.0, "threshold": 500.0},
{"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
]
}
]
}]
}
```
#### No hosts in membership
When the membership resolver returns an empty host list, the response
includes a service-level message but no hosts.
```json
{
"state": "HEALTH_STATE_NOT_SERVING",
"services": [{
"service": "history",
"state": "HEALTH_STATE_NOT_SERVING",
"message": "no available hosts in membership"
}]
}
```
#### Membership resolver failure (INTERNAL_ERROR)
When the frontend can't resolve the membership ring at all
(infrastructure failure), the response includes `INTERNAL_ERROR` with
the resolver error.
```json
{
"state": "HEALTH_STATE_INTERNAL_ERROR",
"services": [{
"service": "history",
"state": "HEALTH_STATE_INTERNAL_ERROR",
"message": "failed to get membership resolver: membership monitor not started"
}]
}
```
### Backward compatibility
- `DeepHealthCheckResponse.state` (field 1) unchanged in both history
and admin protos
- New fields (`checks`, `services`) are additive (field 2) — old clients
simply ignore them
- `GetState()` continues to work as before
### Related
- saas-control-plane PR #12203 — `HealthReport` + `CellHealthEvent`
(consumer side, ready to use these fields)
- Runbooks PR #1231 — end-to-end flow documentation
## Test plan
- [x] All existing `TestHealthCheckerSuite` tests pass (19 tests)
- [x] New tests: `Test_Check_ServiceDetail_Populated`,
`Test_Check_HostChecks_Propagated`, `Test_Check_GetResolver_Error`
(INTERNAL_ERROR + message), `Test_Check_No_Available_Hosts` (message)
- [x] Full `go build ./...` passes
- [ ] Verify saas-control-plane can import
`go.temporal.io/server/common/health` constants
- [ ] Integration test with actual history service
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
|
||
|
David Reiss
|
83174dfb02 |
Fairness counter: persist top K keys (#9188)
## What changed? The counts of some top keys are persisted in task queue metadata so they're preserved on queue movement or reloads. ## Why? More accurate fairness ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) |
||
|
Sean Kane
|
0fbc386c55 |
Include transient and speculative WFT events in GetWorkflowExecutionHistoryResponse (#9325)
## What changed? Re-does #9138 which was incidentally merged. Include transient and speculative WFT events in `GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made request. * Adds `transient_or_speculative_events` back to `GetMutableStateResponse` * Reserve `transient_workflow_task` in `HisotryCOntinuation` token * Add validation helpers * Add query-compare-query for transient events at request start and end Re-implements #7732 ## Why? Fix "premature end of stream" errors when workers request history after cache eviction w/ transient/speculative workflow tasks present. This adds transient & speculative WFT events in `GetWorkflowExecutionHistory` (already in `PollWorkflowTask`). Worker cache eviction w/ speculative workflow tasks causes the expected and actual event counts to be different. #7732 passed transient events through continuation tokens, which could become stale during pagination. This PR implements mutable state querying at both start and end of pagination and compares transient event IDs to detect if WFT state changed during pagination and return a retryable error. ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks Same risks from #7732 |
||
|
Roey Berman
|
e0d9f48c84 |
System nexus endpoint (#9002)
## Overview
This feature introduces a special internal Nexus endpoint called
`__temporal_system` that enables adding functionality to workflows
**without requiring new workflow commands and events**. Operations on
this endpoint are routed internally within Temporal's history service
rather than via external HTTP calls.
## Key Components
### 1. System Endpoint Infrastructure
- **Endpoint Name**: `__temporal_system` (constant in
`common/nexus/constants.go:6`)
- **Callback URL**: `temporal://system` for internal routing
- **New History Service RPCs**
(`proto/internal/temporal/server/api/historyservice/v1/service.proto:433-437`):
- `StartNexusOperation` - Starts operations on the system endpoint
- `CancelNexusOperation` - Cancels operations on the system endpoint
### 2. Operation Processor Framework
A new processor pattern (`chasm/nexus_operation_processor.go`) that
allows CHASM libraries to:
- **Validate and transform input**: Processors can validate operation
inputs and set default values
- **Determine routing**: Each processor returns a routing key that
determines which history shard handles the operation
- **Re-serialize input**: Mutated inputs can be re-serialized to persist
default values
**Routing strategies**:
- `NexusOperationRoutingKeyExecution` - Routes to the shard owning a
specific workflow execution
- `NexusOperationRoutingKeyRandom` - Routes to a random shard
### 3. CHASM Library Integration
CHASM libraries can now provide (`chasm/library.go:16-19`):
- **`NexusServices()`**: Regular Nexus service handlers (implement the
actual operation logic)
- **`NexusServiceProcessors()`**: Input processors for validation and
routing
Example from test library (`chasm/lib/tests/nexus_service.go`):
```go
// Service handler - implements the actual operation
TestOperation = nexus.NewSyncOperation("TestOperation",
func(ctx context.Context, input string, options nexus.StartOperationOptions) (string, error) {
return "Hello, " + input, nil
})
// Processor - validates input and determines routing
func (o testOperationProcessor) ProcessInput(ctx chasm.NexusOperationProcessorContext, input string)
(*chasm.NexusOperationProcessorResult, error) {
return &chasm.NexusOperationProcessorResult{
RoutingKey: chasm.NexusOperationRoutingKeyExecution{
NamespaceID: ctx.Namespace.ID().String(),
BusinessID: input, // Route based on input
},
}, nil
}
```
### 4. Execution Flow
When a workflow schedules a Nexus operation on `__temporal_system`
(`components/nexusoperations/executors.go:233-238`):
1. **Input Processing**: The processor validates input and determines
routing
2. **Internal RPC**: Instead of HTTP, calls
`HistoryClient.StartNexusOperation` with the target shard ID
3. **Handler Execution**: The history service invokes the registered
Nexus handler (`service/history/handler.go:2707-2768`)
4. **Result Handling**: Supports both sync (immediate result) and async
(operation token) responses
5. **Workflow Completion**: Results flow back through the same
completion path as external Nexus operations
### 5. Benefits
✅ **No schema changes**: Add functionality without new commands/events
in workflow history
✅ **Consistent API**: Uses existing Nexus operation semantics
(sync/async, callbacks, links)
✅ **Proper routing**: Operations are intelligently routed to the correct
shard
✅ **Input validation**: Type-safe input validation and default value
handling
✅ **Future extensibility**: Foundation for direct client invocation (not
yet implemented)
### 6. Technical Details
- **Error handling** (`components/nexusoperations/executors.go:444`):
Non-retryable service errors are properly handled and fail operations
immediately
- **Metrics**: System operations are tracked separately with
`DestinationTag` set to the endpoint name
- **Link conversion**: Helper functions convert between Nexus SDK links
and protobuf links (`common/nexus/util.go:17-46`)
- **Operation token handling**: Moved link converters to common package
for reuse (`common/nexus/link_converter.go`)
### 7. Current Limitations
- Only accessible from workflows (via `ScheduleNexusOperation` command)
- Direct client invocation not yet implemented
- Headers not supported for system endpoint operations
## Test Coverage
New test (`tests/nexus_workflow_test.go:2763-2843`) demonstrates:
- Scheduling operation on `__temporal_system` endpoint
- Synchronous operation completion
- Result propagation back to workflow
## Architecture
This architecture provides a clean, extensible way to add internal
functionality while maintaining compatibility with Temporal's existing
workflow execution model. The system endpoint acts as a bridge between
workflows and internal CHASM components, enabling:
- **Extensibility**: New operations can be added by implementing CHASM
libraries
- **Type safety**: Input validation happens before operations are routed
- **Scalability**: Intelligent routing ensures operations land on the
correct shard
- **Consistency**: Same execution model as external Nexus operations
|
||
|
Sean Kane
|
bd19bc6059 |
Revert "Include transient and speculative WFT events in GetWorkflowExecutionHistoryResponse" (#9322)
Reverts temporalio/temporal#9138 |
||
|
Sean Kane
|
aad69c365b |
Include transient and speculative WFT events in GetWorkflowExecutionHistoryResponse (#9138)
## What changed? Include transient and speculative WFT events in `GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made request. * Adds `transient_or_speculative_events` back to `GetMutableStateResponse` * Reserve `transient_workflow_task` in `HisotryCOntinuation` token * Add validation helpers * Add query-compare-query for transient events at request start and end Re-implements #7732 ## Why? Fix "premature end of stream" errors when workers request history after cache eviction w/ transient/speculative workflow tasks present. This adds transient & speculative WFT events in `GetWorkflowExecutionHistory` (already in `PollWorkflowTask`). Worker cache eviction w/ speculative workflow tasks causes the expected and actual event counts to be different. #7732 passed transient events through continuation tokens, which could become stale during pagination. This PR implements mutable state querying at both start and end of pagination and compares transient event IDs to detect if WFT state changed during pagination and return a retryable error. ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [ ] added new unit test(s) - [ ] added new functional test(s) ## Potential risks Same risks from #7732 |
||
|
Kannan Rajah
|
c79beb7cbf |
Eagerly remove worker from poller history on worker shutdown (#9289)
## What changed? Eagerly remove worker from pollerHistory during graceful shutdown so DescribeTaskQueue doesn't show stale pollers. **Note:** This PR subtly changes how pollerHistory is updated `before` and `after` a poll request. Before: - UpdatePollerInfo called at poll START - defer UpdatePollerInfo called at poll END (always, regardless of how poll ended) After: - UpdatePollerInfo called at poll START - UpdatePollerInfo called at poll END only if ctx.Err() != context.Canceled Skipped on cancellation (shutdown/disconnect) to avoid re-adding entry after RemovePoller The defer was originally added (PR #2811) to keep timestamps fresh, but HasPollerAfter() first checks currentPolls > 0 for active polls, so the timestamp is a secondary check **Main changes** API - request_response.proto: Added worker_identity field to CancelOutstandingWorkerPollsRequest Implementation - matching_engine.go: As part of CancelOutstandingWorkerPolls, also invoke RemovePoller on the task queue partition manager. - task_queue_partition_manager.go: Forward the removal to default and versioned task queues. ## Why? Previously, pollers lingered in pollerHistory until TTL expired (~5 min), showing workers that already shut down. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) ## Potential risks Controlled by dynamic config: EnableCancelWorkerPollsOnShutdown |
||
|
Roey Berman
|
53a81d6c18 |
Overhaul Nexus error model (#9290)
## What changed? Replace the limited Nexus HandlerError proto with full Temporal Failure objects for transmitting errors across Nexus operation boundaries. This preserves failure chains, stack traces, and typed failure info (ApplicationFailure, CanceledFailure, etc.) end-to-end. Care was taken to ensure Temporal->Temporal communication works as expected as well as support for non-Temporal Nexus implementations (to maintain support for external endpoints). This ended up being more refactoring that I had originally anticipated but I think the unification of error serialization across the codebase and the more straightforward conversion logic was worth it. The code was validated across variations of new and old caller and handler workers and servers using a [harness](https://github.com/bergundy/nexus-error-compat-tests) that was built specifically to validate the behavior. The harness was also run with an SDK setup that encoded failure attributes to confirm failures are properly transmitted over the different boundaries. Proto changes: - Add `failure` field to DispatchNexusTaskResponse carrying a Failure with NexusHandlerFailureInfo, deprecate `handler_error` - Add `capabilities` field to nexus Request for feature negotiation Failure conversion (`common/nexus/failure.go`): - Rewrite TemporalFailureToNexusFailure and NexusFailureToTemporalFailure with support for recursive cause chains and stack traces - Add special-case handling for NexusHandlerFailureInfo (serialized as nexus.HandlerError type) and OperationError (mapped to CanceledFailure/ApplicationFailure based on state) Nexus SDK layer (`common/nexus/nexusrpc/`): - Inline the failure converter from the upstream SDK with a custom FailureConverter interface that round-trips Temporal failure metadata through Nexus Failure objects - Add a completion client to abstract away HTTP and provide consistent failure conversion across all APIs - Support `unwrap-error` metadata marker so Temporal-to-Temporal calls can unwrap the OperationError envelope and recover the original cause Frontend and backend handlers: - Route responses through new failure path when caller sends `temporal-nexus-failure-support` header - Handle both new `failure` and deprecated `handler_error` response types in nexus_handler.go and nexus_http_handler.go - Update completion handling in history handler and mutable state to produce Temporal Failures instead of Nexus HandlerErrors - Update matching engine to forward the new failure field Dependencies: - Bump `nexus-rpc/sdk-go` to pre-release with StackTrace, Cause, and OriginalFailure fields on Failure and OperationError ## Why? Part of getting Nexus to GA in all SDKs, this change fixes a couple of notable issues with errors in Nexus+Temporal applications: - Nexus SDK errors did not have a way to set an error message, which made them difficult to use in various languages - The protocol used a custom failure format that is diverges from proxy expectations and may result in failures not being encrypted ## Potential risks - Metric label values for callback outcomes now use Nexus handler error values instead of HTTP response codes. Alerts that check these outcomes will need adjustment. |
||
|
Carly de Frondeville
|
e23830cf7d |
Set TargetVersionChanged instead of SuggestCaN when TargetVersionChanged (#9239)
## What changed?
Set TargetVersionChanged instead of SuggestCaN when TargetVersionChanged
https://github.com/temporalio/api/pull/709
## Why?
Setting SuggestContinueAsNew=true for Pinned workflows whenever their is
a new Target Version available for that workflow causes Pinned workflows
to hit that condition much more frequently than they expect. Users who
are currently doing: if workflow_info.suggestContinueAsNew{ do
continue-as-new } in their Pinned workflow code would need to change
that code to protect themselves from running into an infinite-CaN-loop,
because the default CaN behavior for a Pinned workflow is to stay
Pinned.
We should not force users to protect themselves from such a situation.
Because upgrading on continue-as-new is opt-in, receiving the suggestion
to continue-as-new-onto-new-target-version should be opt-in as well. If
people are forced to check the new suggest-continue-as-new-reasons field
to "opt out," that is unsafe, because inevitably some people will forget
to do so or misunderstand, and then get hit by this unexpected footgun.
Much safer and still ergonomical to let upgrade-on-can be opt-in on both
fronts, as proposed here. With this change, the people who are currently
doing if workflow_info.suggestContinueAsNew{ do continue-as-new } won't
see any change in semantics, regardless of their versioning behavior.
People who consciously know that they want to do upgrade-on-can /
Trampolining will have to change their CaN options anyway, so it's easy
enough to teach them to pay attention to this new
TargetWorkerDeploymentVersionChanged flag.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches workflow task started event generation/persistence and
versioning-related signaling, which can affect worker behavior and
history compatibility; changes are gated by dynamic config and covered
by tests.
>
> **Overview**
> Stops using `SuggestContinueAsNew` (and its reason tags) to signal
pinned workflows that a newer target worker deployment version exists,
and instead introduces an explicit
`TargetWorkerDeploymentVersionChanged` boolean on `WorkflowTaskStarted`
events and persisted `WorkflowExecutionInfo`.
>
> Adds namespace dynamic config `EnableSendTargetVersionChanged`
(default on) and a new metric `workflow_target_version_changed_count`
emitted when this flag is set; updates the workflow task state machine,
mutable state plumbing/mocks, proto/pb persistence, and functional tests
accordingly. Also bumps `go.temporal.io/api` to pick up the new event
attribute.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
|
||
|
Kannan Rajah
|
abdc95a763 |
Add CancelOutstandingWorkerPolls RPC to matching service (#9202)
## What changed? Adds infrastructure to cancel all outstanding polls for a worker instance during shutdown. Key Changes: 1. request_response.proto / service.proto - New CancelOutstandingWorkerPolls RPC that cancels polls by worker_instance_key instead of individual poller_id 2. matching_engine.go: - Added workerInstancePollers map to track pollers by worker instance key - pollTask() now registers pollers in both outstandingPollers (by pollerID) and workerInstancePollers (by worker key) - CancelOutstandingWorkerPolls() cancels all pollers for a worker instance and returns count ## Why? To support eager cancellation of outstanding polls when worker call ShutdownWorker. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) ## Potential risks None. No changes to existing functionality. |
||
|
Alex Stanfield
|
56c648a3b0 |
add MigrateSchedule proto and conversion functions (#9058)
## What changed? Added proto definitions and conversion functions for V1 to V2 schedule migration. ## Why? This is the foundation for the V1→V2 schedule migration feature. The `MigrateScheduleRequest` proto captures all scheduler state needed to recreate a schedule in the CHASM architecture. |
||
|
Roey Berman
|
f911e1e7a5 |
Nexus caller timeouts (#9153)
Reviving #9033 with better debuggability and fixes to issue found in the nightly pipelines. |
||
|
Prathyush PV
|
10caf69fdf |
Send raw history events from matching to frontend service (#8829)
## What changed? This PR extends the raw history optimization to pass raw history bytes from History Service → Matching Service → Frontend without deserialization in Matching Service. **Key changes:** 1. History Service: - When `SendRawHistoryBetweenInternalServices` is enabled, sets `RawHistoryBytes` (field 21) with raw proto-encoded history batches 2. Matching Service: - Passes raw history bytes through to frontend via `PollWorkflowTaskQueueResponseWithRawHistory` - Uses wire-compatible proto messages so gRPC auto-deserializes `[][]byte` → `History` on the client side 3. Frontend: - Receives raw history in `RawHistory` field (auto-deserialized by gRPC) - Processes search attributes for raw history since it bypasses history service's normal processing 4. Proto definitions: - Added `raw_history_bytes` (field 21) to `RecordWorkflowTaskStartedResponse` - Added `PollWorkflowTaskQueueResponseWithRawHistory` message with wire-compatible layout - Added `raw_history` (field 22) to `PollWorkflowTaskQueueResponse` ## Why? When `history.sendRawHistoryBetweenInternalServices` is enabled, the previous implementation only avoided deserialization from persistence → History Service. However, Matching Service was still deserializing history events (via gRPC auto-deserialization) and re-serializing them when forwarding to Frontend. This change eliminates that unnecessary serialization/deserialization cycle in Matching Service by: 1. Having History Service send raw bytes directly 2. Having Matching Service forward these raw bytes without parsing 3. Having Frontend receive the bytes which gRPC auto-deserializes This reduces CPU usage in Matching Service for workflows with large histories. ## How did you test it? - [x] built - [x] covered by existing tests - [x] added new unit test(s) - [x] added new functional test(s) (`tests/workflow_task_test.go`) ## Potential risks SendRawHistoryBetweenInternalServices must be disabled when rolling back from this version to an older version. --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Alan Wu
|
c407dc6e29 |
Implement detached component (#9086)
## What changed? Implement detached component as a Field option and Registrable Component option. Add detached boolean value to ComponentAttributes persistence proto definition. ## Why? Allow detached components to continue updates and task execution even if parent node lifecycle is closed. ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [X] added new unit test(s) - [ ] added new functional test(s) |
||
|
Shivam
|
6143b6a4af |
Pass deployment state and version state in forceCAN signals (#9100)
## What changed?
- WISOTT
## Why?
- Operational readiness. There could be a world where we want to "reset"
a user's workflow state given that these workflows CAN so frequently.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- None
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduces optional state override when force-continuing-as-new.
>
> - New proto messages `ForceCANDeploymentSignalArgs` and
`ForceCANVersionSignalArgs` with `override_state` fields; generated
helper/pb code updated
> - Deployment and Version workflows now receive `forceCAN` signals with
args and, if provided, apply `override_state` before continue-as-new
> - Added tests verifying override is honored (e.g., manager identity
and metadata) after continue-as-new
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
|
||
|
Yichao Yang
|
8728c4b1a2 |
Fix migration workflow breaking change for OSS (#9085)
## What changed? - Use plain go struct inside migration workflow activity input output - Customize the json encoding/decoding for migration execution info and make it backward & forward compatible. - This will temporary break cloud and the fix is in https://github.com/temporalio/temporal/pull/9097 - This PR needs to be part of oss v1.30 release. ## Why? - Backward compatibility and less confusion over how the encoding will be done. ## How did you test it? - [ ] built - [ ] run locally and tested manually - [ ] covered by existing tests - [x] added new unit test(s) - [ ] added new functional test(s) |