## What changed?
Added support for Nexus workflow update completion callbacks via CHASM.
This allows a Nexus caller to be notified when a workflow update
completes by attaching completion callbacks to the update request.
## Why?
Nexus operations that target workflow updates need a way to receive
completion notifications. Without this, a Nexus caller that sends an
update has no async mechanism to learn when the update finishes.
Completion callbacks enable the same async notification pattern that
already exists for workflow-level Nexus operations.
## How did you test it?
- [ ] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Touches speculative workflow updates, they are always hard to reason
about. Tried to compensate with lots of test coverage.
Note: Needs this API PR
https://github.com/temporalio/api/pull/742/changes
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **High Risk**
> Touches workflow update state machine and mutable state event handling
to persist/trigger per-update callbacks, including close/retry/reset
paths, which is complex and can affect correctness of update outcomes
and callback delivery.
>
> **Overview**
> Adds **workflow update completion callbacks** via CHASM so Nexus
callers can register callbacks on `UpdateWorkflowExecution` and have
them fired on update completion or workflow close.
>
> This introduces a `WorkflowUpdate` CHASM component with new
`UpdateState` protobuf (including persisted `rejection_failure`), stores
update callbacks under `Workflow.Updates`, and extends callback
processing to handle *update-level* callbacks on update completion,
rejection (including reset/reapply), and on run transitions
(retry/timeout/continue-as-new) where update callbacks must fire even if
workflow-level callbacks are inherited.
>
> It also adds dynamic config gates/limits
(`EnableWorkflowUpdateCallbacks`, `MaxCallbacksPerUpdateID`), updates
`DescribeWorkflow` to surface update callback triggers, extends mutable
state/history builder APIs to carry per-update callback options in
`WorkflowExecutionOptionsUpdated`, and adds `Update.AttachCallbacks`
logic to persist/flush callbacks (including buffering while `stateSent`,
request-id dedup, and stricter validation requiring `request_id` when
callbacks are present).
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
4484fee104. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: long-nt-tran <long.tran@temporal.io>
## What changed?
Added a method to generate an event load token in:
- `MutableState`
- `MutableStateImpl`
- `chasm.NodeBackend`
- `chasm.MSPointer`
- `workflow.Workflow`
## Why?
This was pointed out during the initial implementation but was not
addressed. The point of having a token is to avoid exposing the
persistence concept of event batches to the application.
## What changed?
NOTE: Recreation of https://github.com/temporalio/temporal/pull/10026
since that PR cannot be re-opened or have its target branch updated.
New metrics:
### Counters
| Metric | Description |
|--------|-------------|
| `nexus_operation_success_count` | Successfully completed operations |
| `nexus_operation_failed_count` | Failed operations |
| `nexus_operation_cancel_count` | Cancelled operations |
| `nexus_operation_terminate_count` | Operations terminated before
completion |
| `nexus_operation_timeout_count` | Operations timed out before
completion |
### Histograms
| Metric | Description |
|--------|-------------|
| `nexus_operation_schedule_to_close_latency` | Time between schedule
and close for sync and async operations |
| `nexus_operation_schedule_to_start_latency` | Time between schedule
and start for sync and async operations |
| `nexus_operation_start_to_close_latency` | Time between start and
close for async operations only |
## Labels
All metrics use the following labels:
| Label | Notes |
|-------|-------|
| `namespace` | |
| `nexus_endpoint` | |
| `nexus_service` | Controlled by Dynamic config |
| `nexus_operation` | Controlled by Dynamic config |
| `workflowType` | |
Latency metrics also include:
| Label | Values |
|-------|--------|
| `outcome` | succeeded, failed, canceled, terminated, timedout |
Timeout Count also includes:
| Label | Values |
|-------|--------|
| `timeout_type` | StartToClose, ScheduleToStart, ScheduleToClose |
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Metrics emitting isn't nil safe, and enrich metrics could panic if
misconfigured.
This PR migrates the Nexus operation invocation task handler from HSM
version to Chasm.
Migrating from HSM to Chasm
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Introduces a new CHASM-based Nexus `StartOperation` execution path
with endpoint lookup, callback URL/token generation, and error
classification; mistakes could cause failed invocations, incorrect
retries/timeouts, or misrouted callbacks. Risk is mitigated somewhat by
strict task validation and added unit coverage, but the change touches
critical workflow/history integration and outbound request handling.
>
> **Overview**
> Migrates Nexus operation invocation execution to CHASM by implementing
`OperationInvocationTaskHandler.Validate/Execute` end-to-end, including
endpoint resolution (ID with name fallback), callback URL selection
(system vs templated), callback token generation, timeout budgeting,
outbound StartOperation calls (HTTP or internal history service),
metrics/logging, and classification of results into operation state
transitions.
>
> Adds supporting plumbing:
`OperationStore.NexusOperationInvocationData` and workflow
implementation that loads invocation input/headers from the scheduled
history event, plus a new
`MSPointer.LoadHistoryEvent`/`NodeBackend.LoadHistoryEvent` API.
Configuration is extended to parse `CallbackURLTemplate` into a
`*template.Template`, add `UseSystemCallbackURL`, and pass
`NumHistoryShards` for internal routing; new helper utilities centralize
callback building, error/failure conversion, and internal/HTTP start
logic.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
4b139770ff. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Ported command handler for Nexus "cancel" command from HSM to CHASM.
CHASM migration.
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Roey Berman <roey.berman@gmail.com>
## What changed?
- Define TerminableComponent and RootComponent interface
- Make metrics handler available through chasm context
- Add support for passing key value pairs with chasm context similar to
context.Context
- Add support for specifying key values pairs that will always be
available in the Context when a component is accessed.
- Add requestID to TerminateComponentRequest
## Why?
- Improve support for library authors to implement the Terminate method.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Replace the limited Nexus HandlerError proto with full Temporal Failure
objects for transmitting errors across Nexus operation boundaries. This
preserves failure chains, stack traces, and typed failure info
(ApplicationFailure, CanceledFailure, etc.) end-to-end.
Care was taken to ensure Temporal->Temporal communication works as
expected as well as support for non-Temporal Nexus implementations (to
maintain support for external endpoints).
This ended up being more refactoring that I had originally anticipated
but I think the unification of error serialization across the codebase
and the more straightforward conversion logic was worth it.
The code was validated across variations of new and old caller and
handler workers and servers using a
[harness](https://github.com/bergundy/nexus-error-compat-tests) that was
built specifically to validate the behavior. The harness was also run
with an SDK setup that encoded failure attributes to confirm failures
are properly transmitted over the different boundaries.
Proto changes:
- Add `failure` field to DispatchNexusTaskResponse carrying a Failure
with NexusHandlerFailureInfo, deprecate `handler_error`
- Add `capabilities` field to nexus Request for feature negotiation
Failure conversion (`common/nexus/failure.go`):
- Rewrite TemporalFailureToNexusFailure and
NexusFailureToTemporalFailure with support for recursive cause chains
and stack traces
- Add special-case handling for NexusHandlerFailureInfo (serialized as
nexus.HandlerError type) and OperationError (mapped to
CanceledFailure/ApplicationFailure based on state)
Nexus SDK layer (`common/nexus/nexusrpc/`):
- Inline the failure converter from the upstream SDK with a custom
FailureConverter interface that round-trips Temporal failure metadata
through Nexus Failure objects
- Add a completion client to abstract away HTTP and provide consistent
failure conversion across all APIs
- Support `unwrap-error` metadata marker so Temporal-to-Temporal calls
can unwrap the OperationError envelope and recover the original cause
Frontend and backend handlers:
- Route responses through new failure path when caller sends
`temporal-nexus-failure-support` header
- Handle both new `failure` and deprecated `handler_error` response
types in nexus_handler.go and nexus_http_handler.go
- Update completion handling in history handler and mutable state to
produce Temporal Failures instead of Nexus HandlerErrors
- Update matching engine to forward the new failure field
Dependencies:
- Bump `nexus-rpc/sdk-go` to pre-release with StackTrace, Cause, and
OriginalFailure fields on Failure and OperationError
## Why?
Part of getting Nexus to GA in all SDKs, this change fixes a couple of
notable issues with errors in Nexus+Temporal applications:
- Nexus SDK errors did not have a way to set an error message, which
made them difficult to use in various languages
- The protocol used a custom failure format that is diverges from proxy
expectations and may result in failures not being encrypted
## Potential risks
- Metric label values for callback outcomes now use Nexus handler error
values instead of HTTP response codes. Alerts that check these outcomes
will need adjustment.
## What changed?
- Allow using `chasm.Context` when accessing `MSPointer`
- Embed `MSPointer` in the `Workflow` component to allow callbacks to
access the exposed mutable state method
- Delete `hsm_invocation` to eliminate dead code
- Don't wrap CHASM callback failures in `UnprocessableTask` errors,
failed callbacks are not supposed to bubble back to the executor anyways
## What changed?
Added a new special `MSPointer` type to the CHASM framework
## Why?
To support components invoking methods from the underlying mutable
state. Specifically, Nexus callbacks will need to get the Nexus
completion from the mutable state.
## How did you test it?
updated unit tests
