## What changed?
- Moves `activity.linkValidator` into `common/links`.
- Moves `callback.Validator` into `common/callbacks`
In addition, this PR performs some minor refactorings for consistency
and clarity.
- Moved some `links.Validator`-specific tests from
`chasm/lib/activity/validator_test.go` elsewhere.
- Introduced a `callbacks.ValidatorConfig` to bundle all of the specific
settings. (Since we'll need to wire 3+ more parameters when updating the
`callbacks.Validator` to support worker callbacks.)
> The singular package names `common/link` or `common/callback` would be
more consistent. But `common/links` already existed, there are other
pluralized ones like `common/enums` or `common/headers`. And IMHO, the
plural seems a little more applicable since the validations are only on
groupings of links or callbacks.
## Why?
The `activity.linkValidator` and `callback.Validator` types are great,
but they aren't able to be used as across other CHASM components as
easily. Moreover, `callback.Validator` uses types that are exposed from
the CHASM `callback` package, it will lead to circular dependencies in
the future. (I'm hitting this now in PRs for landing worker callbacks.)
Moving the `commonpb` protobuf validation into `common/` means we can
better separate the the distinction between validation logic and the
CHASM executions that rely on it.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
This should just be a standard refactoring. There should not be any new
validation checks enabled on codepaths where they weren't already
present. (Or in test cases, we initialize fields of
`callback.ValidatorConfig` that weren't used before.)
## What changed?
- Reorganize `chasm/lib/activity`
## Why?
- Improve navigability and codebase comprehensibility
## How did you test it?
- [x] covered by existing tests
Go 1.27 prerequisite that applies the `slicesbackward` Go fixer for
reverse iteration.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
Added Temporal Nexus attributes to spans.
## Why?
Domain attributes make Nexus traces more useful.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Makes Standalone Activity conflict updates idempotent by recording the
`requestID` when attaching callbacks or links and recognizing duplicate
request IDs. I needed to add a dedicated CHASM error for
## Why?
This prevents a successful attachment whose response was lost from
failing on retry or duplicating/replacing callbacks and links.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
### What changed
Sets gobreaker's OnStateChange hook on the outbound queue circuit
breaker pool, logging every transition.
### Why
Obtain more details for debugging curcuit breaker in production.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- Generator execution clamps its processed time range to
`LastProcessedTime`, but `computeFutureActionTimes` (used by both
`Describe` and `UpdateFutureActionTimes`) only floored its starting time
at `max(now, UpdateTime)`, ignoring the watermark.
- This let Describe/`FutureActionTimes` advertise occurrences at or
before `LastProcessedTime` that the generator had already processed or
silently skipped.
- Floors the starting time at `LastProcessedTime` as well, so every
advertised future action is guaranteed to be strictly after the
generator's high water mark.
- Reuses `common/util.MaxTime` for this floor and for the existing
`UpdateTime`-vs-`LastProcessedTime` clamp in
`GeneratorTaskHandler.Execute`, instead of hand-rolled `if X.After(Y) {
Y = X }` checks.
## Test plan
- Added
`TestGeneratorTask_FutureActionTimesRespectLastProcessedTimeWatermark`
in `chasm/lib/scheduler/generator_tasks_test.go`, which pushes
`LastProcessedTime` ahead of "now" and asserts all advertised future
times are strictly after it. Verified it fails without the fix and
passes with it.
- Built on a new scheduler-specific `chasmtest` helper
(`newSchedulerTestEngine` + `updateScheduler`/`readScheduler` in
`helper_test.go`), adapted from PR 0 of the `sch-readable` stack, rather
than the `newTestEnv` rapid harness, so the watermark mutation and task
execution cross the same transaction/read boundaries as production. Left
out the parts of that PR not applicable here (frontend-client plumbing,
the generic side-effect-task firing helper), since
`GeneratorTaskHandler` is a pure task handler that doesn't touch the
frontend client.
- `go test -tags test_dep ./chasm/lib/scheduler/...` passes.
## What changed?
1. Binds httpCaller after setting httpClient
2. Uses TransitionStarted.Possible for complete-before-start
3. Records request time for nexus operation cancel
## Why?
1. Binds httpCaller after setting httpClient
When clusterID isn't set, a non-nil httpCaller with a nil receiver gets
passed to `nexusrpc.NewHTTPClient`. Since the httpCaller is non-nil, it
will skip the check that sets the nil httpCaller to the default caller.
2. Uses TransitionStarted.Possible for complete-before-start
Matches HSM, for if a start response is lost, and a completion lands
while while the operation is in BACKING_OFF.
3. Records request time for nexus operation cancel
This just seems like it wasn't being recorded.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Instrumented local frontend, CHASM callback, and external Nexus
operation HTTP clients with the shared OpenTelemetry transport.
Legacy HSM callbacks and cross-cluster forwarding are intentionally out
of scope.
## Why?
Outbound Nexus HTTP calls need to carry trace context so callbacks and
internal frontend calls remain connected to their originating spans.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
To editorialize, the presence and difficulty of spotting these bugs
makes me want to refactor this code more, but I'm going to defer that
for now. Anyway, checked the V1 code's equivalent and also checked that
the integration-test catches the problem before applying fix.
## LLM Summary
- include an existing deferred `BUFFER_ONE` start in overlap resolution
- keep the earliest buffered occurrence and reject later arrivals
- retain catchup-window expiry for an already-deferred occurrence
## Problem
`BUFFER_ONE` permits at most one pending occurrence while an action
workflow is running. CHASM represents buffered-start lifecycle with
`Attempt`:
- `0`: newly enqueued and not processed;
- `-1`: processed but deferred while another workflow is running;
- `1+`: executing or retrying.
Before this change, `InvokerProcessBufferTask` passed only `Attempt ==
0` starts to overlap resolution. This produced the following sequence:
1. A workflow is running.
2. Occurrence A becomes due under `BUFFER_ONE`.
3. A is retained and marked deferred with `Attempt == -1`.
4. Occurrence B becomes due before the workflow closes.
5. Processing ignores A and presents only B to the shared overlap
resolver.
6. The resolver sees no occupied one-element buffer and retains B as
well.
The invoker can therefore contain both A and B, violating `BUFFER_ONE`
and potentially executing an unexpected workflow.
## Fix
Include a deferred start in the pending overlap set when its effective
overlap policy resolves to `BUFFER_ONE`:
```go
return start.Attempt == 0 ||
(start.Attempt == -1 &&
scheduler.resolveOverlapPolicy(start.GetOverlapPolicy()) ==
enumspb.SCHEDULE_OVERLAP_POLICY_BUFFER_ONE)
```
This lets the existing shared V1 `ProcessBuffer` logic see the occupied
buffer, retain the earliest occurrence, and reject later arrivals. The
effective policy is resolved because a buffered start may store
`UNSPECIFIED` and inherit the schedule current policy.
The special handling is limited to `BUFFER_ONE`; other deferred starts
should not be broadly reprocessed merely because a new occurrence
arrived.
## How this was found
The Schedule V1-to-CHASM production-history replay initially reported
5,355 V1 actions versus 4,961 CHASM actions, suggesting CHASM had
skipped 394 workflows.
Tracing the first independently different action showed the opposite:
CHASM emitted one extra occurrence. Because that workflow did not exist
in V1 history, the replay had no completion event to apply to it. It
remained running in simulated CHASM state and blocked hundreds of later
starts. The apparent 394-action deficit was a downstream alternate-chain
cascade, not 394 independent defects.
Inspecting state at the first difference revealed one running workflow,
one deferred `BUFFER_ONE` start (`Attempt == -1`), one new start
(`Attempt == 0`), and only the new start participating in overlap
resolution. A focused unit test reproduced that exact state.
After the fix, the representative converged to 5,355 actions on both
sides with identical workflow identities; only observation-time
differences remained.
## Impact
The direct product impact is one additional retained and potentially
executed workflow. This is high severity because `BUFFER_ONE` explicitly
bounds pending work, and an unexpected workflow may perform externally
visible or non-idempotent actions.
## Testing
- `TestProcessBufferTask_BufferOneKeepsExistingDeferredStart` verifies
that the first deferred occurrence occupies the buffer and the later
occurrence is rejected.
- `TestProcessBufferTask_BufferOneDropsDeferredStartPastCatchupWindow`
verifies that a deferred occurrence is still dropped when its catchup
deadline has expired.
```sh
go test -tags test_dep ./chasm/lib/scheduler \
-run "^TestProcessBufferTask_BufferOne" -count=1
```
## Summary
- `computeFutureActionTimes` computed `count = min(RemainingActions,
recentActionCount)` and passed it directly to `make()` as a slice
capacity. A negative `RemainingActions` (e.g. corrupted/invalid
persisted or operator-supplied schedule state) produced a negative
capacity and panicked with `makeslice: cap out of range` instead of
degrading gracefully.
- Clamp `count` to zero so invalid negative action counts no longer
panic scheduler reads or tasks.
- Added a shared `newTestEngineContext` helper for
CHASM-test-engine-backed scheduler tests and reused it in the existing
tests that were each hand-rolling the same registry/engine setup.
## What changed?
Validate links on callbacks consistently.
## Why?
Some links on callbacks for some requests are not being validated
properly
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
…s skipped (#10539)"
This reverts commit b11795a993.
## What changed?
Revert skip mutable state transaction if chasm nodes unchanged in
transaction.
## Why?
Needs more testing.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
Outbound nexus operation and callback requests used transports without
HTTP/2 health checks: the CHASM nexus transport provider returned the
global http.DefaultTransport, and both callback HTTPCallerProviders used
a zero-value http.Client. A silently broken connection (peer dropped it
without a GOAWAY or RST) therefore stays in the pool and stalls every
request multiplexed onto it until their individual deadlines expire.
Add common.NewHTTPTransport, which builds a transport with
http.DefaultTransport's field values and sets ReadIdleTimeout=15s /
PingTimeout=5s via http2.ConfigureTransports, and use it for:
- chasm/lib/nexusoperation (also covers components/nexusoperations,
which bridges to the same ClientProvider)
- chasm/lib/callback and components/callbacks
- RPCFactory.createLocalFrontendHTTPClient, used for worker-target nexus
calls and local callbacks
- FrontendHTTPClientCache.newClientForCluster, used for cross-cluster
callbacks
The last two already hand-rolled the same copy of DefaultTransport's
fields, so they now share it.
The helper takes the TLS config as a parameter instead of letting
callers assign it afterwards: h2 is negotiated via ALPN, so replacing
TLSClientConfig after http2.ConfigureTransports would silently fall back
to HTTP/1.1. It clones the config, since both frontend clients pass one
cached by localStoreTlsProvider and shared with gRPC clients, and
enabling h2 appends to NextProtos. Cert rotation is unaffected as it
goes through the GetClientCertificate callback, which Clone carries
over.
Note that cleartext h2c is not negotiated, so for the two frontend
clients the keepalive only applies when TLS is configured.
Also drop components/nexusoperations.ResponseSizeLimiter along with the
LimitedReadCloser it was the only user of. It had no callers: the CHASM
clientProviderFactory is what wraps the transport now, via its own
responseSizeLimiter built on http.MaxBytesReader. Being exported, the
dead code escaped the unused linter. ErrResponseBodyTooLarge moves to
executors.go, which still raises it from the payload size check.
## What changed?
- `WithRequestID` now applies to `UpdateComponent`, enabling
execution-level idempotency guarding via request ID.
- When a request ID is passed as part of an `UpdateComponent` call (via
API handler), it is persisted upon successful updateFn call. If it is
already present, instead, `UpdateComponent` fails with a
`FailedPrecondition`.
- When a request ID is not passed in, a generated ID is still created
for error tracing purposes, but it is not written to mutable state.
- On transaction close, mutable state will sweep the oldest RequestIDs
(with an `attach_time`) upon hitting the configured limit.
- This will sweep both entries below a configurable max age, as well as
past a certain hard length limit.
## Why?
- Scheduler's `UpdateSchedule` and `PatchSchedule` are implemented as
handlers that persist a signal in V1. V1 signals provide idempotency via
their request IDs. Scheduler V2 doesn't make use of signals, so instead,
it must record request IDs explicitly.
- We reuse the existing map within mutable state.
- We *must* fail with an explicit error (`FailedPrecondition`) instead
of simply returning a zero value (as Signals would on repeated
successful requests). This is because `UpdateComponent` can apply to API
models that include response values (which we don't record, therefore,
we can't return on subsequent calls).
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Fred Tzeng <fred.tzeng@temporal.io>
## What changed?
Updated the activity code to use a CHASM context API that was added
after the original code was written.
## Why?
Prevent this pattern from being copied to other libraries.
## What changed?
Add a `__temporal_system_payload = "true"` metadata field to outer
`Payload` returned by system nexus operations when the operations
protobuf result embeds a nested `commonpb.Payload(s)` field.
## Why?
System Nexus operation results are serialized into a single outer
`Payload`. When the underlying proto message itself contains a nested
`Payload(s)` field, that nested payload's bytes get hidden inside the
outer payload's opaque `Data`. Downstream consumers (payload
codec/visitor logic) have no way to know they need to unwrap the outer
payload before they can reach and process what's inside.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
NA
## Summary
Forward `VersioningOverride` from schedule actions into workflow start
requests for both the legacy and CHASM schedulers. Schedule validation
now rejects structurally invalid overrides before they are persisted.
## Root cause and impact
Both schedulers manually built `StartWorkflowExecutionRequest` and
omitted `VersioningOverride`, so schedules accepted and retained the
setting but started workflows using ordinary version routing.
## What changed?
- Fence Backfiller tasks with a persisted task sequence value instead of
comparing task execution time with the backfill HWM.
- Accept unnumbered tasks created by an older binary and restore task
numbering when a new binary executes one.
- Add lifecycle coverage and CHASM test support for firing due persisted
pure tasks.
## Why?
`LastProcessedTime` tracks progress through the requested schedule
range, while a task's `ScheduledTime` controls when that task runs.
Comparing the two can either keep an already-processed historical task
valid or reject a forward-dated task before it runs.
## How did you test it?
- [x] covered by existing tests
- [x] added new unit test(s)
## Potential risks
The immediate task `N` created with a Backfiller is scheduled and
executed in one transaction. Mixed-version risk begins with delayed task
`N+1`.
If `N+1` is handled by a 160 binary:
- For a forward-dated backfill, the old HWM validator can remove `N+1`
before execution, leaving the Backfiller alive with no task to make
further progress.
- For a historical backfill, the old validator can accept `N+1` again
after it has already advanced the HWM, so duplicate execution remains
possible during rollout or rollback.
## What changed?
- Increment the CHASM scheduler conflict token when pause-on-failure
changes the persisted paused state and notes.
- Add a regression test covering a stale Describe token used by Update
after the automatic pause commits.
## Why?
A token-protected Update could previously replace the entire schedule
using a token captured before pause-on-failure, silently clearing the
automatic pause and its explanatory notes. Invalidating the token keeps
optimistic concurrency behavior aligned with the fields Update replaces
and with the V1 scheduler.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
```
go test -tags test_dep ./chasm/lib/scheduler -run 'Test(HandleNexusCompletion_PauseOnFailure|PauseOnFailureInvalidatesConflictToken)$' -count=1
```
## Potential risks
Token-protected updates already in flight when pause-on-failure commits
will now fail with a conflict-token mismatch and must be retried.
Updates that intentionally omit the conflict token remain unconditional.
## Migration: give same-time pending starts unique identities
### Description
`convertBufferedStartsLegacyToCHASM` derives each migrated start's
`RequestId` and `WorkflowId` as pure functions of its timestamps plus
batch-constant inputs (namespace, schedule, conflict token, base
workflow id). Two pending starts with the same `NominalTime` and
`ActualTime` therefore get identical identities. The loop index was
available but unused.
### User experience
Two or more pending actions can legitimately share a nominal/actual time
(e.g. under `ALLOW_ALL` overlap, or overlapping backfill/trigger
requests). When such a schedule is migrated V1→V2, the collision
silently loses actions: the shared `WorkflowId` means only one workflow
starts (`REJECT_DUPLICATE`), and the shared `RequestId` makes completion
routing and dedup treat the pair as one — the invoker keys
`CompletedStarts`/`FailedStarts`/retries by request ID, so the second
start is indistinguishable from the first. No error is surfaced.
### How it occurs
`GenerateRequestID` = `sched-<backfillID>-<sha1(ns, sched, token,
nominalMs, actualMs)>` and `GenerateWorkflowID` =
`<base>-<nominalSecond>`. For a conversion batch everything except the
timestamps is constant, so equal timestamps → identical IDs.
### How it's fixed
Disambiguate with the per-action loop index, at each identity's natural
seam:
- **Request ID** — the index rides in the existing `backfillID` tag
(`"migrated-0"`, `"migrated-1"`, …). That tag is the literal prefix of
`sched-<tag>-<uuid>`, so the IDs differ without touching the hash;
`convertRunningWorkflowsToBufferedStarts` already uses the same
mechanism with the run ID. `GenerateRequestID` itself is unchanged, so
no other caller's IDs move.
- **Workflow ID** — suffixed only past the first start (`i > 0`). Unlike
the request ID this one is user-visible, and dedup against an action the
V1 scheduler had already started depends on it matching, so the common
case of a single pending action keeps the ID that both V1 and native V2
would give it.
Both apply only to regenerated (empty) IDs; identities carried over from
V1 are preserved.
### Test
- `TestSameTimePendingStartsReceiveUniqueIdentities` — fails before,
passes after. Also pins the first start's workflow ID to the undecorated
`GenerateWorkflowID` output.
- `TestMigratedStartsPreserveExistingIdentities` — V1-supplied
identities are neither regenerated nor suffixed.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
Cap the failure kept in LastFailureDetails when an attempt will be
retried, for parity with MutableStateImpl.truncateRetryableActivityFailure
of workflow activities.
## Why?
This is for parity between workflow activity and CHASM based activity
(currently standalone activity) implementation. The workflow activity
limits the retryable failure data kept in the mutable state, so similar
behavior is added for standalone activity. Non-retryable failures are
left intact since they are reported back to the caller, again similar to
workflow activity. It uses the same dynamic config that is used in
workflow activities size limit.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Fred Tzeng <41805201+fretz12@users.noreply.github.com>
## What changed?
Include LastDeploymentVersion in ActivityExecutionInfo returned by
buildActivityExecutionInfo.
## Why?
TransitionStarted persists the poller’s deployment version for the
started activity,
but buildActivityExecutionInfo never assigns the LastDeploymentVersion
field.
Thus Describe always reports nil even when the state contains the value.
Although
the LastDeploymentVersion field is currently not used/set for standalone
activity,
but can be/is only set for workflow activity, this change fixes a
potential bug in
the future when deployment version gets set for standalone activity too.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
- Do not reset heartbeats by default
- Honor `reset_heartbeat` flag
## Why?
- Parity with WFA
- This product behavior makes sense: a user with a long-running activity
using exponential backoff on attempt 10 may wish to reset the attempt
counter in order that the next retry backoff is short, and yet preserve
their checkpointed progress.
## How did you test it?
- [x] modified existing functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes activity reset semantics and persisted state for heartbeat
checkpoints across immediate and deferred reset paths; behavior is
well-covered by tests but affects long-running activity retry/reset
workflows.
>
> **Overview**
> Activity reset now **rewinds the attempt counter** but **keeps
persisted heartbeat checkpoint details by default**, matching
workflow-activity behavior. Clearing heartbeats is **opt-in** via
`reset_heartbeat` / `ResetHeartbeat` on reset APIs.
>
> CHASM activity state adds `reset_should_clear_heartbeat` for resets
requested while a worker is still running; clearing runs when the
attempt yields (same deferred pattern as `restore_original_options`).
Immediate reset paths (`reset`, `resetKeepPaused`) and
cancel-on-reset-request clear that deferred flag only when the flag is
set.
>
> Standalone activity reset forwarding no longer forces `ResetHeartbeat:
true`; it passes the client request. Model/events add
`ResetClearingHeartbeat`; parity and functional tests cover keep vs
clear for scheduled and started activities.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
09fb7a0bd1. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
- Apply InitialPatch pause and unpause state before generator and
backfiller work is armed.
- Add CHASM test-engine coverage for initial pause and unpause creation.
- Give the CHASM test engine a default local namespace entry.
## Why?
CHASM-backed schedules previously ignored InitialPatch.Pause and
InitialPatch.Unpause, unlike workflow-backed schedules.
## What changed?
Removed the UpdateOptionsAllowedAfterDeferredRestoreSupersededByCancel
functional test and added focused unit coverage verifying that
TransitionCancelRequested clears ResetRestoreOptions.
Also corrected the nearby comment claiming that UpdateOptions was
permitted in RESET_REQUESTED when no restore was pending.
## Why?
PR #11394 disallowed UpdateOptions in CANCEL_REQUESTED and
RESET_REQUESTED. PR #11358 later added a conflicting test expecting an
update to succeed after cancel superseded a deferred restore.
The test was removed instead of changed to expect an error because that
result is already covered by UpdateWhileCancelRequestedFails. It also
could not verify whether the deferred restore flag was cleared: both a
cleared and stale flag produce the same FailedPrecondition while the
activity is CANCEL_REQUESTED.
## How did you test it?
- [X] built
- [] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
* Deduplicate cancel retries before terminal-state validation
* Added tests covering terminal-state retries and run-qualified retries
after operation ID reuse.
## Why?
A delayed retry can arrive after the original activity has closed.
Deduplication must still recognize that retry, and callers must pin
cancellation to a run_id so activity ID reuse cannot
redirect the request to a replacement execution.
See also: https://github.com/temporalio/temporal/pull/11344
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Moved activity failure retry classification into a shared helper used by
standalone activities, workflow activities, and workflow retries.
Standalone activities now use the same retry classification as workflow
activities for failures reported through
`RespondActivityTaskFailedById`. This includes retryable
`ServerFailure`s, worker-reported start-to-close and heartbeat timeouts,
and otherwise unrecognized failure variants. Schedule-to-start and
schedule-to-close timeouts remain non-retryable.
## Why?
Standalone activities previously treated non-application failures as
non-retryable. Sharing the existing workflow retry classifier keeps
retry behavior consistent across SAA, WFA, and workflow retries.
---------
Co-authored-by: Fred Tzeng <fred.tzeng@temporal.io>
## What changed?
Added user_metadata summary and details size validation to
StartActivityExecution, using the existing namespace-specific limits and
matching standalone Nexus operation behavior.
## Why?
Standalone activities previously persisted user_metadata without
enforcing its configured size limits. This made the limits inconsistent
across top-level executions and allowed oversized metadata to reach
persistence.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Reject SAA unpause requests unless an unpause transition is possible.
## Why?
A no-op unpause could be recorded and later retried, obscuring invalid
state transitions.
Context: follow-up to [standalone activity operator-request
idempotency](https://github.com/temporalio/temporal/commit/d8b84b8aac).
## How did you test it?
- [x] covered by existing tests
- [x] added new functional test(s)
## Potential risks
- Clients that unpause non-paused activities now receive
`FailedPrecondition` instead of a successful no-op.
- This also affects clients unpausing activities that *were*
legitimately paused: the request-ID dedup that makes the rejection
retry-safe only works when the client supplies `RequestId`. When it's
omitted, the server mints a fresh UUID per attempt, so a retry never
matches `LastUnpauseRequestId`. Concretely: a client unpauses a `PAUSED`
activity, the mutation commits, the RPC times out on the way back
(activity is now `SCHEDULED`), the client retries — on `main` that retry
was a benign no-op, now it returns `FailedPrecondition`. Worth
confirming the SDK/CLI populate `RequestId` before this ships.
- The same `UnpauseActivityExecution` RPC against a workflow-owned
activity still silently no-ops when the activity isn't paused
(`service/history/api/unpauseactivity/api.go:120`), so behavior now
diverges by activity kind (standalone vs. workflow-owned) for the same
RPC.
- Separately, `UpdateActivityExecutionOptions` now also rejects with
`FailedPrecondition` while a deferred `Reset(RestoreOriginalOptions)` is
pending (previously it would silently apply and then be clobbered when
the deferred restore landed). Callers that previously succeeded here now
fail while the restore is pending.
---------
Co-authored-by: Dan Davison <dandavison7@gmail.com>
## What changed?
Do not permit `UpdateOptions` in `CANCEL_REQUESTED` or `RESET_REQUESTED`
## Why?
- Hard to define semantics: if `UpdateOptions` lands after
`Reset(restore_original_options)` then should the update be silently
overridden when honoring the reset on attempt end?
- We opt to simplify the combinatorial possibilities now and retain the
possibility of evolving the API to allow it in the future.
- It is unclear whether these transitions should be allowed.
## How did you test it?
- [x] covered by existing tests
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes public API semantics for activities with pending cancel/reset;
clients that relied on mid-flight option updates will now get
FailedPrecondition.
>
> **Overview**
> **`UpdateActivityExecutionOptions` is no longer allowed** while an
activity is in **`CANCEL_REQUESTED`** or **`RESET_REQUESTED`**. Those
statuses are now treated like other non-updatable states and return
**`FailedPrecondition`** with the same message pattern as terminal
statuses.
>
> This replaces the prior behavior where options could still be updated
on a running attempt with a pending cancel, and where updates during
**`RESET_REQUESTED`** could bump the attempt stamp and re-issue timeout
tasks. Standalone activity tests were flipped to expect refusal and to
assert timeouts and run state stay unchanged.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
b4c607a9a8. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
`Activity.HandleFailed` now treats a `RespondActivityTaskFailed` request
with an omitted `Failure` as retryable, matching WFA behavior.
Previously a nil failure was treated as non-retryable and the SAA closed
as `FAILED`.
## Why?
This achieves parity for SAA with workflow activities.
## How did you test it?
- [x] covered by existing tests
- [x] added new functional test(s)
## Potential risks
This is a behavioral change, but was a bug in the original
implementation
## What changed?
- Emit schedule-to-start latency metric when SAA starts
- The metric distribution (which is per-task queue) will now contain
data points from both SAA and WFA. This is reasonable because there's
nothing about SAA that implies that its time in matching backlog should
have a different distribution.
## Why?
- Required metric; WFA parity
## How did you test it?
- [X] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Adds observability only on the activity-start path after a successful
transition; no auth, persistence, or matching behavior changes. Shared
metric name with workflow tasks is filtered by operation/tags in tests.
>
> **Overview**
> **Standalone activities (SAA)** now emit
`task_schedule_to_start_latency` when History accepts an activity task
start in `HandleStarted`, matching workflow-embedded activity (WFA)
behavior and filling a required metrics gap.
>
> Latency is **started time minus attempt dispatch time** (via
`dispatchTimeForAttempt`), not raw schedule time—so retries measure
backlog from the current attempt’s dispatch, excluding prior attempts
and backoff.
>
> Samples use the same per-task-queue partition scope as WFA
(`HistoryRecordActivityTaskStartedScope`, activity task type,
`MetricsBreakdownByTaskQueue` → real task queue name vs `__omitted__`).
Idempotent `RecordActivityTaskStarted` replays do not record again.
>
> Unit tests in `activity_test` assert sample count and latency for
first start and retry; functional parity tests cover SAA vs WFA for
first attempt and retry across task-queue breakdown settings.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
e702b65ccd. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Added an explicit precondition check for standalone activity
cancellation responses. RespondActivityTaskCanceled now requires the
activity to be in CANCEL_REQUESTED and returns the established
ErrActivityTaskNotCancelRequested error otherwise.
## Why?
Standalone activities previously exposed an internal invalid-transition
error when a worker reported cancellation without a prior cancellation
request. Workflow activities return a stable InvalidArgument API error
for the same scenario.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
See API change https://github.com/temporalio/api/pull/846
## What changed?
- Drop `reset_attempts` and `reset_heartbeat` from
`UnpauseActivityExecution`
## Why?
- We have so far been unable to assign desirable and consistent
semantics to them during implementation: for example if
`Unpause[resetAttempts]` is received during retry backoff it is unclear
whether to honor the remaining delay time, because this is how Unpause
usually behaves, or dispatch immediately, because this is how Reset
behaves.
- No known user demand
- They are confusing: they mix `Unpause` and `Reset` functionality in a
confusing way
- They can be added later
## How did you test it?
- [x] covered by existing tests
## Breaking changes
- This API has always been rejected by the server. When server starts to
accept it, an old client could submit these options and they would be
ignored. Operator API is not GA.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes activity unpause scheduling semantics for the execution API
and drops reset-on-unpause behavior that was only partially implemented;
low user impact if the API was not GA and had no known callers.
>
> **Overview**
> Aligns the server with the **UnpauseActivityExecution** API change:
**`reset_attempts`** and **`reset_heartbeat`** are no longer part of
unpause for standalone (CHASM) activities.
>
> **CHASM activity unpause** no longer resets attempt count, retry
interval, or heartbeat state on unpause, and always considers the
pending retry backoff when scheduling dispatch (the branch that skipped
that when `reset_attempts` was set is removed). Workflow-embedded
unpause forwarding via **`UnpauseActivityExecution`** no longer passes
those fields to the legacy **`UnpauseActivity`** history call (jitter
and identity only).
>
> **`go.temporal.io/api`** is bumped to the revision that removes the
fields from **`UnpauseActivityExecutionRequest`**.
>
> **Tests** are updated so unpause helpers no longer take a reset flag;
reset-on-unpause coverage stays on legacy **`UnpauseActivity`** only
(execution API skips that case). Standalone tests for
**`UnpauseWithResetAttempts`** and **`UnpauseWithResetHeartbeat`** on
**`UnpauseActivityExecution`** are removed.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
7a41d507ca. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Exposed RetryState on standalone activity execution outcomes and
persisted it in activity state. Retry evaluation now records terminal
reasons including retry policy not set, cancellation requested,
non-retryable failure, maximum attempts reached, and timeout.
## Why?
Standalone activities previously collapsed retry decisions into a
boolean, preventing callers from distinguishing why an activity stopped
retrying. This brings standalone activity behavior and observability
into parity with workflow activities.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
During a rolling upgrade, activities closed by an older server may
return RETRY_STATE_UNSPECIFIED. Existing closed activities also remain
unspecified because retry state was not previously persisted. Older
clients safely ignore the new protobuf field.
---------
Co-authored-by: Dan Davison <dandavison7@gmail.com>
## What changed?
- Persist payload sent with attempt failure as last heartbeat details
- Emit metrics associated with that codepath for WFA parity
## Why?
- The first is a relatively bad bug: an activity attempt should be able
to have the latest checkpoint data sent in and persisted with a
retryable failure, but SAA was not persisting it
- SAA vs WFA metrics parity
## How did you test it?
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes activity failure and retry persistence where workers rely on
checkpoint data; scope is narrow with new SAA/WFA parity tests, but
incorrect handling could affect retries or observability.
>
> **Overview**
> Standalone activities (SAA) now **persist `LastHeartbeatDetails` from
`RespondActivityTaskFailed`** before deciding whether to retry or fail
terminally. Previously that checkpoint lived only on the terminal
`TransitionFailed` path, so **retryable failures dropped the worker’s
final progress payload**.
>
> Heartbeat handling on failure now mirrors a normal heartbeat: update
last-heartbeat state (details, recorded time, count) and record metrics
via a shared **`emitHeartbeatMetrics`** helper used by
**`RecordHeartbeat`** as well.
>
> Trace drivers and parity tests gain **`HasHeartbeatDetails`** on
failed-respond events, plus coverage that WFA and SAA expose the same
stored heartbeat details (including terminal SAA failures).
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
a013d1627f. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Updated the CHASM scheduler's DescribeSchedule response to resolve the
effective catchup window using the namespace's scheduler tweakables.
DescribeSchedule now reports:
- nil, zero, or negative values as `DefaultCatchupWindow`
- positive values below the minimum as `MinCatchupWindow`
- values at or above the minimum unchanged
The resolution is applied to a cloned schedule, leaving persisted
scheduler state unchanged.
## Why?
DescribeSchedule previously used a hard-coded one-year default and did
not consistently report the same effective catchup window used during
schedule processing.
Using the existing catchup-window resolver keeps DescribeSchedule
consistent with runtime behavior and namespace-specific dynamic
configuration.
## V1 and V2 behavior
V1 and V2 currently differ when the configured catchup window is zero or
negative:
- V1 treats zero or negative values as below the minimum and resolves
them to `MinCatchupWindow`.
- V2 treats zero or negative values as unset and resolves them to
`DefaultCatchupWindow`.
- Both implementations clamp positive values below the minimum to
`MinCatchupWindow`.
This PR changes only the CHASM/V2 DescribeSchedule path and does not
modify V1 behavior.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
Commands run:
- `go test -tags test_dep ./chasm/lib/scheduler`
- `go test -tags test_dep ./tests -run
'TestScheduleCHASM/TestDescribeCatchupWindowAfterCreateAndUpdate'
-count=1`
- `make lint-code`
`testBasics` already performs create→describe and update→describe,
including the unset/default case, but it is shared by V1 and V2. Since
zero/negative semantics currently differ between the implementations,
adding those cases there would break V1 coverage. A focused CHASM-only
functional test was added instead.
The CHASM-only test covers create with an unset catchup window, followed
by updates with zero, negative, positive-below-minimum, and
above-minimum values.
Local testing:
1. http://localhost:3000/ + UI side override to allow <10 secs
2. Create schedule with catchup window 0 secs. Load it and verify it
shows as 10 secs.
3. DC changes to switch to CHASM. Repeat(2) to verify its set as 1 year.
## Potential risks
DescribeSchedule now returns the effective catchup window rather than
the raw persisted value for non-positive and below-minimum values. This
matches the value used by CHASM schedule processing.
### V1 and V2 migration
This PR changes only the CHASM DescribeSchedule response. It does not
normalize the persisted schedule policy or change migration payloads, so
the existing migration behavior remains:
- **V1 → V2:** V1 eagerly normalizes zero or negative values to an
explicit `MinCatchupWindow`. Migration copies that positive duration, so
V2 continues using the minimum.
- **V2 → V1:** V2 persists the original zero or negative value and
treats it as unset/default at runtime. Migration currently copies that
raw value. V1 then resolves it to `MinCatchupWindow`, potentially
changing the effective behavior from the V2 default to the V1 minimum.
- **Unset:** The target implementation resolves the unset value using
its own default. Behavior could change if the source and target defaults
differ.
- **Positive below minimum:** The target implementation applies its own
minimum. Behavior could change if the source and target minimums differ.
- **At or above minimum:** The explicit value is preserved across
migration.
Resolving or persisting the effective catchup window during migration is
outside the scope of this PR.
## What changed?
- Port SAA/WFA `CompleteById` tests to declarative framework
## Why?
- We will gain additional test assertions when the declarative tests are
wired up to the spec (model)
- Easier to read and reason about the tests: 225 LOC reduction
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Changes are limited to test vocabulary, test drivers, and parity
tests; no production activity completion logic is modified.
>
> **Overview**
> Extends the activity parity **trace model** with `CompleteByID`
(`RespondCompletedByIDType`) and teaches the workflow-activity and
standalone-activity drivers to issue `RespondActivityTaskCompletedById`
when that event appears in a trace.
>
> Replaces two long, hand-written parity tests
(`TestCompleteByID_BeforeAnyWorkerStarts` and
`TestCompleteByID_WhilePaused`) with a single table-driven
`TestCompleteByID` that drives the same scenarios via traces
(`CompleteByID` alone, or `Pause` then `CompleteByID`) for both WFA and
SAA. Standalone activity still asserts `LastStartedTime` is set after
force-complete without a worker poll.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
79ecf3099e16db51d593b71237164770064116d8. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Enable standalone activity start delay by default. Remove unnecessary
test overrides.
## Why?
Start delay to be enabled by default for standalone activities GA
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
In the CHASM (V2) scheduler, treat a non-positive (zero or negative)
schedule
catchup window the same as unset: return the default catchup window
instead of
clamping up to the minimum. Only a positive value below the minimum is
clamped up.
Change is in `chasm/lib/scheduler/spec_processor.go` (`catchupWindow`).
## Why?
Previously only a `nil` catchup window fell back to the default; a zero
or
negative value slipped through to `max(cw, MinCatchupWindow)` and was
silently
clamped up to the minimum. A non-positive value is effectively
"unset/invalid"
and should resolve to the default.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
Added a component level functional test (Generator) for now.
Ideally, we'd add a server-level test that creates schedules with
different catchup window values and verifies the result via
DescribeSchedule. However, that doesn't currently validate the intended
behavior because the describe logic overwrites the catchup window in
some cases (see:
https://github.com/temporalio/temporal/blob/main/chasm/lib/scheduler/scheduler.go#L697-L699).
Will work on this fix next as it needs more plumbing and also add this
specific test in the next PR.
## Potential risks
Behavior change for any schedule that explicitly sets a catchup window
<= 0:
it now resolves to DefaultCatchupWindow instead of MinCatchupWindow.
## What changed?
SAA terminal timeout failures now chain the previous attempt’s failure
as their cause.
## Why?
When retries ended in a timeout, SDK users could see only the timeout
and not the application failure that drove the retries. Preserving the
cause exposes the useful underlying error and matches Workflow Activity
behavior.
## How did you test it?
- [x] added new unit test(s)
- [x] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes activity timeout failure protobuf shape and terminal outcome
logic on a user-visible error path; scope is limited to SAA timeout
handling with strong test coverage.
>
> **Overview**
> **Standalone activity (SAA) terminal timeouts now set `Failure.Cause`
to the last attempt’s stored failure** (typically the application error
that triggered retries), so clients see the underlying error via
`TimeoutError.Unwrap()` instead of only the timeout wrapper—aligned with
workflow-embedded activities.
>
> `TransitionTimedOut` reads `priorAttemptFailure` from
`LastFailureDetails` **before** recording the current timeout, then
passes it into schedule-to-start/close outcome failures and sets `Cause`
on start-to-close and heartbeat terminal failures. When a per-attempt
timeout exhausts the schedule-to-close retry window
(`RETRY_STATE_TIMEOUT`), the final schedule-to-close outcome still
chains that prior failure even though the per-attempt timeout was
written to attempt state first.
>
> Coverage adds a state-machine unit test for the retry-window path and
SAA/WFA parity tests (including a check that **retryable** timeouts do
not chain causes on `LastFailure`). Test helpers use a stable
`TestFailure` application failure type for assertions.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
89ae0251ee. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
This PR refactors the validation logic used for SANO from a collection
of loose functions, into methods on an unexported `validator` type.
(Similar to how `chasm/lib/callback/validator.go` is structured.)
The same checks have all been preserved, although I did fix up one error
message string to be consistent with others.
## Why?
The motivation for this refactoring is to make it easier to land the
"worker callbacks" feature. That will require expanding the validation
checks, and bundling all the dependent parameters on the `type validator
struct` is cleaner than needing to wire through a new parameter at every
callsite.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
I am relying on GenAI in its assertion that the existing validation
checks are essentially identical with these changes. Worst case
scenario, this alerts which types of SANO requests are accepted or
rejected.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
- Deduplicate cancel and pause retries before terminal-state validation.
- Added tests covering terminal-state deduplication and run-qualified
retries after activity ID reuse.
## Why?
A delayed mutation retry can arrive after the original activity has
closed. Deduplication must still recognize that retry, and callers must
pin mutations to a run_id so activity ID reuse cannot redirect the
request to a replacement execution.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## What changed?
- During V2->V1 migration (rollback), ALLOW_ALL starts are now excluded
from the `RunningWorkflows` array.
## Why?
- `RunningWorkflows` in V1 will block a schedule if it isn't set to an
`ALLOW_ALL` policy itself. This ensures that the workflows are copied to
`RecentActions`, but not eligible for `WatchWorkflow` to block on.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Added request-ID-based idempotency for standalone activity unpause,
reset, and update-options operations. Successful request IDs are
persisted in activity state and duplicate requests are handled as
no-ops.
## Why?
These APIs can be retried after timeouts or transient failures.
Persisting the latest successful request ID prevents duplicate
mutations, including delayed unpause retries undoing a later pause.
Workflow-backed activity support will follow separately.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
---------
Co-authored-by: Dan Davison <dandavison7@gmail.com>
## What changed
- Adds SAA payload-size and heartbeat-count metrics.
- Bring SAA metric tags into parity with WFA
- Bring SAA timeout metric behavior into parity with WFA by omitting
start-to-close latency when an attempt times out.
## Why
- Correctness / parity with de-facto correct WFA
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes observable metrics and timeout latency recording in the
activity execution path; parity tests reduce regression risk but
dashboards or alerts keyed on old SAA timeout latency may shift.
>
> **Overview**
> Standalone activity (SAA) metrics are brought in line with
workflow-embedded activity (WFA) behavior and coverage.
>
> **Handler split:** Metrics use a **base** handler (namespace +
`operation` only) for payload-size and heartbeat counters, and an
**enriched** handler (per-activity tags: activity type, task queue,
workflow type, etc.) for success/fail/latency counters. Complete and
fail responses record **`activity_payload_size`** on the base handler
from result/failure serialized size; heartbeats record
**`activity_heartbeat_count`** (with `has_details`) and payload size
when details are present. New standalone activities emit payload size
for schedule input on **`RecordActivityTaskStarted`**.
>
> **Timeout parity:** Terminal and retryable attempt timeouts no longer
emit **`activity_start_to_close_latency`** on the timed-out path
(matching WFA). Attempt-level timeout counters are unchanged.
>
> **Verification:** Integration **`TestWFASAAMetricsParity`** drives
shared event traces against WFA and SAA and compares metric names, tags,
and values; test drivers gain RPCs for heartbeat, complete, terminate,
unpause, and update-options.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
df982df381. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
A small documentation update to correct a slightly misleading comment
## Why?
The current comment gives a slightly misleading view that transient
termination / cancellation errors will just result in it being dropped,
whereas the history-client should be provided wrapped with an internal
retrier, so in practice any failure will get a couple of retries before
being dequeued.
As I understand it, this is arguably different than the v1 behaviour
(local activity, retries for quite a while), however, it's not super
clear to me that it warrants a p1/bugfix.
I think there's an interesting question about whether or not this
architectural pattern of a CHASM task doing multiple things in a single
handler is a good idea; I'm quite tempted to say that each individual
RPC should probably be its own task. However, for now imho this is ok
as-is.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Backfill: capacity stall must not skip part of the requested range
### Description
`processBackfill` uses `Attempt > 0` to decide whether to resume from
the high-water mark. But `Attempt` is a buffer-full back-off counter,
not a progress marker: `Execute` increments `Attempt` via a `defer` even
on the buffer-full early-return where no work happens, while
`LastProcessedTime` stays at its creation-time default (set to "now" in
`addBackfiller`).
### User experience
A backfill request (e.g. "re-run the last N hours") returns success, but
if the invoker buffer was full on the backfiller's first task execution,
some or all of the requested actions silently never run. Timing/load
dependent, no error surfaced — a silent data-completeness bug.
### How it occurs
1. `addBackfiller` creates the backfiller with `Attempt=0` and
`LastProcessedTime=now`.
2. The first task sees `limit <= 0` (buffer full), takes the early
return — no range processed, `LastProcessedTime` untouched — but the
`defer` bumps `Attempt` to 1.
3. On retry, `processBackfill` sees `Attempt > 0` and resumes from
`LastProcessedTime` (= creation "now"). For a past range that is
at/after the range end, so the whole range is skipped and the backfiller
completes having produced nothing.
### How it's fixed
Resume from the high-water mark only when it reflects genuine progress
strictly within the requested range (`start < HWM < end`); otherwise
start from the range start. A capacity-only stall (HWM at its
creation-time default) is no longer mistaken for durable progress.
Single-attempt and legitimate mid-range resume behavior are unchanged.
### Test
`TestBackfillCapacityStallDoesNotSkipRange` — fails before the fix,
passes after.
### Risks:
This is actually not backwards compatible, strictly speaking, but I
think the window during which the lastUpdate has not been recorded is
going to be sub-second for any backfill that's unfortunate enough to be
started in a mixed-brain scenario, so my feeling is that the risk is
probably acceptable. see
https://github.com/temporalio/temporal/pull/11162/changes#r3636260022
If a backfill were to be created at precisely the right time as to use
lastUpdatedTime == now, but then use the new codepath (because it hadn't
had a chance to make progress yet) it might skip to the end. But as far
as I am aware this is only a gap between creation and the CHASM task
completion. As soon as a backfill has made any progress, it should be
fine afaik.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
`RespondActivityTaskCompletedById` can now force-complete an SAA from
the `Scheduled` state.
## Why?
Workflow activities allow force completing an activity before any worker
starts it.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [X] added new functional test(s)
- Added a new file `tests/activity_parity_test.go` to hold all tests
related to parity of workflow activities and standalone activities.
-
## Potential risks
NA
