Add a note that the worker command control queue uses
`TASK_QUEUE_KIND_WORKER_COMMANDS`.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
Add a new `docs/architecture/circuit-breaker.md` file to explain the
circuit breaker component, and in particular how it relates to Nexus
requests.
The content was initially GenAI slop, with multiple revisions and
hand-crafted/artisanal edits to make it more concise and readable. There
are still some tell tale signs, e.g. "This key is the crux of the
isolation story." But to be honest, that seemed to help digest the
document. But I'm happy to revise as desired.
## Why?
This was something that I wanted to understand more deeply, and took a
non-zero amount of time to track down and understand some of the "how
and why" for Nexus.
## How did you test it?
NA, doc addition only.
## Potential risks
NA.
## What changed?
In `docs/architecture/history-service.md`, links pointed at
`service/history/workflow_task_handler_callbacks.go`. That file no
longer exists on `main` — its handlers were extracted into
`service/history/api/respondworkflowtaskcompleted/` (PRs #5902, #5904,
#5905, #5908, #5931, May 2024).
Repointed the dead links to the current code in
`service/history/api/respondworkflowtaskcompleted/api.go`, pinned to
commit `28dd23a` (current `main`), consistent with the doc's
pinned-commit convention:
**Line 171 — "Code entrypoints":**
| Anchor text | Old (`workflow_task_handler_callbacks.go`) | New
(`respondworkflowtaskcompleted/api.go`) |
| --- | --- | --- |
| handler | `handleWorkflowTaskCompleted` L371 | `Invoke` L110 |
| accumulates updates | `handleCommands` call L562 | `handleCommands`
call L416 |
| calls | `UpdateWorkflowExecutionWithNewAsActive` L690 |
`UpdateWorkflowExecutionWithNewAsActive` L623 |
**Line 310 — "Call sites":**
| Anchor text | Old | New |
| --- | --- | --- |
| `handleWorkflowTaskCompleted` → `Invoke` |
`workflow_task_handler_callbacks.go` L690 |
`respondworkflowtaskcompleted/api.go` L623 |
## Why?
The old blob links were stale link-rot pointing at code that has since
moved, so readers following them landed on a deleted file path on
`main`.
## How did you test it?
- [x] verified each new anchor resolves to the intended symbol at the
pinned commit (`git show origin/main:.../api.go` lines 110/416/623)
- [x] covered by existing tests (docs-only change)
## Potential risks
Docs-only change. Links are pinned to a commit SHA, so line numbers stay
stable, but they will drift from `HEAD` over time like the rest of the
doc's pinned links.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Gokhan Tekkaya
Co-authored-by: Claude Opus 4.8
Reworks CHASM scheduler's tasks to harden them against being silently
dropped, and fixes several edge-case bugs we've recently hit. Most of
these only surface in combination, so the fixes land together.
The logging here is just a few breadcrumbs — once the EventLog PR merges
I'll do a full logging/eventlogging pass, so don't read too much into
the individual log statements.
## What changed
### Generator
- `GeneratorTask` now keeps running while paused (advances the HWM,
updates `FutureActionTimes`, drops actions at the source) instead of
stopping — an improvement over V1.
- New `isManualOnly` / `isHeldOpen` helpers carve out schedules with no
automated wakeup source (empty spec, manual-only, pending backfill) so
the idle timer can't silently close them.
### Idle task
- Reworked around `isHeldOpen` plus an `.After()` deadline compare, so
sub-precision drift no longer drops tasks that should still fire.
- Fixes a non-monotonicity bug in `recentActions` (a closing workflow
could regress observed time).
### Invoker
- `ExecuteTask` captures the framework clock at read time and threads it
through `applyBackoff`, keeping `BackoffTime` consistent with
`LastProcessedTime` (no more wall-clock drift in tests/replay).
- `ProcessBufferTask` checks the catchup window before consuming a
`LimitedActions` slot.
- `recordExecuteResult` is idempotent per `RequestId`: a duplicate
`CompletedStart` / `RetryableStart` whose live entry already has a
`RunId` is dropped, without stomping `RunId`/`StartTime`/`HasCallback`
or rewinding `Attempt`/`BackoffTime`.
- `addTasks` is consolidated into a single-pass classifier; the
broadened re-arm gate fixes the stranded-retry bug.
### Backfiller
- `hasMoreAllowAllBackfills` was a misreading of the original code and
is gone — any pending backfill (not just `ALLOW_ALL`) holds a paused
schedule open while idling out.
- On completion, the backfiller revives the Generator so it can
re-evaluate idle/close eligibility.
## Tests
This PR carries only the minimal tests that go red→green with the fixes:
- **Functional** (`tests/schedule_test.go`): `TestPausedDropsCatchup`
and `TestFutureActionTimesAdvanceWhilePaused` — both fail on `main`; the
always-on Generator (advancing the HWM while paused) is what turns them
green.
- **Unit** (`chasm/lib/scheduler/`): targeted guards for each invoker
fix — idempotency, framework-clock, eligibility boundary, re-arm, and
catchup-before-limit — plus the modifications to pre-existing
idle/sentinel tests the rework requires.
Additional regression testing will live in a separate PR.
## Risks
This touches a fair amount of scheduler surface area, but the fixes are
individually validated and the diff is now mostly the rework itself.
## What changed?
Use the new dynamic config instead of leaving it as a placeholder and
update its documentation.
Also bumps the server version constant to `1.32.0`. This was supposed to
be done after the `1.31.0` release and was missed.
## Why?
Remove duplication as we migrate the code to the CHASM backed
implementation.
## Potential risks
This config is only relevant for external endpoints which are
experimental or older server versions, needs to be called out in the
release notes.
## What
Adds an architecture doc for the worker commands framework — how
commands flow from history through matching to workers, and how to add
new command types.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Sean Kane <spkane31@gmail.com>
## What changed?
Deleted `"system.enableNexus"`.
## Why?
Nexus has been GA since Dec 2024.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Re-does #9138 which was incidentally merged.
Include transient and speculative WFT events in
`GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made
request.
* Adds `transient_or_speculative_events` back to
`GetMutableStateResponse`
* Reserve `transient_workflow_task` in `HisotryCOntinuation` token
* Add validation helpers
* Add query-compare-query for transient events at request start and end
Re-implements #7732
## Why?
Fix "premature end of stream" errors when workers request history after
cache eviction w/ transient/speculative workflow tasks present. This
adds transient & speculative WFT events in `GetWorkflowExecutionHistory`
(already in `PollWorkflowTask`). Worker cache eviction w/ speculative
workflow tasks causes the expected and actual event counts to be
different. #7732 passed transient events through continuation tokens,
which could become stale during pagination. This PR implements mutable
state querying at both start and end of pagination and compares
transient event IDs to detect if WFT state changed during pagination and
return a retryable error.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Same risks from #7732
## What changed?
Include transient and speculative WFT events in
`GetWorkflowExecutionHistoryReponse` response, unless UI or CLI made
request.
* Adds `transient_or_speculative_events` back to
`GetMutableStateResponse`
* Reserve `transient_workflow_task` in `HisotryCOntinuation` token
* Add validation helpers
* Add query-compare-query for transient events at request start and end
Re-implements #7732
## Why?
Fix "premature end of stream" errors when workers request history after
cache eviction w/ transient/speculative workflow tasks present. This
adds transient & speculative WFT events in `GetWorkflowExecutionHistory`
(already in `PollWorkflowTask`). Worker cache eviction w/ speculative
workflow tasks causes the expected and actual event counts to be
different. #7732 passed transient events through continuation tokens,
which could become stale during pagination. This PR implements mutable
state querying at both start and end of pagination and compares
transient event IDs to detect if WFT state changed during pagination and
return a retryable error.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Same risks from #7732
## What changed?
- Puts an upper bound on how many concurrent backfillers can be active
in a scheduler.
## Why?
- Prevents a case of unbounded growth.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- I've set too low a limit, and on rollout, we break someone expecting a
higher number of concurrent backfillers. But I'd be pretty surprised if
anyone has a use case for 100+ backfillers..
## What changed?
- Fix issues in architecture doc diagrams
## Why?
- Fix issues in architecture doc diagrams
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Documentation Only. Update Nexus Readme to remove experimental language
and improve callback configuration recommendation.
## Why?
Nexus is Generally Available it is no longer experimental
Want to improve security posture for recommended configurations
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Roey Berman <roey@temporal.io>
Co-authored-by: Roey Berman <roey.berman@gmail.com>
## What changed?
- Adds an up-to-date design doc on the CHASM scheduler.
- Recommend using GitHub's [View
File..](b8d2cc7361/docs/architecture/schedules.md)
link to see it rendered.
## Why?
- Suppose you put every engineer who was familiar with the CHASM
scheduler on a single bus...
---------
Co-authored-by: Vladyslav Simonenko <vlad.simonenko@temporal.io>
## What changed?
Context cancelations are getting incorrectly translated to `Unavailable`
service errors, which lead to Visibility Availability Alert false
positives.
This code change adds branches to check context error types before
casting to the resulting `serviceerror` for both ElasticSearch and SQL
visibility stores.
## Why?
Separate client and service side errors.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
---------
Co-authored-by: Rodrigo Zhou <rodrigo.zhou@temporal.io>
## What changed?
Improve the error message for an Update that is aborted by a closing
workflow.
## Why?
To make it easier for users to understand what happend; and for devs to
debug it easier.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Very low as the service error _type_ is still the same (NotFound).
## What changed?
Workflow Update: update doc to reflect recent changes in #7296.
## Why?
Keep internal doc up to date.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
<!-- Describe what has changed in this PR -->
Workflow Update: fix doc on when `WorkflowUpdateAbortedErr` is returned.
## Why?
<!-- Tell your future self why have you made these changes -->
Make doc in-sync with code.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
N/A
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
N/A
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
It is it.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
Drop speculative WFT even if it had events.
## Why?
<!-- Tell your future self why have you made these changes -->
Special compatibility flag was
[added](https://github.com/temporalio/api/pull/467) to
`RespondWorkflowTaskCompletedRequest`. If this flag is set to true, then
server can drop speculative WFT with update rejections, even if it
shipped events to the worker. SDK support will be added later.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Added new unit tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
No risks. Compatibility flag is used to support backward compatibility
and don't drop speculative WFT if worker doesn't support it (old go
SDKs).
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
Yes, updated.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
Delay Update abortion with effects package.
## Why?
<!-- Tell your future self why have you made these changes -->
To address race condition case when Update is accepted and WF is
completed on the same WFT. In this scenario Update result waiters should
have similar experience as if Update was failed with error on the same
WFT, meaning that even waiters for `Accepted` stage should get Update
result (which is abort failure). Previous implementation had race
condition and those waiter might get Update failure, or "Update is
accepted" response.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Added unit tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
No risks.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
Yes, updated.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
Abort updates with failure if update was accepted but workflow
completes.
## Why?
<!-- Tell your future self why have you made these changes -->
Accepted Updates are failed with special server failure because if a
client knows that Update has been accepted, it expects any following
requests to return an Update result (or failure) but not an error. There
can be different types of Update failures coming from worker and a
client must handle them anyway. It is easier and less error-prone for a
client to handle only Update failures instead of both failures and not
obvious `NotFound` errors in case if the Workflow completes before the
Update completes.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Modified existing tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
It is some sort of breaking change for very narrow case though.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
TBD
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
Workflow Update: update docs with recent code changes in #6477 and
#6485. Also fixed some minors.
## Why?
<!-- Tell your future self why have you made these changes -->
Keep doc up to date.
## What changed?
<!-- Describe what has changed in this PR -->
Move versioning docs to the previous place as it is referenced in the
public docs.
We can move/delete this file once versioning goes to public preview.
## Why?
<!-- Tell your future self why have you made these changes -->
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Document Workflow Update internals.
## Why?
<!-- Tell your future self why have you made these changes -->
For better visibility on how Workflow Update is implemented.
---------
Co-authored-by: Stephan Behnke <stephanos@users.noreply.github.com>
…oaded
## What changed?
MatchingService supports gRPC ForceLoadTaskQueuePartition. Whenever
matching engine loads a root partition, the RPC is called for all
non-root, non-sticky partitions of the same taskQueue.
## Why?
This addresses a unique situation where a TaskQueue has acquired a
backlog, and has been unloaded because there are no workers polling the
TaskQueue. When a worker comes back online, it can take a series of
polls before it is load balanced to all partitions, forcing them to
load. This can delay non-root partitions from attempting a sync-match
with the poller. Forcing the partitions to load when the root partition
is loaded can prevent this entirely because the root partition will be
forced to load when the poll triggers a sync-match.
## How did you test it?
Start workflow with sleep. Allow worker to handler workflow initially,
then kill it. Wait for partitions to be unloaded. Start worker back up,
watch task get picked up immediately.
## Potential risks
The new gRPC method and triggering code could be implemented incorrectly
and cause a cascade of calls. Worst case scenario would be perpetual
loop from root-partition to root-partition.
Additionally, we could have an unforeseen case where we call this RPC
far more often than necessary. Metrics have been installed to help us
keep an eye on that.
## Documentation
Matching Service documentation has been expanded to cover task queue
partitions.
## Is hotfix candidate?
nope.
## What changed?
<!-- Describe what has changed in this PR -->
Added a diagram to the retries.md docs.
## Why?
<!-- Tell your future self why have you made these changes -->
To make it clearer what's going on.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Adding a short doc to give a basic overview of the gRPC retry behavior.
Note that this could/should also include the task processing retry
behavior, but I'm not familiar enough with it to write that right now.
## What changed?
<!-- Describe what has changed in this PR -->
I reorganized our docs into `docs/{admin,architecture,development}`.
## Why?
<!-- Tell your future self why have you made these changes -->
We discussed this internally, but I believe this is a better
segmentation for the different audiences looking at our docs. I plan on
adding more stuff to docs/admin like metrics and dynamic config docs.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
I'm mainly looking for broken links.
- I went through each markdown file in docs manually.
- I looked through all inspection errors in the IDE.
- I looked at all references to "docs/" or "develop/" in our code.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
