## Problem
A standby child workflow's `CloseExecutionTask` verifies its parent
recorded the completion, and past
`MaxLocalParentWorkflowVerificationDuration` also resends the parent
from the active cluster. That resend is a cross-cluster state sync plus
a possibly paginated history backfill — minutes of work — but the whole
call was bounded by the standby task's hard-coded **3s** `taskTimeout`.
Measured at the active cluster's history shard, the deadline arriving
there was `2.999s`. So the resend never completed.
## Change
- **Run the resend in the background**, bounded by
`ReplicationTaskApplyTimeout` — the setting that already bounds this
same work on the replication stream. The verify RPC returns immediately
and the standby task retries until the parent lands, so it never holds a
transfer-queue worker for the sync.
- The background context is **detached from the request** (gRPC cancels
that when the handler returns) and **rooted at the shard lifecycle**, so
the work stops with the shard.
- **One in-flight resend per parent**, tracked in a shard-level map, and
at most `history.parentWorkflowResendMaxInFlight` (8) concurrent resends
per shard. Callers retry while an earlier resend runs; without this the
test measured 5 full state fetches where 1 suffices. The cap bounds the
goroutines this path can create.
- **Lifted two client ceilings** so the deadline can actually propagate
— `admin.SyncWorkflowState` (was 10s) and `history.SyncWorkflowState`
(was 30s) now share a `DefaultStateSyncTimeout` backstop. This also
fixes the same 10s cap on the replication stream's
`ExecutableTaskImpl.SyncState`, where production's 5m setting was never
reachable either.
- Metrics:
`parent_workflow_resend_{attempts,skipped,limited,failures,latency}`.
Async failures reach no caller, so `_failures` is the alert signal;
`_limited` means the shard is shedding resends. The background goroutine
recovers panics, which would otherwise take down the process.
Also fixes the history-client codegen template, which hardcoded
`createContext` and silently ignored the timeout-tier field.
## Rollout
`history.enableAsyncParentWorkflowResend`, **default false**. Disabled =
the previous inline behavior, bounded by the caller's task deadline. Opt
in per cell.
## Testing
Unit tests cover the inline, async, and per-parent-dedup paths.
An xdc test (added in abae9cec, removed in b6eb6631) withholds the
parent's replication tasks so the child *must* pull it, asserts the
parent is absent from the standby, then stalls the active cluster's
`SyncWorkflowState` for 4 minutes:
```
--- PASS: TestChildPullsParentWhenParentReplicationIsWithheld (286.03s)
incoming-ctx-remaining: 4m59.999859334s (2.999s before this change)
sync-state-calls: 1 (5 without the per-parent guard)
dropped-parent-tasks: 9
```
During the stall, 4 verify RPCs reached the standby parent shard (t+0,
+50s, +101s, +169s) and exactly 1 `SyncWorkflowState` reached the active
cluster: the task retried and the guard turned the retries away.
4 minutes exceeds every deadline that previously bounded this path (3s /
10s / 30s) with ~1m headroom against the 5m setting, so the setting is
demonstrably what governs.
To reproduce: `git revert b6eb6631`, then
`go test -tags test_dep ./tests/xdc/ -run
TestVerifyChildCompletionParentResendSuite -timeout 30m`
## Known gaps
- Concurrency across *distinct* parents is unbounded (ordinary fan-out,
not amplification).
- When the parent is deleted on the source, the async path can't report
that back, so the child retries to the 15m discard instead of finishing
immediately. The `workflowNotFoundCache` TODO already in this file would
address it.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
Managing search attributes has been moved to operator handler.
## Why?
`tctl` which used this API is deprecated and no longer maintained.
Instead of removing the endpoints at this moment, forward to operator
handler.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
- Applied the server-owned subset of small gofix analyzer fixes.
- Exact commands that were run:
```sh
make fmt-gofix
make goimports
make fmt
git diff --check
```
- No manual or AI changes were made.
- Some fixes caused lint errors; those were reverted again.
- Changes were all reviewed by me.
## What
Add `CountWorkers` RPC to count workers matching a query filter without
retrieving full worker details.
## Why
The UI needs to display a worker count in places where listing isn't
necessary. A dedicated count API follows the existing pattern
(`CountWorkflowExecutions`, `CountSchedules`).
## How did you test it?
- [x] new unit tests: count all, count with query filter, count with no
matches, invalid query error
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
Enhance matching clients to select random partition for Nexus poll and
dispatch requests.
## Why?
Before this change all Nexus matching request were going to the root
partition only.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None
## Overview
This feature introduces a special internal Nexus endpoint called
`__temporal_system` that enables adding functionality to workflows
**without requiring new workflow commands and events**. Operations on
this endpoint are routed internally within Temporal's history service
rather than via external HTTP calls.
## Key Components
### 1. System Endpoint Infrastructure
- **Endpoint Name**: `__temporal_system` (constant in
`common/nexus/constants.go:6`)
- **Callback URL**: `temporal://system` for internal routing
- **New History Service RPCs**
(`proto/internal/temporal/server/api/historyservice/v1/service.proto:433-437`):
- `StartNexusOperation` - Starts operations on the system endpoint
- `CancelNexusOperation` - Cancels operations on the system endpoint
### 2. Operation Processor Framework
A new processor pattern (`chasm/nexus_operation_processor.go`) that
allows CHASM libraries to:
- **Validate and transform input**: Processors can validate operation
inputs and set default values
- **Determine routing**: Each processor returns a routing key that
determines which history shard handles the operation
- **Re-serialize input**: Mutated inputs can be re-serialized to persist
default values
**Routing strategies**:
- `NexusOperationRoutingKeyExecution` - Routes to the shard owning a
specific workflow execution
- `NexusOperationRoutingKeyRandom` - Routes to a random shard
### 3. CHASM Library Integration
CHASM libraries can now provide (`chasm/library.go:16-19`):
- **`NexusServices()`**: Regular Nexus service handlers (implement the
actual operation logic)
- **`NexusServiceProcessors()`**: Input processors for validation and
routing
Example from test library (`chasm/lib/tests/nexus_service.go`):
```go
// Service handler - implements the actual operation
TestOperation = nexus.NewSyncOperation("TestOperation",
func(ctx context.Context, input string, options nexus.StartOperationOptions) (string, error) {
return "Hello, " + input, nil
})
// Processor - validates input and determines routing
func (o testOperationProcessor) ProcessInput(ctx chasm.NexusOperationProcessorContext, input string)
(*chasm.NexusOperationProcessorResult, error) {
return &chasm.NexusOperationProcessorResult{
RoutingKey: chasm.NexusOperationRoutingKeyExecution{
NamespaceID: ctx.Namespace.ID().String(),
BusinessID: input, // Route based on input
},
}, nil
}
```
### 4. Execution Flow
When a workflow schedules a Nexus operation on `__temporal_system`
(`components/nexusoperations/executors.go:233-238`):
1. **Input Processing**: The processor validates input and determines
routing
2. **Internal RPC**: Instead of HTTP, calls
`HistoryClient.StartNexusOperation` with the target shard ID
3. **Handler Execution**: The history service invokes the registered
Nexus handler (`service/history/handler.go:2707-2768`)
4. **Result Handling**: Supports both sync (immediate result) and async
(operation token) responses
5. **Workflow Completion**: Results flow back through the same
completion path as external Nexus operations
### 5. Benefits
✅ **No schema changes**: Add functionality without new commands/events
in workflow history
✅ **Consistent API**: Uses existing Nexus operation semantics
(sync/async, callbacks, links)
✅ **Proper routing**: Operations are intelligently routed to the correct
shard
✅ **Input validation**: Type-safe input validation and default value
handling
✅ **Future extensibility**: Foundation for direct client invocation (not
yet implemented)
### 6. Technical Details
- **Error handling** (`components/nexusoperations/executors.go:444`):
Non-retryable service errors are properly handled and fail operations
immediately
- **Metrics**: System operations are tracked separately with
`DestinationTag` set to the endpoint name
- **Link conversion**: Helper functions convert between Nexus SDK links
and protobuf links (`common/nexus/util.go:17-46`)
- **Operation token handling**: Moved link converters to common package
for reuse (`common/nexus/link_converter.go`)
### 7. Current Limitations
- Only accessible from workflows (via `ScheduleNexusOperation` command)
- Direct client invocation not yet implemented
- Headers not supported for system endpoint operations
## Test Coverage
New test (`tests/nexus_workflow_test.go:2763-2843`) demonstrates:
- Scheduling operation on `__temporal_system` endpoint
- Synchronous operation completion
- Result propagation back to workflow
## Architecture
This architecture provides a clean, extensible way to add internal
functionality while maintaining compatibility with Temporal's existing
workflow execution model. The system endpoint acts as a bridge between
workflows and internal CHASM components, enabling:
- **Extensibility**: New operations can be added by implementing CHASM
libraries
- **Type safety**: Input validation happens before operations are routed
- **Scalability**: Intelligent routing ensures operations land on the
correct shard
- **Consistency**: Same execution model as external Nexus operations
## What changed?
Add a new dynamic config for Auto Enabling fairness and priority if we
see the relevant tasks coming in.
## Why?
Seamlessly start to transition users who start using the fields over to
the new code path.
## How did you test it?
- [X] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Due to storing this in the userdata we are using that interface a bit
more, we also need to change the initialization such that we start it
before being able to substantiate the defaultQ, this change in
initialization might have unintended side effects that I'm not currently
seeing.
## What changed?
Allow frontend->matching poll requests to retry up to their context
timeout instead of just once.
## Why?
On matching service deployments, a busy new matching node may hit its
persistence rps limit trying to acquire new task queues and be unable to
accept polls.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
* Add implementation for DescribeWorker API (to both frontend and
matching)
* Add feature flag
* Add all quotas/etc.
* Add unit tests
* Add functional tests
## Why?
Part of worker commands work.
Users should be able to directly get worker info for the worker they
need.
Corresponding API PR: https://github.com/temporalio/api/pull/622
## How did you test it?
- [X] add unit tests
- [X] add functional tests
## Potential risks
None, behind feature flag
## What changed?
+ Introduced a new Update TaskQueueConfigApi.
+ Implemented the corresponding handlers in the frontend service and the
matching service.
+ Persistence of TaskQueueConfig from the UpdateTaskQueueConfigApi.
+ Return the config response as part of the DescribeTaskQueue api.
## Why?
+ This is the first part of the UpdateTaskQueueConfig Implementation.
+ Goal is to persist the TaskQueueConfig and handle nil values in the
update request.
+ Next steps would be to attach the corresponding rate limiters with the
persisted configs.
## How did you test it?
- [x] built
- [x] run locally and tested manually
## What changed?
* Add stubs for matching for RecordWorkerHeartbeat/ListWorkers
* Add routing, based on namespace id
* Route those calls to matching from frontend
## Why?
Part of worker visibility work. Current implementation will be
"in-memory", and all worker heartbeats for the same namespace should be
on the same service instance.
## How did you test it?
- [X] built
- [X] run locally and tested manually
## What changed?
Refactor code generators:
1. Extracted templates to separate files.
2. Extracted common helpers to `codegen` package.
3. Simplified code generators.
All generated files are almost unchanged (removed leading empty line).
## Why?
Better maintainability.
## How did you test it?
- [ ] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
<!-- Describe what has changed in this PR -->
Remove license header from every file. Because it is really hard to
follow in this PR here is the summary:
1. License header is removed from all `.go` and `.proto` files
:fireworks::fireworks:🎆.
2. `LICENSE` file in the root directory has only Temporal and Uber
copyrights.
3. 5 other `LICENSE` files added to the packages which have copyrights
different from Temporal and Uber: Datadog, Xargin, "Mat Ryer, Tyler
Bunnell and contributors".
4. `license_file` flag is removed from all code generation tools.
5. `copyright_file` flag is removed from `go:generate mockgen`
directive.
6. All copyright related targets are removed from `Makefile`.
7. Updated Temporal copyright year to 2025 everywhere.
## Why?
<!-- Tell your future self why have you made these changes -->
I double checked with legal department that it is not needed to have
license header in every file. One file per repo is enough. I put all
copyrights to the root `LICENSE` file and removed header from all other
files. Also updated tools and `Makefile`.
## What changed?
The deployment workflow waits for user data to propagate to all task
queue partitions before updating its state.
## Why?
We should ensure that the desired dispatch semantics will be in effect
on all task queue partitions.
## How did you test it?
existing tests, new unit test
## What changed?
<!-- Describe what has changed in this PR -->
Linter to enforce import aliases for protobuf imports.
Enforced rules across entire codebase.
## Why?
<!-- Tell your future self why have you made these changes -->
Consistency. Relief reviewers from pointing it out.
We had 100+ violations.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Running
```
make lint-code
.bin/golangci-lint-dafd65537336fdce063c492a7ab2a68cc89f8d52 run --config=.golangci.yml | grep importas
```
comes up empty (compared to say `grep typecheck`).
Plus compiler and tests ofc.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
Me smuggling in an easter egg into the code (I didn't, I swear 🙃)
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
- Sync deployments to task queue user data.
- Various protos reorganization.
## Why?
So the data is available in matching.
## How did you test it?
not yet
## What changed?
- Added a proto message options extension to express request routing.
- Route to a random shard instead of using round robin for requests that
are shard agnostic.
## Why?
Original PR (#6575) modified DLQ request routing to use
executeWithRedirect in a way that is unpredictable, this approach
improves the codegen experience and lets the RPC author control routing.
## How did you test it?
Ran existing tests and verified generated client output.
## What changed?
<!-- Describe what has changed in this PR -->
Rename `rpcwrappers` to `genrpcwrappers`.
## Why?
<!-- Tell your future self why have you made these changes -->
For consistency with other generators.
