## Problem
A standby child workflow's `CloseExecutionTask` verifies its parent
recorded the completion, and past
`MaxLocalParentWorkflowVerificationDuration` also resends the parent
from the active cluster. That resend is a cross-cluster state sync plus
a possibly paginated history backfill — minutes of work — but the whole
call was bounded by the standby task's hard-coded **3s** `taskTimeout`.
Measured at the active cluster's history shard, the deadline arriving
there was `2.999s`. So the resend never completed.
## Change
- **Run the resend in the background**, bounded by
`ReplicationTaskApplyTimeout` — the setting that already bounds this
same work on the replication stream. The verify RPC returns immediately
and the standby task retries until the parent lands, so it never holds a
transfer-queue worker for the sync.
- The background context is **detached from the request** (gRPC cancels
that when the handler returns) and **rooted at the shard lifecycle**, so
the work stops with the shard.
- **One in-flight resend per parent**, tracked in a shard-level map, and
at most `history.parentWorkflowResendMaxInFlight` (8) concurrent resends
per shard. Callers retry while an earlier resend runs; without this the
test measured 5 full state fetches where 1 suffices. The cap bounds the
goroutines this path can create.
- **Lifted two client ceilings** so the deadline can actually propagate
— `admin.SyncWorkflowState` (was 10s) and `history.SyncWorkflowState`
(was 30s) now share a `DefaultStateSyncTimeout` backstop. This also
fixes the same 10s cap on the replication stream's
`ExecutableTaskImpl.SyncState`, where production's 5m setting was never
reachable either.
- Metrics:
`parent_workflow_resend_{attempts,skipped,limited,failures,latency}`.
Async failures reach no caller, so `_failures` is the alert signal;
`_limited` means the shard is shedding resends. The background goroutine
recovers panics, which would otherwise take down the process.
Also fixes the history-client codegen template, which hardcoded
`createContext` and silently ignored the timeout-tier field.
## Rollout
`history.enableAsyncParentWorkflowResend`, **default false**. Disabled =
the previous inline behavior, bounded by the caller's task deadline. Opt
in per cell.
## Testing
Unit tests cover the inline, async, and per-parent-dedup paths.
An xdc test (added in abae9cec, removed in b6eb6631) withholds the
parent's replication tasks so the child *must* pull it, asserts the
parent is absent from the standby, then stalls the active cluster's
`SyncWorkflowState` for 4 minutes:
```
--- PASS: TestChildPullsParentWhenParentReplicationIsWithheld (286.03s)
incoming-ctx-remaining: 4m59.999859334s (2.999s before this change)
sync-state-calls: 1 (5 without the per-parent guard)
dropped-parent-tasks: 9
```
During the stall, 4 verify RPCs reached the standby parent shard (t+0,
+50s, +101s, +169s) and exactly 1 `SyncWorkflowState` reached the active
cluster: the task retried and the guard turned the retries away.
4 minutes exceeds every deadline that previously bounded this path (3s /
10s / 30s) with ~1m headroom against the 5m setting, so the setting is
demonstrably what governs.
To reproduce: `git revert b6eb6631`, then
`go test -tags test_dep ./tests/xdc/ -run
TestVerifyChildCompletionParentResendSuite -timeout 30m`
## Known gaps
- Concurrency across *distinct* parents is unbounded (ordinary fan-out,
not amplification).
- When the parent is deleted on the source, the async path can't report
that back, so the child retries to the 15m discard instead of finishing
immediately. The `workflowNotFoundCache` TODO already in this file would
address it.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed?
1. add a max skip field to TimeSkippingConfig
2. add TimeSkippingInfo to DescribeWorkflowExecution (contains virtual
current time and running status)
3. add PollWorkflowExecutionTimeSkipping for fast-forward completion
## Why?
1. a generic mechanism to stop endless retries or schedules
2. to give clients easier access to time skipping state changes
related API change: https://github.com/temporalio/api/pull/835
## What changed?
Managing search attributes has been moved to operator handler.
## Why?
`tctl` which used this API is deprecated and no longer maintained.
Instead of removing the endpoints at this moment, forward to operator
handler.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
This branch implements the activity operator commands feature — a set of
server-initiated control APIs (PauseActivityExecution,
UnpauseActivityExecution, ResetActivityExecution,
UpdateActivityExecutionOptions) for both workflow-embedded and
standalone activities.
- Pause, Unpause, Reset and UpdateOptions for standalone activities plus
idempotency via RequestId
- common/activityoptions package (common/activityoptions/merge.go):
extracted mergeActivityOptions from the update-options handler into a
shared package (now also used by CHASM activity component).
- Metric renames: ActivityPauseRequests → ActivityPause,
ActivityResetRequests → ActivityReset, ActivityUnpauseRequests →
ActivityUnpause, ActivityUpdateOptionsRequests → ActivityUpdateOptions.
- RPC boilerplate, proto generation, and matching/frontend wiring for
the new APIs.
## Why?
Activity operator APIs existed for workflow-embedded activities but were
not wired up for standalone activities.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
Minimal, this is a new feature so it won't break users.
---------
Co-authored-by: Dan Davison <dan.davison@temporal.io>
Co-authored-by: Fred Tzeng <fred.tzeng@temporal.io>
## What changed?
Give the history client an explicit `Stop()` driven by fx `OnStop`
instead of `runtime.AddCleanup`: the connection pool's membership
watcher runs on a `goro.Handle`, and `Stop()` cancels it and closes the
pooled gRPC connections. Wired through the redirector, client wrappers,
the client bean, and the CHASM client generator. Removes the
`history.watchMembershipForClose` leak-test ignore. Mirrors #10816
(matching).
## Why?
The watcher's shutdown was tied to GC, which never fired (an open
`*grpc.ClientConn` is rooted by its own goroutines), so the goroutine
leaked per cluster and OOM-killed the test suite.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new functional test(s)
`TestClusterShutdownLeak` passes with the ignore removed (0 after
teardown); `client/history` + `chasm/lib` unit tests pass.
## What changed?
- New `system.retryUnboundedOnSystemResourceExhausted` global dynamic
config (default `false`). When `true`, frontend calls to
history/matching retry past the 2-attempt cap on system-scoped
`ResourceExhausted`, bounded only by the policy's 1-minute expiration
and the caller's context.
- New `backoff.ConditionalRetryPolicy` that delegates to one of two
policies based on a per-attempt predicate.
## Why?
Today's 2-attempt cap surfaces transient capacity events as user-visible
failures. We can hide transient system level resource exhausted errors.
## What changed?
- Applied the server-owned subset of small gofix analyzer fixes.
- Exact commands that were run:
```sh
make fmt-gofix
make goimports
make fmt
git diff --check
```
- No manual or AI changes were made.
- Some fixes caused lint errors; those were reverted again.
- Changes were all reviewed by me.
## What
Add `CountWorkers` RPC to count workers matching a query filter without
retrieving full worker details.
## Why
The UI needs to display a worker count in places where listing isn't
necessary. A dedicated count API follows the existing pattern
(`CountWorkflowExecutions`, `CountSchedules`).
## How did you test it?
- [x] new unit tests: count all, count with query filter, count with no
matches, invalid query error
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
Both downstream client caches now close a cached `*grpc.ClientConn` when
its host leaves the membership ring, instead of holding it until gRPC's
idle timeout:
- History `connectionPool` subscribes to the history ring and closes
departed hosts' conns.
- Matching `ClientCache` gains `Evict` + a per-entry release fn;
`matching.NewClient` subscribes to the matching ring and evicts departed
hosts.
Both wait a configurable drain delay before closing — new
`history.connectionCloseDelay` / `matching.connectionCloseDelay`
settings (default 30s) — so in-flight RPCs can finish. The eviction
goroutines are tied to the owning client via `runtime.AddCleanup`.
## Why?
After #9277 removed the RPCFactory cache, these downstream caches still
held `*grpc.ClientConn` forever, so #8719's dial-timeout log spam just
moved down a layer. Closing conns on ring departure fixes it at the
source.
## How did you test it?
- [x] built
- [x] covered by existing tests
## Potential risks
`CachingRedirector` (non-default;
`HistoryClientOwnershipCachingEnabled=false`) is unchanged and still
relies on gRPC's idle timeout for stale-conn cleanup, same as today.
---------
Co-authored-by: David Reiss <david@temporal.io>
## What changed?
Updated cmd/tools/check-dependencies to use main (not master) as the
default branch for both go.temporal.io/api and go.temporal.io/sdk when
resolving pseudo-version origins.
## Why?
Both temporalio/api-go and temporalio/sdk-go have renamed their default
branch from master to main.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
This PR adds `SignalWithStartWorkflowExecution` as a synchronous Nexus
operation exposed via `__temporal_system endpoint`, allowing workflows
to signal-with-start other workflows through the CHASM Nexus operation
framework.
Key changes:
- `chasm/lib/workflow/nexus_service.go`: `workflowServiceNexusHandler`
implements the `SignalWithStartWorkflowExecution` Nexus sync operation
by resolving the namespace and delegating to the History service. A
`SignalWithStartOperationProcessor` handles input enrichment (namespace,
request ID, links) and routing via CHASM's
`NexusOperationProcessorResult`.
- `chasm/lib/workflow/library.go` — library now holds the
`workflowServiceNexusHandler`, the workflow `Config`, the SA mapper
provider, and the SA validator. `newLibrary` (used by fx) takes those
dependencies, while the public `NewLibrary` keeps its old signature for
external callers. Adds `NexusServices()` so the library registers its
Nexus service via CHASM.
- `chasm/lib/workflow/validator.go`: `RequestValidator` consolidates the
`SignalWithStartWorkflowExecution` validation logic (previously inlined
in `WorkflowHandler`) into a reusable, injectable struct. This same
validator is used by both the frontend handler and the new CHASM
processor.
- common/dynamicconfig/constants.go — adds
`EnableSignalWithStartFromWorkflow` (namespace-scoped, default false).
- `service/frontend/workflow_handler.go`: Removed the `SignalWithStart`
validation block
- `service/history/fx.go`: Provides a `HistoryServiceServerProvider` so
the CHASM workflow library can call the history handler directly.
- `temporal/fx.go`: Removes the now-redundant `ChasmLibraryOptions`
grouping; each service module registers its own CHASM libraries.
- `components/nexusoperations/workflow/commands.go`: `NotFound` and
`InvalidArgument` errors during Nexus command handling are now surfaced
as workflow task failures instead of being treated as transient handler
errors.
- `common/payloads`: Adds `EncodeSingle`, `MustEncodeSingle`, and
`MustEncode` helpers used in tests.
- `cmd/tools/getproto`: Adds support for nexus-proto-annotations proto
imports.
- `tests/signal_with_start_from_workflow_test.go`: Functional test suite
covering the happy path, duplicate detection, conflict policies, and
validation rejection for the new Nexus operation.
## Why?
This functionality is one of our most requested GitHub issues.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## Potential risks
The history service now directly exposes a `HistoryServiceServer`
interface via `fx` for injection into the CHASM workflow library. This
tight coupling between the CHASM workflow library and the history
handler could complicate future layering — callers outside the history
service should not adopt this pattern. The feature is gated by
`history.enableSignalWithStartFromWorkflow` for rollout.
---------
Co-authored-by: Roey Berman <roey.berman@gmail.com>
## What changed?
- Codegen to automatically generate code for extracting businessID for
workflowservice methods based on "temporal-resouce-id" proto option
## Why?
- With codegen, any new API will automatically have the logic for
businessID extraction and avoids the case where businessID extractor
logic forget to be updated.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
Rewrites the release dependency check as a Go tool and extends it to
cover the main branch.
The original check was a shell script using `grep`/`awk` to extract
versions from `go.mod`. Moving to Go lets us use
`golang.org/x/mod/modfile` (proper AST parsing),
`module.IsPseudoVersion`/`PseudoVersionRev` (pseudo-version
decomposition), and `semver.IsValid` — none of which are feasible to
replicate reliably in shell. Using a Go tool is also consistent with the
pattern in `cmd/tools/`.
The rewrite also extends validation: the original script only enforced
tagged releases on `release/*` and `cloud/*` branches. The new tool adds
a `main` branch policy: pseudo-versions are allowed on main, but the
referenced commit must be on the dependency's default branch (not a
feature branch or a fork).
## Policies enforced
- `release/*` and `cloud/*`: must be tagged semver releases
- `main`: tagged releases accepted; pseudo-versions must reference a
commit on the dependency's default branch
- other branches: skipped
## Why
If an API or SDK references a commit that's not on the main branch or a
tag, it creates problems when bumping the version later on. There was a
recent occurrence of this.
## Running locally
```
go run ./cmd/tools/check-dependencies --base-branch main
```
Pass the branch you're targeting as `--base-branch`. For example, to
simulate a PR against a release branch:
```
go run ./cmd/tools/check-dependencies --base-branch release/v1.31
```
## What changed?
Added completion callback support to standalone activities:
- Callback lifecycle: When a standalone activity reaches a terminal
state (completed, failed, canceled, terminated, timed out), it now fires
any registered Nexus completion callbacks — the same mechanism workflows
already use.
- Frontend validation: Callback URL length, header size, and endpoint
allowlist validation is now shared between workflows and standalone
activities via callbacks.ValidateCallbacks, refactored out of the
workflow handler.
- Describe response: DescribeActivityExecution now returns callback
state (CallbackInfo) including trigger, status, attempt count, and
failure details.
- Start response: StartActivityExecutionResponse now includes a
Link_Activity_ identifying the started (or reused) activity.
- Config: Renamed MaxCHASMCallbacksPerWorkflow to
MaxCallbacksPerExecution since it now applies to both workflows and
standalone activities.
## Why?
The v2 scheduler needs to start standalone activities on a schedule and
be notified when they complete, so it can track action results, handle
overlap policies, and support pause-on-failure. Workflows already have
this via CompletionCallbacks + Nexus callback delivery. This PR gives
standalone activities the same capability, reusing the existing callback
library rather than reimplementing the
delivery/retry/backoff logic. This is a prerequisite for the scheduler's
Invoker to call StartActivityExecution with a callback pointing back to
the Scheduler component.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [X] added new functional test(s)
## What changed?
Enhance matching clients to select random partition for Nexus poll and
dispatch requests.
## Why?
Before this change all Nexus matching request were going to the root
partition only.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
None
## What changed?
Extend genrpcserverinterceptors to support `ActivityID` and
`OperationID` and `ChasmRunID`
## Why?
Fix and improve logging.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add chasm logical task type dynamic config filter and standby task
discard delay dynamic config flag.
## Why?
Standby tasks have a configurable dynamic config (per task type) that
specifies the timeout for discarding the task and running the
post-discard function. Currently, this is not configurable for CHASM
logical tasks. By default, CHASM tasks need a higher discard timeout
since they are not regenerated from pending tasks in MS, which can lead
to abandoned tasks. For tasks that can be safely dispatched to Matching,
they can be configured with a lower value. (See Standalone activity
tasks).
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add a new cli tool to run the matching fairness simulator, and some
tests for basic fairness behavior.
## Why?
So users can tell how it will behave for their workloads.
## What changed?
* Convert the flake report we use to monitor functional test reliability
from invoking the `tringa` cli tool and a Python script to only a Go
script that fetches the artifacts, parses the JUnit, and creates the
report.
* This also fixes an issue where `tringa` only found the last 20 tests,
the Go script will iterate through all failures in the last `N` days
## Why?
Everything is in a single executable making it easier to test and
extend.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Minimal, this is only a tooling change.
## What changed?
Integrate Go 1.26's new `go fix` into workflow.
NOTE that the changes caused our linter to fire; a [separate
commit](1b23f787ae)
addresses those.
## Why?
Ensure Go code is standardized/modernized.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Overview
This feature introduces a special internal Nexus endpoint called
`__temporal_system` that enables adding functionality to workflows
**without requiring new workflow commands and events**. Operations on
this endpoint are routed internally within Temporal's history service
rather than via external HTTP calls.
## Key Components
### 1. System Endpoint Infrastructure
- **Endpoint Name**: `__temporal_system` (constant in
`common/nexus/constants.go:6`)
- **Callback URL**: `temporal://system` for internal routing
- **New History Service RPCs**
(`proto/internal/temporal/server/api/historyservice/v1/service.proto:433-437`):
- `StartNexusOperation` - Starts operations on the system endpoint
- `CancelNexusOperation` - Cancels operations on the system endpoint
### 2. Operation Processor Framework
A new processor pattern (`chasm/nexus_operation_processor.go`) that
allows CHASM libraries to:
- **Validate and transform input**: Processors can validate operation
inputs and set default values
- **Determine routing**: Each processor returns a routing key that
determines which history shard handles the operation
- **Re-serialize input**: Mutated inputs can be re-serialized to persist
default values
**Routing strategies**:
- `NexusOperationRoutingKeyExecution` - Routes to the shard owning a
specific workflow execution
- `NexusOperationRoutingKeyRandom` - Routes to a random shard
### 3. CHASM Library Integration
CHASM libraries can now provide (`chasm/library.go:16-19`):
- **`NexusServices()`**: Regular Nexus service handlers (implement the
actual operation logic)
- **`NexusServiceProcessors()`**: Input processors for validation and
routing
Example from test library (`chasm/lib/tests/nexus_service.go`):
```go
// Service handler - implements the actual operation
TestOperation = nexus.NewSyncOperation("TestOperation",
func(ctx context.Context, input string, options nexus.StartOperationOptions) (string, error) {
return "Hello, " + input, nil
})
// Processor - validates input and determines routing
func (o testOperationProcessor) ProcessInput(ctx chasm.NexusOperationProcessorContext, input string)
(*chasm.NexusOperationProcessorResult, error) {
return &chasm.NexusOperationProcessorResult{
RoutingKey: chasm.NexusOperationRoutingKeyExecution{
NamespaceID: ctx.Namespace.ID().String(),
BusinessID: input, // Route based on input
},
}, nil
}
```
### 4. Execution Flow
When a workflow schedules a Nexus operation on `__temporal_system`
(`components/nexusoperations/executors.go:233-238`):
1. **Input Processing**: The processor validates input and determines
routing
2. **Internal RPC**: Instead of HTTP, calls
`HistoryClient.StartNexusOperation` with the target shard ID
3. **Handler Execution**: The history service invokes the registered
Nexus handler (`service/history/handler.go:2707-2768`)
4. **Result Handling**: Supports both sync (immediate result) and async
(operation token) responses
5. **Workflow Completion**: Results flow back through the same
completion path as external Nexus operations
### 5. Benefits
✅ **No schema changes**: Add functionality without new commands/events
in workflow history
✅ **Consistent API**: Uses existing Nexus operation semantics
(sync/async, callbacks, links)
✅ **Proper routing**: Operations are intelligently routed to the correct
shard
✅ **Input validation**: Type-safe input validation and default value
handling
✅ **Future extensibility**: Foundation for direct client invocation (not
yet implemented)
### 6. Technical Details
- **Error handling** (`components/nexusoperations/executors.go:444`):
Non-retryable service errors are properly handled and fail operations
immediately
- **Metrics**: System operations are tracked separately with
`DestinationTag` set to the endpoint name
- **Link conversion**: Helper functions convert between Nexus SDK links
and protobuf links (`common/nexus/util.go:17-46`)
- **Operation token handling**: Moved link converters to common package
for reuse (`common/nexus/link_converter.go`)
### 7. Current Limitations
- Only accessible from workflows (via `ScheduleNexusOperation` command)
- Direct client invocation not yet implemented
- Headers not supported for system endpoint operations
## Test Coverage
New test (`tests/nexus_workflow_test.go:2763-2843`) demonstrates:
- Scheduling operation on `__temporal_system` endpoint
- Synchronous operation completion
- Result propagation back to workflow
## Architecture
This architecture provides a clean, extensible way to add internal
functionality while maintaining compatibility with Temporal's existing
workflow execution model. The system endpoint acts as a bridge between
workflows and internal CHASM components, enabling:
- **Extensibility**: New operations can be added by implementing CHASM
libraries
- **Type safety**: Input validation happens before operations are routed
- **Scalability**: Intelligent routing ensures operations land on the
correct shard
- **Consistency**: Same execution model as external Nexus operations
## What changed?
Instead of storing the test sharing salt in a repo var, store it in a
file.
## Why?
https://github.com/temporalio/temporal/pull/9236 tried to use the repo's
variable for storing the test sharding salt; but that is not accessible
to PRs from forks.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
PRs could fail due to flaky tests. But then the next run will delete the
branch (thereby closing the PR) and open a new one.
## What changed?
Adds script and workflow to auto-update functional test sharding salt.
It downloads the previous JUnit XMLs and finds a salt that balances them
well.
Also added a new variable to store the salt:
<img width="812" height="64" alt="Screenshot 2026-02-05 at 5 59 42 PM"
src="https://github.com/user-attachments/assets/27316baa-03d7-4d0f-8700-9ead9cf322e8"
/>
## Why?
Balanced test run shards lead to lower PR test time.
## How did you test it?
- [ ] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
AFAIK there's no way to run the new GitHub action before merging ... but
I ran the script locally successfully.
## Potential risks
Given that we have some flakiness, the PR to update it might fail;
requiring human intervention.
## What changed?
This PR extends the raw history optimization to pass raw history bytes
from History Service → Matching Service → Frontend without
deserialization in Matching Service.
**Key changes:**
1. History Service:
- When `SendRawHistoryBetweenInternalServices` is enabled, sets
`RawHistoryBytes` (field 21) with raw proto-encoded history batches
2. Matching Service:
- Passes raw history bytes through to frontend via
`PollWorkflowTaskQueueResponseWithRawHistory`
- Uses wire-compatible proto messages so gRPC auto-deserializes
`[][]byte` → `History` on the client side
3. Frontend:
- Receives raw history in `RawHistory` field (auto-deserialized by gRPC)
- Processes search attributes for raw history since it bypasses history
service's normal processing
4. Proto definitions:
- Added `raw_history_bytes` (field 21) to
`RecordWorkflowTaskStartedResponse`
- Added `PollWorkflowTaskQueueResponseWithRawHistory` message with
wire-compatible layout
- Added `raw_history` (field 22) to `PollWorkflowTaskQueueResponse`
## Why?
When `history.sendRawHistoryBetweenInternalServices` is enabled, the
previous implementation only avoided deserialization from persistence →
History Service. However, Matching Service was still deserializing
history events (via gRPC auto-deserialization) and re-serializing them
when forwarding to Frontend.
This change eliminates that unnecessary serialization/deserialization
cycle in Matching Service by:
1. Having History Service send raw bytes directly
2. Having Matching Service forward these raw bytes without parsing
3. Having Frontend receive the bytes which gRPC auto-deserializes
This reduces CPU usage in Matching Service for workflows with large
histories.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s) (`tests/workflow_task_test.go`)
## Potential risks
SendRawHistoryBetweenInternalServices must be disabled when rolling back
from this version to an older version.
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## What changed?
Add a new dynamic config for Auto Enabling fairness and priority if we
see the relevant tasks coming in.
## Why?
Seamlessly start to transition users who start using the fields over to
the new code path.
## How did you test it?
- [X] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
Due to storing this in the userdata we are using that interface a bit
more, we also need to change the initialization such that we start it
before being able to substantiate the defaultQ, this change in
initialization might have unintended side effects that I'm not currently
seeing.
## What changed?
Alert company slack when the main build fails after a PR merge
## Why?
Help us combat our issues with flaky tests.
## How did you test it?
- [ ] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
No, not running in production
## What changed?
Use unique.Handle for dynamicconfig.Keys instead of strings.
## Why?
Force keys to always be lowercase so we don't have to convert at lookup
time, and also speed up lookups by interning.
## How did you test it?
- [x] covered by existing tests
## What changed?
- Frontend workflow_handler can now route to CHASM scheduler, depending
on experiment flag/dynamic config
- Scheduler functional test has been updated to exercise both V1 and V2
scheduler
- Small fixes throughout scheduler codebase to address differences
noticed during functional testing
- Nexus callback mechanisms updated to allow the CHASM internal callback
URL
I plan to address the TODO around LastCompletionResult's assertion in a
separate follow-up PR.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- Existing V1 tests are preserved, to show that the V1 codepath still
works as-is. All new CHASM handling logic falls back to V1 in absence of
a positive response. CHASM can only be opted in for a schedule during
creation, controlled by dynamic config/experimental header.
## What changed?
- Rename Entity to Execution
## Why?
- We agreed on the new naming.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Added Chasm RPC handler foundation and interceptor. Added test as use
case demo.
## Why?
The Chasm RPC handler will be used as the abstraction layer for future
work. A grpc interceptor was added to intercept requests to process
boilerplate code so the handlers don't have to regurgitate it. Please
look at the added test as it demonstrates how the RPC handler is
registered and a request gets processed.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
There is a [future
ticket](https://temporalio.atlassian.net/browse/ACT-69) to refactor all
history RPC calls to start using the interceptor as to avoid boilerplate
code. This does need a deeper discussion with the team and we also need
to figure out how to identify the incoming RPC request so we can extract
namespace etc. properly.
---------
Co-authored-by: Roey Berman <roey.berman@gmail.com>
## What changed?
Added a protoc plugin to generate client for internal CHASM requests to
the history service.
A CHASM library can now define its own set of gRPC services and register
a handler for those.
I had to modify the history client files to be generic, which broke the
mock generation, so I had to hand code the tests there, which IMHO is
actually nicer than the way they were with mocks.
Also opted out of mock generation for CHASM services, there's just not
enough value there.
**NOTE**: Instead of creating three separate clients, I generated a
single layered client, with a constructor that is DI friendly. There
doesn't seem to be a good enough reason to break this out into separate
clients or make client construction configurable (for now).
**NOTE**: We can also get rid of the genrpcwrappers script eventually
and use the protoc plugin approach for all services, I decided not to do
that here though to reduce scope.
For a preview of the generated file see:
-
3a89614a90/chasm/lib/activity/proto/v1/service.proto
-
3a89614a90/chasm/lib/activity/gen/activitypb/v1/service_client.pb.go
## Why?
Keep all CHASM library functionality contained.
## How did you test it?
- [ ] built
## What changed?
- Revive and update original PR #2977 for Elasticsearch CLI tool
Replaces manual curl invocations with a proper CLI tool that leverages
Temporal's built-in Elasticsearch auth providers and provides better
error handling and logging.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None, net new.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduces `temporal-elasticsearch-tool` for ES schema/index
management and ping, extends ES client APIs, embeds ES schema, and
updates Makefile to use the tool.
>
> - **Tools**:
> - New `temporal-elasticsearch-tool` CLI with commands: `setup-schema`,
`update-schema`, `create-index`, `drop-index`, `ping`; supports AWS auth
and uses embedded schema files.
> - Adds entrypoint `cmd/tools/elasticsearch`, README, and basic tests.
> - **Elasticsearch Client**:
> - Extends `CLIClient` with `ClusterPutSettings`, `IndexPutTemplate`,
`IndexPutMapping`, `Ping` and implements them (v7) using raw requests
where needed; allows custom HTTP client from config (e.g., AWS-signed).
> - **Schema**:
> - Embeds ES v7 cluster settings and index template
(`schema.Embedded...` accessors).
> - **Build/Makefile**:
> - Adds build target and binary cleanup for
`temporal-elasticsearch-tool`; updates `install-schema-es` and
`install-schema-xdc` to use the CLI instead of curl.
> - Includes binary in `.goreleaser.yml`; excludes it in
`.dockerignore`.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
b37864809a. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
~Enforce `temporal/cmd/tools/protogen` to set the PATH to point to the
locally installed tools in `.bin`.~
Enforce `temporal/cmd/tools/protogen` to use locally installed protoc
plugins.
## Why?
Without this, it will use whatever is on the PATH; which might not be
the same as in `.bin`.
## How did you test it?
- [ ] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
Without this change, I get different `.pb.go` files emitted as my global
proto-gen-go is used.
## What changed?
This PR aims to avoid usage of
[`strings.Split`](https://pkg.go.dev/strings#Split) where possible in
favor of better string splitting techniques, speficially:
[`strings.SplitN`](https://pkg.go.dev/strings#SplitN) and
[`strings.SplitSeq`](https://pkg.go.dev/strings#SplitSeq) where
appropriate.
There was also a [`strings.Fields`](https://pkg.go.dev/strings#Fields)
change I made to use
[`strings.FieldsSeq`](https://pkg.go.dev/strings#FieldsSeq) instead, and
another for S3 to use the [`path`](https://pkg.go.dev/path) package
instead of [`strings.Split`](https://pkg.go.dev/strings#Split).
## Why?
[`strings.SplitN`](https://pkg.go.dev/strings#SplitN) and
[`strings.SplitSeq`](https://pkg.go.dev/strings#SplitSeq) are often
better options in many cases, and can be _partially_ detected using
[`modernize`](https://pkg.go.dev/golang.org/x/tools/gopls/internal/analysis/modernize):
> `stringsseq`: replace Split in "for range strings.Split(...)" by
go1.24's more efficient `SplitSeq`, or `Fields` with `FieldSeq`.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
There are lots of potentially subtle behaviors from the `strings.Split`
(and `strings.Fields`) usage that should be accounted for. If our
existing tests don't cover those subtleties, there's risk for
introducing an unintended bug. More intricate handling/parsing
previously using the `strings` package should get extra attention from
reviewers. I've attempted to break up my changes into logical commit
chunks to aid in review / help spot potentially concerning changes.
## What changed?
Allow frontend->matching poll requests to retry up to their context
timeout instead of just once.
## Why?
On matching service deployments, a busy new matching node may hit its
persistence rps limit trying to acquire new task queues and be unable to
accept polls.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Fill in support for subscriptions to dynamic config values with
constrained defaults.
## Why?
We'd like to use this combination of functionality.
## How did you test it?
- [x] added new unit test(s)
## What changed?
Log softassert warnings if dynamic config settings are registered with
default values with shared structure.
## Why?
This is very likely unintended and may lead to unexpected behavior of
settings (values will be parsed on top of a copy of the default).
## How did you test it?
- [x] run locally and tested manually
- [x] added new unit test(s)
## What changed?
Added support for defining protos in chasm libs.
## Why?
Keep everything local to the library.
## How did you test it?
- [x] built
- [x] run locally and tested manually
## What changed?
- Split implementation of "constrained default" settings from "plain
default" settings. This is more code and the diff looks complex, but the
individual paths are both simpler than the mixed version.
- Add conversion cache using a weak map.
- Remove GlobalCachedTypedValue.
- Use "raw" values for subscription dispatch deduping to avoid
unnecessary conversions.
- Deep copy default values when using mapstructure, to avoid problems
with merging over shared default values.
## Why?
- Fixes#6756
- Performance improvement for "plain default" settings (almost all of
them)
- Performance improvement for settings with complex converters
- Remove footgun in defaults that aren't scalar values
## How did you test it?
existing+new unit tests
## What changed?
* Add implementation for DescribeWorker API (to both frontend and
matching)
* Add feature flag
* Add all quotas/etc.
* Add unit tests
* Add functional tests
## Why?
Part of worker commands work.
Users should be able to directly get worker info for the worker they
need.
Corresponding API PR: https://github.com/temporalio/api/pull/622
## How did you test it?
- [X] add unit tests
- [X] add functional tests
## Potential risks
None, behind feature flag
## What changed?
Add stubs for worker commands.
Corresponding API PR: https://github.com/temporalio/api/pull/612
## Why?
Unblock SDK team.
## How did you test it?
- [X] built
- [X] covered by existing tests
## What changed?
+ Introduced a new Update TaskQueueConfigApi.
+ Implemented the corresponding handlers in the frontend service and the
matching service.
+ Persistence of TaskQueueConfig from the UpdateTaskQueueConfigApi.
+ Return the config response as part of the DescribeTaskQueue api.
## Why?
+ This is the first part of the UpdateTaskQueueConfig Implementation.
+ Goal is to persist the TaskQueueConfig and handle nil values in the
update request.
+ Next steps would be to attach the corresponding rate limiters with the
persisted configs.
## How did you test it?
- [x] built
- [x] run locally and tested manually
