## What changed
Kept dynamic data out of Nexus log messages and moved it to tags. Added
a review guideline requiring static logger messages and structured tags
for all dynamic content.
## Why
Ensures that Nexus logs are aggregatable.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
## What changed?
`GetWorkflowExecutionHistory` and `GetWorkflowExecutionHistoryReverse`
now check `branch_token` in the page token against the token in mutable
state.
## Why?
To confirm if it is still the correct branch after conflict resolution.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
### What changed
Sets gobreaker's OnStateChange hook on the outbound queue circuit
breaker pool, logging every transition.
### Why
Obtain more details for debugging curcuit breaker in production.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed
Adds logs tags for failures on the Nexus frontend path.
## Why
Have more details to correlate issues with requests.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What changed
Adds a few Nexus-specific log tags to the handler-side frontend logger.
## Why
Mainly for the request ID to debug Nexus calls across namespaces better.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What
The slow-callback warning in `(*registry).RegisterStateChangeCallback`
formatted the callback key with `fmt.Sprintf("%v", key)`:
```go
r.logger.Warn(
"Namespace registry callback slow",
tag.Key(fmt.Sprintf("%v", key)),
tag.Duration("duration", duration),
)
```
Callback keys are arbitrary values and are usually the registrant itself
(e.g. `*WorkflowHandler`, `*historyEngineImpl`, the queue scheduler).
`%v` deep-prints the entire struct via reflection, and when that
traversal iterates an internal map that another goroutine is writing,
the Go runtime aborts the **whole process**:
```
fatal error: concurrent map iteration and map write
...
fmt.Sprintf(...)
nsregistry.(*registry).RegisterStateChangeCallback.func1.1() registry.go:298
```
This is a `fatal()`, not a recoverable panic, so the `goro.Handle`
recover at the top of the goroutine cannot catch it — the frontend
crashes.
## Fix
Replace the format call with `formatCallbackKey`, which:
- prints scalar keys (string/number/bool) by value,
- prints `fmt.Stringer` keys (e.g. `uuid.UUID`, used by the
namespace-handover interceptor) via `String()`,
- renders everything else as `type(0xADDR)` for pointer-like kinds, or
just the type name otherwise — **never traversing fields**.
This keeps the log useful (you can still tell which registrant was slow)
while eliminating the reflection walk that caused the race.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
The `Nexus StartOperation/CancelOperation request failed` logs for CHASM
only attached `tag.Error(callErr)` and are missing helpful context.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
- Add host level rate limiter for controlling RPS of admin batch
operations across all namespaces and all admin batch workflows.
## Why?
- When running admin batch operations we only care about controlling the
total RPS, not really RPS for a given namespace or a given admin batch
workflow.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add scale manager component.
## Why?
Part of dynamic partitioning.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s) - to come in future PRs
## Potential risks
No behavior changes yet.
## What changed?
1) `taskGenerator`:
- add regenerate tasks for time skipping
2) `mutableState`:
- new methods: add key methods to transition time
- `IsStateDirty`: captures changes of time-skipping related fields
- `closeTransaction`: add closeTransactionHandlerTimeSkipping
3) event: new time-skipping runtime event added
4) persistence: new runtime data added to execution info
## Why?
add the foundational mechanism of how time skipping works in the runtime
of a workflow execution
**for reviewers:**
this is a foundation of t-s runtime, so it works without any granular
and extended features as bound, replication, external-transfer tasks,
etc. **Over 50% of lines are tests/mocks/pb-gen. can focus on code
first, then functional tests, and last ut.**
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
## What changed?
Functional tests write debug-level logs to a file rather than inline in
the job. Info and above are still visible and the debug logs are
available for download at the end of each test job.
## Why?
Debug level logs make parsing the functional test run logs very
difficult, sometimes too large to even download the logs. This makes
debugging flaky test runs even more difficult.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
- [X] manually verified in CI
## Potential risks
Minimal, debug logs are still available to download and parse. If there
are logs that are missing for debugging we should move those from
`debug` to `info` level in the future
## What changed?
Extend genrpcserverinterceptors to support `ActivityID` and
`OperationID` and `ChasmRunID`
## Why?
Fix and improve logging.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Do not throttle panics and fatals in throttled logger
## Why?
The doc block for NewThrottledLogger already says:
```
// Fatal/Panic logs are always emitted without any throttling
```
but this is not respected in the code. Changed the code to match this.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Integrate Go 1.26's new `go fix` into workflow.
NOTE that the changes caused our linter to fire; a [separate
commit](1b23f787ae)
addresses those.
## Why?
Ensure Go code is standardized/modernized.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
This updates the server code to use the shorthand log tag constructors
introduced in #9174.
As part of this it _does_ make a breaking change to the `Bool`
constructor: it now takes in the key as a string to be consistent.
The only inconsistent one is now `Error`, but that's used so heavily
that changing it is likely not worth the time.
## Why?
Consistency!
## How did you test it?
Existing tests
## Potential risks
The only risk is that I _have_ introduced a breaking change to the Bool
constructor. I'm happy to undo that if my reviewers desire: my goal is
minimal breaking changes.
I'd prefer none, but I made this change to stir up discussion
## What changed?
This PR adds _consistent_ shorthand tag constructors for all supported
zap tags.
I also added `Uint64` (and `NewUint64`) as we've wanted it in other
projects.
## Why?
There are two main reasons:
- I (and others I've talked to) are lazy and dislike typing
`log.New<Type><TAB>` when `log.<Type>` would have done it
- Consistency! Some tags are just the type name (`Bool`, `Error`), some
have a prefix of New (`NewInt64`), and some have a prefix _and_ a suffix
of Tag (`NewStringTag`).
The new constructors are entirely consistent: no prefixes and no
suffixes; just the type's name.
## How did you test it?
I didn't, yet. These wrap existing APIs.
## Potential risks
None.
## What changed?
- Add panic wrapper for library logic
- Also provides a logger through the Context interface
## Why?
- Capture panic error so that it's easier to tell where the panic
actually happens. Otherwise a panic can cause another panic in some
unrelated deferred function making debugging harder.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Do a final staleness check after reload
## Why?
- Due to data loss or force failover, it's possible that even after
reload the mutable state may still be stale.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Rename the tag from `worker-build-id` -> `worker_version`.
Use `:` instead of `/` in the tag value.
## Why?
Consistent terminology and delimiter.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Self hosted clusters relying on the old tag name and values would have
to adopt new tag. This should be ok because versioning is not GA yet.
Consolidated and refactored activity request validations to the chasm
package.
Added additional tests for the validator.
Existing workflow activities and standalone activities should share the
same validation code. Standalone activities also directly process the
frontend request and therefore has additional fields to validate
compared to embedded activities.
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
This refactors the existing stack, but the validation is basically
exactly copied over and all relevant tests are passing. The standalone
start activity request is now cloned before sanitizing the request
attributes to preserve idempotent behavior during retries, but can
potentially impact performance if there are large inputs.
---------
Co-authored-by: Roey Berman <roey.berman@gmail.com>
## What changed?
- Use archetypeID everywhere in history service and pass them to
persistence
## Why?
- Required for separate ID space work.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Recording unpause event
- Regenerating workflow and activity tasks
- Made EVENT_TYPE_WORKFLOW_EXECUTION_UNPAUSED a buffered event (since
pause event is buffered)
- Added unit tests (Functional tests are added along with the API
changes)
## Why?
This is needed to implement pause/unpause operation APIs.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduce workflow "unpaused" history event end-to-end, buffer
pause/unpause consistently, and reschedule activities on unpause with
tests.
>
> - **History Events**:
> - Add `CreateWorkflowExecutionUnpausedEvent` and
`HistoryBuilder.AddWorkflowExecutionUnpausedEvent`.
> - Update event buffering: buffer both `WORKFLOW_EXECUTION_PAUSED` and
`WORKFLOW_EXECUTION_UNPAUSED` in `historybuilder/event_store.go`.
> - **Mutable State**:
> - Add `AddWorkflowExecutionUnpausedEvent` and
`ApplyWorkflowExecutionUnpausedEvent` to transition to `RUNNING`, clear
`PauseInfo`, bump activity stamps, and (re)generate activity tasks as
needed.
> - Extend `interfaces.MutableState` + mocks with unpause methods.
> - Rebuilder applies `WORKFLOW_EXECUTION_UNPAUSED` events.
> - **Logging/Tags**:
> - Add `tag.WorkflowActionWorkflowUnpaused`.
> - **Tests**:
> - Add unit tests for pause/unpause flow, stamps, and buffering
behavior; adjust existing buffering test expectations.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
f13902700e. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Sean Kane <spkane31@gmail.com>
## What changed?
**Note**: This depends on https://github.com/temporalio/api/pull/653.
Sending it for some early feedback.
- Made changes to `WorkflowPauseInfo` (in WorkflowExecutionInfo).
Reusing unused proto fields.
- Added mutablestate.IsWorkflowExecutionPaused()
- Added mutablestate.AddWorkflowExecutionPausedEvent()
## Why?
- These changes are needed to implement pause/unpause features.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Adds workflow-level pause/unpause with new `WorkflowPauseInfo` fields,
client endpoints, history event handling, state/status validation for
`PAUSED`, and tests; removes activity-level pause info.
>
> - **Protocol/API**:
> - Redefine `persistence.v1.WorkflowPauseInfo` to `{pause_time,
identity, reason, request_id}`; remove `ActivityPauseInfo` and related
helpers.
> - Bump dependency `go.temporal.io/api` and adjust generated code
indexes.
> - Update RPC metadata, redirection maps, quotas, and log tags for
`PauseWorkflowExecution`/`UnpauseWorkflowExecution`.
> - **Frontend Clients**:
> - Add `PauseWorkflowExecution` and `UnpauseWorkflowExecution` to
`client_impl`, `metric_client`, `retryable_client`, and mocks.
> - **History/State**:
> - Add `WorkflowExecutionPaused` event creation
(`Create/Add...PausedEvent`) with buffering rules (paused allowed to
buffer; unpaused not buffered).
> - Implement `ApplyWorkflowExecutionPausedEvent`: set status `PAUSED`,
populate `executionInfo.PauseInfo`, invalidate pending activities and
workflow task via stamps.
> - Rebuilder applies paused event; state transition validation supports
`PAUSED` across CREATED/RUNNING/ZOMBIE.
> - **Validation/Tests**:
> - Allow `WORKFLOW_EXECUTION_STATUS_PAUSED` in validators; expand unit
tests for create/update state/status and paused behavior.
> - **Tooling**:
> - `buf.yaml` breaking ignore for `executions.proto`.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
951c9c4a53. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Changed callers of softasserts that used a dynamic message to use a
static message; and pass rest in through tags.
NOTE: I've experimented with https://github.com/quasilyte/go-ruleguard
to enforce it; but that is too brittle.
## Why?
It makes grouping in logs (and Antithesis) much easier as the identifier
is static.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Enforce archetype when loading mutable state
## Why?
- Avoid workflow operation get applied to non-workflow archetypes.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Route heartbeats, if present, from Poll...and ShutdownWoker frontend
handle to the corresponding matching instance.
## Why?
We piggyback on Poll/Shutdown for heartbeats to reduce the number of
separate API calls.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Latency increate? One of the possible solution will be to make those
calls async in "fire and forget" style.
## What changed?
this PR changes the semantics of common/log/Logger to apply With(tags)
as upserts for duplicate keys, rather than appending duplicates. This
matches the behavior of the TestLogger
## Why?
Having multiple values for the same Tag key in the logs is much more
confusing than helpful.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
- It's possible someone is dependent on the existing behavior and may
need to adjust. However, this shouldn't affect functionality.
- Each Logger instance consumes a bit more memory
## What changed?
This changes all methods that fail tests on the test logger to
**forcibly** fail it by calling `t.Fatalf`.
It adds the stack trace to the failure message as we'd otherwise have no
information on the _path_ to the failure, which is why we initially used
`panic`
## Why?
While testing a separate project I determined that panics from OSS' test
logger (and the internal one it is based on!) were being caught by our
application code, obscuring issues that should cause tests to fail.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
This may fail more tests than we currently do. This is a Good Thing.
## What changed?
Updated (most) log statements that log variables inline to use
structured logging. There were occurrences that I left untouched, as I
felt that the inline logging added siginficant clarity over a tag.
Prompt history (`claude-3.7-sonnet`):
> Although our logger supports structured logging via its tags, many of
our error messages instead emit variable data directly into the text
content of the logged message.
> Can you find occurrences of messages being logged with variable data
directly in the textual portion, and instead move those variables into
tags on the message, to facilitate structured logging?
>
> Please limit changes you make to precisely this ask; if no tags are
changed on a log statement, don't rewrite the message.
>
> The scope should be the *entire* codebase.
...
> Great - are there any other places in the whole codebase you can see
where we're dynamically formatting error strings with variables in log
statements?
...
> Can you make those changes?
...
> Complete that codebase-wide refactoring initiative.
...
> Great. Are there any other places that are still logging with format
strings that include variables, anywhere in the codebase?
...
Bonus robot chatter:
<img width="379" alt="Screenshot 2025-05-14 at 10 51 05 AM"
src="https://github.com/user-attachments/assets/8f1c8a73-3c9b-4d5e-bccd-b37ef8165d9c"
/>
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Change Deployment Version SA delimiter from `.` to `:`.
## Why?
New delimiter is used in:
- `TemporalWorkerDeploymentVersion` SA
- `BuildIds` SA, for the `pinned:<deployment name>:<build id>` format
- Deployment Version workflow ID
Old delimiter is used in:
- v31 version string values in external API
- version string values in internal entity workflows and APIs
We'll refactor code to clean up usage of old delimiter and string
version fields later. For now, the priority is to make public preview
external APIs consistent.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
If Versioning V0.31 pre-release users directly use these SAs, they wont
be able to find old or new wfs if they filter using only one of the
delimiters. (This affects "Go to Workflow" links in UI until UI is
updated)
## What changed?
I've changed _some_ of the tags I altered in #7738 back to StringTags
from StringerTags and added usage recommendations to the StringerTag
(and StringersTag) methods.
## Why?
@prathyushpv rightly pointed out that some of the tags I changed in
#7738 are applied to the loggers themselves and not just the messages.
This means that the `String()` method will be invoked _every time_ a log
is to be emitted. This incurs extra allocations etc. etc.
## What changed?
This PR adds easy access to Zap's Stringer and Stringers fields.
## Why?
`Stringer` fields are more efficient than `String` field. In the OSS
repo alone we have 22 separate invocations of `tag.NewStringTag(...,
foo.String())` which would be better served by using a Stringer tag.
This only calls `String` (which can be expensive) when the line is
actually emitted.
Meaning fewer wasted cycles when calling `String()` for debug-level
logs.
## How did you test it?
- [x] I'm doing this manually elsewhere
## What changed?
<!-- Describe what has changed in this PR -->
Remove license header from every file. Because it is really hard to
follow in this PR here is the summary:
1. License header is removed from all `.go` and `.proto` files
:fireworks::fireworks:🎆.
2. `LICENSE` file in the root directory has only Temporal and Uber
copyrights.
3. 5 other `LICENSE` files added to the packages which have copyrights
different from Temporal and Uber: Datadog, Xargin, "Mat Ryer, Tyler
Bunnell and contributors".
4. `license_file` flag is removed from all code generation tools.
5. `copyright_file` flag is removed from `go:generate mockgen`
directive.
6. All copyright related targets are removed from `Makefile`.
7. Updated Temporal copyright year to 2025 everywhere.
## Why?
<!-- Tell your future self why have you made these changes -->
I double checked with legal department that it is not needed to have
license header in every file. One file per repo is enough. I put all
copyrights to the root `LICENSE` file and removed header from all other
files. Also updated tools and `Makefile`.
## What changed?
<!-- Describe what has changed in this PR -->
Upgraded mockgen.
## Why?
<!-- Tell your future self why have you made these changes -->
Dependency hygiene.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
CI.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
fixed some code to reduce versioning flakes, mainly:
[TestVersioning3FunctionalSuite/TestUnpinnedWorkflowWithRamp_ToUnversioned](https://github.com/temporalio/temporal/actions/runs/13818885947/job/38659266639)
(with_logger data-race ✅)
[TestWorkerDeploymentSuite/TestSetWorkerDeploymentRampingVersion_Unversioned_VersionedCurrent](https://github.com/temporalio/temporal/actions/runs/13818885947/job/38659261367)
( context timeout - maybe fixed by having a separate go-routine ✅ )
[TestDeploymentVersionSuite/TestDrainageStatus_SetCurrentVersion_YesOpenWFs](https://github.com/temporalio/temporal/actions/runs/13818885947/job/38659258964)
(with_logger data-race ✅)
[TestVersioningFunctionalSuite/TestDispatchActivityFailCrossTq (retry
2)](https://github.com/temporalio/temporal/actions/runs/13712084173/job/38350437309)
(with_logger data-race ✅)
## Why?
<!-- Tell your future self why have you made these changes -->
- To stop these from appearing as flakes in the test reports.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- Ran this locally with `-race` and now see no errors (errored out
previously)
- Ran this on CI a bunch of times without errors/flakes
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Added additional tracing logs for forwarded Nexus HTTP requests.
## Why?
<!-- Tell your future self why have you made these changes -->
Forwarded requests create a new request so they were not copying over
the `httptrace.ClientTrace` set in the executors.
## What changed?
<!-- Describe what has changed in this PR -->
Ports `TEMPORAL_TEST_LOG_FORMAT` and `TEMPORAL_TEST_LOG_LEVEL` to
testlogger.TestLogger.
## Why?
<!-- Tell your future self why have you made these changes -->
To not break developer setups.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
- Added a new function `softassert.That` that logs an error if the
provided condition is false
- Added `testlogger.Testlogger` (copied and slightly modified from our
internal repository) to functional tests
## Why?
<!-- Tell your future self why have you made these changes -->
To allow developers to verify pre- and post-conditions in their code.
**Why not panic?** Maybe in the future. For now, we're happy with
finding these failed assertions in functional tests.
**What about nightlies/long-haul tests?** I've filed a ticket to detect
a failed assertion there and surface them as an alert.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Planted a failing assertion.
**The error log:**
```
logger.go:146: 2025-03-04T20:50:08.578Z ERROR log/zap_logger.go:154 Unexpected Error log encountered: failed assertion: wake called twice {"host": "127.0.0.1:35427", "component": "matching-engine", "wf-task-queue-name": "/_sys/temporal-sys-processor-parent-close-policy/1", "wf-task-queue-type": "Workflow", "wf-namespace": "temporal-system", "worker-build-id": "_unversioned_", "failed-assertion": true, "logging-call-at": "/home/runner/work/temporal/temporal/common/log/with_logger.go:72"}
go.temporal.io/server/common/log.(*zapLogger).Error
/home/runner/work/temporal/temporal/common/log/zap_logger.go:154
go.temporal.io/server/common/testing/testlogger.(*TestLogger).Error
/home/runner/work/temporal/temporal/common/testing/testlogger/testlogger.go:359
go.temporal.io/server/common/log.(*withLogger).Error
/home/runner/work/temporal/temporal/common/log/with_logger.go:72
go.temporal.io/server/common/log.(*withLogger).Error
/home/runner/work/temporal/temporal/common/log/with_logger.go:72
go.temporal.io/server/common/log.(*withLogger).Error
/home/runner/work/temporal/temporal/common/log/with_logger.go:72
go.temporal.io/server/common/log.(*withLogger).Error
/home/runner/work/temporal/temporal/common/log/with_logger.go:72
go.temporal.io/server/common/softassert.That
/home/runner/work/temporal/temporal/common/softassert/softassert.go:50
go.temporal.io/server/service/matching.(*waitableMatchResult).wake
/home/runner/work/temporal/temporal/service/matching/matcher_data.go:560
```
https://github.com/temporalio/temporal/actions/runs/13662479940/job/38196953773?pr=7411#step:9:1638
**The test result:**
```
=== Failed
=== FAIL: tests TestPriorityFairnessSuite/TestPriority_Activity_Basic (0.00s)
functional_test_base.go:301: Running TestPriorityFairnessSuite/TestPriority_Activity_Basic in test shard 2/3
functional_test_base.go:373:
Error Trace: /home/runner/work/temporal/temporal/tests/testcore/functional_test_base.go:373
/home/runner/work/temporal/temporal/tests/testcore/functional_test_base.go:272
/home/runner/work/temporal/temporal/tests/testcore/functional_test_base.go:254
/home/runner/work/temporal/temporal/tests/testcore/functional_test_sdk_suite.go:97
/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.10.0/suite/suite.go:192
Error: Failing test as unexpected error logs were found.
Look for 'Unexpected Error log encountered'.
Test: TestPriorityFairnessSuite/TestPriority_Activity_Basic
```
https://github.com/temporalio/temporal/actions/runs/13662479940/job/38196953773?pr=7411#step:9:1053
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
---------
Co-authored-by: David Reiss <david@temporal.io>
Co-authored-by: David Reiss <dnr@dnr.im>
Co-authored-by: Tim Deeb-Swihart <409226+tdeebswihart@users.noreply.github.com>
## What changed?
<!-- Describe what has changed in this PR -->
Added additional HTTP tracing logs for the first 3 Nexus request
retries.
Both minimum and maximum attempts to add additional tracing info to are
configurable.
## Why?
<!-- Tell your future self why have you made these changes -->
To aid in debugging.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
- Always carry over the embedded `zapLogger`.
- Ensure `handler` is never `nil`.
- Allow extracting an underlying `slog.Logger`.
## Why?
Realized there are some issues using custom loggers, like the one used
by the CLI.
## What changed?
<!-- Describe what has changed in this PR -->
Add Matching and History changes to properly route unpinned workflow
tasks.
## Why?
<!-- Tell your future self why have you made these changes -->
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Tests will come in separate PR.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
- Implement UpdateWorkflowExecutionOptions API
## Why?
So users of versioning can override versioning behavior and deployment
for specific workflow executions for debugging or finer grained control.
## How did you test it?
- Unit tests for field mask logic
- Mutable state suite tests to confirm that Adding the event gets
applied correctly
- Todo: functional tests in a side branch using versioning-3 sdk, when
it's ready
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
---------
Co-authored-by: ShahabT <shahab.tajik@temporal.io>
Co-authored-by: Shahab Tajik <shahab@temporal.io>
## What changed?
<!-- Describe what has changed in this PR -->
**(1)** Adds the following OpenTelemetry (OTEL) span attributes:
- `temporalWorkflowID`
- `temporalRunID`
These are the same OTEL attribute keys the SDKs emit. It makes sense to
me to make them the same for consistency.
**(2)** Replaces the deprecated approach of OTEL interceptors with
`stats.Handler`s
**(3)** If there are no exporters, disables the `stats.Handlers`
completely. Right now we run through the OTEL processing even if it's
disabled.
## Why?
<!-- Tell your future self why have you made these changes -->
Right now there is no way to query all spans for a workflow together.
The newly added span attribute allows to correlate spans that belong to
the same workflow execution.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
(1) Added unit tests
(2) Manually
Using
```
docker run -p 127.0.0.1:4317:4317 -p 127.0.0.1:55679:55679 otel/opentelemetry-collector:0.106.1 2>&1 | tee collector-output.txt
```
it outputs
<img width="591" alt="image"
src="https://github.com/user-attachments/assets/0a1dfcc3-b3ae-49b7-bd0b-9667e8ef3945">
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
It appears to be **no** breaking change. (ie same OTEL attributes as
before)
<img width="2011" alt="Screenshot 2024-10-26 at 10 00 41 AM"
src="https://github.com/user-attachments/assets/c871eeac-ed7b-428e-9c0a-425c796d8761">
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
Add more logging around task queue user data propagation RPCs.
Also add a unit test that runs through the basic propagation logic.
## Why?
Add visibility to the process to help debug unusual behavior.
## How did you test it?
Added a unit test.
## What changed?
<!-- Describe what has changed in this PR -->
- Switch to uber gomock
- Generate workflowservice and operatorservice mock inside server repo,
instead of using imports from api-go to avoid making breaking changes
there.
## Why?
<!-- Tell your future self why have you made these changes -->
- google gomock is no longer maintained and doesn't support generics
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
Make import formatting a part of the build, and test validity with
"ensure-no-changes" target
## Why?
So we never discuss import order again (hopefully)
## How did you test it?
its a build change, so the build will test it!
## Potential risks
As long as build works, the only thing I risk is the ire of coworker who
want imports to be different ;)
## Documentation
N/A
## Is hotfix candidate?
No
## What changed?
This PR adds BackfillHistoryTask when SyncVersionedTransitionTask is not on current transition history.
## Why?
This PR is part of SyncVersionedTransitionTask.
## How did you test it?
unittest.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Clear sticky task queue on speculative WFT error.
## Why?
<!-- Tell your future self why have you made these changes -->
If, while completing WFT, error is occurred and returned to the worker
then worker will clear its cache. New WFT will also be created on sticky
task queue and sent to worker. Because worker doesn't have a workflow in
cache, it needs to replay it from the beginning, but sticky WFT has only
partial history. Worker will request full history using
`GetWorkflowExecutionHistory` API. This history doesn't have speculative
WFT events. If WFT is speculative, then worker will see inconsistency
between history from it and full history, and will fail WFT.
To prevent unexpected WFT failure, server clears stickiness for this
workflow, next WFT will go to normal task queue, and will have full
history attached to it.
This is NOT 100% bulletproof solution because this write operation may
also fail.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Run in test environment with fault injection enabled.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
Not 100% bulletproof solution. Future work to improve
`GetWorkflowExecutionHistory` is needed.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
No.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
