## What changed?
Added a missing `d.syncSummary()` call in
`syncVersionDataToComputeStatus`, so it now notifies the parent
Deployment workflow after pulling a compute status from WCI.
## Why?
Without this, the pull only updates the Version workflow's own state.
The Deployment workflow (which
`ListWorkerDeployments`/`DescribeWorkerDeployment` actually read from)
is not updated, so `computeStatus` can stay permanently missing from the
API even when the data is available.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Flipped the activity.enableStandalone dynamic config default from false
to true in chasm/lib/activity/config.go. Since standalone activity is
now on by default (alongside the already-defaulted-true
history.enableTransitionHistory and history.enableChasm), removed the
now-redundant explicit overrides for all three settings from the
development config files: development-cass.yaml, development-sql.yaml,
and development-xdc.yaml.
## Why?
Standalone activity is ready to be on by default, so the server default
is updated to reflect that. With all three settings now defaulting to
true in code, the explicit value: true overrides in the dev configs no
longer do anything and only add noise, so they're removed to keep the
dev configs minimal and avoid drift from the code defaults.
## How did you test it?
- [X] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Use the new dynamic config instead of leaving it as a placeholder and
update its documentation.
Also bumps the server version constant to `1.32.0`. This was supposed to
be done after the `1.31.0` release and was missed.
## Why?
Remove duplication as we migrate the code to the CHASM backed
implementation.
## Potential risks
This config is only relevant for external endpoints which are
experimental or older server versions, needs to be called out in the
release notes.
## What changed?
Deleted `"system.enableNexus"`.
## Why?
Nexus has been GA since Dec 2024.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add dynamic config to toggle standalone activity functionality.
Refactored frontend dynamic config into activity config. Removed
standalone activity dc prefix `chasm`
## Why?
We need the ability to toggle standalone activity functionality as we go
to prod. We also agreed to remove the `chasm` prefix from any chasm
originated dynamic config keys
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Change dev environment to enable Chasm
## Why?
Chasm should be enabled for development by default so devs don't have to
explicitly set the config.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Add revision number to Worker Deployment Version Data that is synced to
Task Queues.
## Why?
This allows making task queue registration also async, but more
importantly, it prevent's race conditions between concurrent
registrations, setCurrent/Ramping, drainage, and deletion.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None.
## What changed?
WISOTT
## Why?
Prevent incompatible changes in the future.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None
## What changed?
- When a deactivated (drained/draining) version gets rollbacked to
becoming active again, the drainage information that was associated with
it will get cleared.
- This is required because when this version were to get deactivated
again, we need the code to start the drainage process on it again. Right
now, it was not doing that since we were not clearing the drainage
information which did not start a fresh drainage tracker for this
version!
*NOTE*: This change requires a patch to be present. This is because when
this scenario is replicated in production, it will result in two new
history calls (syncSummary as well as a CAN of the version workflow).
This will specifically happen when the rolled-back version gets
deactivated eventually, over
[here](ada5d0bbee/service/worker/workerdeployment/version_workflow.go (L546))
- note how previously, since the drainage information was not cleared,
[this](ada5d0bbee/service/worker/workerdeployment/version_workflow.go (L261))
was never executed!)
*How did I verify that this requires a patch*:
- Ran the replay test script before the fix introduced in this PR and
noticed:
<img width="793" height="79" alt="image"
src="https://github.com/user-attachments/assets/bb975d26-4ad1-4847-b477-b74c7d2f55e4"
/>
- Ran the replay test script after the fix introduced in this PR and
noticed:
<img width="911" height="99" alt="image"
src="https://github.com/user-attachments/assets/ad3ff3e1-666a-49b4-bda4-e05f3412d984"
/>
In other words, the total number of our entity workflows went up after
introducing this change and thus requires a patch gate!
## Why?
- we found a bug in the versioning entity workflows and this is an
effort to fix this!
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
## Potential risks
- yes, there is a risk of NDE's happening in production but I mitigated
that risk by placing the fix against a version gate.
## What changed?
- Enable transition history in dev env and tests
- Reset state transition count when reporting nexus operation completion to a reset run
## Why?
- Exercise transition history and state-based replication path more
often
- Make chasm development and testing easier since it relies on
transition history
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
<!-- Describe what has changed in this PR -->
- Enable execution timeout timer by default
## Why?
<!-- Tell your future self why have you made these changes -->
- it's been several server releases since this feature is merged, it's
safe to enable it by default now without any backward compatibility
concerns
- Will remove the flag after next oss release if not issue reported
about it.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- Existing tests
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- Very low. Already enabled in all production envs
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
- A better more robust replay tester covering the different API's made
possible in worker-versioning 3.1
## Why?
<!-- Tell your future self why have you made these changes -->
- To ensure that NDE changes made in these entity wf's can be caught in
the near future
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- These are tests
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
UserDataManger in child partitions constantly fetches data from the
parent partition. There is a safely mechanism to prevent busy loop in
case the parent returns result immediately instead of only when the data
is updated.
The safety mechanism did not work well with Versioning 3 because many
updates are involved in a single tests. With this update we relax the
safety check to only apply when the user data version remains the same
and continue to fetch the data asap as long as each time it gets new
data.
Here, also I'm relaxing the "requested task queue user data for version
greater than known version" errors to be thrown only for the root
partition, as they can normally happen for child partitions.
## Why?
<!-- Tell your future self why have you made these changes -->
So that back-to-back Versioning 3 APIs do not have to wait for a long
time.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Existing tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
Yes.
## What changed?
<!-- Describe what has changed in this PR -->
Enable Update Workflow (sync and async) by default.
Note that I'm leaving the flag and the code checks for now, in case we
need to disable it again.
## Why?
<!-- Tell your future self why have you made these changes -->
For public preview.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Added a new per-namespace dynamic config for allowed callback addresses.
This dynamic config is a list of maps that currently supports two
entries:
* `Pattern:string` The host:port that this rule applies to. It is a
pattern because wildcard (`*`) matching is supported.
* `AllowInsecure:bool` Indicates whether `https` is required for this
address.
Added validation when processing StartWorkflow requests to ensure that
any attached callback URLs have a matching allowed address entry and
that the scheme matches the `AllowInsecure` setting.
## Why?
<!-- Tell your future self why have you made these changes -->
To be able to white-list allowed callback endpoints.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Updated functional tests.
This is a re-revert of https://github.com/temporalio/temporal/pull/5797
now that all changes moving forward will be part of 1.25 release.
## Why?
The new logic to proxy all FE history read operations to the history
service has been part of release 1.23 and it is now safe (w.r.t. FE and
history services deployment ordering) as part of release 1.25 to start
deprecating the legacy code.
## What changed?
We had added a configuration flag to make MutableState cache a
host-level cache.
Changing dynamic config to enable this flag by default and adjusting
cache size to 128K. We found that this size is appropriate for most
workloads we see.
Also setting host-level cache size to be 8K in development dynamic
config.
## Why?
Host level cache is better for cache utilization and it prevents the
problem of a smaller shard-level cache getting full with pinned
elements.
## How did you test it?
Running the cluster with load.
## Potential risks
None
## Documentation
## Is hotfix candidate?
No
- Combined all Nexus related feature flags into a single
`system.enableNexus` dynamic config.
Dynamic configs that were _removed_ (some were namespace specific):
- `frontend.enableNexusAPIs`
- `frontend.enableCallbackAttachment`
- `history.enableMutableStateTransitionHistory`
- `history.outboundProcessorEnabled`
- `component.nexusoperations.enabled`
- Added support for callback header attachment and propagation
- WIP Support for `Worker` endpoint target - localhost only
- Added dynamic config for `cass` and `sql`:
```
- component.nexusoperations.callback.endpoint.template:
- value:
http://localhost:7243/api/v1/namespaces/{{.NamespaceName}}/nexus/callback
```
- Properly handle gRPC errors coming from matching in the frontend nexus
handler
- Consider invalid Nexus responses a downstream error
- Fix panic when trying to get event ID for `StateMachineTimerTask`
(used for logging)
- Don't return `errNoRecentPoller` for nexus tasks
- Remove namespace wait in functional tests (not Nexus specific)
## Why?
The new logic to proxy all FE history read operations to the history
service has been part of release 1.23 and it is now safe (w.r.t. FE and
history services deployment ordering) as part of release 1.24 to start
deprecating the legacy code.
## What changed?
<!-- Describe what has changed in this PR -->
- Add new timer task type for workflow execution timeout & carry over
execution timeout timer state across runs in a workflow chain.
- This change depends on
https://github.com/temporalio/temporal/pull/5531 so that only one timer
task will be created on the standby side as well.
- New behavior is controlled via a feature flag.
## Why?
<!-- Tell your future self why have you made these changes -->
- For a workflow chain, we only need one execution timeout timer task
for the entire chain. Before this change, each run in the chain will
create a run timeout timer. If user set runTimeout == execution timeout
(which is the default if only execution timeout is specified!), then the
timeout timer for all the runs will fire at the exact same time (because
run timeout will be shortened to respect the execution timeout). This
can result in a timer burst for a single workflowID.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- Added unit tests
- Tested locally with cron workflows.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- In worst case, workflow may not timeout properly.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
- N/A, no change on behavior.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- No.
## What changed?
<!-- Describe what has changed in this PR -->
- When workflow in the source cluster completes and starts a new run in
the same transaction. Replication should also apply that in one
transaction in standby cluster. Today this only happens when the current
workflow closes with continue as new event, but there are other cases
like cron and retry where workflow can close with
completed/timedout/failed event and also has a new run.
## Why?
<!-- Tell your future self why have you made these changes -->
- Transactional guarantee should be preserved by replication
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- New unit tests
- Run canary using global namespace.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- Replication task may go to dlq in worst case
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
- N/A
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- No.
## What changed?
Adding a sampled logging when a workflow query task times out.
## Why?
Need this to be able to debug issues in the future when workflow queries
timeout.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
This PR is intended to merge 4 months of work in the `nexus` feature
branch into `main`.
The functionality it brings is:
- Dispatching Nexus Tasks by namespace and task queue
- Internal implementation of the Incoming Service Registry - not yet
exposed
- Attaching workflow close callbacks on `StartWorkflowExecutionRequest`
and processing of those callbacks
There's more to come and some of the callback code will be refactored
into a new plugin architecture
(https://github.com/temporalio/temporal/pull/5446) soon.
---------
Co-authored-by: PJ Doerner <pj.doerner@temporal.io>
<!-- Describe what has changed in this PR -->
All new code is gated behind the frontend.accessHistoryFraction dynamic
config. The new code is meant to be a port of the original FE code, but
primarily executed by the History service moving forward. Because this
change affects both FE and History, it requires a prolonged deprecation
process where:
- upon the 1st minor release (upcoming), the dynamic config must remain
disabled by default
- by the 2nd minor release, the dynamic config to be switched to be
enabled by default
- starting from the 3rd minor release, the deprecated code will be
removed
This change is staged into the following 4 major steps:
- DEPRECATE - refactor FE logic/tests into deprecated files (no logic
change)
- DUPLICATE - dup deprecated logic to history service (verbatim
copy/paste)
- PLUMBING - introduce rpc and handlers along with basic necessities
(hooking up)
- IMPORTANT - call-by-call replacement (requires careful review)
<!-- Tell your future self why have you made these changes -->
This is a big refactor to move business logic from FE to History service.
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- Ported FE unit tests to the History service layer
- Spot checked by running in dev environment
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
