## What changed?
<!-- Describe what has changed in this PR -->
Added namespace-filtered dynamic configuration that allows us to enable
additional error logging for a specific set of namespaces to aid in
investigatory work.
## Why?
<!-- Tell your future self why have you made these changes -->
When investigating certain issues, additional request error logging is
beneficial. Having the logging enabled at all times, however, is too
noisy.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
unit tests
---------
Co-authored-by: Yichao Yang <yichao@temporal.io>
## What changed?
<!-- Describe what has changed in this PR -->
Added dynamic configs to allow controlling the metrics breakdown based
on task queue name, partition ID, and Build ID.
Refactored all the places in the code that used these tags to consider
the dc value.
## Why?
<!-- Tell your future self why have you made these changes -->
- Task Queue cardinality may be a problem for some self-hosted users.
These configs allow to reduce cardinality.
- The dynamic config is per task queue so it allows to turn it on for
one or few task queues under investigation while disabled for everything
else.
- Simply excluding the tag value is not a good enough solution because
the gauges such as backlog size or age will be mixed across task queues
and become invalid. In this change, I disable the gauge completely if
the configs do not allow necessary breakdown.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Added few unit tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
- same as https://github.com/temporalio/temporal/pull/6316 but for
sequential scheduler
## Why?
<!-- Tell your future self why have you made these changes -->
- Simplify logic, avoid additional goroutines for polling dc update
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
Add `history.outboundQueuePendingTasksMaxCount` and
`history.outboundQueuePendingTaskCriticalCount` dynamic configs.
## Why?
The outbound queue runs tasks for multiple destinations with varying
latency and availability. We may need more control over the number of
pending tasks to prevent situations where tasks for unavailable
destinations block tasks that should be okay to process.
## What changed?
<!-- Describe what has changed in this PR -->
Add health check on history service role
## Why?
<!-- Tell your future self why have you made these changes -->
This health check checks on persistence latency and error rate.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
- Use DC subscriptions in fifo scheduler
- I will do sequential scheduler in a separate PR.
## Why?
<!-- Tell your future self why have you made these changes -->
- Simplify logic, avoid additional goroutines for polling dc update
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Clear all workflow contexts, found in the cache, that belong to a
History Shard that is being closed.
## Why?
<!-- Tell your future self why have you made these changes -->
Right now, when a History Shard is closed, any non-durable Update that
is still in-flight will be "lost", meaning that there is no way to
access it, cancel it or complete it anymore. The client that issued the
Update is not informed about this either (neither is the Frontend).
A better experience is to forcibly cancel any in-flight Updates
belonging to the History Shard that is closing. That way, the Frontend
can re-issue the Update request (which will land on a new History
Shard). The client won't even know that the History Shard was closed.
However, if the retry happens towards the end of the Update timeout, the
client's request might still time out after all.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Added unit tests and functional test.
Before going live, this will have to be validated by thorough end-to-end
testing.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
### Code Complexity
Since the workflow cache already has all the workflow contexts, I had
considered adding a by-shardID index a method to iterate over them
there. However, its cache implementation is based on `cache.Cache` (ie a
generic cache) it would have been awkward to add it there. Therefore, I
decided to track the workflow contexts in a separate place, the
Finalizer, which adds to the complexity.
### Memory Leak
There is no way to list the workflow contexts that belong to a History
Shard. Therefore, a new "Finalizer" was introduced that maintains a list
of cleanup callbacks: one callback per workflow context, grouped by
History Shard.
But since the workflow cache will eventually evict workflow contexts
that are no longer needed, if the Finalizer holds on to these cleanup
callbacks past the lifetime of the workflow context, the GC won't be
able to free them from memory. Therefore, a lifecycle hook was
introduced that removes any expired workflow contexts from the
Finalizer.
### Delayed History Shard Closure
The step to close the History Shard should not lock up if some/any of
the workflow contexts cannot be cleared. To clear them, the lock for the
workflow needs to be obtained. To prevent this from blocking
indefinitely, a timeout was introduced. It's better to make sure the
History Shard closes eventually than to clear all workflow contexts.
PS: Logging has been added to be able to monitor the timeouts and
completion rate of the Finalizer.
PPS: If any production issues should occur, the Finalizer's cleanup step
can be disabled entirely by setting the dynamic config for the timeout
to zero. However, the registering/de-registering of callbacks cannot be
disabled.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
Use dynamic config subscription for circuit breaker settings
## Why?
Faster response to changes, simpler code
## How did you test it?
existing tests
## What changed?
<!-- Describe what has changed in this PR -->
Wrap retryable errors from the outbound standby queue tasks with
`DestinationDownError`.
## Why?
<!-- Tell your future self why have you made these changes -->
`DestinationDownError` will trigger the circuit breaker, and minimizing
discarding standby tasks prematurely.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
- Add a feature flag for workflowIDReuse start time validation, default
to **_disabled_**
## Why?
<!-- Tell your future self why have you made these changes -->
- The start time validation introduces one more db read for start
workflow operation. This can be avoided by adding the start time to
workflow execution state.
- Also we should only get workflow start time when TerminateIfRunning
policy is used.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- Existing functional test `TestStartWorkflowExecution_Terminate`
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- Yes
## What changed?
Building on #6065, this PR introduces the bulk of scheduler logic to the
HSM-based prototype. This makes the prototype able to correctly start
workflows according to a schedule, but does not yet port the original
long-polling logic. The plan is to replace long-polling with HSM-based
callbacks in the subsequent PR. HSM also makes most of the side-effects
or activities based logic in the original scheduler obsolete.
## Why?
#6065
## How did you test it?
Unit tests & Functional Tests
## Potential risks
#6065
## Documentation
None
## Is hotfix candidate?
No
---------
Co-authored-by: Roey Berman <roey.berman@gmail.com>
## What changed?
<!-- Describe what has changed in this PR -->
Make WorkflowIdReuseMinimalInterval to be per-namespace, rather then
single global setting.
Also remove ContinueAsNewMinInterval setting (by reusing
WorkflowIdReuseMinimalInterval setting).
This effectively makes ContinueAsNewMinInterval setting be per-namespace
as well
## Why?
<!-- Tell your future self why have you made these changes -->
To have more flexibility in configuring this setting.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Update unit-tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
N/A
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No
## What changed?
<!-- Describe what has changed in this PR -->
1. Add ShardFirstUpdateInterval config setting
2. Initialize shard lastUpdated according to that setting
3. Slightly change the logic in updateShardInfo
## Why?
<!-- Tell your future self why have you made these changes -->
In a situations when shard is (re)created again and again shard update
(and thus storing shard progress) can be delayed.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Add batching on resend
## Why?
<!-- Tell your future self why have you made these changes -->
To reduce state transition and tasks created during resend
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Unit test
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
No risk.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
n/a
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
yes.
## What changed?
Modified the state machine node and ref definitions to support both
transition history and "old" staleness checks.
## Why?
Transition history requires a larger effort to sync across clusters and
should not block Nexus work.
## How did you test it?
Existing + new tests.
## What changed?
Only one replication task is generated even if there are multiple event
batches in one transaction on the active side.
## Why?
We need make sure events that belong to the same transaction on the
active side also belong to the same transaction on the standby side.
## How did you test it?
Tested locally. Added new tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
Added a new experimental code path that starts the scheduler logic using
the HSM framework. For now this code path simply replaces the workflow
with a dummy state machine that periodically prints. The plan is to
incrementally migrate the original scheduler logic over time.
## Why?
This showcases the capability of the HSM framework and can potentially
facilitate simplification of the history service logic by removing
internal workflows.
## How did you test it?
Unit tests
## Potential risks
Minimal as new code is hidden behind an experimental flag via dynamic
config. All changes take effect only under that flag.
## Documentation
N/A for now
## Is hotfix candidate?
No
---------
Co-authored-by: Roey Berman <roey.berman@gmail.com>
## What changed?
Use typed dynamic configs where we currently use maps that are converted
to structs.
## Why?
Simplify code, remove boilerplate.
## How did you test it?
existing tests
## Potential risks
1. If the conversion provided by mapstructure doesn't do the exact same
thing as the hand-written conversion code, some values may change
meaning.
2. In a couple cases I deliberately changed the semantics to match what
seemed to be intended behavior (noted in PR comments).
## What changed?
<!-- Describe what has changed in this PR -->
Main change - add some (configurable, default is 1 sec) grace period
when restarting workflow with the same runID.
If new run is starting too fast - return error rather then terminating
old run.
## Why?
<!-- Tell your future self why have you made these changes -->
This is not typical/expected behavior, but can negatively affect out
system.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Add specific unit tests.
Update existing unit/functional tests to cover new behavior.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
N/A
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
N/A
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
N/A
---------
Co-authored-by: Stephan Behnke <stephanos@users.noreply.github.com>
## What changed?
<!-- Describe what has changed in this PR -->
- Remove shard ownership assertion logic on workflow not found error
## Why?
<!-- Tell your future self why have you made these changes -->
- Persistence implementations in this repo don't need this assertion
logic as they always talk to DB directly.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- Existing logic
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- The feature flag for this has already been disabled for some time.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
- N/A
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Made `shardIOTimeout` configurable via dynamic config.
## Why?
<!-- Tell your future self why have you made these changes -->
Depending on the persistence backend, a longer timeout may be necessary
in some cases.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Existing tests should cover this change, and since the defaults are the
same as the prior values, no impact is expected. We will also test this
in a test environment.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
An improperly configured timeout could cause poor performance and hide
issues with the persistence layer.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- initial commit
- Initial implementation
- Fix
## What changed?
<!-- Describe what has changed in this PR -->
Add implementation for Replication stream flow control
## Why?
<!-- Tell your future self why have you made these changes -->
To allow receiver back pressure to sender
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
integration test
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
n/a
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
n/a
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
no
This is a re-revert of https://github.com/temporalio/temporal/pull/5797
now that all changes moving forward will be part of 1.25 release.
## Why?
The new logic to proxy all FE history read operations to the history
service has been part of release 1.23 and it is now safe (w.r.t. FE and
history services deployment ordering) as part of release 1.25 to start
deprecating the legacy code.
## What changed?
<!-- Describe what has changed in this PR -->
- added EnableShadowReadMode config (default to false)
- if shadow read mode is enabled, perform shadow read from the secondary
visibility store for visibility read requests
## Why?
<!-- Tell your future self why have you made these changes -->
- to measure latency metrics from both visibility stores
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- tested locally, and verified that visibility read requests from both
visibility stores are getting reasonably same latencies
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- Combined all Nexus related feature flags into a single
`system.enableNexus` dynamic config.
Dynamic configs that were _removed_ (some were namespace specific):
- `frontend.enableNexusAPIs`
- `frontend.enableCallbackAttachment`
- `history.enableMutableStateTransitionHistory`
- `history.outboundProcessorEnabled`
- `component.nexusoperations.enabled`
- Added support for callback header attachment and propagation
- WIP Support for `Worker` endpoint target - localhost only
- Added dynamic config for `cass` and `sql`:
```
- component.nexusoperations.callback.endpoint.template:
- value:
http://localhost:7243/api/v1/namespaces/{{.NamespaceName}}/nexus/callback
```
- Properly handle gRPC errors coming from matching in the frontend nexus
handler
- Consider invalid Nexus responses a downstream error
- Fix panic when trying to get event ID for `StateMachineTimerTask`
(used for logging)
- Don't return `errNoRecentPoller` for nexus tasks
- Remove namespace wait in functional tests (not Nexus specific)
## What changed?
add dynamic config for history.ReplicationTaskApplyTimeout
## Why?
We encountered "context deadline exceeded" when retrieving events from
tiered storage. Make this timeout configurable to unblock migration.
## How did you test it?
## Potential risks
## Documentation
## Is hotfix candidate?
- Initial commit
- Implement tiered replication sender
## What changed?
<!-- Describe what has changed in this PR -->
Implement tiered replication sender
## Why?
<!-- Tell your future self why have you made these changes -->
To allow sender to send task based on priority
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
unit test
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
n/a
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
n/a
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
no
## What changed?
<!-- Describe what has changed in this PR -->
Implement tiered replication stream receiver:
1. Change `receiver.receive` to handle original (single ACK level) and
new (prioritized ACK level) properly
2. Change `receiver.ACK` to properly ACK back to sender with proper ACK
level.
## Why?
<!-- Tell your future self why have you made these changes -->
To prioritize different category of replication tasks.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
integration test.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
Replication stack will go down.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
n/a
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
no
## What changed?
<!-- Describe what has changed in this PR -->
Circuit breaker with dynamic settings: it takes a function that is
evaluated whenever `Allow` is called, and if any change happened, then
it replaces the circuit breaker with the updated changes (previous state
is lost). This makes possible to have the circuit breaker automatically
update after a dynamic config change.
Added dynamic config for outbound queue circuit breaker.
## Why?
<!-- Tell your future self why have you made these changes -->
Be able to config by destination.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Dynamic config for the outbound queue host scheduler task rps
## Why?
<!-- Tell your future self why have you made these changes -->
Be able to config by destination.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Dynamic config for the outbound queue group limiter settings
## Why?
<!-- Tell your future self why have you made these changes -->
Have the config dynamic instead of hard typed in the code.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
Dynamic config improvements:
- Typed settings to prevent misuse
- Moved defaults to where settings are defined
- Code generation to make it easy to add new types and filters
- Shorter names for readability
## Why?
Making it easier to use, preventing misuse, making it easy to extend,
easier to parse and generate documentation from, base for future
enhancements (e.g. validation at load time)
## How did you test it?
modified existing tests
## Potential risks
typo or missed something in all the merges and refactors
## What changed?
<!-- Describe what has changed in this PR -->
- Add UnexpectedErrorAttempts tag to some logs.
- Reduce default value for history.TaskDLQUnexpectedErrorAttempts to 70
## Why?
<!-- Tell your future self why have you made these changes -->
More consistent logs. Also 100 attempts take +2hrs which is to long. 50
attempts take about 16 minutes.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
None
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
none
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
none
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
no
## What changed?
<!-- Describe what has changed in this PR -->
- Deprecate persistence priority rate limiting flag
## Why?
<!-- Tell your future self why have you made these changes -->
- Those flags have been enabled by default for several releases and is
stable.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- Existing tests
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- Won't be able to disable the feature in prod.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
- N/A
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- No.
## Why?
The new logic to proxy all FE history read operations to the history
service has been part of release 1.23 and it is now safe (w.r.t. FE and
history services deployment ordering) as part of release 1.24 to start
deprecating the legacy code.
## What changed?
Enable streaming replication by default.
## Why?
<!-- Tell your future self why have you made these changes -->
## How did you test it?
Tested locally.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
Move some code around to avoid import cycles with future changes:
- `common` depends on `common/dynamicconfig` (not changing)
- I want to move dynamic config default values from all over into
`common/dynamicconfig`
- Some constants in `common` are used as default values. Moving them
into `common/dynamicconfig` would create an import cycle.
- So move the constants into `common/primitives` instead and reference
them there.
Also move `GetDefaultRetryPolicyConfigOptions`, `DefaultRetrySettings`,
`FromConfigToDefaultRetrySettings`, `EnsureRetryPolicyDefaults`,
`ValidateRetryPolicy` and associated unit test into new package
`common/retrypolicy` with shorter names.
Arguably more constants should move out of `common`, that package is too
big. But these are the only ones that needed to be moved for now.
## Why?
Split off from future PR to make reviews easier.
## How did you test it?
existing tests
## What changed?
<!-- Describe what has changed in this PR -->
- Add new timer task type for workflow execution timeout & carry over
execution timeout timer state across runs in a workflow chain.
- This change depends on
https://github.com/temporalio/temporal/pull/5531 so that only one timer
task will be created on the standby side as well.
- New behavior is controlled via a feature flag.
## Why?
<!-- Tell your future self why have you made these changes -->
- For a workflow chain, we only need one execution timeout timer task
for the entire chain. Before this change, each run in the chain will
create a run timeout timer. If user set runTimeout == execution timeout
(which is the default if only execution timeout is specified!), then the
timeout timer for all the runs will fire at the exact same time (because
run timeout will be shortened to respect the execution timeout). This
can result in a timer burst for a single workflowID.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- Added unit tests
- Tested locally with cron workflows.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- In worst case, workflow may not timeout properly.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
- N/A, no change on behavior.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- No.
## What changed?
<!-- Describe what has changed in this PR -->
- Allow configuring persistence rate limiter burst ratio, default is
still 1, same as today.
## Why?
<!-- Tell your future self why have you made these changes -->
- It's hard to set rate limit value without burst support. We allow
incoming traffic to burst, but call to persistence can't burst, which
often lead to persistence rate limited errors.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- Existing tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- N/A, default burst is still 1, no behavior change by default.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
- N/A
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- No
## What changed?
<!-- Describe what has changed in this PR -->
- Deprecate EnableAPIGetCurrentRunIDLock flag and always perform the
locking.
## Why?
<!-- Tell your future self why have you made these changes -->
- Flag already enabled in prod for a long time and believe to be safe to
always enable the locking logic.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- N/A
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- N/A
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
- No changed required for OSS user.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- No.
## What changed?
<!-- Describe what has changed in this PR -->
- When workflow in the source cluster completes and starts a new run in
the same transaction. Replication should also apply that in one
transaction in standby cluster. Today this only happens when the current
workflow closes with continue as new event, but there are other cases
like cron and retry where workflow can close with
completed/timedout/failed event and also has a new run.
## Why?
<!-- Tell your future self why have you made these changes -->
- Transactional guarantee should be preserved by replication
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- New unit tests
- Run canary using global namespace.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- Replication task may go to dlq in worst case
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
- N/A
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- No.
## What changed?
Adding a new config history.DLQErrorSubStrings to specify substrings in
task processing error strings.
If the error contains this substring, this task will be enqueued to DLQ.
## Why?
Gives the ability to send tasks to DLQ based on errors.
## How did you test it?
Unit tests
## Potential risks
None
## Documentation
None
## Is hotfix candidate?
No
---------
Co-authored-by: Tim Deeb-Swihart <409226+tdeebswihart@users.noreply.github.com>
## What changed?
<!-- Describe what has changed in this PR -->
Dynamic config to suppress error when setting system search attributes
## Why?
<!-- Tell your future self why have you made these changes -->
Create a way to prevent errors when new system search attributes are
added, but there is already a custom search attribute registered with
the same name.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Added unit test.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed and why?
> // EnableMutableStateTransitionHistory controls whether to record
state transition history in mutable state records.
// The feature is used in the hierarchical state machine framework and
is considered unstable as the structure may
// change with the pending replication design.
## What changed?
Adding code for limiting MutableState cache using total size of entries.
Currently MutableState cache limit is specified by entry count.
This PR adds a new config `history.cacheSizeBasedLimit`. If this is set
to true, MutableState cache will limit entries using the total size of
the entries.
When `history.cacheSizeBasedLimit` is set to true, values of the flags
`history.cacheMaxSize` and `history.hostLevelCacheMaxSize` are the total
size of the cache in bytes instead of entry count.
Also adding a metric for persisted mutable state size.
## Why?
It is easier to control the total cache size by specifying size in bytes
rather than entry count. The size of mutable state varies largely and it
makes difficult to set a size limit using entry count.
## How did you test it?
Unit tests
## Potential risks
None
## Documentation
N/A
## Is hotfix candidate?
No
## Why?
Decided to extend the purpose of this queue for both operations and
callbacks.
Any immediate that task has an external destination should be scheduled
on this queue.
Category was unused and confusing and seems like there's no benefit in
having it for the foreseeable future.
## How did you test it?
Existing tests.
## Notes for reviewers
I don't consider this a final approach but I do think it's a step in
the right direction, we need to model more state machines on top of this
to form a more solid API.
Note that mutable state itself is registered as a state machine in the HSM framework and used as the root node in the HSM tree.
It's a bit annoying that all of the unit tests need to register the workflow state machine but I think this is ended up being the cleanest approach from the HSM framework perspective.
There's still some follow-up work, like collapsing timer tasks and support for interacting with history event, that I'll take on later.
Replication support for the framework will also come at a later time.
- See discussions in related PRs (#5446, #5495)
## What changed?
- Modified the statemachines abstraction to be a bit more generic
- Rewrote callbacks as a plugin using this framework
## Why?
This centralizes most the callback code in the plugin directory instead
of having it spread out the entire project moving common concerns such
as staleness checks, task generation, and (in the future) replication
into a framework and should generally help speed up feature delivery and
maintainability.
I plan to leverage this framework when implementing Nexus operations.
## How did you test it?
Existing tests from the feature branch and added unit tests.
