Github/temporal
1
0
Fork 0
mirror of https://github.com/temporalio/temporal.git synced 2026-08-30 18:41:49 -07:00
Code Issues Packages Projects Releases Wiki Activity

Make multi-cursor predicate shrink threshold dynamic and emit resolution-loss metric (#10489)

Browse Source 
## What changed?

- Replace the hardcoded `shrinkPredicateMaxPendingKeys = 3` with a new
dynamic config `history.queueShrinkPredicateMaxPendingKeys` (default
10), wired into all history queues.
- Emit a new `queue_predicate_resolution_loss` counter, tagged by
`reason`, in the two places a slice gives up exact predicate resolution:
when the shrink is skipped (`reason=max_pending_keys`) and when the
predicate falls back to the universal predicate
(`reason=predicate_size`).

## Why?

The shrink threshold was a magic constant with no way to tune it in
production, and there was no signal when a slice loses resolution and
starts reprocessing extra tasks. The metric makes both cases observable.

## How did you test it?
- [x] built
- [x] covered by existing tests
This commit is contained in:
Prathyush PV
2026-06-25 16:26:28 -07:00
committed by GitHub
parent 7610450278
commit 365117576e
13 changed files with 91 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
service/history/configs/config.go
View File

@@ -103,13 +103,14 @@ type Config struct {
StandbyTaskMissingEventsDiscardDelay dynamicconfig.DurationPropertyFnWithTaskTypeFilter
ChasmStandbyTaskDiscardDelay dynamicconfig.DurationPropertyFnWithChasmTaskTypeFilter
QueuePendingTaskCriticalCount dynamicconfig.IntPropertyFn
QueueReaderStuckCriticalAttempts dynamicconfig.IntPropertyFn
QueueCriticalSlicesCount dynamicconfig.IntPropertyFn
QueuePendingTaskMaxCount dynamicconfig.IntPropertyFn
QueueMaxPredicateSize dynamicconfig.IntPropertyFn
QueueMoveGroupTaskCountBase dynamicconfig.IntPropertyFn
QueueMoveGroupTaskCountMultiplier dynamicconfig.FloatPropertyFn
QueuePendingTaskCriticalCount dynamicconfig.IntPropertyFn
QueueReaderStuckCriticalAttempts dynamicconfig.IntPropertyFn
QueueCriticalSlicesCount dynamicconfig.IntPropertyFn
QueuePendingTaskMaxCount dynamicconfig.IntPropertyFn
QueueMaxPredicateSize dynamicconfig.IntPropertyFn
QueueShrinkPredicateMaxPendingKeys dynamicconfig.IntPropertyFn
QueueMoveGroupTaskCountBase dynamicconfig.IntPropertyFn
QueueMoveGroupTaskCountMultiplier dynamicconfig.FloatPropertyFn
TaskDLQEnabled dynamicconfig.BoolPropertyFn
TaskDLQUnexpectedErrorAttempts dynamicconfig.IntPropertyFn
@@ -525,13 +526,14 @@ func NewConfig(
StandbyTaskMissingEventsDiscardDelay: dynamicconfig.StandbyTaskMissingEventsDiscardDelay.Get(dc),
ChasmStandbyTaskDiscardDelay: dynamicconfig.ChasmStandbyTaskDiscardDelay.Get(dc),
QueuePendingTaskCriticalCount: dynamicconfig.QueuePendingTaskCriticalCount.Get(dc),
QueueReaderStuckCriticalAttempts: dynamicconfig.QueueReaderStuckCriticalAttempts.Get(dc),
QueueCriticalSlicesCount: dynamicconfig.QueueCriticalSlicesCount.Get(dc),
QueuePendingTaskMaxCount: dynamicconfig.QueuePendingTaskMaxCount.Get(dc),
QueueMaxPredicateSize: dynamicconfig.QueueMaxPredicateSize.Get(dc),
QueueMoveGroupTaskCountBase: dynamicconfig.QueueMoveGroupTaskCountBase.Get(dc),
QueueMoveGroupTaskCountMultiplier: dynamicconfig.QueueMoveGroupTaskCountMultiplier.Get(dc),
QueuePendingTaskCriticalCount: dynamicconfig.QueuePendingTaskCriticalCount.Get(dc),
QueueReaderStuckCriticalAttempts: dynamicconfig.QueueReaderStuckCriticalAttempts.Get(dc),
QueueCriticalSlicesCount: dynamicconfig.QueueCriticalSlicesCount.Get(dc),
QueuePendingTaskMaxCount: dynamicconfig.QueuePendingTaskMaxCount.Get(dc),
QueueMaxPredicateSize: dynamicconfig.QueueMaxPredicateSize.Get(dc),
QueueShrinkPredicateMaxPendingKeys: dynamicconfig.QueueShrinkPredicateMaxPendingKeys.Get(dc),
QueueMoveGroupTaskCountBase: dynamicconfig.QueueMoveGroupTaskCountBase.Get(dc),
QueueMoveGroupTaskCountMultiplier: dynamicconfig.QueueMoveGroupTaskCountMultiplier.Get(dc),
TaskDLQEnabled: dynamicconfig.HistoryTaskDLQEnabled.Get(dc),
TaskDLQUnexpectedErrorAttempts: dynamicconfig.HistoryTaskDLQUnexpectedErrorAttempts.Get(dc),