59d3bccb2c
…oaded ## What changed? MatchingService supports gRPC ForceLoadTaskQueuePartition. Whenever matching engine loads a root partition, the RPC is called for all non-root, non-sticky partitions of the same taskQueue. ## Why? This addresses a unique situation where a TaskQueue has acquired a backlog, and has been unloaded because there are no workers polling the TaskQueue. When a worker comes back online, it can take a series of polls before it is load balanced to all partitions, forcing them to load. This can delay non-root partitions from attempting a sync-match with the poller. Forcing the partitions to load when the root partition is loaded can prevent this entirely because the root partition will be forced to load when the poll triggers a sync-match. ## How did you test it? Start workflow with sleep. Allow worker to handler workflow initially, then kill it. Wait for partitions to be unloaded. Start worker back up, watch task get picked up immediately. ## Potential risks The new gRPC method and triggering code could be implemented incorrectly and cause a cascade of calls. Worst case scenario would be perpetual loop from root-partition to root-partition. Additionally, we could have an unforeseen case where we call this RPC far more often than necessary. Metrics have been installed to help us keep an eye on that. ## Documentation Matching Service documentation has been expanded to cover task queue partitions. ## Is hotfix candidate? nope.
1.9 KiB
Matching Service
[see API definition]
Task Queues
Matching Service instances manage Task Queues being polled by Temporal Worker processes. Long-poll requests from Temporal Workers are received by the Frontend Service, which routes them to the Matching Service instance responsible for the requested Task Queue. The Matching Service instance responds by sending Workflow Tasks and Activity Tasks from the requested Task Queue. A single Task Queue is responsible for delivering tasks relating to many Workflow Executions.
Task Queue Partitions
Matching Service splits Task Queues into partitions to provide higher throughput overall. Default is 4, but there can be less or more. Partition ownership can be reassigned, and partitions' metadata and task backlog can be loaded/unloaded from storage.
When throughput of tasks is low or polling by workers is rare, a polling worker can be "forwarded" from an empty partition to its parent partition. This is also true for a task on a partition which is not being polled, the task can be "forwarded" to a parent partition, hoping to find a poller. For small numbers of partitions, the root partition is the direct parent of all children, but with more partitions, the parent relationship forms a tree of depth > 2, converging at the root partition. If a root partition of a Task Queue is loaded, this will force all other partitions of that Task Queue to also load. This ensures that forwarding can occur between a child partition with a task in its backlog and a long-awaited poller.
Additional Documentation of Matching Service internals is not yet available.