mirror of
https://github.com/temporalio/temporal.git
synced 2026-08-30 18:41:49 -07:00
WCP 1/X: Split history event batches by MaximumEventBatchSizeInBytes (#10357)
## What changed? New `MaximumEventBatchSizeInBytes` dynamic config, disabled by default. When set, EventStore rolls the current in-memory batch and starts a new one before appending an event that would push the cumulative serialized size over the threshold. A single oversized event still gets its own batch. If enabled things will break because of the incorrect event batch id references, so there is a big warning. ## Why? This is an initial step for support of workflow tasks completion requests larger than gRPC limit. Today, for such large requests the size of the batch could easily exceed the configured tx limit system.transactionSizeLimit. Because of that, we have to generate smaller batches, which this PR enables to do. There are many places in the code, where we assume that the event lands in the same batch as the corresponding WorkflowTaskCompleted event. So currently, if MaximumEventBatchSizeInBytes is set to >0, things will break. But adding this early on allows to find the issues in the code faster, by setting MaximumEventBatchSizeInBytes to a small value. ## How did you test it? - [X] built - [X] run locally and tested manually - [X] covered by existing tests - [X] added new unit test(s) - [ ] added new functional test(s) ## Potential risks This change doesn't have any effect unless MaximumEventBatchSizeInBytes is tuned from the default value of 0. The config comment explicitly says that this is experimental and things will break if it is enabled.
This commit is contained in:
committed by
GitHub
parent
25d19f4f8f
commit
e601ccd5e4
@@ -196,6 +196,7 @@ type Config struct {
|
||||
MaximumBufferedEventsBatch dynamicconfig.IntPropertyFn
|
||||
MaximumBufferedEventsSizeInBytes dynamicconfig.IntPropertyFn
|
||||
MaximumSignalsPerExecution dynamicconfig.IntPropertyFnWithNamespaceFilter
|
||||
MaximumEventBatchSizeInBytes dynamicconfig.IntPropertyFn
|
||||
|
||||
// ShardUpdateMinInterval is the minimum time interval within which the shard info can be updated.
|
||||
ShardUpdateMinInterval dynamicconfig.DurationPropertyFn
|
||||
@@ -637,6 +638,7 @@ func NewConfig(
|
||||
MaximumBufferedEventsBatch: dynamicconfig.MaximumBufferedEventsBatch.Get(dc),
|
||||
MaximumBufferedEventsSizeInBytes: dynamicconfig.MaximumBufferedEventsSizeInBytes.Get(dc),
|
||||
MaximumSignalsPerExecution: dynamicconfig.MaximumSignalsPerExecution.Get(dc),
|
||||
MaximumEventBatchSizeInBytes: dynamicconfig.MaximumEventBatchSizeInBytes.Get(dc),
|
||||
ShardUpdateMinInterval: dynamicconfig.ShardUpdateMinInterval.Get(dc),
|
||||
ShardFirstUpdateInterval: dynamicconfig.ShardFirstUpdateInterval.Get(dc),
|
||||
ShardUpdateMinTasksCompleted: dynamicconfig.ShardUpdateMinTasksCompleted.Get(dc),
|
||||
|
||||
Reference in New Issue
Block a user