WCP 1/X: Split history event batches by MaximumEventBatchSizeInBytes (#10357)

## What changed?
New `MaximumEventBatchSizeInBytes` dynamic config, disabled by default.
When set, EventStore rolls the current in-memory batch and starts a new
one before appending an event that would push the cumulative serialized
size over the threshold. A single oversized event still gets its own
batch. If enabled things will break because of the incorrect event batch
id references, so there is a big warning.

## Why?
This is an initial step for support of workflow tasks completion
requests larger than gRPC limit. Today, for such large requests the size
of the batch could easily exceed the configured tx limit
system.transactionSizeLimit. Because of that, we have to generate
smaller batches, which this PR enables to do.

There are many places in the code, where we assume that the event lands
in the same batch as the corresponding WorkflowTaskCompleted event. So
currently, if MaximumEventBatchSizeInBytes is set to >0, things will
break. But adding this early on allows to find the issues in the code
faster, by setting MaximumEventBatchSizeInBytes to a small value.

## How did you test it?
- [X] built
- [X] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)

## Potential risks
This change doesn't have any effect unless MaximumEventBatchSizeInBytes
is tuned from the default value of 0. The config comment explicitly says
that this is experimental and things will break if it is enabled.
This commit is contained in:
Vladyslav Simonenko
2026-06-05 09:14:06 -07:00
committed by GitHub
parent 25d19f4f8f
commit e601ccd5e4
8 changed files with 226 additions and 49 deletions

View File

@@ -196,6 +196,7 @@ type Config struct {
MaximumBufferedEventsBatch dynamicconfig.IntPropertyFn
MaximumBufferedEventsSizeInBytes dynamicconfig.IntPropertyFn
MaximumSignalsPerExecution dynamicconfig.IntPropertyFnWithNamespaceFilter
MaximumEventBatchSizeInBytes dynamicconfig.IntPropertyFn
// ShardUpdateMinInterval is the minimum time interval within which the shard info can be updated.
ShardUpdateMinInterval dynamicconfig.DurationPropertyFn
@@ -637,6 +638,7 @@ func NewConfig(
MaximumBufferedEventsBatch: dynamicconfig.MaximumBufferedEventsBatch.Get(dc),
MaximumBufferedEventsSizeInBytes: dynamicconfig.MaximumBufferedEventsSizeInBytes.Get(dc),
MaximumSignalsPerExecution: dynamicconfig.MaximumSignalsPerExecution.Get(dc),
MaximumEventBatchSizeInBytes: dynamicconfig.MaximumEventBatchSizeInBytes.Get(dc),
ShardUpdateMinInterval: dynamicconfig.ShardUpdateMinInterval.Get(dc),
ShardFirstUpdateInterval: dynamicconfig.ShardFirstUpdateInterval.Get(dc),
ShardUpdateMinTasksCompleted: dynamicconfig.ShardUpdateMinTasksCompleted.Get(dc),