mirror of
https://github.com/temporalio/temporal.git
synced 2026-08-30 18:41:49 -07:00
5406711d68
> **Part 3 of a 5-PR series** building to replication stream namespace
isolation (a restructuring of #10147): read buffer → reader group → lane
protocol → isolation manager → sender isolation.
> #11302 (reader group) has merged, so this PR's diff is now standalone
against `main`. · **Next in series: #11304** (isolation manager).
## What changed?
The wire-level building blocks for per-namespace lane isolation,
receiver side only:
- **Proto**: `SyncReplicationState` gains `throttle_high_namespace_ids`
(namespaces the receiver reports as overwhelming the HIGH lane — the
priority lives in the field name, so a future LOW extension adds its own
field rather than widening this one), `isolated_lane_states` (per-lane
applied watermarks keyed by namespace ID — with the documented caveat
that a missing key is ambiguous between "not tracked yet" and "retired
and drained", so a sender must never treat absence alone as drain
proof), and `supports_namespace_isolation` (capability advertisement, so
a sender never emits lane-tagged traffic to a receiver that would
misroute it). `WorkflowReplicationMessages` gains
`isolated_namespace_id` — when set, the batch belongs to that
namespace's dedicated lane — and `retire_isolated_lane`, marking a
lane's final message.
- **Receiver**: lane-tagged batches route to lazily-created
per-namespace task trackers. Each lane is its own monotonic stream for
the life of the connection — there is no rewind or rotation machinery,
because the sender-side design (later in the series) gives every lane a
single owner cursor that never goes backwards. Member-lane watermarks
fold into the overall ack minimum (cleanup safety) and are reported per
lane; member-lane backlogs count toward HIGH flow control. Lane
lifecycle is defensive about ordering: a batch's tasks are tracked
BEFORE its retire flag is applied (so the concurrent ack loop can never
delete a lane whose final batch is mid-track), a retiring lane is only
dropped once it is drained AND has tracked at least one batch, and
non-retire traffic arriving on a retiring lane revives it (the sender
re-isolated the namespace before the lane drained). Lane-tagged traffic
at any priority other than HIGH is a protocol violation and fails the
stream rather than silently mis-acking (isolation splits the HIGH lane
only). Lanes created concurrently with `Stop()` are pre-cancelled so no
tasks run after shutdown.
- **`NamespaceThrottler`** interface (default: noop, via fx) observes
per-namespace HIGH-priority task load and decides which namespaces to
report.
The sender does not tag lanes yet, so this is inert until the
sender-side isolation lands.
## Why?
Isolation needs a wire contract before the sender can use it: capability
advertisement, per-lane routing and progress reporting, and the
throttled-namespace feedback channel. Landing the receiver first makes
mixed-version clusters safe by construction. Compared to #10147, lanes
are per-namespace rather than shared per severity tier — which is what
eliminates that design's cursor rewinds and the
watermark-regression/tracker-rotation protocol this PR previously needed
to compensate for them.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s) — lane routing (priority routing when
unset, per-namespace tracker identity, non-HIGH rejection), retirement
lifecycle (drop once drained, never-tracked retiring lane survives the
ack snapshot, revive on re-isolation traffic, fresh lane after drop),
and post-Stop lane creation being pre-cancelled
- [x] added new functional test(s) — exercised end-to-end by the xdc
test in the final PR of the series
## Potential risks
Inert until a sender emits `isolated_namespace_id`, which is gated
behind both a config flag and the capability advertisement.
Receiver-side lane state is bounded by the sender's isolation cap (final
PR).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches replication ack watermarks and stream failure paths on the
passive cluster; lane mis-handling could stall cleanup or mis-ack,
though lane-tagged traffic is not sent until follow-up sender work.
>
> **Overview**
> Adds the **wire contract and receiver behavior** for per-namespace
HIGH-lane isolation before the sender starts tagging traffic.
>
> **Proto:** `SyncReplicationState` now carries
`throttle_high_namespace_ids`, per-namespace `isolated_lane_states`, and
`supports_namespace_isolation`. `WorkflowReplicationMessages` adds
`isolated_namespace_id` and `retire_isolated_lane` so batches can be
routed and retired on dedicated lanes.
>
> **Receiver:** Lane-tagged HIGH batches use lazily created
per-namespace task trackers (monotonic per connection). Member-lane
watermarks fold into the overall ack minimum and are reported per
namespace; member backlog counts toward HIGH flow control. Acks include
shard-scoped throttled namespace IDs via a new **`NamespaceThrottler`**
(noop by default). Lane lifecycle handles retire/drain/revive and
rejects non-HIGH lane traffic.
>
> Sender emission of lane tags is not in this PR, so behavior stays
inert until a later change gates on capability advertisement.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
688a5173f0. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>