> **Part 3 of a 5-PR series** building to replication stream namespace
isolation (a restructuring of #10147): read buffer → reader group → lane
protocol → isolation manager → sender isolation.
> #11302 (reader group) has merged, so this PR's diff is now standalone
against `main`. · **Next in series: #11304** (isolation manager).
## What changed?
The wire-level building blocks for per-namespace lane isolation,
receiver side only:
- **Proto**: `SyncReplicationState` gains `throttle_high_namespace_ids`
(namespaces the receiver reports as overwhelming the HIGH lane — the
priority lives in the field name, so a future LOW extension adds its own
field rather than widening this one), `isolated_lane_states` (per-lane
applied watermarks keyed by namespace ID — with the documented caveat
that a missing key is ambiguous between "not tracked yet" and "retired
and drained", so a sender must never treat absence alone as drain
proof), and `supports_namespace_isolation` (capability advertisement, so
a sender never emits lane-tagged traffic to a receiver that would
misroute it). `WorkflowReplicationMessages` gains
`isolated_namespace_id` — when set, the batch belongs to that
namespace's dedicated lane — and `retire_isolated_lane`, marking a
lane's final message.
- **Receiver**: lane-tagged batches route to lazily-created
per-namespace task trackers. Each lane is its own monotonic stream for
the life of the connection — there is no rewind or rotation machinery,
because the sender-side design (later in the series) gives every lane a
single owner cursor that never goes backwards. Member-lane watermarks
fold into the overall ack minimum (cleanup safety) and are reported per
lane; member-lane backlogs count toward HIGH flow control. Lane
lifecycle is defensive about ordering: a batch's tasks are tracked
BEFORE its retire flag is applied (so the concurrent ack loop can never
delete a lane whose final batch is mid-track), a retiring lane is only
dropped once it is drained AND has tracked at least one batch, and
non-retire traffic arriving on a retiring lane revives it (the sender
re-isolated the namespace before the lane drained). Lane-tagged traffic
at any priority other than HIGH is a protocol violation and fails the
stream rather than silently mis-acking (isolation splits the HIGH lane
only). Lanes created concurrently with `Stop()` are pre-cancelled so no
tasks run after shutdown.
- **`NamespaceThrottler`** interface (default: noop, via fx) observes
per-namespace HIGH-priority task load and decides which namespaces to
report.
The sender does not tag lanes yet, so this is inert until the
sender-side isolation lands.
## Why?
Isolation needs a wire contract before the sender can use it: capability
advertisement, per-lane routing and progress reporting, and the
throttled-namespace feedback channel. Landing the receiver first makes
mixed-version clusters safe by construction. Compared to #10147, lanes
are per-namespace rather than shared per severity tier — which is what
eliminates that design's cursor rewinds and the
watermark-regression/tracker-rotation protocol this PR previously needed
to compensate for them.
## How did you test it?
- [x] built
- [x] covered by existing tests
- [x] added new unit test(s) — lane routing (priority routing when
unset, per-namespace tracker identity, non-HIGH rejection), retirement
lifecycle (drop once drained, never-tracked retiring lane survives the
ack snapshot, revive on re-isolation traffic, fresh lane after drop),
and post-Stop lane creation being pre-cancelled
- [x] added new functional test(s) — exercised end-to-end by the xdc
test in the final PR of the series
## Potential risks
Inert until a sender emits `isolated_namespace_id`, which is gated
behind both a config flag and the capability advertisement.
Receiver-side lane state is bounded by the sender's isolation cap (final
PR).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches replication ack watermarks and stream failure paths on the
passive cluster; lane mis-handling could stall cleanup or mis-ack,
though lane-tagged traffic is not sent until follow-up sender work.
>
> **Overview**
> Adds the **wire contract and receiver behavior** for per-namespace
HIGH-lane isolation before the sender starts tagging traffic.
>
> **Proto:** `SyncReplicationState` now carries
`throttle_high_namespace_ids`, per-namespace `isolated_lane_states`, and
`supports_namespace_isolation`. `WorkflowReplicationMessages` adds
`isolated_namespace_id` and `retire_isolated_lane` so batches can be
routed and retired on dedicated lanes.
>
> **Receiver:** Lane-tagged HIGH batches use lazily created
per-namespace task trackers (monotonic per connection). Member-lane
watermarks fold into the overall ack minimum and are reported per
namespace; member backlog counts toward HIGH flow control. Acks include
shard-scoped throttled namespace IDs via a new **`NamespaceThrottler`**
(noop by default). Lane lifecycle handles retire/drain/revive and
rejects non-HIGH lane traffic.
>
> Sender emission of lane tags is not in this PR, so behavior stays
inert until a later change gates on capability advertisement.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
688a5173f0. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
## What changed?
Adds `buf format` as a Makefile target; and integrates it into `make
fmt`.
All `.proto` changes are from running `make fmt`.
## Why?
Consistent protobuf file style.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Use plain go struct inside migration workflow activity input output
- Customize the json encoding/decoding for migration execution info and
make it backward & forward compatible.
- This will temporary break cloud and the fix is in
https://github.com/temporalio/temporal/pull/9097
- This PR needs to be part of oss v1.30 release.
## Why?
- Backward compatibility and less confusion over how the encoding will
be done.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Use archetypeID everywhere in history service and pass them to
persistence
## Why?
- Required for separate ID space work.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
No-oping close transfer tasks for SyncWorkflowState tasks
## Why?
We need non state based replication to be eligible for this
optimization.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
`go test -v -tags test_dep ./tests/xdc -run
TestStreamBasedReplicationTestSuite/DisableTransitionHistory/TestCloseTransferTaskAckedReplication
-timeout 10m -count=1`
shows
`2025-10-20T08:15:08.290-0700 info Skipping close transfer task
generation - already acked on active cluster {"cluster-name":
"standby_aadnd", "host": "127.0.0.1:57179", "shard-id": 1, "address":
"127.0.0.1:57179", "wf-namespace-id":
"e550305e-0b43-4bcd-a490-8e3223f51ce1", "wf-id":
"test-replication-e2c094d3-c34f-42d9-a166-a967d4e7f602", "wf-run-id":
"019a0230-299a-74ba-a31f-ac247c05f2b9", "logging-call-at":
"/Users/michaely520/projects/temporal/service/history/workflow/task_generator.go:206"}
stream_based_replication_test.go:975: Verified IsCloseTransferTaskAcked
and IsForceReplication flags in SyncWorkflowStateTask`
## What changed?
1. Impl to write IsCloseTransferTaskAcked and IsForceReplication flags
on replication stream.
2. Impl for a replication stream interceptor, which fully intercepts
client and server and stores calls in-memory for us to assert against.
Also writes to log files so we can reason about/follow logs without
touching server code (this impl does not immediately log to file for
perf reasons).
3. Accompanying tests (unit and functional)
## Why?
This is part of a larger change to optimize passive task processing by
no-oping side-effects of task processing depending on if we can deem
that the task has already been processed on the parent. Will expand
further in the next PR.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [x] added new functional test(s)
`michaely520 ~/projects/temporal % go test -v -tags test_dep ./tests/xdc
-run
TestStreamBasedReplicationTestSuite/EnableTransitionHistory/TestCloseTransferTaskAckedReplication
-timeout 10m -count=1`
## What changed?
Adding in new field for SyncVersionedTransition to denote if close
transfer has been executed on source side.
## Why?
Enabling this flag will allow us to optimize replication of closed
workflows during forced replication to not reexecute closed child
workflows from verifying against their parent.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
use activity start event version to verify SDK's activity update request
## Why?
The version in activity task token is the version when the activity task
is started. However, when checking sdk activity requests, server uses
activity info's version which is the last write version. If there is a
failover, ai.version might be updated to new active's failover version.
The sdk activity request check will fail.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Fix State based replication first task handling
## Why?
Previously, we made an assumption that it is always safe to return
mutation for first replication task, but it is not true.
When there is replication lag, the workflow may make some progress and
tombstone batch is capped. If we still blindly return mutation,
completed statemachine may still looked pending on the passive side.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
No risk, the feature is not launched yet.
## What changed?
<!-- Describe what has changed in this PR -->
Remove license header from every file. Because it is really hard to
follow in this PR here is the summary:
1. License header is removed from all `.go` and `.proto` files
:fireworks::fireworks:🎆.
2. `LICENSE` file in the root directory has only Temporal and Uber
copyrights.
3. 5 other `LICENSE` files added to the packages which have copyrights
different from Temporal and Uber: Datadog, Xargin, "Mat Ryer, Tyler
Bunnell and contributors".
4. `license_file` flag is removed from all code generation tools.
5. `copyright_file` flag is removed from `go:generate mockgen`
directive.
6. All copyright related targets are removed from `Makefile`.
7. Updated Temporal copyright year to 2025 everywhere.
## Why?
<!-- Tell your future self why have you made these changes -->
I double checked with legal department that it is not needed to have
license header in every file. One file per repo is enough. I put all
copyrights to the root `LICENSE` file and removed header from all other
files. Also updated tools and `Makefile`.
## What changed?
<!-- Describe what has changed in this PR -->
Add the following activity retry policy properties to the activity
replication logic:
* InitialInterval
* MaximumInterval
* MaximumAttempts
* BackoffCoefficient
## Why?
<!-- Tell your future self why have you made these changes -->
Before there was assumption that activity retry policy can't change.
Because of that it was not replicated.
Now we can change activity retry policy via UpdateActivity API. Those
changes should be replicated.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Add func tests that check few updated properties.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
Increasing replication size.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No
## What changed?
Add PuaseActivity API implementation
* Add Paused field to the pending activity info
* Add replication for that field
* Add populating search attributes
* basic testing
## Why?
Working on activity API
## How did you test it?
Add basic unit tests. Func tests will come later.
## What changed?
* Change the logic. Now we regenerate retry task for every call.
* Add `RegenerateRetryTask`. Pls check the logic.
* Add replication for all recent fields
* Propagate activity `stamp` as a part of activity task
* "Reject" activity start in case of stamp mismatch from history service
* Update "standby" timer task processed to take into account `stamp` for
retry tasks
(and do nothing for timeout tasks)
## Why?
Continue workfing on UpdateActivityOptions API
Known issues: need more functional tests.
- initial commit
- some fix
## What changed?
<!-- Describe what has changed in this PR -->
Add raw sync versioned transition task for state based replication
## Why?
<!-- Tell your future self why have you made these changes -->
For state based replication
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
unit test
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
no risk, the feature flag is disabled
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
n/a
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
no
## What changed?
<!-- Describe what has changed in this PR -->
Consolidate proto for sync state replication task
## Why?
<!-- Tell your future self why have you made these changes -->
For easier usage
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
unit test
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
no. feature not enabled.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
no
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
no
## What changed?
<!-- Describe what has changed in this PR -->
Implement Sync Versioned Transition logic
## Why?
<!-- Tell your future self why have you made these changes -->
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
test will be added later.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Add sync workflowstate api
## Why?
<!-- Tell your future self why have you made these changes -->
To support state based replication
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
unit test will be added after implementation is agreed
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
no risk, not used in prod
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
n/a
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
no
## What changed?
Implement SyncVersionedTransitionTask conversion logic. There will be more changes to complete the whole conversion logic.
## Why?
For stated-based replication.
## How did you test it?
unit test.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
Add a host level replication progress cache. The cache will be updated whenever a SyncVersionedTransition replication task is sent or SyncState API is called.
## Why?
To reduce the duplication of replication tasks.
## How did you test it?
unittest.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
---------
Co-authored-by: Will Duan <xinw.duan@gmail.com>
Co-authored-by: Roey Berman <roey@temporal.io>
Co-authored-by: Yichao Yang <yichao@temporal.io>
## What changed?
<!-- Describe what has changed in this PR -->
Add VerifyVersionedTransition replication task
## Why?
<!-- Tell your future self why have you made these changes -->
For state replication project
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
unit test
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
no risk. Not in use for now.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
n/a
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
no
## What changed?
This PR adds BackfillHistoryTask when SyncVersionedTransitionTask is not on current transition history.
## Why?
This PR is part of SyncVersionedTransitionTask.
## How did you test it?
unittest.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
1. Add invalidate cache api on namespace registry
2. Make sure the namespace failover version is up-to-date when
processing replication task
## Why?
<!-- Tell your future self why have you made these changes -->
State based replication requires namespace failover version is
up-to-date, so it can determine if buffered events should be flushed or
not.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Will add unit test when agreed on the implementation
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
no risk
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
n/a
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
no
---------
Co-authored-by: Yichao Yang <yichao@temporal.io>
## What changed?
<!-- Describe what has changed in this PR -->
- Add a new replication task type for sync HSM states when there's no
history events associated with the state transition.
- This task is only supported in streaming replication. It will go
directly to DLQ with the old replication stack.
- This task will only be used for the short term nexus replication
solution and will be deprecated once the state based replication is
fully implemented.
Some minor fixes:
- Fixed empty startEventID check in getRawHistoryV2 api
- Rename helper methods in ActivityStateReplicatorImpl from testXXX to
compareXXX.
## Why?
<!-- Tell your future self why have you made these changes -->
- Sync state only hsm changes
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- Unit test
- Functional tests will be added in a separate PR
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- No
## What changed?
Only one replication task is generated even if there are multiple event
batches in one transaction on the active side.
## Why?
We need make sure events that belong to the same transaction on the
active side also belong to the same transaction on the standby side.
## How did you test it?
Tested locally. Added new tests.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- initial commit
- Initial implementation
- Fix
## What changed?
<!-- Describe what has changed in this PR -->
Add implementation for Replication stream flow control
## Why?
<!-- Tell your future self why have you made these changes -->
To allow receiver back pressure to sender
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
integration test
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
n/a
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
n/a
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
no
## What changed?
<!-- Describe what has changed in this PR -->
- Send BuildIdRedirectInfo from Matching to History on
Record*TaskStarted call containing information about redirect intention.
- History validates the redirect info against current MS and fails the
request if the redirect does not have source build if equal to the
current assigned build of the workflow.
- If redirect is valid, workflow is assigned to the new build id and all
pending but not started task are rescheduled to be sent to the new build
id.
## Why?
<!-- Tell your future self why have you made these changes -->
To prevent the following problems from happening for workflows with
concurrent tasks when redirect rules are deleted or not fully propagated
yet:
1. Assign wf back to an old build id after processing task using a newer
build id
2. Interleaved Starts: for some duration tasks are dispatched to a mix
of old or new build IDs
3. New activity output being fed to old wf
4. Execution gets stuck after being (partially) redirected
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Functional test. More unit tests to be added in a followup PR.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
As it is right now, in rare situations when a redirect rule is applied
to a WF with concurrent activities and some of them are in backoff
period due to failure, we may schedule (and start) them on the newer
build without waiting for the backoff to finish. This is planed to be
improved in the futuer.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
None.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
Implement tiered replication stream receiver:
1. Change `receiver.receive` to handle original (single ACK level) and
new (prioritized ACK level) properly
2. Change `receiver.ACK` to properly ACK back to sender with proper ACK
level.
## Why?
<!-- Tell your future self why have you made these changes -->
To prioritize different category of replication tasks.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
integration test.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
Replication stack will go down.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
n/a
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
no
## What changed?
- Refactored Nexus Registry code to merge incoming and outgoing into one
registry
- Added separate concept of "endpoints" - which is roughly analogous to
what was previously called "services"
- Upgraded the Nexus Go SDK and API dependencies accordingly
- Command handler interface was modified to accept a context given new
requirements
- Note that invoking a worker endpoint is not yet supported, there will
be some minor restructuring required to enable that
- Requires this unmerged API PR
https://github.com/temporalio/api/pull/403. The two PRs should be merged
together.
## Why?
Part of late redesign. See also https://github.com/nexus-rpc/api/pull/5
## How did you test it?
Modified existing and added new tests.
## What changed?
<!-- Describe what has changed in this PR -->
Change task priority to use enum, so no more int convert and error
handling. The new attribute is not used yet, so it is safe to change the
type directly.
## Why?
<!-- Tell your future self why have you made these changes -->
Change task priority to use enum, so no more int convert and error
handling.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Unit test/integration test.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
n/a
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
n/a
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
no
## What changed?
<!-- Describe what has changed in this PR -->
Add priority attribute to Replication Task and SyncWorkflowState task
## Why?
<!-- Tell your future self why have you made these changes -->
To distinguish priority on task priority on replication stack
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Unit test and integration test
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
Failed to serailize/deserailize. Current tests should be able to capture
it.
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
n/a
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
no
In addition I've:
- Extended the DC to disable Nexus APIs entirely
(`frontend.enableNexusAPIs`)
- Fixed a path where outgoing service registry could be lost
- Incremented namespace config version that was missed in the original
PR
- Changed uses of `docker-compose` to `docker compose` to work around CI
failures
## What changed?
<!-- Describe what has changed in this PR -->
- When workflow in the source cluster completes and starts a new run in
the same transaction. Replication should also apply that in one
transaction in standby cluster. Today this only happens when the current
workflow closes with continue as new event, but there are other cases
like cron and retry where workflow can close with
completed/timedout/failed event and also has a new run.
## Why?
<!-- Tell your future self why have you made these changes -->
- Transactional guarantee should be preserved by replication
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
- New unit tests
- Run canary using global namespace.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
- Replication task may go to dlq in worst case
## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->
- N/A
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
- No.
## What changed?
I've upgraded the version of our Go API in order to allow invalid UTF-8
data in our proto strings.
Please review
5cf979058d
only
## Why?
The version of gogo/protobuf we used allowed invalid UTF-8 in strings
which is a violation of the proto3 spec. We were unaware of this until
our new google/protobuf based structs began failing to deserialize data
we'd used previously
## How did you test it?
I added a new end-to-end test to verify that we can create, describe,
and delete a namespace that contains invalid UTF-8 in multiple
attributes
## Potential risks
This is lower risk than commits prior to this as temporal DBs may
already contain invalid UTF-8 data
## Documentation
I adjusted our docs to mention this behavior and the `protolegacy` tag
## Is hotfix candidate?
Yes: this will be going out as part of our 1.23.0 release. In fact, the
release is blocked on it!
---------
Co-authored-by: David Reiss <david@temporal.io>
## What changed?
I replaced protoc.sh with
[protogen](https://github.com/temporalio/api-go/pull/139)
## Why?
I've copied some variant of this protoc.sh tool into three repos now.
This is unsustainable, so I've added a little build tool to api-go that
will handle proto generation and post-processing for us.
## How did you test it?
[I ran `make proto` in the PR stacked on this
one](https://github.com/temporalio/temporal/pull/5211)
## Potential risks
None
## Is hotfix candidate?
No
**What changed?**
I rebuilt all proto files and bumped the version of our protoc plugin
**Why?**
So as to not pollute the PRs of others
**How did you test it?**
CI
**Potential risks**
None
**Is hotfix candidate?**
No.
**What changed?**
I added the magic MIT License header to all proto files and regenerated
the code
**Why?**
Turns out we didn't include the header our licensegen script checked
for, so when the new protoc plugin prepending the license our licensegen
script would prepend it a second time
**What changed?**
gogo/protobuf has been replaced with Google's official go compiler.
**Why?**
gogo/protobuf has been deprecated for some time and the community is
moving on, building new tools (like vtproto) atop google's v2 compiler.
**How did you test it?**
`make test`
**Potential risks**
1. The change from embedded gogo-generated-structs to
google-generated-pointers-to-structs created a risk of nil pointer
exceptions. I've fixed all the ones our tests found but it's possible
there are more lurking in the new code.
2. This change may cause our performance to decrease. Certainly
encoding/deconding of proto objects will become slower, but the overuse
of pointers by the google compiler may negatively affect our overall
performance. We'll need to keep an eye on the GC stats
3. This breaks the HTTP API. We will not support [shortand payload
encoding](https://github.com/temporalio/proposals/blob/master/api/http-api.md#payload-formatting)
in this first pass; that will come once this initial work is in testing.
**Breaking changes for developers**
- `*time.Time` in proto structs will now be
[timestamppb.Timestamp](https://pkg.go.dev/google.golang.org/protobuf@v1.31.0/types/known/timestamppb#section-documentation)
- `*time.Duration` will now be
[durationpb.Duration](https://pkg.go.dev/google.golang.org/protobuf/types/known/durationpb)
- V2-generated structs embed locks, so you cannot dereference them. `go
vet` will scream at you about this. If you need a copy, use
`proto.Clone`.
- If the performance of this sucks then I will either update our code
generator to add shallow-clone methods or hand-roll the ones we need
- Proto enums will, when formatted to JSON, now be in
`SCREAMING_SNAKE_CASE` rather than `PascalCase`. We decided (in
discussion with the SDK team) that now was as good a time as any to rip
the bandage off.
- Proto objects, or objects embedding protos, cannot be compared using
`reflect.DeepEqual` or _anything_ that uses it. This includes `testify`
and `mock` equality testers!
- You will need to use the `common/testing/protorequire`,
`common/testing/protoassert`, or `common/testing/protomock` packages
instead. I've implemented proto-compatible matchers and assertions there
for all cases I've encountered
- If you need `reflect.DeepEqual` for any reason you can use
`go.temporal.io/api/temporalproto.DeepEqual` instead
Note that history loading will not be impacted by the JSON changes: I
rewrote history loading to dynamically fix incoming history JSON data
(like all our other sdks); you can find this code in [my fork of our go
API](https://github.com/tdeebswihart/temporal-api-go/blob/master/internal/temporalhistoryv1/load.go)
alongside its tests.
**🚨Sharp Edges Introduced🚨**
Beware `*timestamppb.Timestamp.AsTime()`. If you need to extract a time
value from a proto time (timestamppb) **always** make sure to check
whether it's nil first. When the proto object is `nil` `AsTime()` will
return a non-zero time at the proto epoch: UTC midnight on January 1,
1970.
I've made this mistake multiple times during this transition and each
time it's been a pain to debug
**Is hotfix candidate?**
No.
<!-- Describe what has changed in this PR -->
**What changed?**
Add new raw replication task
<!-- Tell your future self why have you made these changes -->
**Why?**
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
