Commit Graph

321 Commits

Author SHA1 Message Date
Alex Shtin
e12c7a1599 Switch visibility queue default to internal (#1235) 2021-01-27 16:26:23 -08:00
Alex Shtin
1428a3da31 Set default ES bulk processor ack timeout to 1 minute (#1210) 2021-01-21 12:06:29 -08:00
Alex Shtin
0070bf06fb Add option to disable visibility queue processor (#1198) 2021-01-18 18:38:28 -08:00
wxing1292
c38d8fcc63 Better retry policy when loading timer tasks (#1193)
* Better retry policy when loading timer tasks

* address comments
2021-01-15 14:45:34 -08:00
wxing1292
2738ee9748 Refactor Cassandra sessions management (#1148) 2020-12-31 10:22:04 -08:00
Alex Shtin
f28f51cef8 Switch visibility queue to dual processor mode (#1146) 2020-12-30 15:43:20 -08:00
Alex Shtin
8591eb45ae Switch default of history.visibilityQueue to internal (#1140) 2020-12-29 15:56:03 -08:00
Alex Shtin
16cd5400c3 Replace bool visibility queue dynamic configs with single string config (#1104) 2020-12-16 15:20:16 -08:00
Alex Shtin
b9adde0c08 Add DisableTransferQueueProcessForVisibility dynamic config flag (#1096) 2020-12-15 21:03:53 -08:00
Alex Shtin
55881bf0b7 Replace Kafka for visibility with internal VisibilityQueue (#988) 2020-12-11 15:47:31 -08:00
wxing1292
5cd1e7dc28 Revert "Revert "Finish migration of infinite workflow timeout (#1030)" (#1068)" (#1070)
This reverts commit 227069b1f6.
2020-12-10 10:38:37 -08:00
wxing1292
227069b1f6 Revert "Finish migration of infinite workflow timeout (#1030)" (#1068)
This reverts commit f67c31b0d1.
2020-12-09 17:50:27 -08:00
wxing1292
f67c31b0d1 Finish migration of infinite workflow timeout (#1030) 2020-12-09 11:35:38 -08:00
wxing1292
f4cda39252 Allow concurrent fetching replication tasks (#1040)
* Allow concurrent fetching replication tasks
* Enable the RPC replication tests
2020-12-03 13:41:04 -08:00
wxing1292
b0c5e6b7ef Adjust RPC replication config (#1039)
* Adjust RPC replication config
* Adjust RPC replication configs
* Fix bug on DLQ message generation
* Sort replication tasks by shard ID, task ID when querying DLQ
* Fix bug on ReplicationDlqAckLevel
2020-12-02 11:54:41 -08:00
wxing1292
c5d0144c32 Make entity with Start / Stop functions idempotent (#1037)
* Make entity with Start / Stop functions idempotent
* Remove misleading / unnecessary dynamic configs related to NDC
* Minor issue fixes

NOTE: 
* when running in `kafka` replication mode, both tx and rx will use Kafka replication stack
* when running in `kafka-to-rpc` replication mode
  * tx will use RPC stack
  * rx will use both RPC stack and Kafka stack
* when running in `rpc` replication mode, both tx and rx will use RPC replication stack
2020-12-01 18:54:13 -08:00
wxing1292
2b63e6f5b6 Allow workflow timeout being infinite (#995)
* Remove unnecessary dynamic config related to workflow max timeout
* Remove unnecessary dynamic config related to workflow task timeout
* Handle tasks TTL in matching service properly
* Handle activity without timeout
2020-11-30 11:45:16 -08:00
wxing1292
622852f5c1 Refactor replication task processor (#1005)
* Refactor replication task processor
* Rewrite some tests
* Update proto definition of shard ID from int64 to int32: `int32, uint32, int64, uint64, and bool are all compatible`
Ref: https://developers.google.com/protocol-buffers/docs/proto#updating
2020-11-18 16:26:55 -08:00
wxing1292
d90f4e0027 Update NDC default config (#956)
* Fix broken NDC yaml config
* Set NDC replication to default to Kafka
* Disable RPC replication test until ready
2020-11-05 13:18:20 -08:00
mastermanu
407734a730 Increases number of go-routines that acquire history shards #930
For Temporal clusters with a large number of shards, a single go-routine that acquires all the shards causes significant latency when a history node is starting up as it loads the shards one-by-one.

This fix increases the number of goroutines to "10" as that should not cause any issues for existing clusters, but will yield a significant benefit to clusters with a larger number of shards.

We verified this on a cluster with a large number of shards and saw dramatic improvement in overall shard acquisition time.

Ideally, the number of goroutines should be calculated based on a ratio of the total number of shards to make this slightly more flexible. This can be done in a future check-in.
2020-10-30 13:27:51 -07:00
wxing1292
78df3471e4 Refactor history service config (#918) 2020-10-29 14:24:03 -07:00