Commit Graph

321 Commits

Author SHA1 Message Date
Rodrigo Zhou
6e7c669ae5 Create dynamic config to enable manual pagination for Elasticsearch (#4314) 2023-05-11 17:42:35 -07:00
Yu Xia
a3ae60298b Add replication dlq metrics (#3876) 2023-05-11 17:33:24 -07:00
Rodrigo Zhou
f34d5b7787 Default disable order by clause support (#4305) 2023-05-10 23:51:52 -07:00
Matt McShane
25b7cbc73f Add a per-workflow limit to in-flight updates (#4307) 2023-05-10 22:32:07 -04:00
Alex Shtin
d0007b065a In memory timeout timer queue for speculative workflow task (#4254) 2023-05-09 13:00:15 -07:00
Michael Snowden
8e6203a45f Event buffer size limit (#4296)
* Event buffer size limit

* Make size limit inclusive

* Make default size limit more readable

* Add history builder test for proto size

* Split db and memory batch for loop

* Use values instead of pointers to avoid parallel test glitch

* Make a copy of the test case for parallel test to work
2023-05-08 14:46:27 -07:00
Rodrigo Zhou
12655c4695 Refactor configs for visibility (#4095)
New dynamic configs for visibility
2023-04-24 09:40:22 -07:00
Rodrigo Zhou
becc2f3d55 Remove DefaultVisibilityIndexName from history config (#4189) 2023-04-19 12:02:50 -07:00
wxing1292
2ef3afd556 Properly handle replication task lifecycle (#4126)
* Properly handle replication task lifecycle by cancel task from task tracker
* Add more unit test for stream monitor
2023-03-31 15:19:45 -07:00
wxing1292
8bd067a9f3 Add poison pill support for replication stream (#4116)
* Add poison pill support for replication stream & UT
2023-03-29 12:33:00 -07:00
wxing1292
8f030dbb81 Wire replication stream E2E (#4097)
* Add replication stream API &UT
2023-03-29 11:45:25 -07:00
Rodrigo Zhou
0d0cad4a17 Change disable order by dynamic config to per namespace (#4063) 2023-03-20 12:17:09 -07:00
David Reiss
d4498d1a81 Expose history size to workflows (#3055) 2023-02-09 12:24:52 -08:00
Yichao Yang
008f056d65 Improve task scheduler rate limiter (#3860) 2023-02-03 17:34:40 -08:00
Roey Berman
6ef7749538 Eager workflow dispatch (#3835)
* Eager workflow dispatch

* Revert returning workflowTaskInfo from NewWorkflowWithSignal

* Fix lint issues

* Minor restructuring

* Support eager start with TERMINATE_IF_RUNNING

* Properly release workflow context

* Add documentation and restructure for better readability

* Fix lint

* Run go-generate

* Fix nolint directive

* More restructuring, get rid of cyclo complexity

* Fix task inflight condition

* Address review comments

* Address review comments

* Fix missing reason tag value

* Add missing nil check

* Address review comments
2023-02-01 17:34:36 -08:00
Yichao Yang
25015d4704 Emit lag metrics for all registered history queues (#3851) 2023-01-26 14:24:19 -08:00
Michael Snowden
1c638ac56a Enable durable archival (#3828) 2023-01-25 11:18:18 -08:00
Yichao Yang
a026002b54 Remove task max retry count config (#3771) 2023-01-04 18:00:29 -08:00
Yichao Yang
a22db81db6 Remove deprecated code from queue implementation (#3770) 2023-01-03 11:32:36 -08:00
Michael Snowden
92b8758896 Add an integration test for durable archival (#3732) 2022-12-28 09:58:51 -08:00
Michael Snowden
4e8601a05b Create an FX Module for Archival (#3726)
Add an archival.Module to be used in both tests and prod
2022-12-28 09:30:56 -08:00
Alex Shtin
c6a89c095b Workflow task state machine: minor renames (#3735) 2022-12-20 13:54:54 -08:00
Alex Shtin
a7240b4dee Fix ES bulk processor commit timeout (#3696) 2022-12-20 08:17:58 -08:00
Yichao Yang
ea698fa1b7 Remove old queue processing logic (#3716) 2022-12-15 14:43:34 -08:00
Michael Snowden
532c422c2e Add an archival queue factory (#3562) 2022-12-13 13:23:13 -08:00
Yu Xia
8ca4b2cc8a Handle data loss error in replication read history (#3704)
* Bypass data corruption error in replication
2022-12-12 16:03:57 -08:00
Alex Shtin
ae223cf1e7 Filter replication task on source cluster (#3641) 2022-12-09 17:27:00 -08:00
Michael Snowden
eb88af41ea Add random delay to ArchiveExecutionTask (#3565) 2022-12-08 09:34:37 -08:00
Yichao Yang
53f4615772 Enable host level pool and multi-cursor by default (#3660) 2022-11-25 17:48:31 -08:00
Yimin Chen
b717bd5305 Enforce min interval for continue as new runs (#3659)
* Enforce min interval for continue as new runs

* temp fix for test

* Update tests
2022-11-24 18:55:29 -08:00
Yimin Chen
5658d5e79b Disable visibility verify close before delete (#3651) 2022-11-23 11:29:46 -08:00
Yichao Yang
32212eabdd Task scheduler rate limiter (#3606) 2022-11-18 16:33:17 -08:00
Rodrigo Zhou
b13ed161f5 Add flag to enable workflow closed clean up (#3603) 2022-11-18 14:31:11 -08:00
Michael Snowden
be7daec26c Add limits on other workflow fields (#3599) 2022-11-18 09:27:17 -08:00
Michael Snowden
25e4a8fb36 Don't log a warning when the number of child workflows is high (#3598) 2022-11-15 20:00:02 -08:00
Michael Snowden
50936e1d5e Add a tool to limit the number of pending child workflows (#3575) 2022-11-14 14:13:06 -08:00
Yu Xia
bbd386e034 Add jitter for retention timer (#3569)
* Add jitter for retention timer
2022-11-08 16:59:00 -08:00
Michael Snowden
b300010dd9 Add some config values for the archival queue (#3554)
Add some config values for durable archival
2022-11-06 18:47:59 -08:00
Michael Snowden
c2d5496e98 Generate ArchiveExecutionTask instead of DeleteHistoryEventTask whenever an execution is closed (#3533)
Produce archival task instead of retention timer task if enabled
2022-11-04 11:24:42 -07:00
Rodrigo Zhou
3a0d879abe Remove SA and memo after close visibility task is completed (#3504) 2022-10-27 10:53:32 -07:00
Yu Xia
0ae4cc5208 Add config filter by task type (#3455) 2022-10-06 10:54:22 -07:00
Michael Snowden
c9e8f49771 Dynamic config (#3390)
Add dynamic config for TransferProcessorEnsureCloseBeforeDelete
2022-09-16 10:07:18 -07:00
Yichao Yang
877c2f526c Update task processing dynamic config defaults (#3374) 2022-09-13 12:07:15 -07:00
Tony Gallotta
0a042d1885 Increase default sendArchiveSignal timeout and make it configurable (#3325)
* Increase default sendArchiveSignal timeout and make it configurable

* Keep default at 300ms
2022-09-07 19:40:14 -07:00
Yichao Yang
948e303a15 Multi-cursor: improve namespace failover (#3279)
- Task priority assignment now only depends on task type, not namespace activeness or # of attempts
- Trigger an immediate reschedule for tasks in the rescheduler upon namespace failover
- Update task channel weight upon namespace failover
- Reset task attempts upon namespace failover
- Support task channel weight update in iwrr scheduler
- Separate config for active and standby namespace task weights
2022-09-01 10:38:55 -07:00
Yichao Yang
5ab8ab804f Multi-cursor: pending task count monitoring & mitigation (#3275) 2022-08-31 13:12:43 -07:00
Yichao Yang
d1afec25ec Multi-cursor: slice count monitoring & mitigation (#3276) 2022-08-31 12:16:47 -07:00
Yichao Yang
de9757cead Multi-cursor: reader watermark monitoring and mitigation (#3253) 2022-08-25 14:40:22 -07:00
Yichao Yang
780bb6f124 Use separate task channel for different namespaces (#3252) 2022-08-23 16:27:01 -07:00
David Reiss
f4d0290f27 Structured defaults in dynamicconfig (#3199) 2022-08-16 17:18:57 -07:00