## What changed?
- Poll CI memory every 1s with cheap process snapshots.
- Capture pprof and process memory diagnostics when memory crosses
threshold, then every 30s while above it.
- Upload memory diagnostics artifacts from post-test reporting.
## Why?
Make OOM failures easier to diagnose before the runner terminates the
test process.
Example:
https://github.com/temporalio/temporal/actions/runs/29111099672/job/86424089913?pr=10742#step:7:77
Cassandra 3.11 reached EOL in July 2023. Upgrade both development and CI
compose files to Cassandra 5.0 for current driver compatibility and
feature parity with ScyllaDB.
Changes:
- develop/docker-compose: cassandra:3.11 -> cassandra:5.0
- develop/github CI compose: cassandra:3.11 -> cassandra:5.0, increase
healthcheck retries from 30 to 60 (5.0 is slower to start)
- macOS docs: update to 5.0, remove stale ARM JNA workaround that is no
longer needed
## What changed?
Big change, but:
1. Cassandra 3.11 is End-of-Life.
2. Allowed me separately to change the Golang CQL driver to ScyllaDB's,
which doesn't support that EOL Cassandra anymore (my fault too, I
removed that support I think).
## Why?
If I wanted (and I do) to switch to ScyllaDB's GoCQL driver, I needed to
perform this change.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
I also did not see any performance regression!
## What changed?
Add health checks for docker compose dependencies.
## Why?
Seeing connection timeouts for cass_es8
[[example](https://github.com/temporalio/temporal/actions/runs/22005015596/job/63897748984?pr=9292)]
that are failing tests.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Might make tests a little bit slower due to extra wait time.
## What changed?
Use only ARM runners for running tests.
## Why?
They are faster and cheaper and are available faster.
<img width="390" height="249" alt="Screenshot 2026-02-12 at 8 02 58 PM"
src="https://github.com/user-attachments/assets/0de5865e-971a-4d7c-8dd1-a57860412f12"
/>
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
Tested explicitly on all databases.
## What changed?
1. Extended existing Go test memory monitor to include goroutine
profile.
2. Changed snapshot to be of moment with _highest_ memory usage (instead
of latest).
3. Unified report into a single one (both printing and disk snapshot).
4. (bonus) added monitor to unit and integration test jobs.
## Why?
Inspect where high goroutine count comes from.
## How did you test it?
Example:
https://github.com/temporalio/temporal/actions/runs/21695994550/job/62566409323?pr=9162#step:10:17
## What changed?
Migrated `TestWorkflowUpdateSuite` away from testify's `Suite`; enabling
parallel test execution.
**How it works**
- a test invokes `testcore.NewEnv(t)` to obtain a new `TestEnv`
- `TestEnv` sets `t.Parallel()` (_intentionally not giving a way to opt
out!_)
- `TestEnv` obtains a test cluster from `clusterPool` (_or blocks if all
are in-use right now_)
- env var `TEMPORAL_TEST_SHARED_CLUSTERS` controls size of the pool
- if a test relies on APIs like InjectHook, a dedicated cluster is used
to prevent overlap
- env var `TEMPORAL_TEST_DEDICATED_CLUSTERS` controls number of
dedicated clusters
**testify suites**
Existing test suites are limited by the same dedicated cluster pool to
prevent creating too many clusters.
**Database connections**
SQLite setup for TestEnv-based func tests (ie only
TestWorkflowUpdateSuite so far) has been changed to a file-based
approach since that supports much better concurrency due to its WAL that
an in-memory SQLite database does not support.
Connection limits for other databases were also raised due to connection
errors.
**Planned follow-ups**
- Migrating the other testify suites should be fairly straight-forward
with the use of AI agents.
- Reduce need for dedicated clusters by leveraging isolated
namespace-per-test more.
- Eliminate all `time.Sleep`s.
- Tweak test cluster pool behavior.
## Why?
1. **Local speedup**: benchmarks show a ~50% speed increase (36.1s →
16.6s) for `TestWorkflowUpdateSuite`.
5. **Namespace isolation**: every test runs in its own namespace. This
greatly reduces the risk of (accidental) collisions and also reduces the
need to craft unique identifiers such as for task queues and workflow
IDs.
6. **Deprecate testify suites**: Long-term strategy to remove use of
testify suites in functional tests (one reason being their inability to
run tests within a suite in parallel).
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential Issues
1. Logs become less useful since there is more interleaving of tests.
2. Higher resource consumption: it requires more concurrent connections
to databases and shows higher memory consumption (see 3 and 4). This
could cause some short-term instability on CI. Note that some other PRs
were merged to add mechanics for monitoring memory usage much better;
which will help here.
4. Until all functional tests are converted, there is an imbalance in
test cluster creation: migrated tests use the shared pool while current
tests create one cluster each. Especially given the fact that some tests
don't allow for test cluster sharing as they use non-parallelizable
actions such as `InjectHook` or dynamic config overrides. With some more
effort the number of these can be reduced.
7. Setup of test clusters was designed around the idea of short-lived
clusters, one per suite. But when re-using them for longer, some of the
assumptions don't hold anymore and increase memory usage. There's a band
aid in place to limit how often a test cluster can be used before it's
torn down. A long-term solution requires some design changes to how test
clusters are started/used/torn down.
8. If there are certain cross-namespace issues or bugs that affect
multiple tests, it might be harder to identify the root cause now.
However; the existing test re-runs should at least mitigate these
short-term.
---------
Co-authored-by: Dan Davison <dandavison7@gmail.com>
## What changed?
Explicitly set the `"missing"` key in ES sorter. This is to address
pagination bug in OpenSearch 2.8.0+ (see
https://github.com/opensearch-project/OpenSearch/issues/8212#issuecomment-1615325095).
Run CI tests with latest version of OpenSearch 2 and OpenSearch 3.
## Why?
Full support of OpenSearch as visibility store.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Enable functional tests with OpenSearch 2 in CICD
## Why?
Run compatibility with OpenSearch 2
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
## What changed?
Run cassandra with smaller heap (2GB) in CI. Default (on standard GHA
workers) comes out to 4GB, so this is 1/2 of what it was before.
## Why?
Try to reduce CI flakiness.
## What changed?
<!-- Describe what has changed in this PR -->
Github workflow to run tests.
I couldn't add tests on `windows-latest` nor `macos-latest` at this moment.
In Windows, there are issues with filepaths in the archiver (eg: `file:///home/` vs `file:///c:\\`, note that in Windows, it needs a forward slash before the actual path `c:\\`, but it messes up with the actual path)
In MacOS, it doesn't have Docker installed by default.
Also, in both Windows and MacOS, some TLS related unit tests are failing.
## Why?
<!-- Tell your future self why have you made these changes -->
Replace buildkite.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? -->
Ran in my fork: https://github.com/rodrigozhou/temporal/actions/runs/7401242980
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this change to production? -->
No.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to be sent to the broader community? (Yes/No) -->
No.
