When using `git worktree` for agentic workflows during development, the
golangci-lint cache is always cold.
That means the first invocation scans the entire codebase (consuming all
CPU cores) unnecessarily, since almost all lints can be found by just
scanning the changed packages. The exceptions are type-based checks, but
they are rare.
This PR speeds up the agentic development cycle greatly for local
development; and anything that slips through is still caught in CI where
the full codebase is checked.
Example benchmark for cold cache with 1 changed file with 1 finding:
- `lint-code-fast`: 8s
- `lint-code`: 14min
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Dan Davison <dan.davison@temporal.io>
Add uber-go/nilaway as a nil-safety static analyzer, scoped initially to
chasm/lib/scheduler so it can be rolled out incrementally.
- Fix the 6 findings nilaway reports in the package: add nil-guards
after the ReadComponent reads in invoker_tasks.go and
scheduler_tasks.go, and restructure the completed-retention trim in
invoker.go to only reslice when there is excess (resolving a nil-slice
reslice warning).
- Add a pinned nilaway install target and a `lint-nilaway` make target,
scoped via NILAWAY_PKGS (overridable to widen coverage later).
- Add a `nilaway` CI job to linters.yml and to the linters-succeed gate.
nilaway exits non-zero on any finding, so this gates at zero.
- Add unit tests for applyCompletedRetention covering the trim boundary.
## Why?
- We want to use Nilaway, but adding it in a widespread manner is going
to be tricky, so just starting here incrementally
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [X] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Should be low, one minor guard added, but otherwise mostly just test
infra
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What changed?
Testcore contains functional testing's common utils, for example cluster
pools, metric captures, etc... Right now these are under
`test/testcore/...`. At the moment, their unit tests don't run because
we filter out the entire `./test` dir which contains functional tests.
I considered moving these to a different location, but that seems like a
slightly more painful codemod, and the current location seems to make
sense (these are used for functional tests).
A side effect of running these in CI is figuring out that the
`tests/testcore/test_cluster_pool_test.go` would be failing, because we
were trying to stand up a onebox temporal cluster without any services,
causing Nexus callback URL set call to fail (it needs frontend address).
Also fixed that unit test in this PR.
## Why?
Run all tests in CI/CD.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Extending the goroutine leak testing to include checking for memory
leaks.
## Why?
We've observed OOM kills on CI jobs that indicate resource leaks. This
is one more way to identify specific leaks, in this case memory, and
ensure no regressions occur.
## How did you test it?
<details><summary>pass example</summary>
<p>
```
object leak report
tracked root objects: 3
retained objects: 65227 total, 65227 expected, 0 unexpected
stale exclusions: 0
unexpected retained objects:
none
expected retained objects:
8002x cluster.testBase.ShardInfo.state.atomicMessageInfo.Desc.Base.L0.ParentFile.fileRaw.builder.TypeResolver.typesByName[*] (*impl.MessageInfo)
3643x cluster.testBase.ShardInfo.state.atomicMessageInfo.Desc.Base.L0.ParentFile.fileRaw.builder.TypeResolver.typesByName[*].Desc.Base.L0.ParentFile.fileRaw.builder.FileRegistry.fileRegistry.descsByName[*]
[...]
```
</p>
</details>
<details><summary>fail example
</summary>
<p>
```
object leak report
tracked root objects: 3
retained objects: 65256 total, 40014 expected, 25242 unexpected
stale exclusions: 0
unexpected retained objects:
2046x cluster.host.fxApps[*].container.scope.gh.nodes[*] (*dig.graphNode)
2046x cluster.host.fxApps[*].container.scope.providers[*][*].orders[key*].gh.nodes[*] (*dig.graphNode)
2046x cluster.host.fxApps[*].container.scope.providers[*][*].orders[key*].gh.nodes[*].Wrapped.orders[key*].gh.nodes[*] (*dig.graphNode)
1701x cluster.host.fxApps[*].container.scope.providers[*][*].orders[key*].gh.nodes[*].Wrapped.orders[key*].gh.nodes[*].Wrapped.orders[key*].gh.nodes[*] (*dig.graphNode)
[...]
```
</p>
</details>
## What Changed?
Adds a goroutine-leak regression test
## Why?
First step in goroutine leak regression testing effort; establishing
baseline.
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
## What changed?
Old behavior was
```
10m GitHub step timeout
5m --total-timeout (derived from -timeout)
5m go test -timeout
```
New behavior is
```
10m GitHub step timeout
9m TEST_RUNNER_TIMEOUT / --total-timeout
5m go test -timeout
```
ie attach the testrunner timeout to the ceiling not the floor.
Example now with logs:
https://github.com/temporalio/temporal/actions/runs/27570791993/job/81507067577?pr=10713#step:9:46
## Why?
(1) We don't see any logs when the testrunner out.
(2) testrunner has no time anymore to do a retry when `go test` times
out.
## What changed?
- Codegen to automatically generate code for extracting businessID for
workflowservice methods based on "temporal-resouce-id" proto option
## Why?
- With codegen, any new API will automatically have the logic for
businessID extraction and avoids the case where businessID extractor
logic forget to be updated.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
- Added `development-jwt.yaml` config that runs Temporal with JWT
authorization enabled using a JWKS file loaded from local disk. Loading
from local disk is added in
https://github.com/temporalio/temporal/pull/9590.
- Added `config/jwt/setup-keys.sh` to generate an RSA key pair and JWKS
file in `/tmp/temporal-jwt-test/`.
- Added `config/jwt/generate-token.sh` to create signed JWTs for
testing.
- Added `make start-jwt` target.
## Why?
Testing JWT authentication currently requires an external identity
provider or manual key/token setup. This provides a self-contained local
workflow: `make start-jwt` generates keys automatically and starts the
server with auth enforced, and `generate-token.sh` produces tokens on
demand.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
Tested this in conjunction with
https://github.com/temporalio/temporal/pull/9582. Was able to list the
workflows by passing the token and dumping their history. When the token
is not used, the auth fails with "Request denied".
## Potential risks
Keys are stored in /tmp/temporal-jwt-test/ which is world-readable. This
is acceptable for local development but the scripts should not be used
in shared or production environments.
## What changed?
Adds `buf format` as a Makefile target; and integrates it into `make
fmt`.
All `.proto` changes are from running `make fmt`.
## Why?
Consistent protobuf file style.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
Use tagged API 1.62.3 for cloud release purposes. This tag includes the
api-go psuedo-version already in the go.mod.
---------
Co-authored-by: Chris St. Pierre <chris.st-pierre@temporal.io>
## What changed?
Add `fairsim` binary (from #8158) to `.gitignore` and `make clean` and
remove it from the repo. (Don't add it to `make bins` since it's not
generally useful.)
## Why
It accidentally got checked in.
## What changed?
Fixed a bug in the gcloud visibility_archiver.go file and added a new
unit test in visibility_archiver_test.go.
## Why?
There is a bug in the visibility-archiver implementation of the Google
Cloud archiver. All archived workflows are being retrieved and displayed
twice. This change fixes that bug.
## How did you test it?
- [X] built
- [X] run locally and tested manually
- [ ] covered by existing tests
- [X] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
No risk since the archiver is completely decoupled from the core
functionality.
## What changed?
Add a new cli tool to run the matching fairness simulator, and some
tests for basic fairness behavior.
## Why?
So users can tell how it will behave for their workloads.
## What changed?
Add new PR check mixed brain that runs
[Omes](https://github.com/temporalio/omes) against PR's server and
latest released server.
## Why?
It's easy to make an incompatible change accidentally. And while we do
have tests to catch these, they don't run on PRs. This causes a
significant delay (and context loss).
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [x] added new functional test(s)
Verified last backwards incompatible change would have been caught:
https://github.com/temporalio/temporal/actions/runs/22204203793/job/64224556383#step:10:38
## Potential risks
I'd like to make this a _non-blocking_ check until we think it's stable
enough. (requires admin AFAIK)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What changed?
1. Tweak on the loop in Makefile target `fmt-gofix` to retry based on
output log.
2. Include `fmt-gofix` in CI.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Integrate Go 1.26's new `go fix` into workflow.
NOTE that the changes caused our linter to fire; a [separate
commit](1b23f787ae)
addresses those.
## Why?
Ensure Go code is standardized/modernized.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Adds and integrates `yamlfmt` (Go-based, by Google).
Anything not annotated was changed by `yamlfmt`.
## Why?
Consistent YAML formatting. Reduce unrelated diffs caused by formatting
in PRs.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What Changed
This PR introduces a new `--config-file` flag (and the
`TEMPORAL_SERVER_CONFIG_FILE_PATH ` environment variable) to remove the
dependency on `dockerize` in the Temporal server Docker image.
When a configuration file is specified using either the CLI flag or the
environment variable, the server will load configuration **only** from
that file.
Users who want templating behavior similar to `dockerize` can enable it
by adding the comment `# enable-template` at the top of the
configuration file.
---
### Key Changes
1. **New `--config-file` flag:**
* Adds a global `--config-file` flag that accepts a path to a single
configuration file (absolute or relative to the project root).
* Can also be set via the `TEMPORAL_SERVER_CONFIG_FILE_PATH `
environment variable.
2. **Deprecated legacy flags:**
* The `--config`, `--env`, and `--zone` flags are now marked as
**deprecated** in CLI help text.
* These flags still work for backward compatibility.
3. **Embedded config template:**
* The `config_template.yaml` file is now embedded in the binary to
support loading configuration from environment variables.
* Templating is supported if the file includes the `# enable-template`
comment at the top.
4. **Templating support:**
* Configuration files can use templating by including `#
enable-template` at the beginning of the YAML file.
---
### Configuration Loading Priority (Highest to Lowest)
1. **`--config-file` specified** → Load that specific file
2. **`--config`, `--env`, or `--zone` specified** → Load from
configuration directory (**deprecated**)
3. **No configuration specified** → Load from embedded template using
environment variables (default)
---
### Expected Behavior
The following examples illustrate how the new configuration loading
logic behaves:
* **Default behavior:**
Running `temporal start` without flags loads configuration from
environment variables only using the embedded template.
* **Using `--config-file`:**
`temporal --config-file=/path/to/config.yaml start` loads configuration
from the specified file path.
* **Using `TEMPORAL_SERVER_CONFIG_FILE_PATH`:**
Setting `TEMPORAL_SERVER_CONFIG_FILE_PATH=/path/to/config.yaml temporal
start` has the same effect as using the flag.
* **Validation and error handling:**
The CLI returns clear error messages when conflicting flags or
environment variables are used, or when a specified file does not exist.
---
## Breaking Change
The default behavior of `temporal start` has changed.
It now loads configuration **from environment variables** instead of
using a default template path.
---------
Co-authored-by: Alex Stanfield <chaptersix@users.noreply.github.com>
Co-authored-by: michaely520 <michaely520@users.noreply.github.com>
Co-authored-by: Yichao Yang <yichao@temporal.io>
Co-authored-by: David Reiss <david@temporal.io>
## What changed?
- Add [errortype](https://github.com/fillmore-labs/errortype) linter to
`lint-code` target.
- Fix two bugs that it found.
## Why?
Checking error types with errors.As can be tricky.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [x] covered by existing tests
## What changed?
Remove pinned dependency on modernc.org/sqlite.
## Why?
We added this version override when we experienced some table not found
errors. This happens when sqlite db connections are closed when
transactions fail.
We have TestSQLiteTransactionContextCancellation which passes
consistently for this new version. So we believe this issue should not
happen again.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Reverts to urlfav/cli v1 as per other tools and uses shared flag aliases
where appropriate for more consistency with the other tools.
Works around Go embed's lack of support for symlinks by hard coding the
path to the index template to use. Adds a test to ensure this doesn't
get out of sync with the latest.
## Why?
Better consistency with other tools and being able to actually find the
index template :)
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Added a protoc plugin to generate client for internal CHASM requests to
the history service.
A CHASM library can now define its own set of gRPC services and register
a handler for those.
I had to modify the history client files to be generic, which broke the
mock generation, so I had to hand code the tests there, which IMHO is
actually nicer than the way they were with mocks.
Also opted out of mock generation for CHASM services, there's just not
enough value there.
**NOTE**: Instead of creating three separate clients, I generated a
single layered client, with a constructor that is DI friendly. There
doesn't seem to be a good enough reason to break this out into separate
clients or make client construction configurable (for now).
**NOTE**: We can also get rid of the genrpcwrappers script eventually
and use the protoc plugin approach for all services, I decided not to do
that here though to reduce scope.
For a preview of the generated file see:
-
3a89614a90/chasm/lib/activity/proto/v1/service.proto
-
3a89614a90/chasm/lib/activity/gen/activitypb/v1/service_client.pb.go
## Why?
Keep all CHASM library functionality contained.
## How did you test it?
- [ ] built
## What changed?
- Revive and update original PR #2977 for Elasticsearch CLI tool
Replaces manual curl invocations with a proper CLI tool that leverages
Temporal's built-in Elasticsearch auth providers and provides better
error handling and logging.
## How did you test it?
- [x] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
None, net new.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduces `temporal-elasticsearch-tool` for ES schema/index
management and ping, extends ES client APIs, embeds ES schema, and
updates Makefile to use the tool.
>
> - **Tools**:
> - New `temporal-elasticsearch-tool` CLI with commands: `setup-schema`,
`update-schema`, `create-index`, `drop-index`, `ping`; supports AWS auth
and uses embedded schema files.
> - Adds entrypoint `cmd/tools/elasticsearch`, README, and basic tests.
> - **Elasticsearch Client**:
> - Extends `CLIClient` with `ClusterPutSettings`, `IndexPutTemplate`,
`IndexPutMapping`, `Ping` and implements them (v7) using raw requests
where needed; allows custom HTTP client from config (e.g., AWS-signed).
> - **Schema**:
> - Embeds ES v7 cluster settings and index template
(`schema.Embedded...` accessors).
> - **Build/Makefile**:
> - Adds build target and binary cleanup for
`temporal-elasticsearch-tool`; updates `install-schema-es` and
`install-schema-xdc` to use the CLI instead of curl.
> - Includes binary in `.goreleaser.yml`; excludes it in
`.dockerignore`.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
b37864809a. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
## What changed?
Ensure that when running `go-generate`, the local gomock is used.
## Why?
Stop wasting people's time chasing errors due to the wrong binary being
used.
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
Added iotamixing linter.
> Checks if iotas are being used in const blocks with other non-iota
declarations.
## Why?
We had a near-incident that was traced back to using iota incorrectly.
## How did you test it?
Ran
```
.bin/golangci-lint-v2.5.0 run --verbose --enable-only=iotamixing --config=.github/.golangci.yml
```
and it passes (after addressing the one issue it found).
## What changed?
~Enforce `temporal/cmd/tools/protogen` to set the PATH to point to the
locally installed tools in `.bin`.~
Enforce `temporal/cmd/tools/protogen` to use locally installed protoc
plugins.
## Why?
Without this, it will use whatever is on the PATH; which might not be
the same as in `.bin`.
## How did you test it?
- [ ] built
- [x] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
Without this change, I get different `.pb.go` files emitted as my global
proto-gen-go is used.
## What changed?
Added a CI step that writes a crash JUnit XML file if the testrunner
doesn't complete.
## Why?
(1) Make the "Summary" helpful again by quickly seeing what went wrong.
(2) Make CodeCov show these crashes.
(3) Make our internal test report show these crashes.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
Example run:
https://github.com/temporalio/temporal/actions/runs/17107215158?pr=8221
<img width="1184" height="1167" alt="image"
src="https://github.com/user-attachments/assets/a1c5445e-4359-450d-83ec-8b7a169c47cd"
/>
## What changed?
Upgrade go to the 1.25.0
## How did you test it?
- [x] built
- [x] run locally and tested manually
~Blocked by #8174~
---------
Signed-off-by: Roman Dmytrenko <rdmytrenko@gmail.com>
Co-authored-by: Stephan Behnke <stephanos@users.noreply.github.com>
Co-authored-by: Stephan Behnke <stephan.behnke@temporal.io>
## What changed?
Upgrade golangci-lint to v2
## How did you test it?
- [x] run locally and tested manually
---------
Signed-off-by: Roman Dmytrenko <rdmytrenko@gmail.com>
## What
- Commit generated scheduler protos.
- Improve `make ensure-no-changes` to detect untracked files.
## Why?
The protos were not generated since the tool was committed in a separate
PR from where the protos were added.
## What changed?
Added support for defining protos in chasm libs.
## Why?
Keep everything local to the library.
## How did you test it?
- [x] built
- [x] run locally and tested manually
## What changed?
Stripped Google Cloud Storage's gRPC dependencies out for builds via the
Makefile.
## Why?
Reduce binary size by 16MB; and reduces build times a little (ie
linking).
**Before**
```
-rwxr-xr-x@ 1 stephan staff 119M Jun 30 11:23 temporal-server
```
**After**
```
-rwxr-xr-x@ 1 stephan staff 103M Jun 30 11:24 temporal-server
```
## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
Yes, since there are no end-to-end tests, if anyone started using
`storage.NewGRPCClient` that would fail silently. I added a (crude) test
to catch that.
## What changed?
Explicitly set `CGO_ENABLED` for test-related make targets.
## Why?
Missing `CGO_ENABLED` can negatively affect build times on some systems
(e.g. adds +10s on OSX).
Also, consistency. Build targets already have it set.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
This changes all methods that fail tests on the test logger to
**forcibly** fail it by calling `t.Fatalf`.
It adds the stack trace to the failure message as we'd otherwise have no
information on the _path_ to the failure, which is why we initially used
`panic`
## Why?
While testing a separate project I determined that panics from OSS' test
logger (and the internal one it is based on!) were being caught by our
application code, obscuring issues that should cause tests to fail.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
This may fail more tests than we currently do. This is a Good Thing.
This reverts commit 5fc3fb21c4.
## What changed?
Reverts the change to return transient and speculative WFT events.
## Why?
Started seeing new error like this
"history_events: premature end of stream, expectedLastEventID=4424 but
no more events after eventID=4422" in pipelines which was not there
before.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
The effective behavior did not consider an ongoing transition (unlike
effective version) and this caused some tasks getting wrong Directive
and be unwontedly dropped.
Plus some other improvements.
## Why?
above
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## Potential risks
---------
Co-authored-by: Carly de Frondeville <carly.defrondeville@temporal.io>
## What changed?
Refactor: remove yaml config files from functional tests. ndc and xdc
configs are still there and I will do it in following PRs.
## Why?
It is much more convenient to have test configuration in code than in
yaml files.
## How did you test it?
- [ ] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [ ] added new unit test(s)
- [ ] added new functional test(s)
## What changed?
<!-- Describe what has changed in this PR -->
Remove license header from every file. Because it is really hard to
follow in this PR here is the summary:
1. License header is removed from all `.go` and `.proto` files
:fireworks::fireworks:🎆.
2. `LICENSE` file in the root directory has only Temporal and Uber
copyrights.
3. 5 other `LICENSE` files added to the packages which have copyrights
different from Temporal and Uber: Datadog, Xargin, "Mat Ryer, Tyler
Bunnell and contributors".
4. `license_file` flag is removed from all code generation tools.
5. `copyright_file` flag is removed from `go:generate mockgen`
directive.
6. All copyright related targets are removed from `Makefile`.
7. Updated Temporal copyright year to 2025 everywhere.
## Why?
<!-- Tell your future self why have you made these changes -->
I double checked with legal department that it is not needed to have
license header in every file. One file per repo is enough. I put all
copyrights to the root `LICENSE` file and removed header from all other
files. Also updated tools and `Makefile`.
