## What changed?
<!-- Describe what has changed in this PR -->
- allow configuring a custom OTEL `service.name` prefix
- allow configuring an OTEL exporter via env vars (following the [OTEL
spec](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md#configuration-options))
**Note that the OTEL exporter from the environment variables will only
be created if there isn't already one from the config.** Creating an
additional one wouldn't make any sense since the env variables would be
applied to both anyways and there would not be any difference between
them.
## Why?
<!-- Tell your future self why have you made these changes -->
Addresses:
- https://github.com/temporalio/temporal/issues/4042
- https://github.com/temporalio/temporal/issues/4041
- our own (increasing) need to enable OpenTelemetry tracing without
touching the Server config (for ad-hoc debugging when working on a SDK,
for example)
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
I ran an OTEL collector to verify it sends the tracing exports.
These are the env vars I set to make it work:
```go
os.Setenv("OTEL_TRACES_EXPORTER", "oltp")
os.Setenv("OTEL_EXPORTER_OTLP_TRACES_INSECURE", "true")
```
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
I doubt anyone would accidentally have these env variables specified and
turn on OpenTelemetry by accident.
Unless these are specified; no behavior change occurs.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No
## What changed?
Now client can request not to append timestamp to scheduled workflow ID.
## Why?
User request.
## How did you test it?
Functional test.
## Potential risks
N/A
## Documentation
To the best of my ability.
## Is hotfix candidate?
No
## What changed?
Update Go SDK to v1.26.0-rc.2
## Why?
Use a tagged Go SDK for the server release
---------
Co-authored-by: Roey Berman <roey@temporal.io>
## What changed?
<!-- Describe what has changed in this PR -->
I updated the docs for the DLQ to include a few more things and made
some structural changes.
## Why?
<!-- Tell your future self why have you made these changes -->
To make the docs more comprehensive and better structured.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What changed?
<!-- Describe what has changed in this PR -->
Switch from `docker-compose` to `docker compose`.
## Why?
<!-- Tell your future self why have you made these changes -->
`docker-compose` was deprecated some time ago.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
Run locally.
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
No risks.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
Update `base-ci-builder` image to 1.10.8.
## Why?
<!-- Tell your future self why have you made these changes -->
It updates `alpine` image to 3.19.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
## What was changed
I bumped the version of api-go and sdk-go and _slightly_ altered our
nettest RPC factory interface to deal with changes in v1.60.0 of go-grpc
## Why?
To fix the protojson DOS vulns recently patched in the upstream
golang/protobuf. See https://github.com/temporalio/api-go/pull/143 for
details
## How did you test it?
I pulled the tests added to the protojson repo into our fork
## Potential risks
N/A
## Is hotfix candidate?
No as it requires all our other proto changes which aren't released
## What changed?
<!-- Describe what has changed in this PR -->
Github workflow to run tests.
I couldn't add tests on `windows-latest` nor `macos-latest` at this moment.
In Windows, there are issues with filepaths in the archiver (eg: `file:///home/` vs `file:///c:\\`, note that in Windows, it needs a forward slash before the actual path `c:\\`, but it messes up with the actual path)
In MacOS, it doesn't have Docker installed by default.
Also, in both Windows and MacOS, some TLS related unit tests are failing.
## Why?
<!-- Tell your future self why have you made these changes -->
Replace buildkite.
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? -->
Ran in my fork: https://github.com/rodrigozhou/temporal/actions/runs/7401242980
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this change to production? -->
No.
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to be sent to the broader community? (Yes/No) -->
No.
## What changed?
<!-- Describe what has changed in this PR -->
Specify patch version of Cassandra image to use in buildkite.
## Why?
<!-- Tell your future self why have you made these changes -->
It seems that the latest version (3.11.16) has some issues with buildkite (`Cassandra 3.0 and later require Java 8u40 or later.`)
## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? -->
## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this change to production? -->
## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to be sent to the broader community? (Yes/No) -->
## What changed?
I replaced the outstandingTasks map with an ordered treemap and
optimized
completeTask to only scan what was necessary to update the ack level.
## Why?
The old implementation of completeTask required a full scan of the task
map in order to move the ack level which had terrible performance.
By storing tasks in an ordered set we can limit the scan's size by
stopping at the first unacked task.
This trades addTask performance for completeTask performance but since
all
added tasks are presumably completed we should be fine with 1/3 the
performance on addTask for 227x the completeTask performance. With this
change both operations run in about the same amount of time.
Before:
```
$ go test -bench=AckManager ./service/matching/... -run=FooBarBaz
goos: darwin
goarch: arm64
pkg: go.temporal.io/server/service/matching
BenchmarkAckManager_AddTask-12 22768 52206 ns/op
BenchmarkAckManager_CompleteTask-12 38 29293019 ns/op
```
After:
```
$ go test -bench=AckManager ./service/matching -run=FooBarBaz
goos: darwin
goarch: arm64
pkg: go.temporal.io/server/service/matching
BenchmarkAckManager_AddTask-12 8127 147226 ns/op
BenchmarkAckManager_CompleteTask-12 8626 136614 ns/op
```
## How did you test it?
I added both tests and benchmarks to ensure the ackManager worked as
before
## Potential risks
None.
## Is hotfix candidate?
No
## What changed?
All `tests/*_test.go` were renamed to `tests/*go`, their `Test*Suite`
functions were moved out to `tests/all_test.go`.
## Why?
Following up on #5192 and #5188 — making test suites available to
third-party packages.
## How did you test it?
Ran tests.
## Potential risks
Merge conflicts in other PRs.
## Is hotfix candidate?
No.
<!-- Describe what has changed in this PR -->
**What changed?**
Adding documentation on how to deal with messages that are added to
DLQs.
<!-- Tell your future self why have you made these changes -->
**Why?**
This will help operators to find messages in DLQ, the reason why it got
enqueued into DLQ etc.
It also has useful tdbg commands for deleting or re-enqueuing the
messages from DLQ
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
**What changed?**
gogo/protobuf has been replaced with Google's official go compiler.
**Why?**
gogo/protobuf has been deprecated for some time and the community is
moving on, building new tools (like vtproto) atop google's v2 compiler.
**How did you test it?**
`make test`
**Potential risks**
1. The change from embedded gogo-generated-structs to
google-generated-pointers-to-structs created a risk of nil pointer
exceptions. I've fixed all the ones our tests found but it's possible
there are more lurking in the new code.
2. This change may cause our performance to decrease. Certainly
encoding/deconding of proto objects will become slower, but the overuse
of pointers by the google compiler may negatively affect our overall
performance. We'll need to keep an eye on the GC stats
3. This breaks the HTTP API. We will not support [shortand payload
encoding](https://github.com/temporalio/proposals/blob/master/api/http-api.md#payload-formatting)
in this first pass; that will come once this initial work is in testing.
**Breaking changes for developers**
- `*time.Time` in proto structs will now be
[timestamppb.Timestamp](https://pkg.go.dev/google.golang.org/protobuf@v1.31.0/types/known/timestamppb#section-documentation)
- `*time.Duration` will now be
[durationpb.Duration](https://pkg.go.dev/google.golang.org/protobuf/types/known/durationpb)
- V2-generated structs embed locks, so you cannot dereference them. `go
vet` will scream at you about this. If you need a copy, use
`proto.Clone`.
- If the performance of this sucks then I will either update our code
generator to add shallow-clone methods or hand-roll the ones we need
- Proto enums will, when formatted to JSON, now be in
`SCREAMING_SNAKE_CASE` rather than `PascalCase`. We decided (in
discussion with the SDK team) that now was as good a time as any to rip
the bandage off.
- Proto objects, or objects embedding protos, cannot be compared using
`reflect.DeepEqual` or _anything_ that uses it. This includes `testify`
and `mock` equality testers!
- You will need to use the `common/testing/protorequire`,
`common/testing/protoassert`, or `common/testing/protomock` packages
instead. I've implemented proto-compatible matchers and assertions there
for all cases I've encountered
- If you need `reflect.DeepEqual` for any reason you can use
`go.temporal.io/api/temporalproto.DeepEqual` instead
Note that history loading will not be impacted by the JSON changes: I
rewrote history loading to dynamically fix incoming history JSON data
(like all our other sdks); you can find this code in [my fork of our go
API](https://github.com/tdeebswihart/temporal-api-go/blob/master/internal/temporalhistoryv1/load.go)
alongside its tests.
**🚨Sharp Edges Introduced🚨**
Beware `*timestamppb.Timestamp.AsTime()`. If you need to extract a time
value from a proto time (timestamppb) **always** make sure to check
whether it's nil first. When the proto object is `nil` `AsTime()` will
return a non-zero time at the proto epoch: UTC midnight on January 1,
1970.
I've made this mistake multiple times during this transition and each
time it's been a pain to debug
**Is hotfix candidate?**
No.
## What changed?
Makes changes summarized in
https://github.com/temporalio/temporal/assets/52205/0660a021-588a-4573-94d3-301cae0c0040
### Empty responses on server-timeout
The behavior of `PollWorkflowExecutionUpdate` and
`UpdateWorkflowExecution` was changed as follows:
- They introduce a "server-imposed deadline" equal to current time plus
`LongPollExpirationInterval`
- If a timeout occurs due to the server-imposed deadline, then they do
not return an error, but return a response with an empty outcome.
- Otherwise, their behavior is unchanged (in particular, if a timeout
occurs due context deadline expiry, they return the error as before)
### Non-blocking poll
- Change the Update Poll endpoint so that omitting the `WaitPolicy` now
means "do not wait; respond with current Update status without
blocking". Previously omitting the `WaitPolicy` was interpreted to mean
"attempt to wait until Completed"
- Also return the `UpdateRef`
## Why?
### Empty responses on server-timeout
- This makes the behavior of the endoint consistent with
`GetWorkflowExecutionHistory`.
- This makes sense, because that endpoint is used in an analogous way
(e.g. `GetWorkflowExecutionHistory` is used for `wfHandle.result()` and
`PollWorkflowExecutionUpdateRequest` is used for
`updateHandle.result()`, and both involve polling for the result).
- In particular, a polling client interprets
timeout-on-server-imposed-deadline to mean that it should continue
polling, and we would prefer not to use an error to communicate this
non-exceptional situation.
### Non-blocking poll
SDK clients may use non-blocking poll to, for example
- Obtain current status of an Update (perhaps `updateHandle.describe()`
- Determine whether an Update with a given ID already exists / is
running
### How did you test it?
- Unit tests
- Modified Typescript SDK to not send `WaitPolicy`, ran a feature test
against the modified server and modified SDK, and confirmed API
communication as expected (this revealed that I needed to stop Frontend
mutating the request to set `WaitPolicy` to `Completed`).
- I changed the poll loop in sdk-typescript to no longer catch
`DeadlineExceeded`, ran the `update/basic_async` feature test against
the local SDK and against server binaries built prior to and with this
commit, and confirmed that the change in this PR fixed the poll loop.
### Potential risks
- Could break Update operations in SDKs
### Is hotfix candidate?
- No
<!-- Describe what has changed in this PR -->
**What changed?**
Replace xwb1989/sqlparser with forked temporalio/sqlparser
<!-- Tell your future self why have you made these changes -->
**Why?**
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**
Unit tests
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
No.
<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
No.
<!-- Describe what has changed in this PR -->
**What changed?**
I upgraded our zap version from v1.24.0 to v1.26.0, which contains
support for pkg/errors. See [this
issue](https://github.com/uber-go/zap/issues/303) and [this
commit](5fc2db7f38).
<img width="448" alt="image"
src="https://github.com/temporalio/temporal/assets/5942963/7d15d91a-27d3-45f6-9628-30bdcac38771">
<!-- Tell your future self why have you made these changes -->
**Why?**
Before this change, our logs would only contain the stack trace from
where the logger itself was invoked, not from the source of where the
error was generated or wrapped. This provided very little useful
information.
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**
I ran a custom [build of
server](https://gist.github.com/MichaelSnowden/c649dfd1efeb92f10bc72a040a792a8d)
which overwrote some deep code in history to return an error. I then set
up docker-compose to output logs -> promtail -> loki -> grafana. Then, I
queried Grafana to verify that the error log contained an "errorVerbose"
field with the stack trace from where my error was generated. As you can
see from the below image, the stack trace does appear under this field,
and if you turn on JSON parsing and newline escaping, you can both see
it rendered correctly, and you can copy-paste the stack trace.
<img width="983" alt="image"
src="https://github.com/temporalio/temporal/assets/5942963/cb9b0c83-b146-4060-9eac-3ccf9b807657">
<img width="683" alt="image"
src="https://github.com/temporalio/temporal/assets/5942963/6fa28f2f-5d04-47ae-b8a3-7512c3dd85e7">
The stack trace from Grafana:
```
oopsie woopsie
main.(*faultyShardEngine).StartWorkflowExecution
/Users/mikey/src/temporalio/temporal/.scratches/main.go:39
go.temporal.io/server/service/history.(*Handler).StartWorkflowExecution
/Users/mikey/src/temporalio/temporal/service/history/handler.go:595
go.temporal.io/server/api/historyservice/v1._HistoryService_StartWorkflowExecution_Handler.func1
/Users/mikey/src/temporalio/temporal/api/historyservice/v1/service.pb.go:1300
...
```
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
The stack traces are pretty deep because of all our gRPC interceptors.
However, we can definitely fix that later if we want by filtering the
`pkg/errors.StackTrace`. I'd rather do that in a follow-up after getting
support for this initial change first, though.
<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
No.
<!-- Describe what has changed in this PR -->
**What changed?**
Allow rate limiting a batch operation. Fixes
https://github.com/temporalio/temporal/issues/4926.
<!-- Tell your future self why have you made these changes -->
**Why?**
Batch operations are run server side and may effect millions of
executions, this in turn may overload workers and disrupt normal
operations.
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**
I started the Server locally and initiated a batch operation from the
CLI ([CLI changes be found
here](https://github.com/temporalio/cli/pull/366)):
- [x] uses provided limit
- [x] uses server limit when not provided
- [x] caps it at server limit
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
Rate limiting to be incorrect and slow down processing or disrupt
cluster.
<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
No
**What changed?**
- In CI, build most of the tests and dependencies and reuse them in
subsequent steps.
- Additionally pre-build dependencies separately on a weekly schedule
(in buildkite configuration) and reuse them when possible.
- Add more test sharding for functional tests (split the main functional
test suite into three).
**Why?**
Increase parallelism, reduce latency of getting test results, reduce
granularity of retries
**How did you test it?**
lots of testing on buildkite
**Potential risks**
**Is hotfix candidate?**
<!-- Describe what has changed in this PR -->
**What changed?**
Using `gotestsum` - [as suggested by
Buildkite](https://buildkite.com/docs/test-analytics/golang-collectors)
- and uploading the artifacts to their Analytics product.
[Results will be available
here](https://buildkite.com/organizations/temporal/analytics/suites/temporal-public?branch=all+branches).
<!-- Tell your future self why have you made these changes -->
**Why?**
We want to have better insight into which tests are flaky and/or slow.
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**
Running this branch in Buildkite.
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
None?
<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
No.
<!-- Describe what has changed in this PR -->
**What changed?**
`Driver` interface (with two implementations) was added to the
postgresql persistence package in order to support other drivers than
`lib/pq`. Fixes https://github.com/temporalio/temporal/issues/1775
Essentially, this PR does
https://github.com/temporalio/temporal/pull/4914 in a
backward-compatible and extendable way
<!-- Tell your future self why have you made these changes -->
**Why?**
Since `lib/pq` does not support multi-node clusters
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**
Existing tests
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
No, since you should explicitly change the driver in your config file.
However, `connectAttributes` depend on the driver (e.g.,
`binary_parameters` is supported only by `lib/pq` and is not a part of
postresql standard), so one should be careful changing drivers in
production.
<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
No
**What changed?**
The minimum go version was bumped from 1.20 to 1.21 in our go module
file.
**Why?**
Nexus would like to use the new `log/slog` package so let's support
that.
**How did you test it?**
Tests passed locally and in CI
**Potential risks**
Given the go team's aversion to breaking changes I'd say none. I've
spoken with @dnr in slack about how we can control the exact version of
go we use in CI to ensure we don't see any weirdness with unknown
version updates going forward. [We can set the new `GOTOOLCHAIN`
environment variable to `local`](toolchain) to prevent Go from
downloading new toolchains without our knowledge
**Is hotfix candidate?**
Nope.
[toolchain]: https://go.dev/doc/toolchain
<!-- Describe what has changed in this PR -->
**What changed?**
Rename leftover `master` to `main`.
<!-- Tell your future self why have you made these changes -->
**Why?**
`master` branch was renamed to `main` but some places still reference
`master`.
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**
N/A
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
No risks.
<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
No.
<!-- Describe what has changed in this PR -->
**What changed?**
<!-- Tell your future self why have you made these changes -->
**Why?**
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
**How did you test it?**
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->
**Potential risks**
<!-- Is this PR a hotfix candidate or require that a notification be
sent to the broader community? (Yes/No) -->
**Is hotfix candidate?**
* Bump base image (temporalio/base-ci-builder) version to 1.7.0
* Run gofmt & goimports
* Bump go.mod go directive to 1.19
* Bump go version for goreleaser workflow
* Change ES client GetMapping to work with ES8
* Add support for ES8 and integration test
* Add ES8 integration test to buildkite coverage report
* Create es_v8_index_template.json
* Add env vars for ES8 container
