mirror of
https://github.com/temporalio/temporal.git
synced 2026-08-30 18:41:49 -07:00
15f3532ea1
## What changed
- Add `worker_deployment_name` and `worker_build_id` labels to Workflow
Task completion and failure metrics.
- Add the split labels to Activity success, failure, cancellation,
timeout, and completion-latency metrics.
- Add the split labels to Workflow Task and Activity schedule-to-start
latency and poll-time task-dispatch latency.
- Continue emitting the existing combined `worker_version` label on
task-dispatch latency for compatibility.
- Reuse `metrics.breakdownByBuildID` as the task-queue-scoped
cardinality gate for the new labels.
## Which metrics changed
Every metric below now emits both `worker_deployment_name` and
`worker_build_id` when `metrics.breakdownByBuildID` is enabled and
deployment attribution is available. When the gate is disabled or
attribution is unavailable—including unversioned tasks and timeouts
before a worker starts—both labels remain present with empty values.
### Workflow Task outcomes
- `workflow_tasks_completed`
- `failed_workflow_tasks`
### Activity outcomes
- `activity_success`
- `activity_fail`
- `activity_task_fail`
- `activity_cancel`
- `activity_timeout`
- `activity_task_timeout`
### Activity latency
- `activity_end_to_end_latency` (deprecated; use
`activity_start_to_close_latency` instead)
- `activity_start_to_close_latency`
- `activity_schedule_to_close_latency`
### Task-routing latency
- `task_schedule_to_start_latency`
- `task_dispatch_latency` (continues to emit the existing combined
`worker_version` label as well)
## Why
- To improve user-experience for worker-versioning by having more
insights
- Worker Deployment name and build ID need to be independently
filterable. Emitting them separately avoids requiring consumers to parse
the combined `worker_version` value. This PR preserves `worker_version`
on `task_dispatch_latency` for compatibility and does not remove or
deprecate it.
## How each metric was tested
The functional tests run real versioned Workflow and Activity workers
with `metrics.breakdownByBuildID` enabled. They assert that emitted
server metrics contain the actual Worker Deployment name and build
ID—not merely that the label keys exist.
| Functional scenario | Metrics verified | What is asserted |
|---|---|---|
| Workflow and Activity task dispatch |
`task_schedule_to_start_latency`, `task_dispatch_latency` | Both
Workflow and Activity task series contain real deployment/build values
when enabled. Disabled and unversioned cases emit empty values. |
| Successful Workflow and Activity | `workflow_tasks_completed`,
`activity_success`, `activity_end_to_end_latency`,
`activity_start_to_close_latency`, `activity_schedule_to_close_latency`
| Successful completion series contain real deployment/build values. |
| Terminal Activity failure | `activity_task_fail`, `activity_fail`,
`activity_end_to_end_latency`, `activity_start_to_close_latency`,
`activity_schedule_to_close_latency` | Both the failed attempt and
terminal-failure series retain real worker attribution. |
| Activity cancellation | `activity_cancel` | The cancellation series is
attributed to the worker that started the Activity. |
| Terminal Activity timeout | `activity_task_timeout`,
`activity_timeout` | Both attempt-level and terminal timeout series
retain the started worker’s deployment/build values. |
| Failed Workflow Task | `failed_workflow_tasks` | A real versioned
Workflow Task is polled and failed; the resulting series contains the
poller’s deployment/build values. |
### Local end-to-end validation
The PR server was also run locally with the Worker Deployment versioning
canary and a `bench-go` workload.
Prometheus was scraped directly to verify that:
- real `worker_deployment_name` and `worker_build_id` values were
emitted;
- versioned and unversioned series were both accepted without
inconsistent-label errors;
- `activity_success`, `activity_task_fail`,
`activity_end_to_end_latency`, `activity_start_to_close_latency`, and
`activity_schedule_to_close_latency` carried the expected real
deployment/build values.
The remaining failure, cancellation, timeout, and failed-Workflow-Task
paths are covered by the functional tests above.
NEW: Also tested each of these 13 metric families in a cloud test cell
with sample metrics pasted here:
https://grafana.tmprl-internal.cloud/d/shdb6gf/new-dashboard?orgId=1&from=2026-08-27T00:00:00.000Z&to=2026-08-27T23:59:59.000Z&timezone=utc
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Broad metrics surface area across History and Matching with
cardinality gated by config; deployment attribution on timeouts and
eager starts changes which tag values appear on existing metric names.
>
> **Overview**
> Adds **`worker_deployment_name`** and **`worker_build_id`** to History
and Matching metrics so worker versioning can be filtered without
parsing the combined **`worker_version`** label.
**`metrics.breakdownByBuildID`** (task-queue scoped) controls whether
those tags carry real values or empty strings;
**`task_dispatch_latency`** still emits **`worker_version`** for
compatibility.
>
> **Workflow tasks:** Completion and failure counters
(`workflow_tasks_completed`, `failed_workflow_tasks`) now go through
shared helpers that attach versioning behavior plus deployment tags from
the poller’s **`DeploymentOptions`**.
>
> **Workflow activities:** Success, failure, cancel, timeout, and
related latency metrics get the same split labels via
**`VersioningMetricContext`** on respond paths (from request deployment
options), timer-driven timeouts (from **`LastDeploymentVersion`** when
an attempt had started), and schedule-to-start latency on
**`RecordActivityTaskStarted`**. Eager activities started during WFT
completion now record the completing worker’s deployment on the started
event.
>
> **Standalone activities (CHASM):** Completion/timeout/cancel paths use
**`completionMetricsHandler`**, which always includes the deployment
label keys with empty values until standalone versioning exists—keeping
Prometheus label-set parity with workflow-embedded activities.
>
> **Other:** **`AddActivityTaskStartedEvent`** clears stored deployment
when the poller is unversioned; Matching **`task_dispatch_latency`**
adds the new tags alongside **`worker_version`**.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
1c60b05f06. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->