Github/temporal
1
0
Fork 0
mirror of https://github.com/temporalio/temporal.git synced 2026-08-30 18:41:49 -07:00
Code Issues Packages Projects Releases Wiki Activity
Files
d69aede7e5d227fec10d3d36ec45f97792686ead
temporal/api/health/v1
History
Lanie Hei 88ea88ce1a Add per-check diagnostics to DeepHealthCheck API (#9350) 
## Summary

Extends the `DeepHealthCheck` API to return **per-check diagnostic
details** alongside the existing `HealthState` enum. When fault
detection triggers a cell failover, operators can now see exactly which
health check failed and why — not just that the cell is unhealthy.

### What changed

- **New proto package `health/v1`** — `HealthCheck` message with
`check_type` (string), `state`, `value`, `threshold`, and human-readable
`message`. `HostHealthDetail` and `ServiceHealthDetail` aggregate
per-host and per-service results.
- **New enum value `HEALTH_STATE_INTERNAL_ERROR`** — for infrastructure
failures like membership resolver errors (previously returned
`UNSPECIFIED`).
- **History handler** now runs all 5 checks unconditionally (gRPC
health, RPC latency, RPC error ratio, persistence latency, persistence
error ratio) and returns each with actual values and thresholds.
Previously it early-returned on first failure.
- **Frontend health checker** collects per-host `HostHealthDetail`
(address, state, checks) and builds a `ServiceHealthDetail` with
diagnostic messages for all paths — including resolver errors and empty
membership.
- **AdminService** passes `ServiceHealthDetail` through to callers.
- **`check_type` uses string constants**
(`common/health/check_types.go`) instead of a proto enum for
extensibility — new check types can be added without proto changes, and
the `message` field provides human-readable context with actual values
(e.g. `"RPC latency 850.00ms exceeded 500.00ms threshold"`).

### How it works

The call chain is: `AdminService.DeepHealthCheck()` →
`HealthChecker.Check()` → fan-out to all history hosts in membership →
`HistoryHandler.DeepHealthCheck()` per host.

Each history host runs **5 independent checks** and returns all results:
1. `grpc_health` — is the gRPC health server serving?
2. `rpc_latency` — average RPC latency vs threshold
3. `rpc_error_ratio` — RPC error rate vs threshold
4. `persistence_latency` — DB latency vs threshold
5. `persistence_error_ratio` — DB error rate vs threshold

The frontend collects results from **all hosts in membership**,
aggregates them, and returns the full breakdown.

### Example responses

#### Healthy cluster (3 hosts, all serving)

```json
{
  "state": "HEALTH_STATE_SERVING",
  "services": [{
    "service": "history",
    "state": "HEALTH_STATE_SERVING",
    "hosts": [
      {
        "address": "10.0.1.5:7234",
        "state": "HEALTH_STATE_SERVING",
        "checks": [
          {"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
          {"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 45.0, "threshold": 500.0},
          {"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.01, "threshold": 0.1},
          {"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 12.0, "threshold": 500.0},
          {"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
        ]
      },
      {
        "address": "10.0.1.6:7234",
        "state": "HEALTH_STATE_SERVING",
        "checks": [
          {"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
          {"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 52.0, "threshold": 500.0},
          {"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1},
          {"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 18.0, "threshold": 500.0},
          {"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
        ]
      },
      {
        "address": "10.0.1.7:7234",
        "state": "HEALTH_STATE_SERVING",
        "checks": [
          {"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
          {"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 38.0, "threshold": 500.0},
          {"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
          {"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 15.0, "threshold": 500.0},
          {"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
        ]
      }
    ]
  }]
}
```

#### Degraded cluster — 1 host with high RPC latency (3 hosts, 1
failing, under threshold)

The failing host clearly shows which check triggered and the actual vs
threshold values. Because only 1/3 hosts failed (33%) and the failure
threshold is 25% but we require at least 2 hosts to fail, the overall
state remains `SERVING`.

```json
{
  "state": "HEALTH_STATE_SERVING",
  "services": [{
    "service": "history",
    "state": "HEALTH_STATE_SERVING",
    "hosts": [
      {
        "address": "10.0.1.5:7234",
        "state": "HEALTH_STATE_NOT_SERVING",
        "checks": [
          {"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
          {"check_type": "rpc_latency", "state": "HEALTH_STATE_NOT_SERVING", "value": 850.0, "threshold": 500.0, "message": "RPC latency 850.00ms exceeded 500.00ms threshold"},
          {"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
          {"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 120.0, "threshold": 500.0},
          {"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
        ]
      },
      {
        "address": "10.0.1.6:7234",
        "state": "HEALTH_STATE_SERVING",
        "checks": [
          {"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
          {"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 52.0, "threshold": 500.0},
          {"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1},
          {"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 18.0, "threshold": 500.0},
          {"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
        ]
      },
      {
        "address": "10.0.1.7:7234",
        "state": "HEALTH_STATE_SERVING",
        "checks": [
          {"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
          {"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 38.0, "threshold": 500.0},
          {"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
          {"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 15.0, "threshold": 500.0},
          {"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
        ]
      }
    ]
  }]
}
```

#### Unhealthy cluster — hosts unreachable (6 hosts in membership, 3
unreachable via RPC)

When the frontend cannot reach a host via RPC, it creates a synthetic
`host_availability` check with the error. The host appears in the
response with `NOT_SERVING` and the RPC error message. With 3/6 hosts
failing (50% > 25% threshold), the overall state is `NOT_SERVING`.

```json
{
  "state": "HEALTH_STATE_NOT_SERVING",
  "services": [{
    "service": "history",
    "state": "HEALTH_STATE_NOT_SERVING",
    "hosts": [
      {
        "address": "10.0.1.5:7234",
        "state": "HEALTH_STATE_SERVING",
        "checks": [
          {"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
          {"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 45.0, "threshold": 500.0},
          {"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.01, "threshold": 0.1},
          {"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 12.0, "threshold": 500.0},
          {"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
        ]
      },
      {
        "address": "10.0.1.6:7234",
        "state": "HEALTH_STATE_SERVING",
        "checks": [
          {"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
          {"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 52.0, "threshold": 500.0},
          {"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1},
          {"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 18.0, "threshold": 500.0},
          {"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
        ]
      },
      {
        "address": "10.0.1.7:7234",
        "state": "HEALTH_STATE_SERVING",
        "checks": [
          {"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
          {"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 38.0, "threshold": 500.0},
          {"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
          {"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 15.0, "threshold": 500.0},
          {"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
        ]
      },
      {
        "address": "10.0.1.8:7234",
        "state": "HEALTH_STATE_NOT_SERVING",
        "checks": [
          {"check_type": "host_availability", "state": "HEALTH_STATE_NOT_SERVING", "message": "failed to reach host for health check: rpc error: code = Unavailable desc = connection refused"}
        ]
      },
      {
        "address": "10.0.1.9:7234",
        "state": "HEALTH_STATE_NOT_SERVING",
        "checks": [
          {"check_type": "host_availability", "state": "HEALTH_STATE_NOT_SERVING", "message": "failed to reach host for health check: rpc error: code = Unavailable desc = connection refused"}
        ]
      },
      {
        "address": "10.0.1.10:7234",
        "state": "HEALTH_STATE_NOT_SERVING",
        "checks": [
          {"check_type": "host_availability", "state": "HEALTH_STATE_NOT_SERVING", "message": "failed to reach host for health check: context deadline exceeded"}
        ]
      }
    ]
  }]
}
```

#### Host voluntarily draining — gRPC health declined (DECLINED_SERVING)

When a host's gRPC health server reports not serving (e.g. during
graceful shutdown), the check returns `DECLINED_SERVING`. If enough
hosts are in this state (exceeding the declined serving proportion
threshold), the overall service state becomes `DECLINED_SERVING`.

```json
{
  "state": "HEALTH_STATE_DECLINED_SERVING",
  "services": [{
    "service": "history",
    "state": "HEALTH_STATE_DECLINED_SERVING",
    "hosts": [
      {
        "address": "10.0.1.5:7234",
        "state": "HEALTH_STATE_DECLINED_SERVING",
        "checks": [
          {"check_type": "grpc_health", "state": "HEALTH_STATE_DECLINED_SERVING", "message": "gRPC health server not serving"},
          {"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 45.0, "threshold": 500.0},
          {"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.01, "threshold": 0.1},
          {"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 12.0, "threshold": 500.0},
          {"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
        ]
      },
      {
        "address": "10.0.1.6:7234",
        "state": "HEALTH_STATE_DECLINED_SERVING",
        "checks": [
          {"check_type": "grpc_health", "state": "HEALTH_STATE_DECLINED_SERVING", "message": "gRPC health server not serving"},
          {"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 52.0, "threshold": 500.0},
          {"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1},
          {"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 18.0, "threshold": 500.0},
          {"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
        ]
      },
      {
        "address": "10.0.1.7:7234",
        "state": "HEALTH_STATE_SERVING",
        "checks": [
          {"check_type": "grpc_health", "state": "HEALTH_STATE_SERVING"},
          {"check_type": "rpc_latency", "state": "HEALTH_STATE_SERVING", "value": 38.0, "threshold": 500.0},
          {"check_type": "rpc_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.02, "threshold": 0.1},
          {"check_type": "persistence_latency", "state": "HEALTH_STATE_SERVING", "value": 15.0, "threshold": 500.0},
          {"check_type": "persistence_error_ratio", "state": "HEALTH_STATE_SERVING", "value": 0.0, "threshold": 0.1}
        ]
      }
    ]
  }]
}
```

#### No hosts in membership

When the membership resolver returns an empty host list, the response
includes a service-level message but no hosts.

```json
{
  "state": "HEALTH_STATE_NOT_SERVING",
  "services": [{
    "service": "history",
    "state": "HEALTH_STATE_NOT_SERVING",
    "message": "no available hosts in membership"
  }]
}
```

#### Membership resolver failure (INTERNAL_ERROR)

When the frontend can't resolve the membership ring at all
(infrastructure failure), the response includes `INTERNAL_ERROR` with
the resolver error.

```json
{
  "state": "HEALTH_STATE_INTERNAL_ERROR",
  "services": [{
    "service": "history",
    "state": "HEALTH_STATE_INTERNAL_ERROR",
    "message": "failed to get membership resolver: membership monitor not started"
  }]
}
```

### Backward compatibility

- `DeepHealthCheckResponse.state` (field 1) unchanged in both history
and admin protos
- New fields (`checks`, `services`) are additive (field 2) — old clients
simply ignore them
- `GetState()` continues to work as before

### Related

- saas-control-plane PR #12203 — `HealthReport` + `CellHealthEvent`
(consumer side, ready to use these fields)
- Runbooks PR #1231 — end-to-end flow documentation

## Test plan

- [x] All existing `TestHealthCheckerSuite` tests pass (19 tests)
- [x] New tests: `Test_Check_ServiceDetail_Populated`,
`Test_Check_HostChecks_Propagated`, `Test_Check_GetResolver_Error`
(INTERNAL_ERROR + message), `Test_Check_No_Available_Hosts` (message)
- [x] Full `go build ./...` passes
- [ ] Verify saas-control-plane can import
`go.temporal.io/server/common/health` constants
- [ ] Integration test with actual history service

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 22:51:58 +00:00
..
message.go-helpers.pb.go
Add per-check diagnostics to DeepHealthCheck API (#9350)
2026-02-24 22:51:58 +00:00
message.pb.go
Add per-check diagnostics to DeepHealthCheck API (#9350)
2026-02-24 22:51:58 +00:00