Refactor deep health check code for history service and add a 60-second wait before returning unhealthy to the DeepHealthCheck (#10093)

## What changed?
Pulled out the DeepHealthCheck into its own file. Added a fixed delay
from startup during which DeepHealthCheck will see NOT_SERVING from the
local health server as SERVING

## Why?
During history service scale-up/scale-downs, individual history hosts
can be detected as "NOT_SERVING" because the shard initialization hasn't
happened yet. This is an expected behavior from the history service pod
and does not indicate an error, but DeepHealthCheck currently treats it
as one.

## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [ ] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)

## Potential risks
This change makes the DeepHealthCheck less likely to expose a problem
that causes an individual history service to reprovision faster than the
suppression interval. The standard HealthCheck for historyservice is
unaffected.

---------

Co-authored-by: Stephen Stanton <stephenstanton10@gmail.com>

service/history/configs/config.go

@@ -408,6 +408,7 @@ type Config struct {
 	HealthPersistenceErrorRatio     dynamicconfig.FloatPropertyFn
 	HealthRPCLatencyFailure         dynamicconfig.FloatPropertyFn
 	HealthRPCErrorRatio             dynamicconfig.FloatPropertyFn
+	HealthHistoryInitializationTime dynamicconfig.DurationPropertyFn
 	BreakdownMetricsByTaskQueue     dynamicconfig.BoolPropertyFnWithTaskQueueFilter
 
 	LogAllReqErrors dynamicconfig.BoolPropertyFnWithNamespaceFilter
@@ -789,6 +790,7 @@ func NewConfig(
 		HealthPersistenceErrorRatio:     dynamicconfig.HealthPersistenceErrorRatio.Get(dc),
 		HealthRPCLatencyFailure:         dynamicconfig.HealthRPCLatencyFailure.Get(dc),
 		HealthRPCErrorRatio:             dynamicconfig.HealthRPCErrorRatio.Get(dc),
+		HealthHistoryInitializationTime: dynamicconfig.HealthHistoryInitializationTime.Get(dc),
 
 		BreakdownMetricsByTaskQueue: dynamicconfig.MetricsBreakdownByTaskQueue.Get(dc),