Reorganize docs (#5535)

## What changed?
<!-- Describe what has changed in this PR -->
I reorganized our docs into `docs/{admin,architecture,development}`.

## Why?
<!-- Tell your future self why have you made these changes -->
We discussed this internally, but I believe this is a better
segmentation for the different audiences looking at our docs. I plan on
adding more stuff to docs/admin like metrics and dynamic config docs.

## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
I'm mainly looking for broken links.
- I went through each markdown file in docs manually. 
- I looked through all inspection errors in the IDE.
- I looked at all references to "docs/" or "develop/" in our code.

## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->

## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->

## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->

CONTRIBUTING.md

@@ -1,7 +1,7 @@
 # Develop Temporal Server
 This doc is for contributors to Temporal Server (hopefully that's you!)
 
-**Note:** All contributors also need to fill out the [Temporal Contributor License Agreement](develop/docs/temporal-cla.md) before we can merge in any of your changes.
+**Note:** All contributors also need to fill out the [Temporal Contributor License Agreement](docs/development/temporal-cla.md) before we can merge in any of your changes.
 
 ## Prerequisites
 
@@ -22,7 +22,7 @@ This doc is for contributors to Temporal Server (hopefully that's you!)
 * [docker](https://docs.docker.com/engine/install/)
 
 > Note: it is possible to run Temporal server without a `docker`. If for some reason (for example, performance on macOS)
-> you want to run dependencies on the host OS, please follow the [doc](develop/docs/run_dependencies_host.md).
+> you want to run dependencies on the host OS, please follow the [doc](docs/development/run-dependencies-host.md).
 
 ### For Windows developers
 For developing on Windows, install [Windows Subsystem for Linux 2 (WSL2)](https://aka.ms/wsl) and [Ubuntu](https://docs.microsoft.com/en-us/windows/wsl/install-win10#step-6---install-your-linux-distribution-of-choice). After that, follow the guidance for installing prerequisites, building, and testing on Ubuntu.

README.md

@@ -20,7 +20,7 @@ Learn more:
 
 - [Courses](https://learn.temporal.io/courses/temporal_101/)
 - [Docs](https://docs.temporal.io)
-- Internal architecture: [docs/](./docs/README.md)
+- Internal architecture: [docs/](./docs/architecture/README.md)
 
 ## Getting Started
 
@@ -59,7 +59,7 @@ This repository contains the source code of the Temporal server. To implement Wo
 
 ## Contributing
 
-We'd love your help in making Temporal great. Please review the [internal architecture docs](./docs/README.md) and our [contribution guide](CONTRIBUTING.md).
+We'd love your help in making Temporal great. Please review the [internal architecture docs](./docs/architecture/README.md) and our [contribution guide](CONTRIBUTING.md).
 
 If you'd like to work on or propose a new feature, first peruse [feature requests](https://community.temporal.io/c/feature-requests/6) and our [proposals repo](https://github.com/temporalio/proposals) to discover existing active and accepted proposals.
 

docs/Makefile

@@ -1,6 +1,6 @@
 .DEFAULT_GOAL := update-diagrams
 COLOR := "\e[1;36m%s\e[0m\n"
-DIAGRAMS := $(wildcard ./docs/assets/*.d2)
+DIAGRAMS := $(wildcard ./docs/_assets/*.d2)
 
 install-d2:
 	@printf $(COLOR) "Install d2..."

docs/README.md

@@ -1,82 +1,20 @@
-These pages contain an introduction to Temporal's system architecture and internal implementation.
+# Temporal Server Documentation
 
-Below we give a high-level overview of how Temporal works; other pages provide more detailed descriptions of the services and of the lifecycle of a Workflow Execution:
+This folder contains docs for those working closely with the Temporal server. 
+If you are more interested in just authoring workflows, see our [Getting Started Guide](../README.md#getting-started).
 
-**Internal services**
+## Core Documentation Sections
 
-- Frontend Service
-- [History Service](./history-service.md)
-- [Matching Service](./matching-service.md)
-- Internal Workers Service
+### Architectural Diagrams (`/architecture`)
+This section contains high-level explanations of Temporal's architecture and core concepts. 
+It is designed to be useful for both server developers and those interested in understanding the technological underpinnings of Temporal. 
+Detailed diagrams and descriptions can be found there.
 
-**Workflow lifecycle**
+### Development Instructions (`/development`)
+Here, you'll find guides to setting up a local development environment, along with potentially more advanced topics such as adding migrations or new Remote Procedure Calls (RPCs). 
+This section is essential for developers looking to contribute to the Temporal codebase or understand its inner workings.
 
-- [Sequence diagrams](./workflow-lifecycle.md)
-- [SDK Workers](https://github.com/temporalio/sdk-core/blob/master/ARCHITECTURE.md)
-
-A Temporal cluster executes units of application logic called Workflows in a durable manner that automatically handles intermittent failures, and retries failed operations.
-The following are some fundamental premises:
-
-**Requirements**
-
-- Workflows are defined as code, in one of the supported [SDK languages](https://docs.temporal.io/dev-guide).
-
-- _Durable execution_ of workflows must be guaranteed: workflows must still execute correctly in the face of transient failures in server processes and user-hosted processes.
-
-- The system can be scaled to handle arbitrarily many concurrent workflow executions.
-
-- User code is executed in environments owned by the user.
-
-**Design decisions**
-
-- The system functions via event sourcing: an append-only history of events is stored for each workflow execution, and all required workflow state can be recreated at any time by replaying this history.
-
-- User code defining workflows is segregated into [Workflow](https://docs.temporal.io/workflows) definitions and [Activity](https://docs.temporal.io/activities) definitions.
-  Workflow code must be deterministic and have no side effects (with specific exceptions), and activity code must either be idempotent or non-retryable (i.e. at least once or at most once).
-
-## High-level architecture
-
-These premises have led to a system architecture which is divided into user-hosted processes, versus the Temporal cluster processes (which may be hosted outside the user's systems):
-
-<!-- https://lucid.app/lucidchart/0202e4b8-5258-4cd6-a6a0-67159300532b/edit -->
-<img src="assets/temporal-high-level.svg">
-
-### User-hosted processes
-
-- The User Application uses one of the [Temporal SDKs](https://docs.temporal.io/dev-guide) as a gRPC client to communicate with the Temporal server to start/cancel workflows, and interact with running workflows.
-
-- In addition, the user segregates some of their application code into Temporal Workflow and Activity definitions, and hosts [Worker](https://docs.temporal.io/workers) processes, which execute their Workflow and Activity code.
-  Workflow and Activity code uses the SDK as a library, and the Worker runtime is implemented by the SDK.
-
-- The worker processes communicate with the Temporal server in two ways: they continuously poll the server for Workflow and Activity tasks, and on completion of each task they send information to the server.
-  (For a Workflow task they send commands to the server specifying what must be done to further advance the Workflow execution, and for an Activity task they send the task result or failure information.)
-  See [Tasks](#Tasks) below.
-
-### Temporal Cluster
-
-- History Service shards manage individual [Workflow Executions](https://docs.temporal.io/workflows#workflow-execution).
-  They handle RPCs originating from the User Application and the Temporal Worker, drive the Workflow Execution to completion by enqueuing Workflow and Activity Tasks in the Matching Service, and store all state required for durable execution of the workflow.
-- The Matching Service manages the [Task Queues](https://docs.temporal.io/workers#task-queue) being polled by Temporal Worker processes.
-  A single task queue holds tasks for multiple Workflow Executions.
-- Users can host and operate the Temporal server and its database themselves, or use [Temporal Cloud](https://temporal.io/cloud).
-
-### Tasks
-
-Temporal Workers poll the Task Queues in the Matching service for tasks. There are two main types of task:
-
-- A **Workflow Task** is processed by resuming execution of the user's workflow code until it becomes blocked (e.g. on a timer or an Activity call), or is complete.
-  On completion of a Workflow Task the worker sends a sequence of commands specifying what is required to advance the workflow (e.g. set a timer, schedule an Activity task).
-
-- An **Activity Task** is processed by attempting to execute an Activity.
-  On completion of an Activity Task (whether success or failure), the worker sends information about the activity outcome to the server.
-
-In addition, Queries are implemented via Query Tasks, which are similar to Workflow Tasks but result only in Query results being sent to the server (do not cause the Workflow to advance).
-
-## Further reading
-
-- Frontend Service
-- [History Service](./history-service.md)
-- [Matching Service](./matching-service.md)
-- Internal Workers Service
-- [Workflow lifecycle sequence diagrams](./workflow-lifecycle.md)
-- [SDK Workers](https://github.com/temporalio/sdk-core/blob/master/ARCHITECTURE.md)
+### Operational Guides (`/admin`)
+This section provides reference materials for administrators responsible for deploying Temporal in a production environment. 
+It is pretty bare for now, but it is intended to covers topics like spinning up a production environment, configuring it, and monitoring it with metrics and dynamic configurations.
+For now, you can find this info on the Temporal docs website at https://docs.temporal.io/self-hosted-guide.

docs/architecture/README.md

@@ -0,0 +1,82 @@
+These pages contain an introduction to Temporal's system architecture and internal implementation.
+
+Below we give a high-level overview of how Temporal works; other pages provide more detailed descriptions of the services and of the lifecycle of a Workflow Execution:
+
+**Internal services**
+
+- Frontend Service
+- [History Service](./history-service.md)
+- [Matching Service](./matching-service.md)
+- Internal Workers Service
+
+**Workflow lifecycle**
+
+- [Sequence diagrams](./workflow-lifecycle.md)
+- [SDK Workers](https://github.com/temporalio/sdk-core/blob/master/ARCHITECTURE.md)
+
+A Temporal cluster executes units of application logic called Workflows in a durable manner that automatically handles intermittent failures, and retries failed operations.
+The following are some fundamental premises:
+
+**Requirements**
+
+- Workflows are defined as code, in one of the supported [SDK languages](https://docs.temporal.io/dev-guide).
+
+- _Durable execution_ of workflows must be guaranteed: workflows must still execute correctly in the face of transient failures in server processes and user-hosted processes.
+
+- The system can be scaled to handle arbitrarily many concurrent workflow executions.
+
+- User code is executed in environments owned by the user.
+
+**Design decisions**
+
+- The system functions via event sourcing: an append-only history of events is stored for each workflow execution, and all required workflow state can be recreated at any time by replaying this history.
+
+- User code defining workflows is segregated into [Workflow](https://docs.temporal.io/workflows) definitions and [Activity](https://docs.temporal.io/activities) definitions.
+  Workflow code must be deterministic and have no side effects (with specific exceptions), and activity code must either be idempotent or non-retryable (i.e. at least once or at most once).
+
+## High-level architecture
+
+These premises have led to a system architecture which is divided into user-hosted processes, versus the Temporal cluster processes (which may be hosted outside the user's systems):
+
+<!-- https://lucid.app/lucidchart/0202e4b8-5258-4cd6-a6a0-67159300532b/edit -->
+<img src="../_assets/temporal-high-level.svg">
+
+### User-hosted processes
+
+- The User Application uses one of the [Temporal SDKs](https://docs.temporal.io/dev-guide) as a gRPC client to communicate with the Temporal server to start/cancel workflows, and interact with running workflows.
+
+- In addition, the user segregates some of their application code into Temporal Workflow and Activity definitions, and hosts [Worker](https://docs.temporal.io/workers) processes, which execute their Workflow and Activity code.
+  Workflow and Activity code uses the SDK as a library, and the Worker runtime is implemented by the SDK.
+
+- The worker processes communicate with the Temporal server in two ways: they continuously poll the server for Workflow and Activity tasks, and on completion of each task they send information to the server.
+  (For a Workflow task they send commands to the server specifying what must be done to further advance the Workflow execution, and for an Activity task they send the task result or failure information.)
+  See [Tasks](#Tasks) below.
+
+### Temporal Cluster
+
+- History Service shards manage individual [Workflow Executions](https://docs.temporal.io/workflows#workflow-execution).
+  They handle RPCs originating from the User Application and the Temporal Worker, drive the Workflow Execution to completion by enqueuing Workflow and Activity Tasks in the Matching Service, and store all state required for durable execution of the workflow.
+- The Matching Service manages the [Task Queues](https://docs.temporal.io/workers#task-queue) being polled by Temporal Worker processes.
+  A single task queue holds tasks for multiple Workflow Executions.
+- Users can host and operate the Temporal server and its database themselves, or use [Temporal Cloud](https://temporal.io/cloud).
+
+### Tasks
+
+Temporal Workers poll the Task Queues in the Matching service for tasks. There are two main types of task:
+
+- A **Workflow Task** is processed by resuming execution of the user's workflow code until it becomes blocked (e.g. on a timer or an Activity call), or is complete.
+  On completion of a Workflow Task the worker sends a sequence of commands specifying what is required to advance the workflow (e.g. set a timer, schedule an Activity task).
+
+- An **Activity Task** is processed by attempting to execute an Activity.
+  On completion of an Activity Task (whether success or failure), the worker sends information about the activity outcome to the server.
+
+In addition, Queries are implemented via Query Tasks, which are similar to Workflow Tasks but result only in Query results being sent to the server (do not cause the Workflow to advance).
+
+## Further reading
+
+- Frontend Service
+- [History Service](./history-service.md)
+- [Matching Service](./matching-service.md)
+- Internal Workers Service
+- [Workflow lifecycle sequence diagrams](./workflow-lifecycle.md)
+- [SDK Workers](https://github.com/temporalio/sdk-core/blob/master/ARCHITECTURE.md)

docs/architecture/history-service.md

@@ -4,7 +4,7 @@ This page is an introduction to the History Service architecture and internal im
 See [Temporal overview](./README.md) for a system architecture overview placing this in context.
 
 <!-- https://lucid.app/lucidchart/0202e4b8-5258-4cd6-a6a0-67159300532b/edit -->
-<img src="./assets/temporal-high-level.svg">
+<img src="../_assets/temporal-high-level.svg">
 
 The History Service handles two main types of request (gRPC) relating to an individual Workflow Execution:
 

docs/architecture/matching-service.md

@@ -3,7 +3,7 @@
 [see [API definition](https://github.com/temporalio/temporal/blob/main/proto/internal/temporal/server/api/matchingservice/v1/service.proto)]
 
 <!-- https://lucid.app/lucidchart/0202e4b8-5258-4cd6-a6a0-67159300532b/edit -->
-<img src="./assets/matching-context.svg">
+<img src="../_assets/matching-context.svg">
 
 Matching Service instances manage [Task Queues](https://docs.temporal.io/workers#task-queue) being polled by Temporal Worker processes.
 Long-poll requests from Temporal Workers are received by the Frontend Service, which routes them to the Matching Service instance responsible for the requested Task Queue.

docs/development/new-rpcs.md

@@ -13,5 +13,5 @@ Here's a list of what you'll need to modify:
 5. For frontend: add definitions to `service/frontend/dcRedirectionHandler.go`.
    (In the future hopefully we can make this generated or use interceptors.)
 6. Add your new methods to `service/<service>/configs/quotas.go`.
-7. Finally implement your new methods in the RPC handler.
+7. Finally, implement your new methods in the RPC handler.
 

docs/development/tracing.md

@@ -161,12 +161,10 @@ benefit of sharing a single mapping function.
 ### Start a span in `common` or other non-service-specific code
 
 *Q:* Given that common code can be called from any service, how can I start a span
-in common library code that is bound the the appropriate service
-(frontend/history/matching/worker)?
+in common library code that is bound the appropriate service (frontend/history/matching/worker)?
 
 *A:* The `TracerProvider` that created the currently active Span can be retrieved
-from that Span itself and the currently active Span can be received from from
-the `context.Context`.
+from that Span itself and the currently active Span can be received from the `context.Context`.
 
 ```
 // DoFoo is a function in the common package
@@ -180,7 +178,7 @@ func DoFoo(ctx context.Context, x int, y string) string {
 
 ### `RecordError` does not imply Span failure
 
-Using `Span.RecordError` is a good idea but not all errors imply failure. Thus
+Using `Span.RecordError` is a good idea but not all errors imply failure. Thus,
 if you want to capture an error _and also_ capture that a span failed, you must
 additionally call `Span.SetStatus(codes.Error, err.Error())`. A
 `FailSpanWithError` utility function might be a good idea.
@@ -188,7 +186,7 @@ additionally call `Span.SetStatus(codes.Error, err.Error())`. A
 ### Propagate TraceContext across things other than function calls
 
 This is taken care of by default for gRPC calls via the otelgrpc interceptors.
-However you may want to propagate tracing information between goroutines or
+However, you may want to propagate tracing information between goroutines or
 other places where the `context.Context` is not passed such as handoffs through
 a Go channel or an external datastore. There are two broad approaches that are
 applicable in different situations:

service/history/README.md

@@ -1 +1 @@
-See [docs/history-service.md](../../docs/history-service.md).
+See [docs/history-service.md](../../docs/architecture/history-service.md).

service/history/queues/executable.go

@@ -523,7 +523,7 @@ func (e *executableImpl) HandleErr(err error) (retErr error) {
 		// or send it to the DLQ if that is enabled.
 		metrics.TaskCorruptionCounter.With(e.taggedMetricsHandler).Record(1)
 		if e.dlqEnabled() {
-			// Keep this message in sync with the log line mentioned in Investigation section of develop/docs/dlq.md
+			// Keep this message in sync with the log line mentioned in Investigation section of docs/admin/dlq.md
 			e.logger.Error("Marking task as terminally failed, will send to DLQ", tag.Error(err), tag.ErrorType(err))
 			e.terminalFailureCause = err // <- Execute() examines this attribute on the next attempt.
 			metrics.TaskTerminalFailures.With(e.taggedMetricsHandler).Record(1)
@@ -539,7 +539,7 @@ func (e *executableImpl) HandleErr(err error) (retErr error) {
 	e.logger.Error("Fail to process task", tag.Error(err), tag.ErrorType(err), tag.UnexpectedErrorAttempts(int32(e.unexpectedErrorAttempts)), tag.LifeCycleProcessingFailed)
 
 	if e.unexpectedErrorAttempts >= e.maxUnexpectedErrorAttempts() && e.dlqEnabled() {
-		// Keep this message in sync with the log line mentioned in Investigation section of develop/docs/dlq.md
+		// Keep this message in sync with the log line mentioned in Investigation section of docs/admin/dlq.md
 		e.logger.Error("Marking task as terminally failed, will send to DLQ. Maximum number of attempts with unexpected errors",
 			tag.Attempt(int32(e.unexpectedErrorAttempts)), tag.Error(err))
 		e.terminalFailureCause = err // <- Execute() examines this attribute on the next attempt.

service/matching/README.md

@@ -1 +1 @@
-See [docs/matching-service.md](../../docs/matching-service.md).
+See [docs/matching-service.md](../../docs/architecture/matching-service.md).