Github/temporal
1
0
Fork 0
mirror of https://github.com/temporalio/temporal.git synced 2026-08-30 18:41:49 -07:00
Code Issues Packages Projects Releases Wiki Activity

Reorganize docs (#5535)

Browse Source 
## What changed?
<!-- Describe what has changed in this PR -->
I reorganized our docs into `docs/{admin,architecture,development}`.

## Why?
<!-- Tell your future self why have you made these changes -->
We discussed this internally, but I believe this is a better
segmentation for the different audiences looking at our docs. I plan on
adding more stuff to docs/admin like metrics and dynamic config docs.

## How did you test it?
<!-- How have you verified this change? Tested locally? Added a unit
test? Checked in staging env? -->
I'm mainly looking for broken links.
- I went through each markdown file in docs manually. 
- I looked through all inspection errors in the IDE.
- I looked at all references to "docs/" or "develop/" in our code.

## Potential risks
<!-- Assuming the worst case, what can be broken when deploying this
change to production? -->

## Documentation
<!-- Have you made sure this change doesn't falsify anything currently
stated in `docs/`? If significant
new behavior is added, have you described that in `docs/`? -->

## Is hotfix candidate?
<!-- Is this PR a hotfix candidate or does it require a notification to
be sent to the broader community? (Yes/No) -->
This commit is contained in:
Michael Snowden
2024-03-18 23:13:02 -07:00
committed by GitHub
parent 3825007f66
commit 676657cbf4
23 changed files with 113 additions and 95 deletions
Download Patch File Download Diff File Expand all files Collapse all files

64
develop/docs/dlq.md
View File

@@ -1,64 +0,0 @@
# Operating the Dead Letter Queue (DLQ)
When a task in the Temporal server encounters a terminal error, it may be enqueued in the Dead Letter Queue (DLQ).
A terminal error is a non-retryable error, such as failing to deserialize data due to data corruption.
To effectively manage these DLQ messages, follow these steps:
## Configuration
### History Replication DLQ
To enable the DLQ for history replication tasks, set `history.enableHistoryReplicationDLQV2` to true.
### History Tasks DLQ
To enable the DLQ for non-replication history tasks, set `history.TaskDLQEnabled` to true.
## Detection
There is a metric `dlq_writes`, which is incremented each time a message is enqueued to the DLQ.
You can use this to determine when a task encountered a terminal error and needs manual resolution.
https://github.com/temporalio/temporal/blob/1de185f3d615bb5e7876804e89aeb7086d16833e/common/metrics/metric_defs.go#L917-L920
## Identification
Search for `Task enqueued to DLQ` in the logs and inspect the `dlq-message-id`, `xdc-source-cluster`, `xdc-target-cluster`, `queue-task-type` and `wf-namespace(-id)?` tags.
The source and target cluster tags are different only for replication tasks.
The namespace tag may have a `-id` suffix if we were unable to determine the namespace name from its ID.
## Investigation
From the task type, get the corresponding DLQ type from this map:
1. transfer: 1
2. timer: 2
3. replication: 3
4. visibility: 4
You can list the DLQ messages using the command:
`tdbg dlq --dlq-version v2 read --dlq-type {type}`. Substitute `{type}` with the integer value from step 3.
You can specify the maximum message ID to read using `--last-message-id` flag.
In case of replication tasks, you can specify the source cluster using the flag `--cluster`.
Search `Marking task as terminally failed, will send to DLQ` in logs and find the terminal error that caused the task to be enqueued to the DLQ.
If you can't find the logs, you can list all DLQs using the command `tdbg dlq --dlq-version v2 list` and find non-empty queues.
This command will list all queues in the decreasing order of message count.
Now these DLQ tasks can either be purged (removed from the DLQ), or merged (re-enqueued to the original queue which will
cause them to be retried).
## Resolution
### Deleting Tasks
To purge a message, execute the command `tdbg dlq --dlq-version v2 purge --dlq-type {type} --last_message_id {message_id}`.
Note that this command will purge all messages with an ID less than or equal to the specified `message_id`.
The output of this command will have a job token which can be used to manage the purge job.
Before executing this command, you can list the messages in the queue using the command mentioned in step 6 above to make sure more messages are not purged by mistake.
### Retrying Tasks
To merge a message, execute the command `tdbg dlq --dlq-version v2 merge --dlq-type {type} --last_message_id {message_id}`.
This command will merge all messages with an ID less than or equal to `message_id` back into the original queue for reprocessing.
The output of this command will have a job token that can be used to manage the merge job.
Once merge or purge command is executed, it will create a DLQ job which will process the DLQ messages.
You can get the status of this job using the command `tdbg dlq --dlq-version v2 job describe --job-token {job-token}`.
The value of job-token will be printed in the output of merge and purge commands.
The output of the describe command will have details like the last processed message ID, number of messages processed, etc.
### Cancelling Jobs
If you want to cancel a specific DLQ job, you can execute the command `tdbg dlq --dlq-version v2 job cancel --job-token {job-token} --reason {reason}`.

22
develop/docs/macos/cassandra.md
View File

@@ -1,22 +0,0 @@
# Run Cassandra v3.11 on macOS
### Install
```bash
brew install cassandra@3.11
```
For MacBook with ARM chip, you need to replace the JNA library with a recent version.
1. Locate the JNA file (`find $(brew --prefix) -name "jna-*.jar"`) and remove it.
2. Download a recent version of JNA jar file (5.8+) from [Maven](https://search.maven.org/artifact/net.java.dev.jna/jna).
3. Move the file you downloaded to the folder in step 1.
### Start
```bash
cassandra -f
```
### Post Installation
Verify Cassandra v3.11 is running and accessible:
```bash
cqlsh
```

68
develop/docs/macos/mysql.md
View File

@@ -1,68 +0,0 @@
# Run MySQL v5.7 on macOS
### Install
```bash
brew install mysql@5.7
```
### Start
```bash
brew services start mysql@5.7
```
### Stop
```bash
brew services stop mysql@5.7
```
### Post Installation
Verify MySQL v5.7 is running and accessible:
```bash
mysql -h 127.0.0.1 -P 3306 -u root
```
Within `mysql` shell, create user and password:
```mysql
ALTER USER 'root'@'localhost' IDENTIFIED BY 'root';
CREATE USER 'temporal'@'localhost' IDENTIFIED BY 'temporal';
GRANT ALL PRIVILEGES ON *.* TO 'temporal'@'localhost';
```
Verify password:
```bash
mysql -h 127.0.0.1 -P 3306 -u root -p
mysql -h 127.0.0.1 -P 3306 -u temporal -p
```
### TLS
[TLS Key / Cert Setup Guide](../tls/tls.md)
```bash
emacs /usr/local/etc/my.cnf
```
setting the variables below to
```
require_secure_transport=ON
ssl-ca=<path to the server-cert.pem>
ssl-cert=<path to the server-cert.pem>
ssl-key=<path to the server-key.pem>
```
```mysql
ALTER USER 'root'@'localhost' IDENTIFIED BY 'root' REQUIRE X509;
ALTER USER 'temporal'@'localhost' IDENTIFIED BY 'temporal' REQUIRE X509;
```
then restart MySQL:
```bash
brew services restart mysql@5.7
```
Verify TLS & password:
```bash
mysql -u root -p \
--ssl-cert=<path to the client-cert.pem> \
--ssl-key=<path to the client-key.pem> \
--ssl-ca=<path to the ca.pem>
```

99
develop/docs/macos/postgresql.md
View File

@@ -1,99 +0,0 @@
# Run PostgreSQL v9.6 on macOS
### Install
```bash
brew install postgresql@9.6
```
### Start
```bash
brew services start postgresql@9.6
```
### Stop
```bash
brew services stop postgresql@9.6
```
### Post Installation
Create a user `postgres`
```bash
createuser -s root
```
Verify PostgreSQL v9.6 is running and accessible:
```bash
psql -h 127.0.0.1 -p 5432 -U root -d postgres
```
Within `psql` shell, add a password:
```postgresql
ALTER USER root WITH PASSWORD 'root';
ALTER USER root WITH SUPERUSER;
CREATE USER temporal WITH PASSWORD 'temporal';
ALTER USER temporal WITH SUPERUSER;
```
Change the following file context:
```bash
emacs /usr/local/var/postgresql@9.6/pg_hba.conf
```
from
```
local all all trust
host all all 127.0.0.1/32 trust
host all all ::1/128 trust
```
to
```
local all all md5
host all all 127.0.0.1/32 md5
host all all ::1/128 md5
```
then restart PostgreSQL:
```bash
brew services restart postgresql@9.6
```
Verify password:
```bash
psql -h 127.0.0.1 -p 5432 -U root -d postgres
psql -h 127.0.0.1 -p 5432 -U temporal -d postgres
```
### TLS
[TLS Key / Cert Setup Guide](../tls/tls.md)
```bash
emacs /usr/local/var/postgresql@9.6/postgresql.conf
```
setting the variables below to
```
ssl=on
ssl_cert_file=<path to the server-cert.pem>
ssl_key_file=<path to the server-key.pem>
ssl_ca_file=<path to the ca.pem>
```
```bash
emacs /usr/local/var/postgresql@9.6/pg_hba.conf
```
changes the configs like below
```
hostssl all all 127.0.0.1/32 md5 clientcert=1
hostssl all all ::1/128 md5 clientcert=1
```
then restart PostgreSQL:
```bash
brew services restart postgresql@9.6
```
Verify TLS & password:
```bash
psql "sslmode=require host=localhost dbname=postgres user=root \
sslkey=<path to the client-key.pem> \
sslcert=<path to the client-cert.pem> \
sslrootcert=<path to the ca.pem>"
```

17
develop/docs/new-rpcs.md
View File

@@ -1,17 +0,0 @@
## Adding new RPCs
Adding new RPCs to any of the server roles is currently a little involved.
Here's a list of what you'll need to modify:
1. Add RPC definitions to proto files. Either in the `api` repo (for frontend)
or here in `proto/internal` (for history/matching).
2. Update `go.temporal.io/api` to pick up the new generated files (for frontend)
or generate them here with `make proto` (for history/matching).
3. Add metric scope defs for client-side metrics to `common/metrics/defs.go`
(this is going away soon).
4. `make go-generate` to generate wrappers in `client`.
5. For frontend: add definitions to `service/frontend/dcRedirectionHandler.go`.
(In the future hopefully we can make this generated or use interceptors.)
6. Add your new methods to `service/<service>/configs/quotas.go`.
7. Finally implement your new methods in the RPC handler.

9
develop/docs/run_dependencies_host.md
View File

@@ -1,9 +0,0 @@
## Run Temporal dependencies on the host
### macOS
While developing Temporal server you may want to run its dependencies locally. One of the reason might be
a bad docker file system performance on macOS. Please follow the doc for the database you use:
[Cassandra](macos/cassandra.md), [MySQL](macos/mysql.md), or [PostgreSQL](macos/postgresql.md).
### Linux
Linux users should use `docker compose` as described in the [contribution guide](../../CONTRIBUTING.md).

66
develop/docs/temporal-cla.md
View File

@@ -1,66 +0,0 @@
**Individual Contributor License Agreement**
In order to clarify the intellectual property license granted with Contributions from any person or entity, Temporal
Technologies Inc., ("Company") must have a Contributor License Agreement ("CLA") on file that has been signed by each
Contributor, indicating agreement to the license terms below. This license is for your protection as a Contributor as
well as the protection of Company; it does not change your rights to use your own Contributions for any other purpose.
You accept and agree to the following terms and conditions for Your present and future Contributions submitted to
Company. Except for the license granted herein to Company and recipients of software distributed by Company, You reserve
all right, title, and interest in and to Your Contributions.
1. Definitions.
"You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this
Agreement with Company. For legal entities, the entity making a Contribution and all other entities that control, are
controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes
of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such
entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares,
or (iii) beneficial ownership of such entity.
"Contribution" shall mean any original work of authorship, including any modifications or additions to an existing work,
that is intentionally submitted by You to Company for inclusion in, or documentation of, any of the products owned or
managed by Company (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal,
or written communication sent to Company or its representatives, including but not limited to communication on
electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of,
Company for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked
or otherwise designated in writing by You as "Not a Contribution."
2. Grant of Copyright License. Subject to the terms and conditions of this Agreement, You hereby grant to Company and to
recipients of software distributed by Company a perpetual, worldwide, non-exclusive, no-charge, royalty-free,
irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform,
sublicense, and distribute Your Contributions and such derivative works.
3. Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to Company and to
recipients of software distributed by Company a perpetual, worldwide, non-exclusive, no-charge, royalty-free,
irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import,
and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are
necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which
such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (
including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have
contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity
under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.
4. You represent that you are legally entitled to grant the above license. If your employer(s) has rights to
intellectual property that you create that includes your Contributions, you represent that you have received
permission to make Contributions on behalf of that employer, that your employer has waived such rights for your
Contributions to Company, or that your employer has executed a separate Corporate CLA with Company.
5. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of
others). You represent that Your Contribution submissions include complete details of any third-party license or
other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware
and which are associated with any part of Your Contributions.
6. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support.
You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in
writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
express or implied, including, without limitation, any warranties or conditions of TITLE, NON- INFRINGEMENT,
MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
7. Should You wish to submit work that is not Your original creation, You may submit it to Company separately from any
Contribution, identifying the complete details of its source and of any license or other restriction (including, but
not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and
conspicuously marking the work as "Submitted on behalf of a third-party: [named here]".
8. You agree to notify Company of any facts or circumstances of which you become aware that would make these
representations inaccurate in any respect.

31
develop/docs/tls/tls.md
View File

@@ -1,31 +0,0 @@
# Temporal Server TLS Key / Cert Setup Guide
## Setup Local CA
```bash
openssl genrsa -aes256 -out ca-key.pem 4096
openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem
```
## Setup Key / Cert for Callee. e.g. Database
```bash
openssl genrsa -out server-key.pem 4096
openssl req -subj "/CN=localhost" -sha256 -new -key server-key.pem -out server.csr
echo subjectAltName = IP:127.0.0.1,DNS:localhost > extfile.cnf
openssl x509 -req -days 365 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -extfile extfile.cnf
```
## Setup Key / Cert for Caller. e.g. Server
```bash
openssl genrsa -out client-key.pem 4096
openssl req -subj '/CN=localhost' -new -key client-key.pem -out client.csr
echo subjectAltName = IP:127.0.0.1,DNS:localhost > extfile.cnf
openssl x509 -req -days 365 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out client-cert.pem -extfile extfile.cnf
```
## Cleanup & Permission
```bash
rm -v client.csr server.csr
chmod -v 0400 ca-key.pem server-key.pem client-key.pem
chmod -v 0444 ca.pem server-cert.pem client-cert.pem
```

227
develop/docs/tracing.md
View File

@@ -1,227 +0,0 @@
# Tracing Temporal Services with OTEL
The Temporal server supports ability to configure OTEL trace exporters to
support emitting spans and traces for observability. More specifically, the
server uses the [Go Open Telemetry
library](https://github.com/open-telemetry/opentelemetry-go) for instrumentation
and multi-protocol multi-model telemetry exporting. This document is intended to
help developers understand how to configure exporters and instrument their code.
A full exploration of tracing and telemetry is out of scope of this document and
the reader is referred to [external reference
material](https://opentelemetry.io/docs/concepts/signals/traces/), [third party
descriptions](https://lightstep.com/opentelemetry/tracing), and the
[specification
itself](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/overview.md#tracing-signal).
## Configuring
No trace exporters are configured by default and thus trace data is neither
collected nor emitted without additional configuration.
In OpenTelemetry, the concept of an "exporter" is
abstract. The concrete implementation of an exporter is determined by a
3-tuple of values: the exporter signal, model, and protocol:
- a "signal" is one of traces, metrics, or logs (in this document we will only deal with traces),
- "model" indicates the abstract data model for the span and trace data being exported,
- and the "protocol" specifies the concrete application protocol binding for the indicated model.
Temporal is known to support exporting trace data as defined by otlp over grpc.
### Configuration File
The server supports an `otel` YAML stanza which is used to configure a
set of process-wide exporters.
A common configuration is to emit tracing data to an agent such as the
[otel-collector](https://opentelemetry.io/docs/collector/) running locally. To
configure such a system add the stanza below to your configuration yaml file(s).
```
otel:
exporters:
- kind:
signal: traces
model: otlp
protocol: grpc
spec:
connection:
insecure: true
endpoint: localhost:4317
```
Another example is pointing Temporal directly at the Honeycomb hosted OTLP
collection service. To achieve such a configuration you will need an API key
from the upstream Honeycomb service and the stanza below.
```
otel:
exporters:
- kind:
signal: traces
model: otlp
protocol: grpc
spec:
connection:
endpoint: api.honeycomb.io:443
headers:
x-honeycomb-team: <a honeycomb API key>
```
Note that the configuration parser supports defining multiple exporters by
supplying additional `kind` and `spec` declarations. Additional configuration
fields can be found in [config_test.go](../../common/telemetry/config_test.go)
and are mostly related to the underlying gRPC client configuration (retries,
timeouts, etc).
### Environment Variables
#### Creating Exporter
An OTEL span exporter can also be configured via environment variables: [OTEL_TRACES_EXPORTER](
https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#exporter-selection)
creates a span exporter.
```
OTEL_TRACES_EXPORTER=otlp
```
Note that if the configuration file already defines a traces exporter, no additional exporter
will be created.
#### Configuring Exporter
The Go OTEL SDK will also read a well-known set of environment variables for the configuration
of the exporter. So if you prefer setting environment variables to writing YAML then you can use the
[variables defined in the OTEL spec](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/).
For example:
```
OTEL_SERVICE_NAME=my-service OTEL_EXPORTER_OTLP_TRACES_INSECURE=true
```
**NOTE: If an environment variable conflicts with YAML-provided configuration then the environment
variable takes precedence.**
## Instrumenting
While the exporter configuration described above is executed and set up at
process startup time, instrumentation code - the creation and termination of
spans - is inserted inline (like logging statements) into normal server
processing code. Spans are created by `go.opentelemetry.io/otel/trace.Tracer`
objects which are themselves created by
`go.opentelemetry.io/otel/trace.TracerProvider` instances. The `TracerProvider`
instances are bound to a single logical service and as such a single Temporal
process will have up to four such instances (for worker, mathcing, history, and
frontend services respectively). The `Tracer` object is bound to a single
logical _library_ which is different than a _service_. Consider that a history
_service_ instance might run code from the temporal common library, gRPC
library, and gocql library.
`Tracer` and `TracerProvider` object management has been added to the server's
`fx` [DI configuration](https://github.com/temporalio/temporal/blob/f86b8d2c5f43907eaea4ad53ea082d70692c38cf/temporal/fx.go#L787-L889)
and thus they are available to be added to any fx-enabled object constructors.
Due the possibility of multiple services being coresident within a single
process, we do not use the OTEL library's capability to host and access a single
global `TracerProvider`.
By default, gRPC clients and servers are instrumented via the open source
[otelgrpc](https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation/google.golang.org/grpc/otelgrpc)
library.
## Instrumentation Tips
### Follow the OTEL attribute naming guidelines
The OpenTelemetry project has published a non-normative set of [guidelines for
attribute naming](https://opentelemetry.io/docs/reference/specification/common/attribute-naming/).
If nothing else, please
1. Always check for an appropriate attribute in
[semconv](https://pkg.go.dev/go.opentelemetry.io/otel/semconv) before
creating your own
1. Always prefix Temporal attributes with `io.temporal`
### Create shared package-appropriate attribute keys
Do not create a single file in common for all attributes
Do not create packages _just_ for OTEL attributes
Do create a set of `attribute.Key`s in the semantically appropriate package and
re-use those to create `attribute.KeyValue`s as needed.
Do create a set of utility functions that can transform frequently used
aggregate types (Tasks, WorkflowExecutions, TaskQueues, etc) into an
`[]attribute.KeyValue`. The association of `attribute.KeyValue`s to a
`trace.Span` can be verbose in terms of the number of lines of code needed so
any reduction in that noise will be a good idea. Not to mention the consistency
benefit of sharing a single mapping function.
### Start a span in `common` or other non-service-specific code
*Q:* Given that common code can be called from any service, how can I start a span
in common library code that is bound the the appropriate service
(frontend/history/matching/worker)?
*A:* The `TracerProvider` that created the currently active Span can be retrieved
from that Span itself and the currently active Span can be received from from
the `context.Context`.
```
// DoFoo is a function in the common package
func DoFoo(ctx context.Context, x int, y string) string {
var span trace.Span
ctx, span = trace.SpanFromContext(ctx).TracerProvider().Tracer("go.temporal.io/server/common").Start("DoFoo")
defer span.End()
return fmt.Sprintf("%v-%v", y, x)
}
```
### `RecordError` does not imply Span failure
Using `Span.RecordError` is a good idea but not all errors imply failure. Thus
if you want to capture an error _and also_ capture that a span failed, you must
additionally call `Span.SetStatus(codes.Error, err.Error())`. A
`FailSpanWithError` utility function might be a good idea.
### Propagate TraceContext across things other than function calls
This is taken care of by default for gRPC calls via the otelgrpc interceptors.
However you may want to propagate tracing information between goroutines or
other places where the `context.Context` is not passed such as handoffs through
a Go channel or an external datastore. There are two broad approaches that are
applicable in different situations:
1. If the object being transferred is not externally durable (e.g. an object put
into a Go channel but _not_ spooled to a database) then you can pull the
`trace.SpanContext` out of the current `trace.Span` with
`trace.SpanContextFromContext(context.Context)` or `Span.SpanContext()` and
pass that object along with the data being transferred. The consuming side
can restore the tracing state with
`trace.ContextWithSpanContext(trace.SpanContext)`.
1. If the tracing state needs to be serialized, the OTEL library provides the
[propagation](https://pkg.go.dev/go.opentelemetry.io/otel@v1.7.0/propagation)
package to convert trace state into a more serialization-friendly type such
as a `map[string]string`. The `propagation.TraceContext` type can be used to
inject and extract trace state into a key-value-ish object.
```
carrier := propagation.MapCarrier(map[string]string{})
propagation.TraceContext{}.Inject(ctx, carrier)
// write the carrier object to a durable store
```
### Trace individual tasks that are processed together in batches
OpenTelemetry Spans can be _linked_ together to form a non-parent-child
relationship. One of the main use cases for linking is so that a batch process
(e.g. a database read that fills a large buffer of work items) can create Spans
for each of the individual work items it creates and those Spans can be linked
back to the parent batch Span without that span becoming their logical parent.
### Still want to log things?
Use `Span.AddEvent` to write messages that will be associated with that `Span`.
From the OTEL manual
> An event is a human-readable message on a span that represents “something happening” during it’s lifetime