Skip to main content

Temporal Cloud and SDKs emit metrics that can be used to monitor performance and troubleshoot errors.

Temporal Cloud emits metrics through a Prometheus HTTP API endpoint which can be directly used as a Prometheus data source in Grafana or to query and export Cloud metrics to any observability platform.

The open-source SDKs require you to set up a Prometheus scrape endpoint for Prometheus to collect and aggregate the Worker and Client metrics.

This article describes how to set up your Temporal Cloud and SDK metrics and use them as data sources in Grafana.

The process for setting up observability includes the following steps:

  1. Create or get your Prometheus endpoint for Temporal Cloud metrics and enable SDK metrics.
  2. Run Grafana and set up data sources for Temporal Cloud and SDK metrics in Grafana. The examples in this article describe running Grafana on your local host where you run your application code.
  3. Create dashboards in Grafana to view Temporal Cloud metrics and SDK metrics. Temporal provides sample community-driven Grafana dashboards for Cloud and SDK metrics that you can use and customize according to your requirements.

If you're following through with the examples provided here, ensure that you have the following:

  • Root CA certificates and end-entity certificates. See Certificate requirements for details.
  • Set up your connections to Temporal Cloud using an SDK of your choice and have some Workflows running on Temporal Cloud. See Connect to a Cluster for details.
  • Prometheus and Grafana installed.

Background: What is a Failure?

A Failure is Temporal's representation of various types of errors that occur in the system.

There are different types of Failures, and each has a different type in the SDKs and different information in the protobuf messages (which are used to communicate with the Temporal Cluster and appear in Event History).

All requests made to the Temporal Cluster by the Client or Worker are gRPC requests. Sometimes, when these frontend requests can't be completed, you'll see this particular error message: Context: deadline exceeded. Network interruptions, timeouts, server overload, and Query errors are some of the causes of this error.

The following sections discuss the nature of this error and how to troubleshoot it.

The Temporal Cluster and SDKs emit metrics that can be used to monitor performance and troubleshoot issues. To collect and aggregate these metrics, you can use one of the following tools:

  • Prometheus
  • StatsD
  • M3

After you enable your monitoring tool, you can relay these metrics to any monitoring and observability platform.

Running into limits can cause unexpected failures, so be mindful when you design your systems. Here is a list of many hard (error) or soft (warn) limits that you could encounter while using the Temporal Platform.