Skip to main content

What is a Data Converter?

A Data Converter is a Temporal SDK component that encodes and decodes data entering and exiting a Temporal Server.

Data Converter encodes and decodes data

Data is encoded before it is sent to a Temporal Server, and it is decoded when it is received from a Temporal Server.

The main pieces of data that run through the Data Converter are arguments and return values:

  • The Client:
    • Encodes Workflow, Signal, and Query arguments.
    • Decodes Workflow and Query return values.
  • The Worker:
    • Decodes Workflow, Signal, and Query arguments.
    • Encodes Workflow and Query return values.
    • Decodes and encodes Activity arguments and return values.

Each piece of data (like a single argument or return value) is encoded as a Payload Protobuf message, which consists of binary data and key-value metadata.

Default Data Converter​

Each Temporal SDK includes a default Data Converter. In most SDKs, the default converter supports binary, JSON, and Protobufs. (In SDKs that cannot determine parameter types at runtime—like TypeScript—Protobufs aren't included in the default converter.) It tries to encode values in the following order:

  • Null
  • Binary
  • Protobuf JSON
  • JSON

For example:

  • If a value is an instance of a Protobuf message, it will be encoded with proto3 JSON.
  • If a value isn't null, binary, or a Protobuf, it will be encoded as JSON. If any part of it is not serializable as JSON (for example, a Date—see JSON data types), an error will be thrown.

The default converter also supports decoding binary Protobufs.

Custom Data Converter​

Applications can create their own custom Data Converters to alter the format (for example using MessagePack instead of JSON) or add compression or encryption.

To use a custom Data Converter, provide it in the following contexts:

Custom Data Converters are not applied to all data:

A custom Data Converter has three parts:

Payload Converter​

Some SDKs have the concept of a Payload Converter that's part of the Data Converter and does the conversion from a value to a Payload and back.

Payload Codec​

In TypeScript, Go, and Python, data conversion happens in two stages:

  1. A Payload Converter converts a value into a Payload.
  2. A Payload Codec transforms an array of Payloads (for example, a list of Workflow arguments) into another array of Payloads.

The Payload Codec is an optional step that happens between the wire and the Payload Converter:

Temporal Server <--> Wire <--> Payload Codec <--> Payload Converter <--> User code

Common Payload Codec transformations are compression and encryption.

In codec implementations, we recommended running the function (whether it be compressing, encrypting, etc) on the entire input Payload, and putting the result in a new Payload's data field. That way, the input Payload's headers are preserved. See, for example:

To view data that's been converted by a codec in the Web UI and tctl, use a Codec Server.

Encryption​

Doing encryption in a custom Data Converter ensures that all application data is encrypted during the following actions:

  • Being sent to/from Temporal Server.
  • Moving inside Temporal Server.
  • Stored by Temporal Server.

Then data exists unencrypted in memory only on the Client and in the Worker Process that is executing Workflows and Activities on hosts that the application developer controls.

Our encryption samples use AES GCM with 256-bit keys:

Failure Converter​

A Failure Converter converts error objects to proto Failures and back.

The default Failure Converter copies error messages and stack traces as plain text. If your errors may contain sensitive information, you can encrypt the message and stack trace by configuring the default Failure Converter to use your Payload Codec, in which case it will move your message and stack_trace fields to a Payload that's run through your codec.

You can make a custom Failure Converter, but if you use multiple SDKs, you'd have to implement the same logic in each.