Workflow architecture

Dapr Workflows allow developers to define workflows using ordinary code in a variety of programming languages. The workflow engine runs inside of the Dapr sidecar and orchestrates workflow code deployed as part of your application. Dapr Workflows are built on top of Dapr Actors providing durability and scalability for workflow execution.

This article describes:

• The architecture of the Dapr Workflow engine
• How the workflow engine interacts with application code
• How the workflow engine fits into the overall Dapr architecture
• How different workflow backends can work with workflow engine

For more information on how to author Dapr Workflows in your application, see How to: Author a workflow.

The Dapr Workflow engine is internally powered by Dapr’s actor runtime. The following diagram illustrates the Dapr Workflow architecture in Kubernetes mode:

To use the Dapr Workflow building block, you write workflow code in your application using the Dapr Workflow SDK, which internally connects to the sidecar using a gRPC stream. This registers the workflow and any workflow activities, or tasks that workflows can schedule.

The engine is embedded directly into the sidecar and implemented using the durabletask-go framework library. This framework allows you to swap out different storage providers, including a storage provider created for Dapr that leverages internal actors behind the scenes. Since Dapr Workflows use actors, you can store workflow state in state stores.

Sidecar interactions

When a workflow application starts up, it uses a workflow authoring SDK to send a gRPC request to the Dapr sidecar and get back a stream of workflow work items, following the server streaming RPC pattern. These work items can be anything from “start a new X workflow” (where X is the type of a workflow) to “schedule activity Y with input Z to run on behalf of workflow X”.

The workflow app executes the appropriate workflow code and then sends a gRPC request back to the sidecar with the execution results.

All interactions happen over a single gRPC channel and are initiated by the application, which means the application doesn’t need to open any inbound ports. The details of these interactions are internally handled by the language-specific Dapr Workflow authoring SDK.

The history the sidecar sends over this stream is incremental: once a worker is warm for an instance, it is sent only the events added since its previous turn rather than the whole history. See the stateful workflow client.

A load balancer or proxy between the app and the sidecar can close a stream that carries no traffic. AWS ALB and nginx, for example, close an idle stream after 60 seconds, which costs the worker a reconnect and a full history resend. To keep the stream open, a worker can advertise the WORKER_CAPABILITY_HEALTH_PING capability when it opens the stream, and the sidecar then sends it a health ping every 30 seconds. Workers built with the 1.19 Python, Java, and .NET SDKs, or with the Go SDK v1.16 (durabletask-go v0.15), advertise it. The sidecar reads a DAPR_WORKFLOW_HEALTH_PING_INTERVAL environment variable that overrides the interval, but it exists for Dapr’s integration tests and is not a supported setting. See the environment variable reference.

Differences between workflow and application actor interactions

If you’re familiar with Dapr actors, you may notice a few differences in terms of how sidecar interactions works for workflows compared to application defined actors.

Workflow distributed tracing

The durabletask-go core used by the workflow engine writes distributed traces using Open Telemetry SDKs. These traces are captured automatically by the Dapr sidecar and exported to the configured Open Telemetry provider, such as Zipkin.

Each workflow instance managed by the engine is represented as one or more spans. There is a single parent span representing the full workflow execution and child spans for the various tasks, including spans for activity task execution and durable timers.

Workflow activity code currently does not have access to the trace context.

Workflow actors

Upon the workflow client connecting to the sidecar, there are two types of actors that are registered in support of the workflow engine:

• dapr.internal.{namespace}.{appID}.workflow
• dapr.internal.{namespace}.{appID}.activity

The {namespace} value is the Dapr namespace and defaults to default if no namespace is configured. The {appID} value is the app’s ID. For example, if you have a workflow app named “wfapp”, then the type of the workflow actor would be dapr.internal.default.wfapp.workflow and the type of the activity actor would be dapr.internal.default.wfapp.activity.

The following diagram demonstrates how workflow actors operate in a Kubernetes scenario:

Just like user-defined actors, workflow actors are distributed across the cluster by the hashing lookup table provided by the actor placement service. They also maintain their own state and make use of reminders. However, unlike actors that live in application code, these workflow actors are embedded into the Dapr sidecar. Application code is completely unaware that these actors exist.

Workflow actors

There are 2 different types of actors used with workflows: workflow actors and activity actors. Workflow actors are responsible for managing the state and placement of all workflows running in the app. A new instance of the workflow actor is activated for every workflow instance that gets scheduled. The ID of the workflow actor is the ID of the workflow. This workflow actor stores the state of the workflow as it progresses, and determines the node on which the workflow code executes via the actor lookup table.

As workflows are based on actors, all workflow and activity work is distributed across all replicas of the application implementing workflows by the actor placement table. By default there is no locality or relationship between where a workflow is started and where each work item is executed. With the workflow fast path preview feature enabled, the host that commits an event also drives the resulting turn, and activity bodies are driven on their target host without a reminder round trip through the Scheduler.

Each workflow actor saves its state using the following keys in the configured actor state store:

The following diagram illustrates the typical lifecycle of a workflow actor.

To summarize:

1. A workflow actor is activated when it receives a new message.
2. New messages then trigger the associated workflow code (in your application) to run and return an execution result back to the workflow actor.
3. Once the result is received, the actor schedules any tasks as necessary.
4. After scheduling, the actor updates its state in the state store.
5. Finally, the actor goes idle until it receives another message. During this idle time, the sidecar may decide to unload the workflow actor from memory.

Activity actors

Activity actors are responsible for managing the state and placement of all workflow activity invocations. A new instance of the activity actor is activated for every activity task that gets scheduled by a workflow. The ID of the activity actor is the ID of the workflow, the task ID of the activity (task IDs start at 0) and a constant trailing 0, joined by ::. For example, if a workflow has an ID of 876bf371 and schedules its third activity, the activity actor’s ID is 876bf371::2::0. The third component is kept for compatibility with earlier releases, which wrote the workflow generation there; it no longer carries information. Task IDs restart from 0 when a workflow uses continue as new, so a later generation reuses the same activity actor IDs; the engine tells the executions apart by the task execution ID carried on the scheduling and completion events.

Activity actors store no state of their own, and instead all resulting data is sent back to the parent workflow actor. The one exception is the workflow fast path, where an activity actor writes a short-lived execution-claim record while actor placement moves a running execution to another host.

The following diagram illustrates the typical lifecycle of an activity actor.

Activity actors are short-lived:

1. Activity actors are activated when a workflow actor schedules an activity task.
2. Activity actors then immediately call into the workflow application to invoke the associated activity code.
3. Once the activity code has finished running and has returned its result, the activity actor sends a message to the parent workflow actor with the execution results.
4. The activity actor then deactivates itself.
5. Once the results are sent, the workflow is triggered to move forward to its next step.

With the workflow fast path enabled, the activity body runs as soon as the dispatch arrives, without the one-shot reminder, whenever the workflow actor certifies that its janitor backstop is armed. During a placement change the activity actor may write a small execution-claim record so that the new owner can tell a live execution from one that died with its host.

Reminder usage and execution guarantees

The Dapr Workflow ensures workflow fault-tolerance by using actor reminders to recover from transient system failures. By default, prior to invoking application workflow code, the workflow or activity actor will create a new reminder. These reminders are made “one shot”, meaning that they will expire after successful triggering. If the application code executes without interruption, the reminder is triggered and expired. However, if the node or the sidecar hosting the associated workflow or activity crashes, the reminder will reactivate the corresponding actor and the execution will be retried, forever. Retries of a failed delivery are paced with a decorrelated jittered exponential backoff between 50 ms and 2 s, so that many instances failing at once do not retry in lockstep.

With the workflow fast path preview feature enabled, the per-event reminders are replaced by a single repeating “janitor” reminder per live workflow instance, which fires every 20 seconds, drives any turn or activity that lost its local driver, and deletes itself when the instance completes. A wake whose local drive keeps failing is left to the janitor; a failed activity drive escalates back to the one-shot run-activity reminder described above. Either way the recovery guarantees are the same on both paths.

An instance whose state was saved but whose start can never be processed (for example because its committed start event was lost) is failed with the error type DAPR_WORKFLOW_UNSTARTABLE_STATE instead of remaining PENDING forever, and a pending start whose reminder is overdue is re-driven when the instance’s status is read.

State store usage

Dapr Workflows use actors internally to drive the execution of workflows. Like any actors, these workflow actors store their state in the configured actor state store. This is done by specifying a state store component in your Dapr configuration and then referencing that state store in the actorStateStore property of the configuration’s actors section. Read the state API reference and the actors API reference to learn more about state stores for actors.

Any state store that supports actors implicitly supports Dapr Workflow.

As discussed in the workflow actors section, workflows save their state incrementally by appending to a history log. The history log for a workflow is distributed across multiple state store keys so that each “checkpoint” only needs to append the newest entries.

The size of each checkpoint is determined by the number of concurrent actions scheduled by the workflow before it goes into an idle state. Sequential workflows will therefore make smaller batch updates to the state store, while fan-out/fan-in workflows will require larger batches. The size of the batch is also impacted by the size of inputs and outputs when workflows invoke activities or child workflows.

Different state store implementations may implicitly put restrictions on the types of workflows you can author. For example, the Azure Cosmos DB state store limits item sizes to 2 MB of UTF-8 encoded JSON (source). The input or output payload of an activity or child workflow is stored as a single record in the state store, so a item limit of 2 MB means that workflow and activity inputs and outputs can’t exceed 2 MB of JSON-serialized data.

Similarly, if a state store imposes restrictions on the size of a batch transaction, that may limit the number of parallel actions that can be scheduled by a workflow.

Workflow state can be purged from a state store, including all its history. Each Dapr SDK exposes APIs for purging all metadata related to specific workflow instances.

State store record count

The number of records which are saved as history in the state store per workflow run is determined by its complexity or “shape”. In other words, the number of activities, timers, sub-workflows etc. The following table shows a general guide to the number of records that are saved by different workflow tasks. This number may be larger or smaller depending on retries or concurrency.

Every instance also carries the metadata and customStatus keys, which do not grow with the history. Child workflows additionally carry a transient parent-notify marker between their terminal commit and the parent’s acknowledgement, and, when history signing is enabled, a creation-input row.

Query Workflow History

dapr workflow --app-id myapp list
dapr workflow --app-id myapp history <instance-id>

Supported State Stores

The workflow engine supports these state stores:

• PostgreSQL
• MySQL
• SQL Server
• SQLite
• Oracle Database
• CockroachDB
• MongoDB
• Redis

Workflow scalability

Because Dapr Workflows are internally implemented using actors, Dapr Workflows have the same scalability characteristics as actors. The placement service:

• Doesn’t distinguish between workflow actors and actors you define in your application
• Will load balance workflows using the same algorithms that it uses for actors

The expected scalability of a workflow is determined by the following factors:

• The number of machines used to host your workflow application
• The CPU and memory resources available on the machines running workflows
• The scalability of the state store configured for actors
• The scalability of the actor placement service and the reminder subsystem

The implementation details of the workflow code in the target application also plays a role in the scalability of individual workflow instances. Each workflow instance executes on a single node at a time, but a workflow can schedule activities and child workflows which run on other nodes.

Workflows can also schedule these activities and child workflows to run in parallel, allowing a single workflow to potentially distribute compute tasks across all available nodes in the cluster.

You can configure the maximum concurrency of workflows and activities using the Dapr configuration as described in the next section.

Workflows don’t control the specifics of how load is distributed across the cluster. For example, if a workflow schedules 10 activity tasks to run in parallel, all 10 tasks may run on as many as 10 different compute nodes or as few as a single compute node. The actual scale behavior is determined by the actor placement service, which manages the distribution of the actors that represent each of the workflow’s tasks.

Workflow latency

In order to provide guarantees around durability and resiliency, Dapr Workflows frequently write to the state store and rely on reminders to drive execution. Dapr Workflows therefore may not be appropriate for latency-sensitive workloads. Expected sources of high latency include:

• Latency from the state store when persisting workflow state.
• Latency from the state store when rehydrating workflows with large histories.
• Latency caused by too many active reminders in the cluster.
• Latency caused by high CPU usage in the cluster.

See the Reminder usage and execution guarantees section for more details on how the design of workflow actors may impact execution latency.

The workflow fast path preview feature removes the per-event reminders and most of the state store commits from each turn. With it enabled, the first and third sources above apply mainly to recovery after a failure rather than to the steady-state turn.

Increasing scheduling throughput

By default, when a client schedules a workflow, the workflow engine waits for the workflow to be fully started before returning a response to the client. Waiting for the workflow to start before returning can decrease the scheduling throughput of workflows. When scheduling a workflow with a start time, the workflow engine does not wait for the workflow to start before returning a response to the client. To increase scheduling throughput, consider adding a start time of “now” when scheduling a workflow. An example of scheduling a workflow with a start time of “now” in the Go SDK is shown below:

client.ScheduleNewWorkflow(ctx, "MyCoolWorkflow", workflow.WithStartTime(time.Now()))

This tip speeds up how quickly the client’s scheduling call returns. To increase the throughput of the turns themselves, see the workflow fast path.

Workflows cluster deployment when using Dapr Shared with workflow

When using Dapr Shared, it can be the case that there are multiple daprd sidecars running behind a single load balancer or service. As such, the instance to which a worker receiving work, may not be the same instance that receives the work result. Dapr uses a third actor type to handle this scenario: dapr.internal.{namespace}.{appID}.executor. The daprd that holds a pending work item registers a process-local waiter for its result. A result that lands on a different daprd is forwarded with a single internal call to the executor actor, which shares its ID with the workflow or activity actor that is waiting, so actor placement resolves it to the waiting host. On a single replica no forwarding call is made. The dapr_runtime_workflow_completion_route_count metric, tagged with task_type and route (wait_local, wait_watch, complete_local, complete_actor), shows which path each result took; a sustained wait_watch rate indicates that results are not landing on the waiting host.

Next steps

Related links

• Workflow overview
• Workflow fast path
• Stateful workflow client
• Workflow API reference
• Try out the Workflow quickstart
• Try out the following examples:
  • Python
  • JavaScript example
  • .NET
  • Java
  • Go example