Context & Why: Managing Distributed Business Logic at Scale
Six months ago, our order processing pipeline began buckling under roughly 45,000 daily transactions. Like many microservice stacks, our checkout flow relied on an ad-hoc combination of Kafka topics, Postgres status flags, and 5-minute cron sweeps.
Whenever Stripe timed out or our inventory service restarted during a checkout, orders stalled in inconsistent states. Our on-call engineers spent 10 to 15 hours each week running manual compensation scripts, untangling dead-letter queues, and reconciling orphaned database rows.
Distributed transactions are notoriously brittle. Networks drop packets, containers restart without warning, and upstream third-party APIs throw intermittent 502 errors. Fixing this manually usually means building custom state machines in SQL, propagating idempotency keys across half a dozen services, and adding bespoke retry loops to every HTTP client. Before long, that defensive boilerplate swallows most of your codebase.
Temporal solves this problem through durable execution. You write straightforward, sequential Go code instead of tracking step transitions and retry counters yourself. Under the hood, the Temporal cluster records state transitions into an append-only event log. If a worker process crashes mid-function or an underlying VM disappears, another worker picks up the execution right away. It reconstructs the execution state and continues from the exact line of code where execution stopped.
After migrating our checkout pipeline to Temporal, ghost transactions dropped from roughly 120 per month down to zero. On-call pages related to order routing practically vanished. Even better, our pull request review cycles sped up by 30% because engineers no longer had to design custom compensation paths for every infrastructure hiccup.
Installation: Setting Up the Temporal Server and Go SDK
Spinning up a local Temporal cluster takes under two minutes. The Temporal CLI bundles a self-contained development server with an embedded SQLite datastore and a web console.
Grab the CLI via the official install script or Homebrew:
# Linux and macOS via curl
curl -sSf https://temporal.download/cli.sh | sh
# macOS via Homebrew
brew install temporalStart the local server instance:
temporal server start-devThis command spins up the gRPC engine on 127.0.0.1:7233 and starts the web UI on http://localhost:8233. Load that URL in your browser to inspect running workflows, audit event timelines, and review task queue throughput.
With the server running, initialize your Go module and install the official SDK:
mkdir temporal-go-orders
cd temporal-go-orders
go mod init temporal-go-orders
go get go.temporal.io/sdkConfiguration: Structuring Workflows, Activities, and Workers in Go
Temporal organizes orchestration around two core building blocks: Workflows and Activities.
- Workflows: House pure orchestration logic. They must stay deterministic—never make raw network calls, query system clocks directly, or generate unseeded random values inside workflow functions.
- Activities: Execute side effects like credit card charges, SQL updates, or external API requests. Temporal retries activities automatically according to your backoff policies.
1. Defining Activities
In activities.go, define the operations that perform external interactions:
package main
import (
"context"
"errors"
"fmt"
"math/rand"
)
type OrderRequest struct {
OrderID string
Amount float64
Email string
}
type OrderActivities struct{}
func (a *OrderActivities) ProcessPayment(ctx context.Context, order OrderRequest) (string, error) {
// Simulate payment gateway latency and intermittent flakiness (30% failure rate)
if rand.Float32() < 0.3 {
return "", errors.New("payment gateway timeout: connection reset by peer")
}
txnID := fmt.Sprintf("TXN-%s-OK", order.OrderID)
return txnID, nil
}
func (a *OrderActivities) ReserveInventory(ctx context.Context, orderID string) error {
// Call internal inventory service
return nil
}
func (a *OrderActivities) SendConfirmationEmail(ctx context.Context, email string, orderID string) error {
// Trigger transactional email via Resend or SendGrid
return nil
}2. Writing the Deterministic Workflow
Next, define your business flow in workflow.go. Notice the explicit timeouts and exponential retry policy set in workflow.ActivityOptions:
package main
import (
"time"
"go.temporal.io/sdk/temporal"
"go.temporal.io/sdk/workflow"
)
func OrderWorkflow(ctx workflow.Context, order OrderRequest) (string, error) {
retryPolicy := &temporal.RetryPolicy{
InitialInterval: time.Second * 1,
BackoffCoefficient: 2.0,
MaximumInterval: time.Second * 30,
MaximumAttempts: 5,
}
actOptions := workflow.ActivityOptions{
StartToCloseTimeout: time.Minute * 2,
RetryPolicy: retryPolicy,
}
ctx = workflow.WithActivityOptions(ctx, actOptions)
var activities *OrderActivities
// Step 1: Charge customer credit card with automatic retries
var transactionID string
err := workflow.ExecuteActivity(ctx, activities.ProcessPayment, order).Get(ctx, &transactionID)
if err != nil {
return "", err
}
// Step 2: Reserve inventory item
err = workflow.ExecuteActivity(ctx, activities.ReserveInventory, order.OrderID).Get(ctx, nil)
if err != nil {
return "", err
}
// Step 3: Dispatch confirmation email
err = workflow.ExecuteActivity(ctx, activities.SendConfirmationEmail, order.Email, order.OrderID).Get(ctx, nil)
if err != nil {
return "", err
}
return transactionID, nil
}3. Building the Worker Process
Workers are long-running Go binaries that poll Temporal task queues via gRPC. Place the following worker implementation in main.go:
package main
import (
"context"
"log"
"go.temporal.io/sdk/client"
"go.temporal.io/sdk/worker"
)
const TaskQueueName = "orders-queue"
func main() {
c, err := client.Dial(client.Options{
HostPort: client.DefaultHostPort,
})
if err != nil {
log.Fatalln("Unable to create Temporal client:", err)
}
defer c.Close()
w := worker.New(c, TaskQueueName, worker.Options{})
act := &OrderActivities{}
w.RegisterWorkflow(OrderWorkflow)
w.RegisterActivity(act)
go func() {
// Trigger a sample workflow execution after the worker boots
workflowOptions := client.StartWorkflowOptions{
ID: "order-sample-1001",
TaskQueue: TaskQueueName,
}
req := OrderRequest{
OrderID: "1001",
Amount: 89.99,
Email: "[email protected]",
}
we, startErr := c.ExecuteWorkflow(context.Background(), workflowOptions, OrderWorkflow, req)
if startErr != nil {
log.Println("Failed to start workflow:", startErr)
return
}
log.Printf("Started WorkflowID: %s, RunID: %s\n", we.GetID(), we.GetRunID())
}()
log.Println("Worker listening on task queue:", TaskQueueName)
err = w.Run(worker.InterruptCh())
if err != nil {
log.Fatalln("Worker encountered fatal error:", err)
}
}Verification & Monitoring: Testing Failures and Observability
Let’s confirm how the system behaves during sudden process failures. Launch the worker:
go run .Open http://localhost:8233 in your browser and select the execution titled order-sample-1001.
Validating Fault-Tolerance
Simulate an ungraceful crash right from your terminal:
- Run
go run .to start the workflow. - Hit
Ctrl+CwhileProcessPaymentruns to crash the worker process instantly. - Check the Temporal web console. The workflow stays in an open, healthy state instead of failing. Temporal simply waits for an active worker to poll the queue.
- Restart the worker with
go run .. Temporal re-assigns the queued task. The workflow resumes from the exact point of interruption without duplicating previous operations.
Production Monitoring and Key Metrics
Running this architecture in high-throughput environments requires monitoring two essential signals:
- Schedule-To-Start Latency: Track the time difference between when Temporal enqueues a task and when a worker claims it. In a healthy cluster, this stays below 50ms. If it creeps past 1 to 2 seconds, scale up your worker instances.
- Activity Failure Rates: Inspect the Temporal UI history tree during upstream provider outages. It provides full stack traces, input payloads, and failure reasons for every retry. You diagnose issues immediately without digging through unstructured logs.
Combining Temporal’s event-sourced state engine with Go’s lightweight concurrency model gives you clean, resilient workflows that survive network drops and server crashes out of the box.
