Distributed Transactions in .NET: Eventual Consistency

A distributed transaction spans multiple services and databases. In a monolith, a single database transaction enforces ACID: all-or-nothing semantics ensure consistency. In microservices, each service owns a separate database; atomic transactions across them are impossible. The solution is the saga pattern: a sequence of local transactions, each in one service, with compensating transactions if a step fails.

A saga is a workflow where each step is a transaction in one service. Order Service creates an order; Inventory Service reserves items; Payment Service charges the card. If any step fails, previous steps are undone via compensation: if payment fails, the order is cancelled and the reservation is released. Unlike atomic transactions, compensations are eventual: the system is inconsistent momentarily, but converges to a consistent state.

The Saga Pattern: Choreography vs Orchestration

Two styles:

Choreography: Services publish events; other services listen and react. Order Service publishes OrderCreated; Inventory Service subscribes, reserves items, and publishes ItemsReserved; Payment Service subscribes to ItemsReserved, charges the card, and publishes PaymentAuthorized. If payment fails, Payment Service publishes PaymentFailed; other services subscribe and compensate (cancel the order, release the reservation).

Orchestration: A saga orchestrator directs the workflow. The orchestrator calls Inventory Service to reserve items; waits for the response; if successful, calls Payment Service; if the payment fails, calls Inventory Service to release the reservation. The orchestrator is the conductor; services are musicians following the baton.

Choreography is simpler and more decoupled; orchestration gives you explicit control. For complex workflows with many steps and conditional logic, orchestration is clearer. For simple workflows, choreography is lighter.

Choreography Example: Order Saga

// Order Service publishes events
public class OrderService
{
    private readonly OrderContext _dbContext;
    private readonly IPublishEndpoint _publishEndpoint;
    private readonly ILogger<OrderService> _logger;

    public async Task<Order> CreateOrderAsync(CreateOrderRequest req)
    {
        var order = new Order
        {
            CustomerId = req.CustomerId,
            Items = req.Items,
            Status = OrderStatus.Pending,
            CreatedAt = DateTime.UtcNow
        };

        await _dbContext.Orders.AddAsync(order);
        await _dbContext.SaveChangesAsync();

        // Publish event; Inventory Service will process it
        await _publishEndpoint.Publish<OrderCreatedEvent>(new
        {
            order.Id,
            order.CustomerId,
            order.Items
        });

        _logger.LogInformation("Order {OrderId} created", order.Id);
        return order;
    }

    // Subscribe to payment failure
    public async Task OnPaymentFailedAsync(PaymentFailedEvent evt)
    {
        var order = await _dbContext.Orders.FindAsync(evt.OrderId);
        if (order != null)
        {
            order.Status = OrderStatus.Cancelled;
            await _dbContext.SaveChangesAsync();

            // Publish event so Inventory can release the reservation
            await _publishEndpoint.Publish<OrderCancelledEvent>(new
            {
                order.Id
            });

            _logger.LogWarning("Order {OrderId} cancelled due to payment failure", order.Id);
        }
    }
}

// Inventory Service subscribes to order creation
public class OrderCreatedEventHandler : IConsumer<OrderCreatedEvent>
{
    private readonly InventoryContext _dbContext;
    private readonly IPublishEndpoint _publishEndpoint;
    private readonly ILogger<OrderCreatedEventHandler> _logger;

    public async Task Consume(ConsumeContext<OrderCreatedEvent> context)
    {
        var evt = context.Message;
        _logger.LogInformation("Received OrderCreatedEvent for order {OrderId}", evt.OrderId);

        try
        {
            // Reserve items in inventory
            var reservation = new Reservation
            {
                OrderId = evt.OrderId,
                Items = evt.Items.Select(i => new ReservedItem { ProductId = i.ProductId, Quantity = i.Quantity }).ToList(),
                CreatedAt = DateTime.UtcNow
            };

            await _dbContext.Reservations.AddAsync(reservation);
            await _dbContext.SaveChangesAsync();

            // Publish success event
            await _publishEndpoint.Publish<ItemsReservedEvent>(new
            {
                evt.OrderId,
                evt.Items
            });

            _logger.LogInformation("Items reserved for order {OrderId}", evt.OrderId);
        }
        catch (InsufficientInventoryException ex)
        {
            _logger.LogWarning(ex, "Could not reserve items for order {OrderId}", evt.OrderId);

            // Publish failure event; Order Service will cancel
            await _publishEndpoint.Publish<ReservationFailedEvent>(new
            {
                evt.OrderId,
                Reason = "Insufficient inventory"
            });
        }
    }
}

// Inventory Service: compensate on order cancellation
public class OrderCancelledEventHandler : IConsumer<OrderCancelledEvent>
{
    private readonly InventoryContext _dbContext;
    private readonly ILogger<OrderCancelledEventHandler> _logger;

    public async Task Consume(ConsumeContext<OrderCancelledEvent> context)
    {
        var evt = context.Message;
        _logger.LogInformation("Received OrderCancelledEvent for order {OrderId}", evt.OrderId);

        var reservation = await _dbContext.Reservations
            .FirstOrDefaultAsync(r => r.OrderId == evt.OrderId);

        if (reservation != null)
        {
            _dbContext.Reservations.Remove(reservation);
            await _dbContext.SaveChangesAsync();

            _logger.LogInformation("Reservation cancelled for order {OrderId}", evt.OrderId);
        }
    }
}

// Payment Service subscribes to items reserved
public class ItemsReservedEventHandler : IConsumer<ItemsReservedEvent>
{
    private readonly PaymentContext _dbContext;
    private readonly IPaymentProvider _paymentProvider;
    private readonly IPublishEndpoint _publishEndpoint;
    private readonly ILogger<ItemsReservedEventHandler> _logger;

    public async Task Consume(ConsumeContext<ItemsReservedEvent> context)
    {
        var evt = context.Message;
        _logger.LogInformation("Received ItemsReservedEvent for order {OrderId}", evt.OrderId);

        try
        {
            // Charge the card
            var payment = await _paymentProvider.AuthorizePaymentAsync(evt.OrderId, evt.Total);

            var transaction = new Transaction
            {
                OrderId = evt.OrderId,
                AuthorizationId = payment.Id,
                Amount = evt.Total,
                Status = TransactionStatus.Authorized,
                CreatedAt = DateTime.UtcNow
            };

            await _dbContext.Transactions.AddAsync(transaction);
            await _dbContext.SaveChangesAsync();

            // Publish success
            await _publishEndpoint.Publish<PaymentAuthorizedEvent>(new
            {
                evt.OrderId
            });

            _logger.LogInformation("Payment authorized for order {OrderId}", evt.OrderId);
        }
        catch (PaymentDeclinedException ex)
        {
            _logger.LogWarning(ex, "Payment declined for order {OrderId}", evt.OrderId);

            // Publish failure; Order Service will cancel
            await _publishEndpoint.Publish<PaymentFailedEvent>(new
            {
                evt.OrderId,
                Reason = "Payment declined"
            });
        }
    }
}

This choreography-based saga flows:

1. Order Service creates order, publishes OrderCreatedEvent.
2. Inventory Service reserves items, publishes ItemsReservedEvent.
3. Payment Service charges card, publishes PaymentAuthorizedEvent.
4. If payment fails, Payment Service publishes PaymentFailedEvent.
5. Order Service cancels order, publishes OrderCancelledEvent.
6. Inventory Service releases reservation in response to OrderCancelledEvent.

Each step is a local transaction. If payment fails, compensation undoes the reservation and order.

Orchestration Example: Saga Orchestrator

For explicit control, use a saga orchestrator:

public class OrderSagaOrchestrator
{
    private readonly OrderServiceClient _orderClient;
    private readonly InventoryServiceClient _inventoryClient;
    private readonly PaymentServiceClient _paymentClient;
    private readonly ILogger<OrderSagaOrchestrator> _logger;

    public async Task<OrderSaga> ExecuteOrderSagaAsync(CreateOrderRequest req)
    {
        var saga = new OrderSaga { OrderId = null, Status = SagaStatus.Started };

        try
        {
            // Step 1: Create order
            var order = await _orderClient.CreateOrderAsync(req);
            saga.OrderId = order.Id;
            _logger.LogInformation("Order {OrderId} created", order.Id);

            // Step 2: Reserve inventory
            var reservation = await _inventoryClient.ReserveItemsAsync(order.Id, req.Items);
            saga.ReservationId = reservation.Id;
            _logger.LogInformation("Reservation {ReservationId} made for order {OrderId}", reservation.Id, order.Id);

            // Step 3: Authorize payment
            var payment = await _paymentClient.AuthorizePaymentAsync(order.Id, req.Total);
            saga.PaymentId = payment.Id;
            _logger.LogInformation("Payment {PaymentId} authorized for order {OrderId}", payment.Id, order.Id);

            saga.Status = SagaStatus.Completed;
            return saga;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Saga failed at step; starting compensations");

            // Compensation: undo in reverse order
            if (saga.PaymentId != null)
            {
                await _paymentClient.ReversePaymentAsync(saga.PaymentId);
                _logger.LogInformation("Payment {PaymentId} reversed", saga.PaymentId);
            }

            if (saga.ReservationId != null)
            {
                await _inventoryClient.ReleaseReservationAsync(saga.ReservationId);
                _logger.LogInformation("Reservation {ReservationId} released", saga.ReservationId);
            }

            if (saga.OrderId != null)
            {
                await _orderClient.CancelOrderAsync(saga.OrderId);
                _logger.LogInformation("Order {OrderId} cancelled", saga.OrderId);
            }

            saga.Status = SagaStatus.Failed;
            throw;
        }
    }
}

public class OrderSaga
{
    public int? OrderId { get; set; }
    public int? ReservationId { get; set; }
    public int? PaymentId { get; set; }
    public SagaStatus Status { get; set; }
}

public enum SagaStatus { Started, Completed, Failed }

The orchestrator calls each service in sequence. If any step fails, it compensates the earlier steps in reverse order. This is more explicit and easier to test than choreography but creates a central coordinator.

Handling Idempotency

If a service handles an event twice (due to a retry or redelivery), it should not create duplicate state. Use idempotency keys:

public class OrderCreatedEventHandler : IConsumer<OrderCreatedEvent>
{
    private readonly InventoryContext _dbContext;

    public async Task Consume(ConsumeContext<OrderCreatedEvent> context)
    {
        var idempotencyKey = context.Message.OrderId.ToString();

        // Check if already processed
        var existing = await _dbContext.ProcessedEvents
            .FirstOrDefaultAsync(e => e.IdempotencyKey == idempotencyKey);

        if (existing != null)
        {
            // Already processed; skip
            return;
        }

        // Process the event
        var reservation = new Reservation { OrderId = context.Message.OrderId };
        await _dbContext.Reservations.AddAsync(reservation);

        // Record that we processed this event
        var processedEvent = new ProcessedEvent { IdempotencyKey = idempotencyKey, ProcessedAt = DateTime.UtcNow };
        await _dbContext.ProcessedEvents.AddAsync(processedEvent);

        await _dbContext.SaveChangesAsync();
    }
}

The ProcessedEvents table tracks which events have been handled. If a duplicate event arrives, the handler checks the table, sees it is already processed, and skips duplicate work.

Monitoring and Observability

Sagas can fail in many ways. Use correlation IDs and distributed tracing:

public async Task<Order> CreateOrderAsync(CreateOrderRequest req)
{
    var correlationId = Guid.NewGuid().ToString();
    var order = new Order { CorrelationId = correlationId };
    // ... create order

    // Include correlation ID in all published events
    await _publishEndpoint.Publish<OrderCreatedEvent>(new { order.Id }, context =>
    {
        context.CorrelationId = correlationId;
    });

    _logger.LogInformation("Order {OrderId} created with correlation {CorrelationId}", order.Id, correlationId);
    return order;
}

The correlation ID flows through all services, allowing you to trace the entire saga in logs. Use tools like Datadog, New Relic, or Application Insights to visualize saga execution and identify failures.

Key Takeaways

• Sagas coordinate multi-service transactions via compensations, not atomic locks.
• Use choreography (event-driven) for simple sagas; use orchestration for complex workflows with conditional logic.
• Ensure each service is idempotent: handling the same request twice produces the same result.
• Use correlation IDs to trace sagas across services for debugging and monitoring.
• Sagas provide eventual consistency, not strong consistency; accept momentary inconsistency.

Frequently Asked Questions

What if a compensation fails?

If a compensation fails, the system is left in an inconsistent state. Monitor for failed compensations and fix them manually (e.g., call the refund API directly). Design compensations to be idempotent so retries succeed.

Should I use sagas or two-phase commit?

Two-phase commit (2PC) enforces strong consistency but is slow, does not scale well, and is difficult in microservices. Sagas accept eventual consistency and scale better. Use sagas for microservices; 2PC is for monoliths or tightly coupled systems.

How do I test sagas?

Test each service independently (unit tests). Write integration tests that trigger the saga and verify compensation logic. Use test containers (Docker) to run dependencies (databases, message brokers) locally.

What if the order was already shipped before the cancellation event arrives?

Design compensations to check state before reversing. If the order is already shipped, cancellation should refund the customer but not un-ship the order. Compensations should be aware of all possible states.

How do I know if a saga succeeded or failed?

Publish a saga completion event. The orchestrator publishes OrderSagaCompletedEvent or OrderSagaFailedEvent. Clients subscribe to these events or poll the order status via the Order Service API.

Further Reading

• Saga Pattern (Chris Richardson)
• MassTransit Saga Implementation
• Azure Durable Functions for Sagas
• Event Sourcing and Sagas for Distributed Transactions (Microsoft)