Instrument Code for Tracing: .NET Guide
Instrumentation is the practice of inserting OpenTelemetry calls into your application code to emit spans. While automatic instrumentation covers framework components, business logic and domain operations require manual spans—the most important traces in any system.
I once spent three hours debugging a customer complaint that "payments fail 50% of the time." Automatic tracing showed the payment service was fine; database traces looked normal. Only after adding manual spans to the payment validation logic did I discover that a third-party fraud check service had a 40% timeout rate. Without instrumentation of that custom code, the issue was invisible. This article teaches you to avoid that trap.
Creating a Tracer and Basic Spans
Every instrumented operation starts with a tracer object, obtained from the TracerProvider:
using OpenTelemetry.Trace;
using System.Diagnostics;
// In your service or controller
private readonly Tracer _tracer;
public OrderService(TracerProvider tracerProvider)
{
_tracer = tracerProvider.GetTracer("OrderService");
}
public async Task ProcessOrderAsync(Order order)
{
// Create a span for the entire operation
using var span = _tracer.StartSpan("ProcessOrder");
span.SetAttribute("order.id", order.Id);
span.SetAttribute("customer.id", order.CustomerId);
span.SetAttribute("total_amount", order.Total);
// Business logic here
await ValidateInventoryAsync(order);
await ChargePaymentAsync(order);
await SendConfirmationAsync(order);
}
The using statement ensures the span is completed and exported even if an exception occurs. Attributes are key-value pairs that enrich the span with context—always include IDs for querying later.
Creating Child Spans for Nested Operations
Spans can have children, forming a tree that shows the causal flow of operations. A parent span starts child spans for sub-operations:
private async Task ValidateInventoryAsync(Order order)
{
// Child span: This is nested under ProcessOrder
using var span = _tracer.StartSpan("ValidateInventory");
span.SetAttribute("order.id", order.Id);
foreach (var item in order.Items)
{
using var itemSpan = _tracer.StartSpan("CheckItemStock");
itemSpan.SetAttribute("sku", item.Sku);
itemSpan.SetAttribute("quantity_requested", item.Quantity);
var available = await _inventoryService.GetStockAsync(item.Sku);
itemSpan.SetAttribute("quantity_available", available);
if (available < item.Quantity)
{
itemSpan.SetStatus(
ActivityStatusCode.Error,
"Insufficient stock"
);
}
}
}
The resulting trace tree shows:
ProcessOrder (100ms total)
├── ValidateInventory (25ms)
│ ├── CheckItemStock: WIDGET-001 (8ms)
│ └── CheckItemStock: GADGET-002 (12ms)
├── ChargePayment (50ms)
└── SendConfirmation (20ms)
This hierarchy is the primary value of tracing—you see exactly where time is spent and the sequence of operations.
Measuring Execution Time and Recording Events
OpenTelemetry automatically records start and end times, but you can add intermediate events for important milestones:
private async Task ChargePaymentAsync(Order order)
{
using var span = _tracer.StartSpan("ChargePayment");
span.SetAttribute("order.id", order.Id);
span.SetAttribute("amount_cents", order.Total * 100);
try
{
// Call payment processor
var startTime = DateTime.UtcNow;
var result = await _paymentProcessor.ChargeAsync(
order.CustomerId,
order.Total
);
var duration = (DateTime.UtcNow - startTime).TotalMilliseconds;
// Record a milestone event
span.AddEvent(new("PaymentAuthorized",
new ActivityTagsCollection
{
{ "transaction_id", result.TransactionId },
{ "processor_latency_ms", duration }
}
));
span.SetAttribute("transaction.id", result.TransactionId);
span.SetStatus(ActivityStatusCode.Ok);
}
catch (PaymentTimeoutException ex)
{
span.AddEvent(new("PaymentTimeout",
new ActivityTagsCollection
{
{ "message", ex.Message }
}
));
span.RecordException(ex);
span.SetStatus(
ActivityStatusCode.Error,
"Payment timeout after 30 seconds"
);
throw;
}
}
Events mark checkpoints within a span (authorization succeeded, timeout occurred). The span's status is set to Ok or Error based on the outcome. Recording exceptions ensures error details are captured.
Working with Baggage and Context Propagation
Baggage is data attached to the trace context and automatically propagated to child spans and across service boundaries. It carries request-scoped values without explicit parameter passing:
using OpenTelemetry.Api;
public async Task<IActionResult> CreateOrderAsync([FromBody] CreateOrderRequest req)
{
var requestId = Guid.NewGuid().ToString();
var correlationId = req.CorrelationId ?? requestId;
// Set baggage (automatically propagated to all child spans)
using var baggage = new BaggageSetterScope("request.id", requestId);
using var _ = new BaggageSetterScope("correlation.id", correlationId);
using var span = _tracer.StartSpan("CreateOrder");
span.SetAttribute("request.id", requestId);
var order = await _orderService.ProcessOrderAsync(req.OrderData);
return Created($"/orders/{order.Id}", order);
}
// Helper class (simplified; use ActivityBaggage in production)
public class BaggageSetterScope : IDisposable
{
private readonly string _key;
private readonly string _previousValue;
public BaggageSetterScope(string key, string value)
{
_key = key;
_previousValue = Baggage.GetBaggage(key);
Baggage.SetBaggage(key, value);
}
public void Dispose() => Baggage.SetBaggage(_key, _previousValue);
}
Baggage ensures every span in the entire trace tree has access to the request ID and correlation ID, enabling later queries like "show me all traces with correlation ID XYZ."
Handling Exceptions and Errors Comprehensively
Proper error instrumentation captures stack traces and categorizes failures:
private async Task SendConfirmationAsync(Order order)
{
using var span = _tracer.StartSpan("SendConfirmation");
span.SetAttribute("order.id", order.Id);
span.SetAttribute("customer.email", order.CustomerEmail);
try
{
var response = await _emailService.SendAsync(
to: order.CustomerEmail,
subject: $"Order {order.Id} Confirmed",
body: GenerateConfirmationEmail(order)
);
span.SetAttribute("email.message_id", response.MessageId);
span.SetStatus(ActivityStatusCode.Ok);
}
catch (SmtpException ex) when (ex.Message.Contains("timeout"))
{
span.RecordException(ex);
span.SetStatus(ActivityStatusCode.Error, "Email timeout");
span.SetAttribute("error.type", "SmtpTimeout");
// Log or retry; do not throw, as confirmation email is not critical
_logger.LogWarning("Email send timeout for order {OrderId}", order.Id);
}
catch (Exception ex)
{
span.RecordException(ex);
span.SetStatus(ActivityStatusCode.Error, ex.GetType().Name);
throw; // Critical error; re-throw
}
}
This pattern distinguishes between transient failures (retryable, log but don't crash) and critical errors (re-throw) while ensuring all exceptions are recorded in the trace.
Key Takeaways
- Always create a tracer from the
TracerProviderand useusingstatements to ensure spans are completed. - Set attributes (order ID, customer ID, amounts) on every span for later filtering and debugging.
- Create child spans for nested operations to build a hierarchical trace tree.
- Record events (checkpoints) and exceptions within spans to capture the full narrative of an operation.
- Use baggage to propagate request context (request ID, correlation ID) automatically across child spans.
Frequently Asked Questions
Should I create a span for every method, or only critical ones?
Critical methods (business operations, I/O, third-party calls). Avoid spans for trivial helpers; it creates noise. A good rule: if the method is async or calls an external service, span it.
What attributes are most important to set?
IDs first (order ID, customer ID, request ID). Then quantitative results (amount, count, duration). Skip redundant attributes if the parent span already has them.
Can I nest spans more than 2 or 3 levels deep?
Yes, but excessive nesting (10+ levels) makes traces hard to visualize. If you find yourself going that deep, refactor to larger, coarser-grained spans.
What is the performance cost of adding a span?
A single span (without export) adds roughly 1–5 microseconds. Batch exporters (article 2) amortize the cost over many spans, making it negligible for most applications.
How do I pass context to async methods across thread boundaries?
OpenTelemetry's automatic context capture handles this. Just ensure you use async/await (not Task.Run) and the context flows correctly.