Tracing and observability

A useful diagnostic record connects the business interaction to the processing step that failed or became slow. Troolio metadata provides correlation, causation, and transaction context. The event provider contributes stream and source position. Application logging and .NET tracing can make those facts searchable and measurable.

Keep business identity alongside diagnostics. “Registration 123 was requested twice” and “this dispatch took two seconds” are different questions, and one trace ID does not answer both automatically.

Use structured scopes and spans

The example instruments application-owned work with an ILogger scope and an ActivitySource span. Its callback supplies the operation being observed.

public static class WorkflowTelemetry
{
    public static readonly ActivitySource Source = new("Workshop.Workflows");

    public static async Task ExecuteAsync(ILogger logger, Metadata headers, Func<Task> work)
    {
        using IDisposable? scope = logger.BeginScope(new Dictionary<string, object?>
        {
            ["CorrelationId"] = headers.CorrelationId,
            ["MessageId"] = headers.MessageId,
            ["CausationId"] = headers.CausationId,
            ["TransactionId"] = headers.TransactionId
        });
        using Activity? activity = Source.StartActivity("registration.process", ActivityKind.Internal);
        activity?.SetTag("troolio.correlation_id", headers.CorrelationId.ToString("D"));
        try
        {
            await work();
            logger.LogInformation("Registration processing completed");
        }
        catch (Exception error)
        {
            activity?.SetStatus(ActivityStatusCode.Error, error.GetType().Name);
            logger.LogError(error, "Registration processing failed");
            throw;
        }
    }
}

StartActivity may return null when no listener requests a span, so instrumentation should tolerate that. Register an appropriate telemetry listener/exporter in your host to collect activities. The scope keeps related IDs available in logs even when trace collection is not enabled.

The official .NET instrumentation guide describes ActivitySource and Activity. A Troolio correlation GUID is an application tag; it does not automatically create a W3C parent context or propagate traceparent across remote calls. Preserve that trace context separately when the transport supports it.

Follow the causal chain

Begin an investigation with the business operation and correlation ID. Find the handled command, emitted events, target orchestration commands, projection work, and effect outcomes. Causation identifies the direct parent when a workflow branches.

For a projection failure, also retain consumer name, source stream/version, and projection version. For a selected command, retain the verified operation identity and route epoch. For an external effect, retain intent identity and provider receipt. Log the outcome category so a transient failure can be distinguished from invalid data or an uncertain provider result.

Protect the diagnostic surface

IDs can still identify people or devices. Restrict log access and retention. Avoid logging private event bodies, connection credentials, prompts, or provider tokens just to correlate messages. Error messages from providers may also contain sensitive details, so shape public errors independently from operator diagnostics.

Do not use arbitrary user, actor, correlation, or message IDs as metric labels. Their cardinality can grow with every request. Metrics can use bounded dimensions such as consumer kind, route, status category, and deployment version, while logs carry the individual identifiers needed for an investigation.

Measure the work after commit

Useful signals include command latency and conflicts, projection apply duration, oldest pending handoff, catch-up progress, quarantined sources, pending effect age, and provider terminal outcomes. A successful command dashboard alone misses a read side that has been stuck for hours.

Measure per meaningful partition where it helps isolate impact. A global ready signal should not conceal one unavailable user's view, and one broken partition should not automatically mark unrelated application reads unavailable.

Add checks at the failure boundary

Exercise process loss before and after event commit, projection commit, source acknowledgement, and provider execution. Inspect whether the retained IDs and statuses let you determine the outcome. A timeout with no operation identity in diagnostics leaves the application unable to decide whether a retry is safe.

Continue with message metadata, correlation and causation, and projection catch-up.