Design guide

Design for partial failure, not the happy path.

Distributed systems trade local simplicity for independent deployment, scale, and resilience. The fundamentals below help you decide where messaging helps—and what your application must still guarantee.

Know when asynchronous messaging helps

Choose asynchronous messaging when the sender can hand off work without an immediate answer, when bursts should be buffered, when several capabilities react independently to the same fact, or when work should survive a temporary outage in another service. It is especially useful at a real ownership boundary where the receiver must control its own pace.

A broker is not a substitute for a well-placed service boundary. If two pieces of code always change together, share one transaction, and cannot be useful independently, a local call or modular monolith is often the better design.

Start with boundaries and ownership

Split services around business capabilities and data ownership, not technical layers. A service should be able to make its own decisions with the data it owns. Crossing a boundary is a business interaction with latency and failure, even when the call looks like an ordinary method invocation.

Make message intent obvious

A command asks one capability to do something. Name it in the imperative, such as SubmitOrder. An event reports a fact that has already happened, such as OrderSubmitted, and may have many subscribers. Queries and request/response are useful when the caller truly cannot proceed without an answer, but they keep the caller temporally coupled to the responder.

Assume delivery can happen more than once

A broker and consumer can disagree about whether work completed—for example, when a consumer commits a database change and loses its connection before the acknowledgement arrives. Design consumers to be idempotent: processing the same operation again must produce the same durable result without repeating an irreversible side effect.

Use a stable business operation identifier, enforce uniqueness in durable storage, and make the side effect and processed marker atomic when possible. The in-memory check shown above explains the shape; production code needs a database constraint or transaction that remains correct across crashes and replicas.

Treat consistency as a workflow

A transaction cannot normally span a service database and a broker safely. If a state change and an outgoing message must either both happen or neither happen, use a transactional outbox or another durable hand-off. MyServiceBus now provides matching PostgreSQL persistence foundations for C# and Java. The high-level bus capture and hosted dispatch layers are still being integrated, so the current boundary is explicit rather than a one-line production option.

See the transactional outbox guide for the working provider surface, verified database behavior, and remaining gates.

From message reactions to explicit workflows

The first workflow-coordination challenge usually arrives as choreography, even when nobody calls it that yet. One service publishes a fact, another reacts, and that reaction produces the next message. Those decisions often live inline with the consumer code that sends or publishes the next contract. Each consumer is understandable on its own, but the broader workflow is spread across contracts, producers, consumers, and independently deployed services.

As the application grows, that implicit workflow becomes difficult to express and verify. Which reactions are intended? Which states or outcomes have no handler? Can a cycle amplify messages? Did a process stall, or is an optional participant simply quiet? The code may remain locally simple while the system-wide story becomes hard to see.

MyServiceBus's choreography work begins there: make decentralized reactions explicit enough to document, compare with configured routing, and observe without moving ownership into a central engine. A declared choreography enriches the inline reactions with workflow identity, ownership, expected outcomes, and operational context; it describes and connects the local knowledge rather than replacing it.

Some workflows reveal that their decisions should not be distributed. When one component genuinely needs to know the broader process, remember its progress, handle timeouts, and decide the next command, orchestration defines that knowledge separately in a coordinator—often a saga state machine hosted by one coordinating service. The saga sits between the participating services in the workflow relationship: it consumes their messages and directs later work through messages without moving their local business capabilities into the coordinator. This can make the process easier to reason about, but the runtime is more demanding: durable state, correlation, concurrency, retries, compensation, and recovery all become part of the coordinator's contract.

Both are ways services relate and coordinate through messages. The difference is where the coordination relationship lives:

1. Choreography — a decentralized coordination relationship: each service keeps workflow knowledge beside its local reaction: which events it handles and which messages it emits. A choreography declaration can connect and enrich that distributed intent, but no participant necessarily owns or understands the whole process.
2. Orchestration — a centralized coordination relationship: one participant—the orchestrator, often implemented separately as a saga or state machine—knows the cross-service process and directs or advances it through messages. It owns the workflow state and decides what should happen next while the participating services retain their own business responsibilities.

“Centralized” here means that process knowledge and decisions have one explicit owner; it does not require one deployment, database, or synchronous call chain. The distinction is the coordination relationship and ownership of workflow knowledge, not whether a broker or messages are used. Both styles communicate through produced and consumed message contracts. The orchestrator is itself a message-driven participant; choreography distributes the decisions among the participants that own each reaction.

These relationships can coexist in the same distributed system. One workflow may remain a decentralized choreography, while another uses an orchestrator because its progress and decisions need one explicit owner. An orchestrated workflow can also publish events that start or contribute to choreographed reactions outside the coordinator's boundary.

You can implement either style yourself using consumers, sends, publications, persistence, and application-owned state. MyServiceBus is building tools and abstractions that make those intents easier to express, verify, and operate in both C# and Java: declared local reactions and causal diagnostics for choreography, and sagas plus state-machine definitions for orchestration. Your services still own the business rules; the framework supplies the coordination vocabulary and runtime behavior. That progression—from discovering an implicit choreography to choosing whether it should remain decentralized or become an explicit orchestration—is the focus of the current work.

Choreography comes first. Its declarations and Dashboard projections live above the messaging protocol as local topology and operational metadata. They do not change the application messages, MassTransit-compatible envelopes, headers, addresses, or broker conventions exchanged between services, so a MassTransit service can continue to participate in the same message flow without understanding the MyServiceBus choreography model.

The Dashboard provides a common place to observe both forms. “Workflow” is the collective catalog and query term, while monitoring keeps choreography records and saga/state-machine records separate and labels every result by implementation kind and lifecycle authority. Shared catalog and run-index projections provide filtering and navigation, then link each declared workflow and observed run to its complete kind-specific page. A choreography map groups reactions by service and reconstructs bounded workflow runs from declarations and exact causal message evidence, while showing where that evidence is incomplete. Selecting a run opens a D3 activity diagram with application swimlanes plus an ordered step breakdown: which application and consumer handled each message, handler and observed handoff duration, what it emitted, and where retries or failures occurred. The view updates as observations arrive but treats live activity, terminal reactions, and inactivity as evidence rather than authoritative lifecycle state. The initial orchestration view instead shows the saga's current state and committed transition timeline; a richer state-machine graph can later highlight the traversed path and annotate pending timeouts, faults, and instance counts. The views may look related, but the Dashboard must never present a reconstructed choreographed outcome as though it were persisted workflow state.

The mixed-language Aspire sample now makes that boundary executable. A C# saga owns an order's cross-service sequence, sends inventory work to a Java participant, advances when Java publishes the result, then coordinates a C# payment participant and publishes completion. Inventory and payment rules remain local to their services; the saga owns only correlation, current state, and the next coordination decision. Both clients export registered state-machine definitions and endpoint attachment to a dedicated monitoring saga catalog and emit payload-free committed-transition evidence to a separate bounded saga-instance query. Neither is stored as choreography or presented as a reconstructed choreography run. The sample repository and transition history are intentionally volatile while durable profiles are developed.

An additional experimental Raven saga! DSL now projects this same orchestration model into a concise state-machine language. It generates the ordinary .NET saga definition and executes through the existing MyServiceBus runtime; it does not alter the portable C# and Java concepts or the messages exchanged between services.

Classify failures before retrying

Retry transient failures such as brief network or dependency outages. Do not retry invalid input, missing business preconditions, or permanent authorization failures: retries only increase load and delay diagnosis. Keep the retry window bounded, use backoff for shared dependencies, and make the consumer idempotent first.

After retries are exhausted, MyServiceBus publishes a Fault<T> and moves the original delivery to the endpoint's _error queue. Treat these as operational signals for diagnosis or compensation; never blindly republish a failed message.

Timeouts create uncertainty, not cancellation

A timeout tells the caller that an answer did not arrive in time. It does not prove the remote work stopped—or even that it failed. MyServiceBus request clients have a default 30-second deadline. Give commands stable operation IDs, make retries safe, and provide a way to query the eventual result when the business operation may outlive the request.

Avoid hidden ordering assumptions

Do not assume a global message order. Retries, multiple consumers, competing replicas, and independent queues can change completion order. Put the current entity version or sequence in the contract when order matters, partition work by the business key when the transport supports it, and make consumers reject or defer stale transitions explicitly.

Carry context across the boundary

Logs alone cannot explain a workflow spread across services. Preserve message, correlation, and trace identifiers; record the endpoint and message type; and measure throughput, latency, retries, faults, and queue age. Use bounded labels for metrics—never message or correlation IDs.

MyServiceBus propagates W3C trace context and exposes runtime monitoring for aggregate flow and failures. Custom headers are useful for bounded business context such as a tenant, but must not contain secrets or unbounded payload data.

Review every message flow

• Who owns the data and the business decision?
• Is this a command, an event, or a query—and does its name make that clear?
• What happens if it is delayed, duplicated, or completed out of order?
• Which failures are transient, and where do exhausted messages go?
• What does a timeout mean to the caller, and how can the final result be found?
• Which consistency guarantees are required, and where is the durable hand-off?
• Can one slow dependency create a retry storm or exhaust consumer capacity?
• Can an operator trace, diagnose, and safely repair the workflow?
• Have the happy path, duplicate, timeout, retry, and poison-message paths been tested?