Similar in behavior to standard load balancers
the NServiceBus Distributor is the key to scaling out
message processing over many machines transparently.

As a standard NServiceBus process, the distributor maintains
all the fault-tolerant and performance characteristics of
NServiceBus but also is designed never to overwhelm any
of the worker nodes configured to receive work from it.

Why use it

When starting to use NServiceBus, you'll see that you can easily run multiple instances of the same process with the same input queue. This may look like scaling-out at first, but is really no different than running multiple threads within the same process. If you try to do this with multiple machines, you'll see that you can't share a single input queue across multiple machines.

The distributor gets around this limitation.

What about MSMQ v4

In version 4 of MSMQ, made available with Vista and Server 2008, there is now the ability to perform 'remote transactional receive' which wasn't possible in previous versions.

What this means is that processes on other machines can transactionally pull work from a queue on a different machine. If the machine processing the message were to crash, the message would roll back to the queue and other machines could then process it.

Even though the distributor provided similar functionality even before Vista was released, there are other reasons to use it even on the newer OS.

The problem with 'remote transactional receive' is that it gets proportionally slower as more worker nodes are added. This is due to the overhead of managing more transactions, as well as the longer period of time that these transactions are open.

In short, the scale-out benefits of MSMQ v4 by itself are quite limited.

How does it work

Worker nodes send messages to the distributor, telling it when they're ready for work. These messages arrive at the distributor via a separate 'control' queue. The distributor stores this information.

When applicative messages arrive at the distributor, it uses the previously stored information to find a free worker node, and sends the message on to it.

If no worker nodes are free, the distributor waits a bit and then repeats the previous step.

This way, all pending work stays in the distributor's queue (rather than building up in each of the workers' queues) giving visibility into how long messages are actually waiting. This is important for complying with time-based service level agreements (SLAs).

For more information on monitoring, see Monitoring NServiceBus Endpoints.

Where is it

Unlike NServiceBus V2.6, there is no specific Distributor process.
Any NServiceBus endpoint maybe configured to work as a Distributor.

To see the Distributor in action you may have a look at the ScaleOut sample.

Distributor Configuration

When Hosting In NServiceBus.Host.exe

If you are running with the NServiceBus.Host.exe, you have the following profiles that will start your endpoint with the Distributor functionality:

To start your endpoint as a distributor, you can run it from the command line as follows:

The NServiceBus.Integration profile makes sure all required resources will be created.

The NServiceBus.Distributor profile, instruct the NServiceBus framework to start a distributor on this endpoint waiting for workers to enlist to it. Unlike the NServiceBus.Master profile, the NServiceBus.Distributor profile, will not execute a Worker on its node.

You may use the NServiceBus.Master to start, on your endpoint a Distributor with a Worker on its endpoint. When specifying the NServiceBus.Master profile, also the gateway runs on endpoint.

When Self-hosting

When you self host your endpoint, the following Configuration are available after configuring AsMasterNode():

• RunDistributor() - This will start your endpoint as a Distributor, waiting for Workers to enlist, and then distribute load to those enlisted workers. In addition, it will start a local worker (working from the same machine as the Distributor).
• RunDistributorWithNoWorkerOnItsEndpoint() - This will start your endpoint as a Distributor, waiting for Workers to enlist, and then load balance.

The following is an example of a Distributor with a Worker on its endpoint:

Configure.With()
    .Log4Net()
    .DefaultBuilder()
    .XmlSerializer()
    .AsMasterNode()
        .RunDistributor()
    .MsmqTransport()
    .UnicastBus()
    .CreateBus()
    .Start(() => Configure.Instance.
        ForInstallationOn<Environments.Windows>().Install());

Replacing .RunDistributor() with with .RunDistributorWithNoWorkerOnItsEndpoint() will run the Distributor without a worker on its endpoint.

Worker Configuration

Any NServiceBus endpoint may run as a Worker node. To "activate it", a handler of the relevant messages should exist and the app.config file should contain routing information of the Distributor. Please see  to see it working.

When Hosting In NServiceBus.Host.exe

If you are hosting your endpoint with NServiceBus.Host.exe, to run as a worker, you may use the following command line:

If queues were already created, you can run it also using the following:

The only thing you need to configure is the name of the master node server. An example app.config is shown below. Note the MasterNodeConfig section

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="MessageForwardingInCaseOfFaultConfig"
         type="NServiceBus.Config.MessageForwardingInCaseOfFaultConfig, NServiceBus.Core" />
    <section name="UnicastBusConfig" 
         type="NServiceBus.Config.UnicastBusConfig, NServiceBus.Core"/>
    <section name="MasterNodeConfig" 
        type="NServiceBus.Config.MasterNodeConfig, NServiceBus.Core" />
  </configSections>

  <MessageForwardingInCaseOfFaultConfig ErrorQueue="error"/>
  <MasterNodeConfig Node="MachineWhereDistriutorRuns"/>
</configuration>

More about the DistributorControlAddress and the DistributorDataAddress the Routing with the Distributor section in this document.

When Self-hosting

When self hosting, you should call the EnlistWithDistributor() configuration extension method.
The Enlist command takes the endpoint name of the worker and sends an "I am ready message" to the control queue of the distributor. See more on this below in the Routing section.

The following is an example of configuration of a Worker node:

Configure.With()
    .Log4Net()
    .DefaultBuilder()
    .EnlistWithDistributor()
    .XmlSerializer()
    .MsmqTransport()
    .UnicastBus()
    .CreateBus()
    .Start(() => Configure.Instance. 
        ForInstallationOn<Environments.Windows>().Install());

Similar to self hosting, the app.config of the worker should contain the MasterNodeConfig section to point to the host name where the master node (and a distributor) are running.

Routing with the Distributor

The distributor makes use of two queues for its runtime operation. The DataInputQueue is the queue that client processes will be sending their applicative messages to. The ControlInputQueue is the queue that worker nodes will be sending their control messages to.

If you need to use other values than the NServiceBus defaults you can override them like displayed in the UnicastBusConfig section below

<UnicastBusConfig
    DistributorControlAddress="distributorControlBus@Cluster1"
    DistributorDataAddress="distributorDataBus@Cluster1">

    <MessageEndpointMappings>
      <!-- regular entries -->
    </MessageEndpointMappings>
  </UnicastBusConfig>

If those settings do not exist, the control queue is assumed to be the endpoint name of the worker, concatenated with the "distributor.control@HostWhereDistributorIsRunning" string.

Similar to standard NServiceBus routing, you wouldn't want high priority messages to get stuck behind lower priority messages, so just as you would have separate NServiceBus processes for different message types, you would also set up different distributor instances (with separate queues) for different message types.

In this case, you could name the queues just like the messages - for example, SubmitPurchaseOrder.StrategicCustomers.Sales. This would be the name of the distributor's data queue and that of the input queues of each of the workers. The distributor's control queue would best be named with a prefix of 'control' as follows: Control.SubmitPurchaseOrder.StrategicCustomers.Sales.

When using the distributor in a full publish/subscribe deployment, what we'll see is a distributor within each subscriber balancing the load of events being published as follows:

Keep in mind that the distributor is designed for load balancing within a single site and should not be used between sites. In the image above, all publishers and subscribers are within a single physical site.

For information on using NServiceBus across multiple physical sites, see the gateway.

High Availability

If the distributor goes down, even if its worker nodes remain running they will not receive any messages. Therefore, it is important to run the distributor on a cluster configuring its queues as clustered resources.

Since the distributor doesn't do CPU or memory intensive work, you can often put several distributor processes on the same clustered server. Be aware the network IO may end up being the bottleneck for the distributor so take into account message sizes and throughput when sizing your infrastructure.

Licensing and Distribution

Basic licensing restricts the number of worker nodes to 2, you can read more about it here.

Where to go from now?

Build a scalable solution using Master node and workers solution that are in the ScaleOut sample.