Time comes when a single worker handling messages is not enough. There is a need to scale out.
The following sample demonstrates how easy it is, using NServiceBus to scale out your existing message processing by adding more workers on different machines.

The sample will start with a sender and a receiver.
After that, using few steps (and no code changes) we will scale out by executing workers to other machines.
We will see that to scale out our workers require no code changes.
Only few configuration changes your worker App.Config file to point to the distributor and using the right profiles at the command line.

Open the ScaleOut sample in a Visual studio that was started.
Run (F5) the solution - you should see two console applications start up. If you get a "queue not found" exception (its a common race condition for first run - when the queues were not created yet), then start the Orders.Handle project and then the Orders.Sender project.
Find the sender application by looking for the one with "Orders.Sender" in its path and press Enter in the window:

Your screen should look something like this:

Now let's go look at the code:

Code Walk-Through

There are three projects in the solution.

1. Orders.Handler handles the PlaceOrder command. Upon receiving the PlaceOrder command it will return PlaceOrderStatus.Ok status code and also publish an OrderPlaced event.

2. Orders.Sender sends the Place order command, receive the Ok status and handles the published event OrderPlaced after the Orders.Handlers completes "processing" the order and publish the event.

3. Orders.Messages containing the PlaceOrder command, the error codes enumeration and the OrderPlaced event.

Scaling Out

Lets assume business is booming, orders keep flowing in, and PlaceOrder commands are stacking up in the Orders.Handler endpoint.

We need to scale out.

Scaling out can be achieved by having the same Orders.Handler project function as a distributor and another copy of Orders.Handler function as a worker.

Starting the Orders.Handler with the Master profile, among other things, turns on the Distributor at this endpoint.

Starting the Orders.Handler with the Worker profile, makes it enlist with the Distributor and function as a worker.

Being a distributor means that Workers can send "I'm ready" message to the control endpoint of the Distributor, and the Distributor will forward messages to them in a round robin manner.
You can find more information about it here.

Steps to scale out

The following are the steps required to scale out our message handlers by deploying more workers on additional machines.

1. Setting up the Master (and Distributor) node

No changes should be done at the Master Orders.Handler.

You can start it directly from Visual studio and specifying the profiles as command line arguments.
Or you can start the Orders.Handler from the command line as follows:

1. Master profile. This will turn on the Distributor within the Orders.Profile. In addition, it will start a worker on that endpoint.

2. Integration profile. The production and integration profiles configures NServiceBus to use Raven for storing the subscriptions.  As stated earlier, this store will be shared among all Orders.Handler endpoints (all workers). You can read more about publish subscribe here.
   If running from visual studio, when configuring NServiceBus.Production profile, NServiceBus is configured to create the queues for you.
   If running from the command line, then NServiceBus.Integration should be used to create the queues.

In order for the Master node to function, the machine needs be accessible to Raven from where the Worker is deployed. By default, Raven is accepting calls on port 8080, you can check access to this machine by trying to get to the Raven management console on http://ip-of-MasterNode:8080. If management page does not load, open the port in your incoming firewall.

2. Setting up additional Worker

To scale out we need another worker to handle PlaceOrder messages. The following steps required to have another Worker up and running:

1. Stop the Orders.Handler.

2. Copy the Orders.Handler bin folder to another machine (or another folder on the same machine if you are just testing the sample). This instance of Orders.Handler is going to be our additional Worker.

3.  The master node configuration should be uncommented in the Orders.Handler.dll.config (two lines should b uncommented):

    <section name="MasterNodeConfig" ype="NServiceBus.Config.MasterNodeConfig, NServiceBus.Core" />

   		<MasterNodeConfig Node="HostNameWhereMasterNodeIsRunning"/>

   The Node should point to where the host name where the MasterNode is running.  If you are running the Worker from the same machine as the Distributor, than Node should equal "localhost".

    

That's it. To start the worker node, type the following at the command line:

1. NServiceBus.Integration. Orders.Handler is a publisher, a publisher stores its subscribers addressing information to know where the published event should go to.
   Since our workers run from different machines, they need to share the subscriptions.
   NServiceBus.Integration and NServiceBus.Production profiles register Raven as their underlying store.

2. NServiceBus.Worker profile. Order.Handler with this profile will send "ready to accept incoming messages"  to the master node.

By specifying the NServiceBus.Integration and NServiceBus.Worker profiles, along with the MasterNodeConfig Node setting in the configuration file, NServiceBus will direct the Orders.Handler Worker to use the same Raven instance as the Orders.Handler Master instance. The Raven database is used for storing the subscription information.
The NServiceBus.Worker profile is used by NServiceBus.Host.Exe to enlist the worker endpoint at the Distributor.

More on profiles, can be found here and here.

3. Setting up the Sender

Nothing have to be done to the Sender.
NServiceBus will do all the distribution work for you, leaving the sender agnostic to the fact that it is sending messages to a bunch of workers. As far as the sender concerns, it is sending the message to a single endpoint.
It is configured to send PlaceOrder command to the Orders.Handler endpoint, if the Distributor runs on a different machine than the Sender, then the queue@machine notation should be used.

<MessageEndpointMappings>
        <add Messages="Orders.Messages" Endpoint="Orders.Handler" />
</MessageEndpointMappings>

After setting up the Orders.Handler as a distributor and having the Orders.Handler starting as a worker on another machine, the following diagram demonstrates the flow of messages and the queues that exists on both machines:

What is going on here?

As can be seen from the diagram, nothing is changed for the Orders.Sender. It is still sending messages to Orders.Handler endpoint.

Orders.Handler that was started with the NServiceBus.Master profile is responsible to receive the messages that arrives at its Orders.Handler endpoint and forward them to its workers.
Since Orders.Handler was started with the NserviceBus.Master profile, besides being a Distributor, it also starts up a Worker, processing messages from the Orders.Handlers.Worker endpoint.

Both this worker and the Order.Handlers that is running on another machine with the NServiceBus.Worker profile send an "I'm ready to process messages" to the Distributor, to its Orders.Handler.Control endpoint.
After the Distributor forward the ProcessOrder command to one of its workers,  the receiving worker is responsible to process the PlaceOrder command (reply with a status to the Orders.Sender, and then publish the OrderProcess event).

When completing handling the ProcessOrder command, workers will send again, the "I'm ready to process messages" to the Distributor control endpoint: Orders.Handler.Control.

The workers are sharing the subscription storage so they can both be able to publish the OrderPlaces event.
NServiceBus is responsible to let all the workers know what/where database to connect to.
The database should be accessible to the workers.

Worker At Work:

The following snapshot shows the Worker console window at work (the worker is running from a different machine than the Distributor).

As you can see from the Worker console window (above), the worker is receiving the Process Order (order2) and replying with 'OK', processing the order and publishes an Order Placed event.

The error messages that can be seen in this sample appears when there is no valid license for the worker and it is working on a different node than the distributor.
This message will be logged for every received message from the remote worker.
In this case a valid NServiceBus License should be obtained.
You can get it here.

Where to go next?

It might be a good idea to read more about the distributor here.