The Gateway And Multi-Site Distribution

The challenges of high availability have greatly
increased the number of multi-site deployments
for enterprise .NET systems.

Another benefit of multi-site distributions is
faster response times for users as the
servers and data they access is closer.

RPC technologies quickly run into trouble in these
environments as they make machines in the same site
and those in remote sites look the same.

Messaging works out better than RPC in these cases but still many developers mistakenly represent physical site boundaries as logical boundaries resulting in problems here as well. NServiceBus prevents developers from going down the wrong path but may leave them wondering how is multi-site communication handled.

Disaster recovery and physical sites

In some cases, physical sites are replicas of each other. This is a common configuration for the purposes of disaster recovery and is largely influenced by technology, cost, and performance.

NServiceBus provides no special facilities for disaster recovery other than to enable developers to plug in their own specific technologies into it. This can be of the form of database replication of subscription information, configuring MSMQ to store its message data on a SAN, etc. The difference in price and performance of the various options is quite large and will not be covered here.

The use of NServiceBus in logically significant physical sites is described in the next section.

Logically significant physical sites

While each branch of a bank or retail store has significance in each domain, when looking at the behaviors of each site we see a great deal of similarity even to the point of identical functionality. This may not be true across all sites, especially when looking at sites which serve as regional centers or headquarters.

The logical services that make up the business solution can have components installed at multiple physical sites - some of the components can be the same, others may be different. It is likely that multiple logical services in the same site will collaborate with each other quite closely, and possibly less closely than with their own components at other sites.

For example, we'd expect the Sales service in a store to talk to the Pricing service in the same store for every transaction. On the other hand, the Pricing service at the headquarters would likely do at most a daily push of updated prices to the stores. Similarly we'd expect an End-Of-Day push of transactions from the Sales service at each store to the headquarters.

Store to headquarters pricing and sales interaction

This approach is not only the most common but is recommended for use in situations where physical sites have logical significance - keep all inter-site communication within logical service boundaries.

Intra-Service Cross-Site Messaging

Notice that when sites have logical significance, the messages passed between them are different than the messages sent within the site. The act of publishing prices from the headquarters has logical significance. The manager of a store explicitly performs an "end of day" operation after all cash has been collected from the tills and counted.

Therefore, design separate classes for those messages passed between sites.

Cross-Site Data Transfer

Depending on your network technology, you can choose to set up a virtual private network (VPN) between your sites. This provides Windows networking visibility of queues in the target site from the sending site. At that point, you'll likely be using standard NServiceBus APIs to direct messages to their relevant targets, in the form of Bus.Send(toDestination, msg);

This model is recommended as it provides all the benefits of durable messaging between machines with an unreliable connection between them between sites the same as within a single site.

You can find a great deal of information on setting up and managing a Windows VPN here.

In the cases where you only have access to HTTP for connection between sites, you can enable the NServiceBus Gateway feature on each site and it will transmit messages from a queue in one site to a queue in another site, including the hash of the messages to ensure that the message was transmitted correctly. The following diagram shows how it works:

The sending process in site A sends a message to the gateway's input queue. The gateway then initiates an HTTP connection to it's configured target site. The gateway in site B accepts HTTP connections, takes the message transmitted, hashes it, and returns the hash to site A. If the hashes match, the gateway in site B transmits the message it received to a configured queue. If the hashes don't match, the gateway in site A retransmits.

Configuration and Code

When configuring the client endpoint, make sure that in the UnicastBusConfig's MessageEndpointMappings element that you have an entry indicating that the relevant message types go to the gateway's input queue.

To send a message top a remote site you would use the SendToSites api call displayed below:

Bus.SendToSites(new[]{"SiteA","SiteB"},new MyCrossSiteMessage());

Did you notice the list of strings as the first parameter? This is the list of remote sites that you want the message(s) to be sent to. While you can put the Urls of the site directly in the call we recommend that you put these settings in you app.config in order for your admins to change them should the need arise. To do this just add the config section below:

<section name="GatewayConfig"
         type="NServiceBus.Config.GatewayConfig, NServiceBus.Core" />

<GatewayConfig>
    <Sites>
      <Site Key="SiteA" Address="http://SiteA.mycorp.com/" ChannelType="Http"/>
      <Site Key="SiteB" Address="https://SiteB.mycorp.com/" ChannelType="Http"/>
    </Sites>
  </GatewayConfig>

NServiceBus will automatically set the required headers that enables you to do the familiar Bus.Reply to send messages back over the gateway. Note that since all cross site interactions is perfomed internally to a service publish and subscribe is not supported across gateways.

Since the gateway is located in the NServiceBus core you can enable it by just flipping a switch. If you runt the NServiceBus host you can enable it by specifying the MultiSite profile ( more on profiles). If you self host NServiceBus you can turn on the gateway be adding a call to Configure.RunGateway() in your configuration.

Securing The Gateway With SSL

In order to provide data encryption for messages transmitted between sites, configure SSL on the machines in each site where the gateway is running.

Follow the steps for configuring SSL found on this page and then make sure to configure the gateway to listen on the appropriate port, as well as to contact the remote gateway on the same port.

Automatic deduplication

Going across alternate channels like Http means that you loose the safety of guaranteed exactly one messaging the MSMQ provides for you. This means that communication errors resulting in retries can lead to messages beeing received more than once. To avoid burdening users with deduplication the NServiceBus gateway support this out of the box. In order to do this deduplication we need to store the message id's of all received messages to be able to detect potential duplicates. By default NServiceBus uses RavenDB to store those id's but there is also support for InMemory and SqlServer storages as well depending on your requirements. To use a storage other than Raven just add Configure.RunGatewayWithInMemoryPersistence() or Configure.RunGateway(typeof(SqlPersistence)) to you configuration.

Incoming channels

When you enable the gateway it will automatically setup a http channel to listen on http://localhost/{name of you endpoint}. If you need to change this Url or add more than one incoming channel you need to configure this in your app.config. The section below describes this syntax:

<GatewayConfig>
    <Channels>
      <Channel Address="https://Headquarter.mycorp.com/" ChannelType="Http" Default="true"/>
      <Channel Address="http://Headquarter.myotherdomain.com/" ChannelType="Http"/>
    </Channels>
  </GatewayConfig>

Note the "Default" on the first channel. This tells the gateway which address to attach on outgoing messages if not specified explicitly by the sender. You can of course add as many channels as you like and also mix between all the supported channels. Currently Http/Https is the only supported channel but there are plans for both Azure, Ftp and Amazon SQS to help users bridge both onsite and cloud sites.

Follow the steps for configuring SSL found on this page and then make sure to configure the gateway to listen on the appropriate port, as well as to contact the remote gateway on the same port.

The gateway in action

I you want to take the gateway for a spin please take a look at the Gateway sample in the Samples\Gateway folder under the NServiceBus package.