Once the primary cluster is setup, backup clusters may be added for real-time or delayed
backup of all data of the primary cluster.

For ease of understanding, let us consider that we have additional 2 nodes that are
independently setup for a cluster

Cluster 1 - nodes

Cluster 2 - nodes

Initial Setup
The cluster to be added as a backup cluster, must first be setup as an independent cluster.
All nodes in the cluster must be empty nodes i.e. no data or user permission changes shall
be carried out in the nodes.

On Node 1 or Node 2

blobcity>add-cluster node3 192.168.1.1 <mode>

Running the above command on either node1 or node2 will add the entire cluster of
node3 and node4 as a realtime replication cluster. All data on nodes 1 & 2 will get copied
to nodes 3 & 4. Post copy, all transactions initiated on cluster 1 will get replicated at
realtime on cluster 2. The reverse is also true wherein any query run on cluster 2 will get
replicated at realtime on cluster 1.

For cluster 2 to become operational could be a long running operation. The time it takes to
fully connect cluster 2 to cluster 1 is dependent on the amount of data to be moved from
cluster 1 to cluster 2 and the network speed available to carry out the move.

Note: You cannot run the add-cluster query on node1 or node2 assumed the two nodes
already have data. Just like add-node the add-cluster query is also forward binding.

The mode parameter can be one of r-rep or d-rep. They stand for realtime replication
and delayed replication respectively. The functioning of each mode is explained below.

Realtime Replication Mode

In case of selecting the realtime replication mode also called the r-rep mode, transactions
on any one cluster are replicated at realtime onto other clusters. This means transactions
on one cluster are blocked until they are carried out on all other r-rep mode clusters.

All Real-time Replicated Clusters are Equal
Each r-rep cluster inside BlobCity is an equal cluster. This means incase of shutdown of
your primary cluster, the backup cluster can take the full transactional load and offer
business continuity. When the primary comes back up, data will be automatically synced
to the cluster which was offline for a while.

When using multiple r-rep clusters, you should use a load balancer to frontend all
requests going into the database. The load balancer should be cluster aware and must
route the queries to a desired cluster. It is advisable that the load balancer route all
queries only to a single cluster at any point in time. In case of failure of any cluster, the
load balancer can route to any other active r-rep cluster. In case the load balancer routes
queries at all times to two or more clusters, your setup may not be partition tolerant.

Delayed Replication Mode

In case of delayed replication mode also called d-rep mode, data from the primary clusters
is synced to the d-rep cluster as fast as possible, but not necessarily in realtime.
Transactions on primary clusters are not blocked until they are performed on the d-rep
cluster. In case of system failures d-rep clusters may or may not hold the latest copies of
data.

d-rep clusters take read-only queries at all times. In case of failure of the primary cluster,
you can read data from the d-rep clusters, but you cannot run data insert, update or delete
transactions on the d-rep cluster.