Failover

There are two basic types of failover: graceful and hard.

Graceful failover must be manually initiated. Hard failover can be manually initiated. Hard failover can also be initiated automatically by Couchbase Server: this is known as automatic failover. The Cluster Manager detects the unavailability of a node, and duly initiates a hard failover, without administrator intervention.

Note that when a node is failed over (as opposed to removed), some replica vBuckets are lost from the surviving nodes; since some are promoted to active status, and are not replaced with new replica-copies. By contrast, removal creates new copies of those replica vBuckets that would otherwise be lost. This maintains the cluster’s previous level of data-availability; but results in greater competition for memory resources, across the surviving nodes.

Ideally, after any failover, rebalance should be performed. This is especially important when a Data Service node has been failed over, since the rebalance will ensure an optimal ratio of active to replica vBuckets across all the remaining Data Service nodes.

Hard failover is performed after a node has failed. It can be initiated either by administrative intervention, or through automatic failover.

When automatic failover is used, the Cluster Manager handles both the detection of failure, and the initiation of hard failover, without administrative intervention: however, the Cluster Manager does not identify the cause of failure. Following failover, administrator-intervention is required, to identify and fix problems, and to initiate rebalance, whereby the cluster is returned to a healthy state.

If manual failover is to be used, administrative intervention is required to detect that a failure has occurred. This can be achieved either by assigning an administrator to monitor the cluster; or by creating an externally based monitoring system that uses the Couchbase REST API to monitor the cluster, detect problems, and either provide notifications, or itself trigger failover. Such a system might be designed to take into account system or network components beyond the scope of Couchbase Server.

When failover has occurred, and active vBuckets have thereby been replaced through the promotion of replicas, the resulting imbalance in the ratio of active to replica vBuckets on the surviving nodes should be corrected, by means of rebalance.

The variations in vBucket numbers and ratios that occur due to outage, failover, and rebalance are demonstrated by the following tables.

Table 1 shows the disposition of data across a cluster of four nodes. One bucket, of 31,591 items, has been configured with three replicas. The ratio of active to replica items is therefore 1:3. (Note that some numbers in the table are approximated, and are therefore shown in italics.)

Table 2 shows the result of failing over node 4.

Active and replica vBuckets for the failed over node, 4, are still counted, but are not available (and so are marked here with asterisks). To compensate, replicas on nodes 1 to 3 have been promoted to active status; this being evident from their modified numbers and ratios. For example, on node 1, the number of active items is now raised (from its former total of 7,932) to 11,000; while the number of replica items is now lowered (from its former total of 23,600) to 20,500.

Subsequent rebalance corrects the ratios, as shown by Table 3, below.

Ratios on nodes 1 to 3 are now 1:2, indicating that rebalance has reduced the number of replicas for the bucket from 3 to 2, in correspondence with the reduced node-count.

For further examples of rebalance in the context of node removal, see Removal.

The efficiency with which failover occurs is related to the cluster’s Master Services. These manage operations with cluster-wide impact; such as failover, rebalance, and adding and deleting buckets. The Master Services are sometimes referred to as the Orchestrator.

At any given time, only one instance of the Master Services is in charge: the instances having negotiated among themselves, to identify and elect the instance. Should the elected instance subsequently become unavailable, another takes over.

When the node that is currently hosting the active Master Services must itself be failed over, this requires greater latency than usual; since time must be taken to ensure that negotiation and election of a replacement occur. In case where the Master Services are co-located with a highly subscribed services (in particular, the Data Service), this can result in unnecessary latency in failing over the node and ensuring that data can be served.

In consequence, the Master Services should ideally not be co-located with any service. Such a configuration is possible in Couchbase Enterprise Server 7.6+: one or more serviceless nodes can be added to the cluster, with the intention of ensuring that the Master Services will occupy such a node, and thereby not be co-located with any service.

Node removal uses the rebalance process to remove a node from a cluster in a controlled fashion. It creates on the remaining nodes new copies of replica vBuckets that would otherwise be lost when the selected node is taken offline. See Removal for a conceptual description of node-removal. For practical steps, see Remove a Node and Rebalance.