Upgrade an Offline Cluster
A multi-node cluster can most simply be upgraded when entirely offline; meaning that it is not serving data.
Understanding Offline Cluster-Upgrade
Offline cluster-upgrade can occur when the cluster is not required to serve data for some period of time. All application-access to the cluster is stopped, and each node in turn is upgraded. Then, the cluster is brought back online, and application-access to it is restored. The page provides step-by-step instructions for upgrading an offline multi-node cluster.
The procedure is presented in three stages, below.
Stage One: Prepare the Cluster
Proceed as follows:
-
Stop all applications that access the cluster. Monitor the disk-write queue on each node, to ensure that all data has been persisted to disk. A command of the following form can be used:
curl -s -u '${USERNAME}:${PASSWORD}' ${NODE_HOSTNAME}:${NODE_MANGEMENT_PORT}/pools/default/buckets/${BUCKET}/stats | jq ".op.samples.disk_write_queue[-1]"
When this command returns
0
, all data has been persisted to disk. -
Back up the cluster’s user-data. Backup can be performed either with cbbackupmgr or with the Backup Service; and should be a full (rather than an incremental) backup.
For example, to use
cbbackupmgr
to configure an archive and repository for the backup, a command of the following form should be entered:cbbackupmgr config --archive ${ABS_PATH_TO_ARCHIVE} --repo ${REPO_NAME}
Here,
ABS_PATH_TO_ARCHIVE
is an absolute path to a filesystem location that will serve as the archive within which the backed up data will reside. TheREPO_NAME
is the name of the repository that will be associated with the location.Once the archive and repository have been created, a command such as the following performs a full backup:
cbbackupmgr backup --archive ${ABS_PATH_TO_ARCHIVE} --repo ${REPO_NAME} --cluster ${CLUSTER_ADDRESS} --username ${USERNAME} --password ${PASSWORD} --full-backup
Here, the
CLUSTER_ADDRESS
is the IP address or domain name of the cluster. The--full-backup
flag ensures that the backup is indeed a full backup.For the equivalent procedure as performed by the Backup Service, see Run an Immediate Backup.
-
Disable auto-failover (to prevent auto-failover from occuring when individual nodes stop communicating with the rest of the cluster, during their upgrade). Disablement is performed by means of Couchbase-Server General Settings, using the UI, the CLI, or the REST API. Disablement needs to be performed only once, on one node.
For example, to disable auto-failover by means of the CLI, enter a command of the following form:
couchbase-cli setting-autofailover -c ${NODE_HOSTNAME} -u ${USERNAME} -p ${PASSWORD} --enable-auto-failover 0
Stage Two: Upgrade Each Individual Node
Each individual node must now be upgraded in turn. Therefore, for each node, proceed as follows:
-
Stop the
couchbase-server.service
service, on the first node that is to be upgraded. Enter the following command:systemctl stop couchbase-server.service
This stops the service; and so allows it to be restarted after reboot. Note that, optionally, at this point, the service can also be disabled; which prevents it from restarting after reboot. This may be useful if additional tasks, such as OS upgrade, need to be performed. If such disabling is desired, enter the following command:
systemctl disable --now couchbase-server.service
Note that to disable and/or stop Couchbase Server Community Edition, in these commands,
couchbase-server-community
should be substituted forcouchbase-server
. -
Manually back up the node’s configuration files. These files reside in
/opt/couchbase/var/lib/couchbase/config
. It is recommended that the path to the backup-location for these files contain the node’s domain name or IP address, to ensure accurate recovery. It is also recommended that the backup-location be on a separate machine.Enter a command such as the following;
cp -r /opt/couchbase/var/lib/couchbase/config ${PATH_TO_A_SAFE_LOCATION}/${NODE_IP}_config_files
-
If using Couchbase-provided repositories/PPAs, the Couchbase-Server version should be unpinned, to allow it to be upgraded. Proceed as follows, for the appropriate platform:
By editing the file
/etc/yum/yum.conf
, ensure that the packagecouchbase-server
(or the packagecouchbase-server-community
, if using Community Edition) is not excluded. This means that the package-name must not appear on the list of one or more package-names that follows theexclude=
statement.For example, the line may initially appear as follows:
exclude=couchbase-server
If so, edit the line so that it appears as follows:
exclude=
Then, save the file.
Run the following command (specifying, if using Community Edition,
couchbase-server-community
, rather thancouchbase-server
):apt-mark unhold couchbase-server
-
Upgrade the Couchbase Server package, using the package manager from the distribution currently running on the cluster.
If using a Couchbase-provided
yum
repository, enter the following:yum update couchbase-server
If Using a Couchbase-provided PPA, enter the following:
apt --only-upgrade install couchbase-server
If using a downloaded package-archive, enter the command appropriate for the platform, as follows:
-
Enable and start the
couchbase-server
(orcouchbase-server-community
) service. (The package installer may already have performed enablement: however, explicit enablement is nevertheless recommended.)Enter the following commands (substituting, if using Community Edition,
couchbase-server-community
forcouchbase-server
).systemctl enable --now couchbase-server.service systemctl is-active --quiet couchbase-server.service || systemctl start couchbase-server.service
-
Repin (or hold) future package-upgrades for Couchbase Server, so that none occurs before the administrator’s next, elective, manually driven upgrade. Proceed as follows for the appropriate platform:
Add the
couchbase-server
(orcouchbase-server-community
) package to theexclude
section of/etc/yum/yum.conf
. The line appears as follows:exclude=couchbase-server
Run the following command (substituting, if running Community Edition,
couchbase-server-community
forcouchbase-server
):apt-mark hold couchbase-server
-
Repeat all prior steps in this section, Upgrade Each Individual Node, for every other node in the cluster.
Stage Three: Bring the Cluster Back Online
Proceed as follows:
-
Wait for the completion of warmup, for all Couchbase buckets. Note that this may take some time, if the buckets contain large amounts of data.
The status of warmup can be checked for each node as follows:
cbstats ${NODE_ADDRESS}:${NODE_KV_PORT} -u ${USERNAME} -p ${PASSWORD} -b ${BUCKET} warmup | grep state
For example:
/opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b travel-sample warmup | grep state
When warmup is complete, the command returns the following:
ep_warmup_state: done
Note that Ephemeral buckets do not require warmup. If an Ephemeral bucket is specified in this command, an error is returned.
-
Following warmup, bring the cluster back online, restarting applications.
This concludes the upgrade process for the offline, multi-node cluster.