cbcollect_info

    +
    The cbcollect_info tool provides detailed statistics for a specific node.

    Syntax

    The basic syntax is:

    cbcollect_info [options] [output-file]

    where output-file is the name of the .zip file containing the diagnostic information.

    Description

    The cbcollect_info command gathers statistics from an individual node in the cluster.

    To collect diagnostic information for an entire cluster, run the command for every node that exists in that cluster. If you are experiencing problems with multiple nodes in a cluster, you may need to run it on all nodes in a cluster.

    A root account is required to run this command and collect all the server information. There are internal server files and directories that this tool accesses that require root privileges.

    To use this command, remotely connect to the machine which contains the Couchbase Server then issue the command with appropriate options. This command is typically run under the direction of technical support at Couchbase and generates a large .zip file. This archive contains files which contain performance statistics and server logs.

    cbcollect_info is one of the most important diagnostic tools. Run cbcollect_info on all nodes in the cluster, and upload all of the resulting files to Couchbase technical support.

    This tool is at the following locations:

    Table 1. Tool locations
    Operating System Location

    Linux

    /opt/couchbase/bin/cbcollect_info

    Windows

    C:\Program Files\Couchbase\Server\bin\cbcollect_info

    Mac OS X

    /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/cbcollect_info

    The tool creates the following log files in the output archive:

    Table 2. Output files
    File Log Contents

    backup_service.log

    Contains information on activities performed by the Backup Service.

    cbcollect_info.log

    Contains information about the performance of cbcollect_info.

    couchbase.log

    Contains system-level information, primarily the result of shell commands. Generated at collection-time.

    ddocs.log

    Contains the design documents for each bucket in JSON format, taken from the view file on disk.

    diag.log

    A diagnostic summary of the cluster manager, obtained by downloading /diag output from the node’s running service.

    kv_trace.json

    Metadata used for Phosphor KV tracing.

    ini.log

    A dump of all ini configuration files.

    master_events.log

    Contains major events that have affected the cluster, like bucket creation, and rebalance information such as vBucket movement.

    memcached.log

    Contains information relating to the core memcached component, including DCP stream requests and slow operations.

    ns_server.analytics.log

    Troubleshooting log for the Analytics service.

    ns_server.analytics_debug.log

    Debug-level troubleshooting log for the Analytics service.

    ns_server.analytics_shutdown.log

    Shutdown log for the Analytics service.

    ns_server.analytics_trace.log

    Trace-level troubleshooting log for the Analytics service.

    ns_server.babysitter.log

    Troubleshooting log for the babysitter process which is responsible for spawning all Couchbase Server processes and respawning them where necessary.

    ns_server.couchdb.log

    Troubleshooting log for the couchdb subsystem which underlies map-reduce.

    ns_server.debug.log

    Debug-level troubleshooting for the cluster management component.

    ns_server.error.log

    Error-level troubleshooting log for the cluster management component.

    ns_server.eventing.log

    Eventing service logs.

    ns_server.fts.log

    Troubleshooting logs for the full-text search service.

    ns_server.goxdcr.log

    Troubleshooting log for the cross datacenter replication (XDCR) component used in Couchbase Server versions after 4.0.

    ns_server.http_access.log

    The admin access log records server requests (including administrator logins) to the REST API or Couchbase Server web console. It is output in common log format and contains several important fields such as remote client IP, timestamp, GET/POST request and resource requested, HTTP status code, and so on.

    ns_server.http_access_internal.log

    The admin access log records internal server requests (including administrator logins) to the REST API or Couchbase Server web console. It is output in common log format and contains several important fields such as remote client IP, timestamp, GET/POST request and resource requested, HTTP status code, and so on.

    ns_server.indexer.log

    Troubleshooting log for the indexing and storage subsystem.

    ns_server.info.log

    Info-level troubleshooting log for the cluster management component.

    ns_server.json_rpc.log

    JSON RPC logs.

    ns_server.mapreduce_errors.log

    JavaScript and other view-processing errors are reported in this file.

    ns_server.metakv.log

    Troubleshooting log for the metakv store, a cluster-wide metadata store.

    ns_server.ns_couchdb.log

    Contains information related to starting up the CouchDB subsystem.

    ns_server.projector.log

    Troubleshooting log for the projector process which is responsible for sending appropriate mutations from Data nodes to Index nodes.

    ns_server.query.log

    Query service logs.

    ns_server.reports.log

    Contains progress and crash reports for the Erlang processes. Due to the nature of Erlang, processes crash and restart upon an error.

    ns_server.ssl_proxy.log

    Troubleshooting log for the ssl proxy spawned by the cluster manager.

    ns_server.stats.log

    Contains periodic statistic dumps from the cluster management component.

    ns_server.views.log

    Troubleshooting log for the view engine, predominantly focussing on the changing of partition states.

    ns_server.xdcr.log

    Troubleshooting log for the cross datacenter replication (XDCR) component used in Couchbase Server versions prior to 4.0.

    ns_server.xdcr_errors.log

    Error-level troubleshooting log for the cross datacenter replication (XDCR) component used in Couchbase Server versions prior to 4.0.

    ns_server.xdcr_target.log

    Log of traces from cross datacenter replication (XDCR) target endpoint handlers.

    ns_server.xcdr_trace.log

    Trace-level troubleshooting log for the cross datacenter replication (XDCR) component used in Couchbase Server versions prior to 4.0. Unless trace-level logging is explicitly turned on this log is empty.

    projector_pprof.log

    Goroutine dump captured for the projector process. A goroutine dump is the equivalent of a full thread dump in other languages and can give valuable insight into what a process is doing at that time.

    stats.log

    Contains output from various cbstats commands. Generated at collection-time.

    syslog.tar.gz

    Archive of various system-level logs.

    systemd_journal.gz

    (Linux only.) The logs from systemd.

    system:completed_requests

    A list of recent, completed requests. See system:completed_requests.

    Options

    The following are the command options:

    Table 3. cbcollect_info options
    Parameter Description

    -h, --help

    Shows help information.

    -r ROOT

    Specifies the root directory. Defaults to the parent directory of the directory containing cbcollect_info, e.g. /opt/couchbase/bin/.. on Linux.

    -v

    Increases the verbosity level: If specified, debugging information will be included in the console display.

    -p

    Specifies that only product-related information should be gathered.

    -d

    Dumps a list of commands required by cbcollect_info.

    --bypass-sensitive-data

    If set to true, skips collecting breakpad crash-dumps.

    --task-regexp=TASK_REGEXP

    Runs only those tasks that match the specified regular expression. (This flag is provided for debugging purposes.)

    --tmp-dir=TMP_DIR

    Specifies the tmp directory that is used during data processing. This setting overrides any existing setting of the TMPDIR environment variable.

    --initargs=INITARGS

    Specifies the server initargs path.

    --log-redaction-level=REDACT_LEVEL

    Specifies the redaction level for the logs collected. The level can be either none (the default) or partial.

    --log-redaction-salt=SALT_VALUE

    Salts the hashing of tagged data. The default value is a random uuid. If this flag is used, a --log-redaction-level value should be specified.

    --upload-host=UPLOAD_HOST

    Specifies the fully-qualified domain name of the host you want the logs uploaded to. The protocol prefix of the domain name, https://, is optional. It is the default-only supported protocol.

    --customer=UPLOAD_CUSTOMER

    Specifies the customer name. This value must be a string whose maximum length is 50 characters. Only the following characters can be used: [A-Za-z0-9_.-]. If any other characters are included, the request is rejected.

    --upload-proxy=UPLOAD_PROXY

    Specifies a proxy for upload.

    --ticket=UPLOAD_TICKET

    Specifies the Couchbase Support ticket-number. The value must be a string with a maximum length of 7 characters, containing only digits in the range of 0-9.

    Examples

    To create a diagnostics .zip file, log onto the node and run the cbcollect_info tool.

    On Linux, run as root or use sudo:

    sudo /opt/couchbase/bin/cbcollect_info <node_name>.zip

    On Windows, run as Administrator:

    C:\Program Files\Couchbase\Server\bin\cbcollect_info <node_name>.zip

    Response

    The following example response shows partial output when running the cbcollect_info command.

    Using temporary dir /tmp
    Log Redaction () - OK
    uname (uname -a) - OK
    time and TZ (date; date -u) - OK
    ntp time (ntpdate -q pool.ntp.org || nc time.nist.gov 13 || netcat time.nist.gov 13) - OK
    ntp peers (ntpq -p) - Exit code 127
    raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - Exit code 1
    raw /etc/timezone (cat /etc/timezone) - OK
    System Hardware (lshw -json || lshw) - OK
    Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK
    
    ...
    
    adding: /tmp/tmpY7p_3T/couchbase.log -> cbcollect_info__20180718-084858/couchbase.log
    adding: /tmp/tmpY7p_3T/ns_server.error.log -> cbcollect_info__20180718-084858/ns_server.error.log
    adding: /tmp/tmpY7p_3T/ns_server.analytics_debug.log -> cbcollect_info__20180718-084858/ns_server.analytics_debug.log
    adding: /tmp/tmpY7p_3T/ns_server.json_rpc.log -> cbcollect_info__20180718-084858/ns_server.json_rpc.log
    adding: /tmp/tmpY7p_3T/cbcollect_info.log -> cbcollect_info__20180718-084858/cbcollect_info.log
    adding: /tmp/tmpY7p_3T/ns_server.ns_couchdb.log -> cbcollect_info__20180718-084858/ns_server.ns_couchdb.log
    adding: /tmp/tmpY7p_3T/ns_server.analytics.log -> cbcollect_info__20180718-084858/ns_server.analytics.log
    adding: /tmp/tmpY7p_3T/stats.log -> cbcollect_info__20180718-084858/stats.log
    adding: /tmp/tmpY7p_3T/memcached.log -> cbcollect_info__20180718-084858/memcached.log
    adding: /tmp/tmpY7p_3T/ns_server.indexer.log -> cbcollect_info__20180718-084858/ns_server.indexer.log
    adding: /tmp/tmpY7p_3T/ns_server.fts.log -> cbcollect_info__20180718-084858/ns_server.fts.log
    adding: /tmp/tmpY7p_3T/ns_server.http_access_internal.log -> cbcollect_info__20180718-084858/ns_server.http_access_internal.log
    adding: /tmp/tmpY7p_3T/ns_server.xdcr_target.log -> cbcollect_info__20180718-084858/ns_server.xdcr_target.log
    adding: /tmp/tmpY7p_3T/ns_server.eventing.log -> cbcollect_info__20180718-084858/ns_server.eventing.log
    adding: /tmp/tmpY7p_3T/syslog.tar.gz -> cbcollect_info__20180718-084858/syslog.tar.gz
    adding: /tmp/tmpY7p_3T/ns_server.reports.log -> cbcollect_info__20180718-084858/ns_server.reports.log
    adding: /tmp/tmpY7p_3T/ns_server.query.log -> cbcollect_info__20180718-084858/ns_server.query.log
    adding: /tmp/tmpY7p_3T/ns_server.couchdb.log -> cbcollect_info__20180718-084858/ns_server.couchdb.log
    adding: /tmp/tmpY7p_3T/ns_server.goxdcr.log -> cbcollect_info__20180718-084858/ns_server.goxdcr.log
    adding: /tmp/tmpY7p_3T/master_events.log -> cbcollect_info__20180718-084858/master_events.log
    adding: /tmp/tmpY7p_3T/ns_server.info.log -> cbcollect_info__20180718-084858/ns_server.info.log
    adding: /tmp/tmpY7p_3T/diag.log -> cbcollect_info__20180718-084858/diag.log
    adding: /tmp/tmpY7p_3T/ns_server.analytics_shutdown.log -> cbcollect_info__20180718-084858/ns_server.analytics_shutdown.log
    adding: /tmp/tmpY7p_3T/systemd_journal.gz -> cbcollect_info__20180718-084858/systemd_journal.gz
    adding: /tmp/tmpY7p_3T/ns_server.analytics_trace.json -> cbcollect_info__20180718-084858/ns_server.analytics_trace.json
    adding: /tmp/tmpY7p_3T/ns_server.stats.log -> cbcollect_info__20180718-084858/ns_server.stats.log
    adding: /tmp/tmpY7p_3T/ns_server.views.log -> cbcollect_info__20180718-084858/ns_server.views.log
    adding: /tmp/tmpY7p_3T/ini.log -> cbcollect_info__20180718-084858/ini.log
    adding: /tmp/tmpY7p_3T/kv_trace.json -> cbcollect_info__20180718-084858/kv_trace.json
    adding: /tmp/tmpY7p_3T/ns_server.mapreduce_errors.log -> cbcollect_info__20180718-084858/ns_server.mapreduce_errors.log
    adding: /tmp/tmpY7p_3T/ns_server.projector.log -> cbcollect_info__20180718-084858/ns_server.projector.log
    adding: /tmp/tmpY7p_3T/ns_server.metakv.log -> cbcollect_info__20180718-084858/ns_server.metakv.log
    adding: /tmp/tmpY7p_3T/ns_server.debug.log -> cbcollect_info__20180718-084858/ns_server.debug.log
    adding: /tmp/tmpY7p_3T/ns_server.babysitter.log -> cbcollect_info__20180718-084858/ns_server.babysitter.log
    adding: /tmp/tmpY7p_3T/ns_server.http_access.log -> cbcollect_info__20180718-084858/ns_server.http_access.log