Please use the form below to provide your feedback. Because your feedback is valuable to us, the information you submit in this form is recorded in our issue tracking system (JIRA), which is publicly available. You can track the status of your feedback using the ticket number displayed in the dialog once you submit the form.

A newer version of this documentation is available.

View Latest

Release Notes

    +

    Release notes for the Kafka Connector.

    Couchbase Kafka Connector 3.4.8 (2020-04-21)

    kafka-connect-couchbase-3.4.8.zip

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.8</version>
</dependency>

    The Sink connector’s Couchbase client now uses mutation tokens to improve performance when the couchbase.durability.replicate_to or couchbase.durability.persist_to config options are used. This also prevents the DocumentConcurrentlyModifiedException that would sometimes occur with the old CAS-based durability checks.

    Enhancements

    Couchbase Kafka Connector 3.4.7 (2020-03-18)

    kafka-connect-couchbase-3.4.7.zip

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.7</version>
</dependency>

    This maintenance release fixes a bug in the Source connector where specifying a custom port in the connection.cluster_address config property would cause a connection failure.

    The Couchbase libraries are also updated to the latest versions.

    Bug Fixes

    • KAFKAC-165 Source: Cannot connect to Couchbase on custom port

    Enhancements

    • KAFKAC-164 Upgrade DCP client from 0.25.0 to 0.26.0

    • KAFKAC-168 Upgrade Couchbase client from 2.7.11 to 2.7.13

    Known Issues

    Unlike the Source connector, the Sink connector still can’t use custom ports. We’re hoping to address this in a future release.

    • KAFKAC-167 Sink: Cannot connect to Couchbase on custom port

    Couchbase Kafka Connector 3.4.6 (2020-01-21)

    kafka-connect-couchbase-3.4.6.zip

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.6</version>
</dependency>

    The connector is now more careful about tracking virtual bucket UUIDs. This makes it much less likely that a rollback to zero will occur when restarting the connector after a Couchbase Server rebalance.

    Bug Fixes

    • KAFKAC-118 Source: vBucketUuid may be incorrect

    • KAFKAC-161 Source: Should include vBucket UUID in saved offset

    Enhancements

    • KAFKAC-162 Upgrade to Couchbase client 2.7.11 and DCP client 0.25.0

    Couchbase Kafka Connector 3.4.5 (2019-08-21)

    kafka-connect-couchbase-3.4.5.zip

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.5</version>
</dependency>

    This maintenance release brings the connector up to date with the latest versions of the Couchbase Client SDK and DCP client. As a result of this upgrade, the connector is now able to connect to Couchbase Server clusters that advertise alternate addresses. This behavior is controlled by the new couchbase.network configuration property which defaults to automatic network selection.

    The alternate address selection feature introduced in this patch version is an "uncommitted" feature, meaning it may change without advance notice in a future patch version.

    Enhancements

    • KAFKAC-154 Expose multi-network config options

    • KAFKAC-155 Upgrade to Couchbase client 2.7.9 and DCP client 0.24.0

    Couchbase Kafka Connector 3.4.4 (2019-05-21)

    kafka-connect-couchbase-3.4.4.zip

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.4</version>
</dependency>

    This maintenance release fixes a bug that could prevent some versions of Couchbase Server from rebalancing when the connector is running.

    Bug Fixes

    • KAFKAC-148 Couchbase Server fails to rebalance if Kafka connector is running

    Couchbase Kafka Connector 3.4.3 (2019-03-19)

    kafka-connect-couchbase-3.4.3.zip

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.3</version>
</dependency>

    This patch release adds out-of-the-box support for publishing Couchbase documents to Kafka as raw JSON using RawJsonSourceHandler.

    The quickstart guide has been updated with an example of how to apply Single Message Transforms to modify documents before the Sink connector writes them to Couchbase.

    The included examples/custom-extensions Maven project now demonstrates how to implement a custom transform.

    The new source handlers and transforms introduced in this patch version are "uncommitted" features, meaning they may change without advance notice in a future patch version.

    Enhancements

    • KAFKAC-141 Add example of a custom Single Message Transform

    • KAFKAC-142 Support publishing raw JSON without having to build example project

    • KAFKAC-143 Upgrade DCP client to 0.22.0

    Couchbase Kafka Connector 3.4.2 (2019-02-19)

    kafka-connect-couchbase-3.4.2.zip

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.2</version>
</dependency>

    This maintenance release improves the behavior of persistence polling and connection keep-alive.

    When connecting to a multi-node Couchbase cluster, connections not used for streaming will no longer be incorrectly flagged as dead.

    Persistence polling no longer repeatedly closes connections to servers not hosting any partitions during a rebalance operation.

    In this release the Couchbase client dependency has been upgraded to version 2.7.2 for security updates in the repackaged Jackson databind library. Because Jackson versions compatible with Java 1.6 are no longer maintained, the connector is no longer compatible with Java 1.6. The connector now requires Java 1.8 or later. Couchbase prefers to make such platform changes in minor releases, but is applying the change here given the nature of the issue.

    Enhancements

    • KAFKAC-136 Require Java 1.8 or later

    • KAFKAC-137 Fix Maven issues with Java 11

    • KAFKAC-138 Upgrade Kafka API dependencies to 1.0.2

    • KAFKAC-139 Upgrade Couchbase client to 2.7.2 and DCP client to 0.21.0

    Bug Fixes

    • JDCP-116 BucketConfigHelper.getHostedPartitions() fails when node hosts no partitions

    • JDCP-117 Dead connection detection gives false positives when no streams are open

    Couchbase Kafka Connector 3.4.1 (2019-01-15)

    kafka-connect-couchbase-3.4.1.zip

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.1</version>
</dependency>

    This maintenance release fixes memory leaks that can occur in distributed mode or under certain error conditions.

    Also, the connector now responds to more error conditions by reporting them to the Kafka Connect framework instead of failing silently.

    Issues resolved

    • KAFKAC-133: [BUGFIX] Buffers not released on all code paths

    Couchbase Kafka Connector 3.4.0 (2018-10-16)

    kafka-connect-couchbase-3.4.0.zip

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.0</version>
</dependency>

    New features

    Version 3.4 introduces rollback mitigation in the form of persistence polling. Messages are published to Kafka only after the database events have been saved to disk on all Couchbase replicas. Rollback mitigation is enabled by default. It may be disabled by setting the persistence polling interval to 0.

    The flow control buffer size is now configurable, with a default size of 128 megabytes. This is an increase over the previous hard-coded value of 20 megabytes.

    New configuration properties

    • couchbase.flow_control_buffer=128m

    • couchbase.persistence_polling_interval=100ms

    Issues resolved

    • KAFKAC-116: [ENHANCEMENT] Add support for rollback mitigation

    • KAFKAC-127: [BUGFIX] Source connector can’t reconnect to Couchbase hosted on Docker Container

    • KAFKAC-124: [BUGFIX] Rollback mitigation causes data loss when opening DCP stream

    Couchbase Kafka Connector 3.4.0-beta.1 (2018-07-06)

    kafka-connect-couchbase-3.4.0-beta.1.zip

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.4.0-beta.1</version>
</dependency>
    Version 3.4.0-beta.1 has a bug that can cause data loss. When rollback mitigation is enabled, some events might not be published to the Kafka topic. See KAFKAC-124 for details. This issue is resolved in the 3.4.0 release.

    When database changes are rolled back due to failover of a Couchbase Server node, the connector will now by default prevent the rolled-back changes from being published to the Kafka topic. This is done by buffering changes until they have been persisted to all replicas in the Couchbase cluster (and are unlikely to be rolled back). The connector polls the database at a configurable interval to determine which changes have been persisted. This feature is enabled by default, and may be disabled by setting the polling interval to 0.

    The flow control buffer size is now configurable, with a default size of 128 megabytes. This is an increase over the previous hard-coded value of 20 megabytes.

    New configuration properties:

    • couchbase.flow_control_buffer=128m

    • couchbase.persistence_polling_interval=100ms

    Issues resolved in this release:

    • KAFKAC-116: [ENHANCEMENT] Add support for rollback mitigation

    Couchbase Kafka Connector 3.3.2 GA (2018-07-02)

    kafka-connect-couchbase-3.3.2.zip

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.3.2</version>
</dependency>

    The source connector now resumes streaming after a Couchbase Server rollback.

    Issues resolved in this release:

    • KAFKAC-72: [BUGFIX] Connector stops streaming after rollback

    • KAFKAC-115: [ENHANCEMENT] Upgrade to DCP client 0.18.0

    Couchbase Kafka Connector 3.3.1 GA (2018-06-19)

    kafka-connect-couchbase-3.3.1.zip

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.3.1</version>
</dependency>

    The sink connector can now set expiration times for documents, and can assign document IDs by composing values from multiple fields.

    The source connector now continues streaming after a graceful failover.

    Issues resolved in this release:

    • KAFKAC-104: [ENHANCEMENT] Support for setting TTL on items being written to Couchbase

    • KAFKAC-106: [ENHANCEMENT] Support assigning document ID from multiple fields

    • KAFKAC-109: [ENHANCEMENT] KAFKAC-109 Upgrade to Java client 2.5.9 and DCP client 0.17.0

    • JDCP-85: [BUGFIX] Client stops receiving some events after failover

    • JDCP-94: [BUGFIX] Should never pass null ChannelFlowController to event handlers

    Couchbase Kafka Connector 3.3.0 GA (2018-05-15)

    kafka-connect-couchbase-3.3.0.zip

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.3.0</version>
</dependency>

    Compression is now enabled by default (requires Couchbase Server 5.5 or later).

    IPv6 is now enabled by default. To disable IPv6 in a dual-stack environment, set the new couchbase.forceIPv4 config property to true.

    Resolved a regression that caused the connector to need a restart after a Couchbase Server restart.

    Couchbase password and SSL keystore password may be specified using via KAFKA_COUCHBASE_PASSWORD and KAFKA_COUCHBASE_SSL_KEYSTORE_PASSWORD environment variables.

    The sink connector can now modify documents using the sub-document API (many thanks to community member Didier Caron).

    Issues resolved in this release:

    • KAFKAC-97: [BUGFIX] Kafka Connector needs a restart after Couchbase Server restart

    • KAFKAC-102: [ENHANCEMENT] Enable compression by default

    • KAFKAC-100: [ENHANCEMENT] Enable IPv6 by default, add first-class config option

    • KAFKAC-96: [ENHANCEMENT] Support setting password via environment variable

    • KAFKAC-98: [ENHANCEMENT] Sink: Support sub-document access (Didier Caron)

    • KAFKAC-103: [ENHANCEMENT] Upgrade Couchbase dependencies

    Couchbase Kafka Connector 3.2.3 GA (2018-02-20)

    Keepalive is now enabled on the Couchbase Server connection. This prevents the server from dropping idle connections, and enables dead connection detection. Thanks to community member Patrik Nordebo (patrikn).

    A new config key "couchbase.log_redaction" controls whether sensitive info in connector log messages is tagged for redaction. Options are "NONE", "PARTIAL", and "FULL".

    A new source config key "couchbase.compression" can be used to enable compression when reading from Couchbase Server 4.5 and later. Options are "DISABLED", "ENABLED", and "FORCED".

    Issues resolved in this release:

    • KAFKAC-89: [ENHANCEMENT] Enable NOOP for dead connection detection (Patrik Nordebo)

    • KAFKAC-82: [FEATURE] Implement log redaction for Kafka Connector

    • KAFKAC-90: [FEATURE] Source: Add config settings to enable compression

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.3</version>
</dependency>

    kafka-connect-couchbase-3.2.3.zip

    Couchbase Kafka Connector 3.2.2 GA (19 December 2017)

    The source connector now does a better job of reporting abnormal termination. Thanks to community member p_mx (tiny1990).

    A new config key "couchbase.stream_from" lets you tell the source connector when in Couchbase history to start streaming from. Options are "BEGINNING", "NOW", "SAVED_OFFSET_OR_BEGINNING", and "SAVED_OFFSET_OR_NOW".

    When the sink connector receives a Kafka message with a null value, it now deletes the Couchbase document whose ID matches the Kafka message key. (Previous versions would terminate when a null value was encountered.)

    You can now specify durability requirements for the sink connector’s write operations via two new config keys: "couchbase.durability.persist_to" and "couchbase.durability.replicate_to".

    Issues resolved in this release:

    • KAFKAC-84: [FEATURE] Sink: Allow setting durability requirements for Couchbase writes

    • KAFKAC-85: [FEATURE] Sink: Support deletion

    • KAFKAC-86: [FEATURE] Source: Restart from a given state / offset

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.2</version>
</dependency>

    kafka-connect-couchbase-3.2.2.zip

    Couchbase Kafka Connector 3.2.1 GA (8 November 2017)

    Fixes a regression in 3.2.0 that prevented the sink connector from working.

    Issues resolved in this release:

    • KAFKAC-83: [BUGFIX] Sink connector fails due to UnsupportedOperationException in JsonBinaryTranscoder.newDocument.

    • KAFKAC-81: [ENHANCEMENT] Document how to customize source handler

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.1</version>
</dependency>

    kafka-connect-couchbase-3.2.1.zip

    Couchbase Kafka Connector 3.2.0 GA (17 October 2017)

    The sink connector is now officially supported.

    Multiple source connector instances reading from the same Couchbase bucket can now manage their state independently. To enable this feature, assign a unique name to each connector and set the new config property compat.connector_name_in_offsets to true.

    The source connector has a new, more flexible SourceHandler extension point intended to replace the Converter interface. By providing a custom SourceHandler, a developer can filter events, route messages to a topic other than the configured default topic, and control the format of the Kafka message.

    The default Kafka message format now includes bucket and vBucketUuid fields, which may be used along with the partition (vBucketID) and bySeqno fields to construct a Couchbase MutationToken.

    The sink connector now allows the Couchbase document ID to come from a field of the document. This feature is controlled by two new config properties, couchbase.document.id and couchbase.remove.document.id.

    The Couchbase client libraries are upgraded to the latest versions, with performance enhancements and other improvements for Couchbase Server 5.0.

    The following classes are scheduled for removal in version 4.0.0 of the connector:

    • com.couchbase.connect.kafka.converter.Converter - Deprecated in favor of SourceHandler.

    • com.couchbase.connect.kafka.converter.SchemaConverter - Deprecated in favor of DefaultSchemaSourceHandler.

    Issues resolved in this release:

    • KAFKAC-69: [FEATURE] Allow Source connector to split DCP stream and write into separate topics.

    • KAFKAC-70: [FEATURE] Allow using connector name in offset storage namespace

    • KAFKAC-77: [FEATURE] Allow setting document ID from message field.

    • KAFKAC-78: [ENHANCEMENT] Upgrade Couchbase java-client to version 2.5.1, dcp-client to version 0.12.0

    • KAFKAC-79: [ENHANCEMENT] Use custom doc transcoder to reduce memory copies.

    • KAFKAC-80: [FEATURE] MutationToken Enabled in Kafka Connector to N1QL at plus.

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.2.0</version>
</dependency>

    kafka-connect-couchbase-3.2.0.zip

    Couchbase Kafka Connector 3.1.3 GA (31 May 2017)

    Version 3.1.3 is maintenance release.

    • KAFKAC-71: Support for RBAC credentials on Couchbase Server 5+. Java DCP Client updated to 0.10.0.

    <dependency>
    <groupId>com.couchbase.client</groupId>
    <artifactId>kafka-connect-couchbase</artifactId>
    <version>3.1.3</version>
</dependency>

    kafka-connect-couchbase-3.1.3.zip

    Couchbase Kafka Connector 3.1.2 GA (14 March 2017)

    Version 3.1.2 is maintenance release.

    • KAFKAC-66: On backfilling from large bucket, it is possible to get OOM when internal queue is not drained quickly enough to relay the data into Kafka.

    kafka-connect-couchbase-3.1.2.zip

    Couchbase Kafka Connector 3.1.1 GA (21 February 2017)

    Version 3.1.1 is maintenance release. It contains fixes for resuming DCP streams after restart.

    • KAFKAC-56: Session state might be left partially initialized, which leads to rolling back to sequence number zero (0) and starting from the beginning (duplicating events in Kafka topic).

    kafka-connect-couchbase-3.1.1.zip

    Couchbase Kafka Connector 3.1.0 GA (03 January 2017)

    Version 3.1.0 is GA release.

    kafka-connect-couchbase-3.1.0.zip

    Couchbase Kafka Connector 3.0.0 GA (14 December 2016)

    Version 3.0.0 is GA release. It brings documentation update.

    kafka-connect-couchbase-3.0.0.zip

    Couchbase Kafka Connector 3.0.0 BETA (22 November 2016)

    Version 3.0.0-BETA is pre-release version of the 3.0.0. It brings documentation update, feature enhancements and bug fixes

    • KAFKAC-52: Support for SSL connections

    • Update dependencies: dcp-client to 0.7.0, and confluent libraries up to versions shipped with 3.1.1

    • Cleanup various configuration workarounds for platform 3.0

    kafka-connect-couchbase-3.0.0-BETA.zip

    Couchbase Kafka Connector 3.0.0 DP4 (5 November 2016)

    Version 3.0.0-DP4 is the fourth developer preview of the 3.0.x series.

    • KAFKAC-54: Create example of using in Kafka Stream to process events from Couchbase

    • Rename internal classes, and make configuration more consistent with other connectors (e.g. instead of timeout_ms, use timeout.ms)

    • Allow to override internal convertor into SourceRecord, and allow to inject Filter class to skip events before writing into Kafka

    kafka-connect-couchbase-3.0.0-DP4.zip

    Couchbase Kafka Connector 3.0.0 DP3 (20 October 2016)

    Version 3.0.0-DP3 is the third developer preview of the 3.0.x series. It implements new features and also includes bug fixes to previous release.

    • KAFKAC-50: Allow to buffer DCP snapshots for consistent writes.

    • KAFKAC-51: Specify key for SourceRecord. Allows to use multiple Kafka partitions.

    • KAFKAC-53: Node-aware distribution of partitions for Tasks. Reduces amount of resources allocated on the server.

    kafka-connect-couchbase-3.0.0-DP3.zip

    Couchbase Kafka Connector 3.0.0 DP2 (24 September 2016)

    Version 3.0.0-DP2 is the second developer preview of the 3.0.x series. It improves configuration. And now can maintain replication state, which allow to resume transmission.

    kafka-connect-couchbase-3.0.0-DP2.zip

    Couchbase Kafka Connector 3.0.0 DP1 (6 September 2016)

    Version 3.0.0-DP1 is the first developer preview of the 3.0.x series.

    kafka-connect-couchbase-3.0.0-DP1.zip

    Parent topic: Kafka Connector

    Previous topic: Couchbase Sample with Kafka Streams