Release Notes

KAFKAC-311: Source If the source bucket is configured to preserve history, the connector publishes every version of a document it sees within the history window, not just the latest version. See Delivery Guarantees for details. Requires Couchbase Server 7.2 or later.

KAFKAC-316: Upgraded Couchbase DCP client from 0.43.0 to 0.44.0.

KAFKAC-317: Upgraded Couchbase Java SDK from 3.4.1 to 3.4.3.

KAFKAC-303: Source The connector now includes DCP snapshot boundaries in its source offsets. This improves the reliability of resuming from saved offsets.

KAFKAC-302: Source The connector now logs more diagnostic information about lifecycle milestones.

KAFKAC-301: Upgraded Couchbase DCP client from 0.42.0 to 0.43.0.

KAFKAC-305: Upgraded Couchbase Java SDK from 3.3.4 to 3.4.1.

JDCP-232: Fixed a race condition that sometimes caused the connector to fail on startup with the message: java.lang.IllegalStateException: Tried to add duplicate channel.

The Couchbase dcp-client library no longer includes a repackaged version of Jackson. If you have written your own custom Filter, SourceHandler, or SinkHandler component that depends on the DCP client’s repackaged Jackson, search for:

import com.couchbase.client.dcp.deps.

import com.couchbase.client.core.deps.

JDCP-224: Use SASL mechanism PLAIN when authenticating with Couchbase on a secure connection. PLAIN is the fastest mechanism, and the only one that works with LDAP users.

JDCP-217: Support Couchbase clusters that advertise only TLS ports.

KAFKAC-299: Upgrade Couchbase DCP client from 0.41.0 to 0.42.0.

KAFKAC-295 Source The new couchbase.collection.to.topic config property lets you specify an arbitrary mapping from Couchbase scope & collection to Kafka topic. This is useful when the couchbase.topic property is not sufficient because the desired topic name differs from the collection name. Thanks to Shahrzad Haji Amin Shirazi.

KAFKAC-293 Sink N1qlSinkHandler now honors the configured destination scope & collection. Thanks to Shahrzad Haji Amin Shirazi.

KAFKAC-294 Sink The new experimental AnalyticsSinkHandler sends documents directly to the Analytics service. Thanks to Shahrzad Haji Amin Shirazi.

KAFKAC-296 Source In extremely low-traffic environments that ignore a majority of Couchbase events, the new couchbase.black.hole.topic config property may reduce the occurrence of rollback to zero. If a topic is specified, the connector sends a tiny record to this topic for each ignored event, to inform Kafka Connect about the event’s source offset.

KAFKAC-297: Upgraded Couchbase Java SDK from 3.3.0 to 3.3.4.

KAFKAC-298: Upgraded DCP client from 0.40.0 to 0.41.0

KAFKAC-290 When connecting to Capella, it is no longer necessary to configure the Certificate Authority certificate. All you need to do is enable TLS.

KAFKAC-288: Upgraded Couchbase Java SDK from 3.2.5 to 3.3.0.

KAFKAC-283: The connector no longer leaks JMX MBeans on shutdown. The leak was a regression in version 4.0.2.

KAFKAC-284 Source Added a new couchbase.enable.dcp.trace config option for enabling a DCP protocol trace to assist with diagnosing connector issues. Also added couchbase.dcp.trace.document.id.regex to narrow the scope of the trace.

KAFKAC-286: Upgraded Couchbase Java SDK from 3.2.4 to 3.2.5.

KAFKAC-287: Upgraded Couchbase DCP Client from 0.39.0 to 0.40.0.

KAFKAC-279: Upgraded Couchbase DCP Client from 0.37.0 to 0.39.0:

KAFKAC-281: Upgraded Couchbase Java SDK from 3.2.3 to 3.2.4.

KAFKAC-282: Upgraded other dependencies to latest versions.

KAFKAC-275: Sink Added the couchbase.retry.timeout config property. If non-zero, the connector retries write failures until the timeout expires. This is better than simply extending the KV timeout; see the documentation for details.

KAFKAC-276: Upgraded Couchbase Java SDK from 3.2.0 to 3.2.3.

KAFKAC-271: Source The default flow control buffer size is now 16 MB instead of 128 MB. This makes it less likely the source connector will run out of memory under heavy load with the default heap size. The documentation now describes how the couchbase.flow.control.buffer config property affects the Kafka Connect worker’s memory requirements.

KAFKAC-272: Source Upgraded DCP client from 0.36.0 to 0.37.0. This upgrade adds a workaround for MB-48655 so all versions of Couchbase now correctly log DCP diagnostic messages from the source connector.

KAFKAC-269: Upgraded the jsoup library from 1.13.1 to 1.14.2

KAFKAC-266: Source Upgraded DCP client from 0.35.0 to 0.36.0 for compatibility with Couchbase Server 7.0.2.

Apache Kafka 2.4.0 is now the minimum required version. For Confluent Platform users, this corresponds to Confluent version 5.4.0.

The config options for working with Couchbase 7 Scopes and Collections are now part of the "committed" API.

All other "uncommitted" config options are promoted to "committed" status as well.

KAFKAC-257: Sink The connector’s behavior is now completely customizable by implementing the SinkHandler interface. The new couchbase.sink.handler config option specifies the class to use. See the custom extensions example code to see how to implement your own handler.

KAFKAC-263: Source Upgraded DCP client from 0.34.0 to 0.35.0.

KAFKAC-258: Sink Upgraded Couchbase Java SDK from 3.1.3 to 3.2.0.

KAFKAC-261: Sink A connector configured to use SUBDOCUMENT mode with an operation of ARRAY_PREPEND or ARRAY_PREPEND could mistakenly ignore updates if Kafka records targeting the same document arrive in rapid succession. This is now fixed.

KAFKAC-262: Sink A connector configured to use N1QL mode could mistakenly ignore updates if Kafka records targeting the same document(s) arrive in rapid succession. This is now fixed.

KAFKAC-250: Sink You can now configure Couchbase Java SDK Settings in the connector config file. This includes KV timeout durations, Response Time Observability settings, and lots more.

KAFKAC-251: Source Upgraded DCP client from 0.33.0 to 0.34.0.

KAFKAC-211: Source A rollback in Couchbase no longer causes the connector to fail.

KAFKAC-234: The Couchbase root CA certificate can now be read directly from a PEM file; it’s no longer necessary to add it to a Java keystore first. The new couchbase.trust.certificate.path config property points to the PEM file.

KAFKAC-242: Source When the new couchbase.log.document.lifecycle config property is set to true, the connector writes detailed log entries as each document flows through the connector.

KAFKAC-245: Sink Upgraded Couchbase client from 3.1.2 to 3.1.3.

KAFKAC-246: Source Upgraded DCP client from 0.32.0 to 0.33.0. OBSERVE_SEQNO events are now logged at TRACE level instead of DEBUG.

KAFKAC-241: When secure connections are enabled, it is now possible to authenticate with Couchbase using an X.509 certificate instead of a username & password. See couchbase.client.certificate.path and couchbase.client.certificate.password for details. (This feature is added as "uncommitted", meaning it may change without notice.)

KAFKAC-238: Sink Upgraded Couchbase client from 3.0.9 to 3.1.2.

KAFKAC-239: Source Upgraded DCP client from 0.31.0 to 0.32.0. Previously, TLS hostname verification was done only for the Couchbase Java client connection; now the DCP client connection is verified as well. If for some reason you need to disable TLS hostname verification, this is now possible by setting the couchbase.enable.hostname.verification config property to false.

JDCP-183: If an invalid stream offset is detected, the connector will now fail fast instead of potentially corrupting the saved checkpoint.

JDCP-184: Resolved an issue that could cause a flow control deadlock when streaming from a subset of a bucket’s collections or scopes.

KAFKAC-152: Documentation - Added a "Monitoring" page which refers users to the Kafka Connect framework documentation for monitoring connectors via JMX.

KAFKAC-232: Source - Metrics specific to the Couchbase source connector are now exposed via JMX under the kafka.connect.couchbase domain.

KAFKAC-110: Source - Added a gauge that reports the Couchbase connection status.

KAFKAC-231: Source - Upgrade DCP client from 0.30.0 to 0.31.0

KAFKAC-227: Source - Added config property couchbase.xattrs (boolean, defaults to false). If set to true, a custom Filter or SourceHandler may inspect a document’s extended attributes by calling DocumentEvent.xattrs().

KAFKAC-226: Renamed the ZIP archive from couchbaseinc-kafka-connect-couchbase to couchbase-kafka-connect-couchbase (removed the "inc" from "couchbaseinc"). This made it easier to publish the connector on Confluent Hub.

KAFKAC-228: Upgraded the Couchbase Java SDK from 3.0.6 to 3.0.9. Bootstrap performance is improved when specifying custom ports, and the connector no longer logs spurious warnings about being unable to fetch collections manifests.

KAFKAC-229: Upgraded the DCP client from 0.29.0 to 0.30.0, adding support for XATTRs.

KAFKAC-225: Fixed a regression in version 4.0.0 that broke alternate address resolution. The connector now handles DNS SRV and alternate addresses correctly, and can connect to Couchbase Capella or other network environments that use alternate addresses.

KAFKAC-192: Renamed the connector config properties to follow the standard Kafka naming convention ("lowercase.with.dots.between.words"). See the Migration Guide for a comprehensive list of changes, and a shell script that can help upgrade your 3.x config files to use the new property names.

KAFKAC-157: The connector is now packaged as a Confluent Hub component. Confluent Platform users can easily install the connector using the confluent-hub install command. The Quickstart Guide has been updated to show how Apache Kafka users can install the connector.

KAFKAC-167: You can now specify custom ports in the list of Couchbase seed nodes. If you specify a port, it must be the port of the Key/Value service (which defaults to 11210 for unencrypted connections).

KAFKAC-207: Sink - You can now specify an "enhanced durability" requirement with the new couchbase.durability config property. Enhanced durability requires Couchbase Server 6.5 or later.

KAFKAC-197: Sink - Added config property couchbase.topic.to.collection which maps Kafka topics to Couchbase collections. Added config property couchbase.default.collection which is used when a message’s topic is not present in the map.

KAFKAC-209: Source - The connector now evenly distributes the workload among all tasks, instead of trying to minimize the total number of Couchbase connections.

KAFKAC-177: Source - The example config files now use RawJsonSourceHandler and publish Kafka messages whose contents are the same as the Couchbase documents.

KAFKAC-212: Source - If you don’t care about the content of the Couchbase document, you can set the new couchbase.no.value config property to true, and the connector will omit the document content from Kafka messages.

KAFKAC-194: Source - A custom SourceHandler can now set headers on the Kafka record.

KAFKAC-223: Source - The connector is now more responsive to "pause" requests.

KAFKAC-220: Source - Custom Filter and SourceHandler components can now access connector configuration properties. These interfaces now have an init(Map<String, String>) method. The connector config is passed to this method when the component is created.

KAFKAC-222: The custom-extensions example project now includes a CustomFilter class that demonstrates how to read properties from the connector config. This example filter accepts or rejects documents based on a field of the document. The target field and the list of acceptable values are both configurable.

KAFKAC-196: Source - Added couchbase.scope and couchbase.collection config properties that let you stream from a specific scope or set of collections.

KAFKAC-195: Source - Modified the couchbase.topic config property to be a format string that supports ${bucket}, ${scope}, and ${collection} placeholders. This makes it easy to publish to different Kafka topics depending on the Couchbase document’s parent collection. The default value is ${bucket}.${scope}.${collection}.

KAFKAC-171: The couchbase.password config property (previously called connection.password) no longer defaults to an empty string.

KAFKAC-175: APIs deprecated in version 3.x have been removed.

Upgraded Kafka Connect API from 1.0.2 to 2.5.0.

Upgraded Couchbase client from 2.7.13 to 3.0.6.

Upgraded DCP client from 0.26.0 to 0.29.0.

KAFKAC-169: Sink - If two Kafka messages with the same key arrive in rapid succession, it’s no longer theoretically possible for them to be written to Couchbase in the wrong order.

KAFKAC-220: Source - Custom Filter and SourceHandler components can now access connector configuration properties. These interfaces now have an init(Map<String, String>) method. The connector config is passed to this method when the component is created.

KAFKAC-222: The custom-extensions example project now includes a CustomFilter class that demonstrates how to read properties from the connector config. This example filter accepts or rejects documents based on a field of the document. The target field and the list of acceptable values are both configurable.

KAFKAC-197: Sink - Added config property couchbase.topic.to.collection which maps Kafka topics to Couchbase collections. Added config property couchbase.default.collection which is used when a message’s topic is not present in the map.

KAFKAC-207: Sink - You can now specify an "enhanced durability" requirement with the new couchbase.durability config property. Enhanced durability requires Couchbase Server 6.5 or later.

KAFKAC-206: Source - Config property couchbase.connector.name.in.offsets now defaults to false again. This property doesn’t do anything useful, and should only be set to true if you previously had compat.connector_name_in_offsets set to true.

KAFKAC-177: Source - The example config files now use RawJsonSourceHandler and publish Kafka messages whose contents are the same as the Couchbase documents.

KAFKAC-209: Source - The connector now evenly distributes the workload among all tasks, instead of trying to minimize the total number of Couchbase connections.

KAFKAC-212: Source - If you don’t care about the content of the Couchbase document, you can set the new couchbase.no.value config property to true, and the connector will omit the document content from Kafka messages.

KAFKAC-205: Removed the unused couchbase.force.ipv4 config property.

KAFKAC-169: Sink - If two Kafka messages with the same key arrive in rapid succession, it’s no longer theoretically possible for them to be written to Couchbase in the wrong order.

KAFKAC-214: Sink - The Couchbase Java SDK has been updated from 3.0.5 to 3.0.6. As a result, setting couchbase.document.expiration to longer than 30 days now works correctly instead of causing immediate expiration. (This was a regression in 4.0.0-dp.1.)

KAFKAC-203: Source - The 3.x → 4.0 migration script now properly converts the old couchbase.flow_control_buffer property to the new name: couchbase.flow.control.buffer.size.

KAFKAC-204: Source - Fixed the invalid value for couchbase.bootstrap.timeout in the quickstart-couchbase-source.json example config file.

KAFKAC-182: Upgraded Kafka Connect API from 1.0.2 to 2.5.0.

KAFKAC-188: Upgraded Couchbase client from 2.7.13 to 3.0.5.

KAFKAC-189: Upgraded DCP client from 0.26.0 to 0.28.0.

KAFKAC-192: Renamed the connector config properties to follow the standard Kafka naming convention ("lowercase.with.dots.between.words"). See the Migration Guide for a comprehensive list of changes, and a shell script that can help upgrade your 3.x config files to use the new property names.

KAFKAC-196: Source: Added couchbase.scope and couchbase.collection config properties that let you stream from a specific scope or set of collections.

KAFKAC-195: Source: Modified the couchbase.topic config property to be a format string that supports ${bucket}, ${scope}, and ${collection} placeholders. This makes it easy to publish to different Kafka topics depending on the Couchbase document’s parent collection. The default value is ${bucket}.${scope}.${collection}.

KAFKAC-194: Source: A custom SourceHandler can now set headers on the Kafka record.

KAFKAC-157: The connector is now packaged as a Confluent Hub component. Confluent Platform users can easily install the connector using the confluent-hub install command. The Quickstart Guide has been updated to show how Apache Kafka users can install the connector.

KAFKAC-167: You can now specify custom ports in the list of Couchbase seed nodes. If you specify a port, it must be the port of the Key/Value service (which defaults to 11210 for unencrypted connections).

KAFKAC-171: The couchbase.password config property (previously called connection.password) no longer defaults to an empty string.

KAFKAC-173: The couchbase.connector.name.in.offsets config property (previously called compat.connector_name_in_offsets) now defaults to true.

KAFKAC-175: APIs deprecated in version 3.x have been removed.