v0.11.5

v0.11.5 is a feature release that adds support for the Kafka Admin API (KIP-4).

Admin API

This release adds support for the Admin API, enabling applications and users to perform administrative Kafka tasks programmatically:

• Create topics - specifying partition count, replication factor and topic configuration.
• Delete topics - delete topics in cluster.
• Create partitions - extend a topic with additional partitions.
• Alter configuration - set, modify or delete configuration for any Kafka resource (topic, broker, ..).
• Describe configuration - view configuration for any Kafka resource.

The API closely follows the Java Admin API:

https://github.com/edenhill/librdkafka/blob/master/src/rdkafka.h#L4495

New and updated configuration

• Added compresion.level configuration option, which allows fine-tuning of gzip and LZ4 comression level (Emmanuel Raulo-Kumagai (@erkoln))
• Implement ssl.curves.list and ssl.sigalgs.list configuration settings (Valentín Gutiérrez (@jvgutierrez))
• Changed queue.buffering.backpressure.threshold default (#1848)

Enhancements

• Callback based event notifications (Federico Giraud (@fede1024))
• Event callbacks may now optionally be triggered from a dedicated librdkafka background thread, see rd_kafka_conf_set_background_event_cb.
• Log the value that couldn't be found for flag configuration options (Andreas Kohn (@ankon))
• Add support for rd_kafka_conf_set_events(conf, ..EVENT_ERROR) to allow generic errors to be retrieved as events.
• Avoid allocating BIOs and copying for base64 processing (Adam Langley (@agl))
• Don't log connection close for idle connections (regardless of log.connection.close)
• Improve latency by using high-precision QPC clock on Windows
• Added make uninstall
• Minor documentation updates from replies to #1794 (Emmanuel Briot (@briot))
• Added rd_kafka_controllerid() to return the current controller.
• INTRODUCTION.md: add chapter on latency measurement.
• Add relative hyperlinks to table of contents in INTRODUCTION.md (#1791, Stanislav Kozlovski (@stanislavkozlovski))
• Improved statistics:
  • Added Hdr Histograms for all windowed stats (rtt, int_latency, throttle) #1798
  • Added top-level totals for broker receive and transmit metrics.
  • Added batchcnt, batchsize histograms to consumer.
  • Added outbuf_latency histograms.
  • STATISTICS.md moved from wiki to source tree.

Fixes

• Fixed murmur2 partitioner to make it compatible with java version (#1816, Shuai Lin (@lins05))
• Fix pause/resume: next_offset was not properly initialized
• Fix a segment fault error in rdkafka_buf with zero-length string (sunny1988)
• Set error string length in rkmessage.len on error (#1851)
• Don't let metadata ERR_UNKNOWN set topic state to non-existent.
• The app_offset metric is now reset to INVALID when the fetcher is stopped.
• consumer_lag is now calculated as consumer_lag = hi_wmark_offset - MAX(app_offset, committed_offset), which makes it correct after a reassignment but before new messages have been consumed (#1878)
• socket.nagle.disable=true was never applied on non-Windows platforms (#1838)
• Update interface compile definitions for Windows using CMake (#1800, Raul Bocanegra Algarra (@raulbocanegra))
• Fix queue hang when queue_destroy() is called on rdkafka-owned queue (#1792)
• Fix hang on unclean termination when there are outstanding requests.
• Metadata: fix crash when topic is in transitionary state
• Metadata: sort topic partition list
• rdkafka_example emitted bogus produce errors
• Increase BROKERS_MAX to 10K and PARTITIONS_MAX to 100K
• Proper log message on SSL connection close
• Missing return on error causes use-after-free in SASL code (Huzaifa Sidhpurwala (@sidhpurwala-huzaifa))
• Fix configure --pkg-config-path=... (#1797, xbolshe)
• Fix -fsanitize=undefined warning for overflowed OP switches (#1789)

v0.11.4

Maintenance release

Default changes

• socket.max.fails changed to 1 to provide same functionality (fail request immediately on error) now when retries are working properly again.
• fetch.max.bytes (new config property) is automatically adjusted to be >= message.max.bytes, and receive.message.max.bytes is automatically adjusted to be > fetch.max.bytes. (#1616)

New features

• Message Headers support (with help from John Dennison (@johnistan))
• Java-compatible Murmur2 partitioners (#1468, barrotsteindev)
• Add PKCS#12 Keystore support - ssl.keystore.location (#1494, AMHIT)

Noteworthy fixes

• Formalise and fix Producer retries and retry-ordering (#623, #1092, #1432, #1476, #1421)
  • Ordering is now retained despite retries if max.in.flight=1.
  • Behaviour is now documented
• Fix timeouts for retried requests and properly handle retries for all request types (#1497)
• Add and use fetch.max.bytes to limit total Fetch response size (KIP-74, #1616). Fixes "Invalid response size" issues.

Enhancements

• Added sasl.mechanism and compression.type configuration property aliases for conformance with Java client.
• Improved Producer performance
• C++: add c_ptr() to Handle,Topic,Message classes to expose underlying librdkafka object
• Honour per-message partition in produce_batch() if MSG_F_PARTITION set (barrotsteindev, closes #1604)
• Added on_request_sent() interceptor
• Added experimental flexible producer queuing.strategy=fifo|lifo
• Broker address DNS record round-robin: try to maintain round-robin position across resolve calls.
• Set system thread name for internal librdkafka threads (Trevor Saunders (@tbsaunde))
• Added more concise and user-friendly 'consumer' debug context
• Add partitioner (string) topic configuration property to set the builtin partitioners
• Generate rdkafka-static.pc (pkg-config) for static linking

Fixes

• Fix producer memory leak on <0.11 brokers when compressed messageset is below copy threshold (closes #1534)
• CRC32C - fix unaligned access on ARM (Soundman32)
• Fix read after free in buf_write_seek
• Fix broker wake up (#1667, Guillaume Duranceau (@gduranceau))
• Fix consumer hang when rebalancing during commit (closes #1605, ChenyuanHu (@ChenyuanHu))
• CMake fixes for Windows (Raul Bocanegra Algarra (@raulbocanegra))
• LeaveGroup was not sent on close when doing final offset commits
• Fix for consumer slowdown/stall on compacted topics where actual last offset < MsgSet.LastOffset (KAFKA-5443)
• Fix global->topic conf fallthru in C++ API
• Fix infinite loop on LeaveGroup failure
• Fix possible crash on OffsetFetch retry
• Incorporate compressed message count when deciding on fetch backoff (#1623)
• Fix debug-only crash on Solaris (%s NULL) (closes #1423)
• Drain broker ops queue on termination to avoid hang (closes #1596)
• cmake: Allow build static library (#1602, proller)
• Don't store invalid offset as next one when pausing (#1453, Matias Fontanini (@mfontanini))
• use #if instead of #ifdef / defined() for atomics (#1592, Marek Vavruša (@vavrusa))
• fixed .lib paths in nuget packaging (#1587)
• Fixes strerror_r crash on alpine (#1580, svante karlsson (@skarlsson))
• Allow arbitrary lengthed (>255) SASL PLAIN user/pass (#1691, #1692)
• Trigger ApiVersionRequest on reconnect if broker.version.fallback supports it (closes #1694)
• Read Fetch MsgAttributes as int8 (discovered by Taras Voinarovskyi (@tvoinarovskyi), closes #1689)
• Portability: stop using typeof in rdkafka_transport.c (#1708, Trevor Saunders (@tbsaunde))
• Portability: replace use of #pragma once with header guards (#1688, Trevor Saunders (@tbsaunde))
• mklove: add LIBS in reverse order to maintain dependency order
• Fix build when python is not available #1358

v0.11.3

Maintenance release

Default changes

• Change default queue.buffering.max.kbytes and queued.max.message.kbytes to 1GB (#1304)
• win32: Use sasl.kerberos.service.name for broker principal, not sasl.kerberos.principal (#1502)

Enhancements

• Default producer message offsets to OFFSET_INVALID rather than 0
• new nuget package layout + debian9 librdkafka build (#1513, Matt Howlett (@mhowlett))
• Allow for calling rd_kafka_queue_io_event_enable() from the C++ world (#1483, Akhilesh Singhania (@akhi3030))
• rdkafka_performance: allow testing latency with different size messages (#1482, Trevor Saunders (@tbsaunde))

Fixes

• Improved stability on termination (internal queues, ERR__DESTROY event)
• offsets_for_times() return ERR__TIMED_OUT if brokers did not respond in time
• Let list_groups() return ERR__PARTIAL with a partial group list (#1508)
• Properly handle infinite (-1) rd_timeout:s throughout the code (#1539)
• Fix offsets_store() return value when at least one valid partition
• portability: rdendian: add le64toh() alias for older glibc (#1463)
• Add MIPS build and fix CRC32 to work on big endian CPUs (Andreas Smas (@andoma), closes #1498)
• osx: fix endian checking for software crc32c
• Fix comparison in rd_list_remove_cmp (closes #1493)
• stop calling cnd_timedwait() with a timeout of 0h (#1481, Trevor Saunders (@tbsaunde))
• Fix DNS cache logic broker.address.ttl (#1491, Daniel Woodlins (@dacjames))
• Fix broker thread "hang" in CONNECT state (#1397)
• Reset rkb_blocking_max_ms on broker DOWN to avoid busy-loop during CONNECT (#1397)
• Fix memory leak when producev() fails (#1478)
• Raise cmake minimum version to 3.2 (#1460)
• Do not assume LZ4 worst (best?) case 255x compression (#1446 by Tudor Bosman (@tudor))
• Fix ALL_BROKERS_DOWN re-generation (fix by ciprianpascu, #1101)
• rdkafka-performance: busy wait to wait short periods of time

v0.11.1

Maintenance release

NOTE: If you are experiencing lousy producer performance, try setting the linger.ms configuration property to 100 (ms).

Noteworthy critical bug fixes

• Fix OpenSSL instability on Windows (fix thread id callback) - the bug resulted in SSL connections being torn down for no apparent reason.
• Fetch response fix: read all MessageSets, not just the first one. (#1384) - Huge performance degradation (compared to v0.9.5) when fetching small batches.

Enhancements

• Add api.version.request.timeout.ms (#1277, thanks to @vinipuh5)
• Point users to documentation when attempting to use Java security properties (#1412)

Fixes

• Adjust log level for partial message reads when debug is enabled (#1433)
• Allow app metadata() requests regardless of outstanding metadata requests (#1430)
• Proper size calculation from flags2str to include nul (#1334)
• Thread-safe rd_strerror() (#1410)
• Proper re-sends of ProduceRequests in-transit on connection down (#1390)
• Producer: invalid offsets reported back when produce.offset.report=false
• Consumer: Message Null Key/Value were messed up (regression in v0.11.0, #1386)
• Fix metadata querying for topics with LEADER_UNAVAIL set (#1313)
• Treat request __TIMED_OUT as a retryable error
• Let timed out ProduceRequests result in MSG_TIMED_OUT error code for messages
• Fix crash on leader rejoin when outstanding assignor metadata (#1371)
• sasl_cyrus: Fix dangling stack ptr to sasl_callback_t (#1329, thanks to Gene Novark (@gnpdt) )

v0.11.0

Feature release

v0.11.0 is a new feature release of librdkafka with support for the new Kafka message format (MsgVersion 2) which makes librdkafka (and any librdkafka-based clients) transparently compatible for use with the EOS (Exactly-Once-Semantics) supporting Java client released with Apache Kafka v0.11.0.

This release also includes enhancements and fixes as listed below.

NOTE: While librdkafka implements the new Message version and features, it does not yet implement the EOS (Exactly-Once-Semantics) functionality itself.

NOTE: The librdkafka C++ API is unfortunately not ABI safe (the API stability is guaranteed though): C++ users will need to recompile their applications when upgrading librdkafka.

Upgrade notes

api.version.request:
The api.version.request property (see https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility) default value has changed from false to true, meaning that librdkafka will make use of the latest protocol features of the broker without the need to set the property to true explicitly on the client.

WARNING: Due to a bug in Apache Kafka 0.9.0.x, the ApiVersionRequest (as sent by the client when connecting to the broker) will be silently ignored by the broker causing the request to time out after 10 seconds. This causes client-broker connections to stall for 10 seconds during connection-setup before librdkafka falls back on the broker.version.fallback protocol features. The workaround is to explicitly configure api.version.request to false on clients communicating with &lt=0.9.0.x brokers.

Producer:
The default value of queue.buffering.max.ms was changed from 1000ms to 0ms (no delay). This property denotes the internal buffering time (and latency) for messages produced.

Features

• Added support for MsgVersion v2 (message format of KIP-98) - message format compatible with EOS clients
• Added support for client interceptors
• Added support for dynamically loaded plugins (plugin.library.paths, for use with interceptors)
• Added SASL SCRAM support (KIP-84)
• Added builtin SASL PLAIN provider (for Win32, #982)

Enhancements

• Deprecate errno usage, use rd_kafka_last_error() instead.
• Deprecate rd_kafka_wait_destroyed().
• Implemented per-partition Fetch backoffs, previously all partitions for the given broker were backed off.
• Added updated Kafka protocol and error enums
• Added rd_kafka_message_latency()
• Added rd_kafka_clusterid() and rd_kafka_type()
• SSL: set default CA verify locations if ssl.ca.location is not specified
• C++: add yield() method
• Added support for stats as events (#1171)
• Build with system liblz4 if available, else fall back on built-in lz4, for improved portability.
• Use SNI when connecting through SSL (Vincent Bernat (@vincentbernat))
• Improve broker thread responsiveness, decreasing internal latency
• Improve OpenSSL config error propagation (#1119)
• Prioritize all relevant user-facing ops (callbacks) over messages on poll queue (#1088)
• Added global->topic config fallthru: default topic config properties
  can now be set effortlessly on global config object.
• Log offset commit failures when there is no offset_commit_cb (closes #1043)
• Add CRC checking support to consumer (#1056)
• C++: Added seek() support to KafkaConsumer
• Added rd_kafka_conf_dup_filter() to selectively copy a config object.

Fixes:

• Avoid _ALIGN re-definition on BSD (#1225)
• rdkafka_performance: exit with code 1 if not all messages were delivered
• Fix endianism issues that were causing snappy to compress incorrectly (#1219, Bob Halley (@rthalley))
• Fix stability on AIX (#1211)
• Document that rd_kafka_message_errstr() must not be used on producer
• Add support for re-queuing half-processed ops to honour yield()
• Handle null Protocol in JoinGroupResponse (#1193)
• Consumer: Proper relative offset handling (#1192, Bob Halley (@rthalley))
• OSX: silence libsasl deprecated warnings
• partition count should be per topic in offset request buffer (closes #1199, Matthew D Jones (@matthew-d-jones))
• fix build on SmartOS (#1186 by Julien Gilli (@misterdjules))
• ERR_remove_thread_state OpenSSL version checking
• Don't emit TIMED_OUT_QUEUE for timed out messages (revert)
• producev() default partition should UA, not 0 (#1153)
• Fix SaslHandshakeRequest timeout to 10s
• SASL: fix memory leak: received SASL auth frames were not freed
• Clean up OpenSSL per-thread memory on broker thread exit
• Properly auto-set metadata.max.age.ms when metadata.refresh.interval.ms is disabled (closes #1149)
• Fix memory alignment issues (#1150)
• configure: auto add brew openssl pkg-config path
• Fix consumer_lag calculation (don't use cached hi_offset)
• rdkafka_example: fix message_errstr usage, not allowed on producer
• Avoid use of partially destroyed topic object (#1125)
• Improve reconnect delay handling (#1089)
• C++: fix conf->get() allocation (closes #1118)
• Use app_offset to calculate consumer_lag (closes #1112)
• Fix retrybuf memory leak on termination when broker is down
• Fix small memory leak in metadata_leader_query
• Fix use-after-free when log.queue and debug was used
• consumer_example: fix crash on -X dump (closes #841)
• Added rd_kafka_offsets_store() (KafkaConsumer::offsets_store) (closes #826)
• Optimize broker id lookups (closes #523)
• Don't log broker failures when an error_cb is registered (closes #1055)
• Properly log SSL connection close (closes #1081)
• Win32 SASL GSSAPI: protection level and message size were not sent
• C++: improved error reporting from Conf::set()
• Flush partition fetch buffer on seek (from decide())
• Fix recursive locking on periodic refresh of leader-less partition (closes #1311)

v0.9.5

Maintenance release

Critical fixes

• q_concat: don't wakeup listeners if srcq is empty (fix by Martijn Veldpaus (@orthrus) in #1121): Fixes idle consumer CPU usage on Windows
• Prioritize commit_cb over messages on poll queue (closes #1088)
• Prioritize all relevant user-facing ops (callbacks) (#1088)
• Fix SaslHandshakeRequest timeout

Fixes

• Properly log SSL connection close (closes #1081)
• Don't log broker failures when an error_cb is registered (closes #1055)
• Fix use-after-free when log.queue and debug was used
• Log offset commit failures when there is no offset_commit_cb (closes #1043)
• Fix retrybuf memory leak on termination when broker is down
• Fix small memory leak in metadata_leader_query
• Use app_offset to calculate consumer_lag (closes #1112)
• Fix consumer_lag calculation (dont use cached hi_offset)
• C++: fix conf->get() allocation (closes #1118)
• Avoid use of partially destroyed topic object (#1125, fixed by benli123)
• sasl win32: protection level and message size not sent (fixed by Kevin Mullins (@zyzil) )
• Properly auto-set metadata.max.age.ms when metadata.refresh.interval.ms is disabled (closes #1149)
• Fix memory alignment issues (#1150)
• consumer_example: fix crash on -X dump (closes #841)
• configure: auto add brew openssl pkg-config path
• producev() default partition should be UA, not 0 (#1153)

Enhancements

• Added global->topic config fallthru: Topic-level configuration properties can now be set on the global configuration object. The property will be applied on the default_topic_conf object (if no such object exists one is created automatically).
• Use SNI when connecting through SSL (by Vincent Bernat (@vincentbernat) )
• Windows performance improvements (use atomics instead of locks, avoid locking in some cases)
• Add lz4/lib sources for in-tree building when external lz4 is not available
• Add CRC checking support to consumer (#1056)
• Added op priority to queues (for #1088)
• Fail known unsupported requests locally (closes #1091)
• Improve reconnect delay handling (#1089)
• Improve OpenSSL config error propagation (#1119)
• C++: improved error reporting from Conf::set()
• Don't emit TIMED_OUT_QUEUE (revert)

v0.9.4

librdkafka v0.9.4 is a small feature release with a horde of enhancements and fixes.

NOTE: The librdkafka C++ API is unfortunately not ABI safe (the API stability is guaranteed though): C++ users will need to recompile their applications when upgrading librdkafka.

New features

• SASL: Windows SASL GSSAPI support using native Windows SSPI (#888)
• Added rd_kafka_offsets_for_times (KIP-79, #842)
• Added rd_kafka_producev() using va-args to construct message fields, including timestamps (#858, #707, #908, #345)
• C++: added produce(..., timestamp, ..)
• Added support for partition-specific queues (Turing Eret (@lambdaknight))
  • allows redirecting partitions to separate queues and separates application threads.
• True low-latency (on linux and osx) by fd-based queue wakeups

Critical fixes

• leader failover error detection and recovery for both producer and consumer
• cgrp: failover robustness improvements
• cgrp: certain failed Heartbeats could mask out future heartbeat requests
• buffer handle callbacks could previously be triggered on wrong thread

Enhancements

• Events: expose EVENT_OFFSET_COMMIT
• Experimental cmake support (thanks to Ruslan Baratov (@ruslo))
• Validate topics in subscribe() (mainly regex validation)
• Added set_log_queue() - allows application thread log handling (#355)
• Add support for Metadata v1..2 (part of KIP-4)
• New metadata request and caching framework:
  • much fewer metadata requests
  • use cached metadata when deemed possible
  • KIP-4 support for lean metadata requests in clusters with many topics
  • improved (faster, more reliable) leader lookups
• C++: added TopicPartition::create(, ..offset)
• C++: added const attribute to TopicPartition getters
• C++: added TopicPartition::destroy(vector) helper
• C++: added commitSync(.., OffsetCommitCb) variant
• C++: Enrichment of Conf getter for callbacks. (#883, vin-d)
• Expose rd_kafka_topic_partition_list_sort()
• Added max.in.flight -> max.in.flight.requests.per.connection alias
• Win32: add lz4 support (using NuGet lz4 package)
• Add cgrp stats: rebalance_age, rebalance_cnt, assignment_size

Fixes

• increase maximum supported topic limit to 1M
• fix static linking on osx
• Fix infinte wait in .._commit_queue() if callback on temp queue was used
• Improved handling of assign() and outstanding rebalance_cb on termination
• commit_queue: avoid global offset_commit_cb if local specified
• Imrpoved Heartbeat error handling
• Improve OffsetCommit failure handling
• Query topics with missing or down leaders from topics_scan()
• Handle rebalance from consumer_close (#1002)
• rd_kafka_producev() would send 0 timestamp by default instead of current time
• queue.buffering.max.ms and fetch.error.backoff.ms now takes precedence over socket.blocking.max.ms (#966)
• flush() fix: msg_cnt not initialized
• rdkafka_performance: fix multi -p partitions allocation error and memory leak
• Refactor query_watermark_offsets to use new leader lookup code
• Proper error propagation on invalid topic names (#977)
• callbacks will no longer cut consumer_poll() timeouts short
• improved queue handling and serving
• rd_kafka_producev(): fix topic/rkt referencing
• Fix ApiVersionRequest timeout (10s)
• rd_kafka_q_concat: fix erroneous unlock() on enq failure
• Speed up termination by emitting TERMINATE ops
• cgrp: enforce metadata update check if matched subscribed topics is empty
• Proper suppression of connect failure logs (#847)
• Proper ATOMIC_OPn (#708)
• fix: use out of scope variable (#974, HeChuan (@HeChuanXUPT))
• C++: Fix undefined references to static const ints (#961, #962, Filipp (@bbfgelman1))
• mklove: lib_check now attempts compilation after pkg-config passed (for false positive of SSL on OSX)
• mklove: fix ./configure --reconfigure
• stats: fix brokers.toppars key duplication

v0.9.3

Maintenance release

Enhancements

• Increase .._TOPICS_MAX to 1M
• Add cgrp stats: rebalance_age, rebalance_cnt, assignment_size
• win32: allow the use of LIBRDKAFKA_STATICLIB to enable static library (#905, svante karlsson (@skarlsson))
• C++: Add support for setting "consume_cb" in Conf with a RdKafka::ConsumeCb and proper handling of that callback object by the underlying C library. (#741, Turing Eret (@lambdaknight))
• Added socket.nagle.disable to disable the Nagle algorithm (#862)
• Added rd_kafka_topic_partition_list_sum()
• Added connect_cb and closesocket_cb to conf

Fixes

• Fix relative offset calculation (#859, Yunjing Xu (@yunjing))
• cgrp: dont get stuck in wait-coord state on reoccuring query errors (issue #844)
• cgrp: fix a number of assign() problems (issues #871, #844, possibly #730)
• cgrp: increased robustness for wait_commit et.al (#785)
• cgrp: time out ops in wait_coord queue (#785)
• cgrp: do async termination by state to avoid recursive locks from timeout_scan
• Dont store initial offset in fetch_start0 (#742)
• Win32 compilation and linking fixes
• queue.buffering.max.kbytes wrapped on platforms with 32-bit size_t (#998)
• Timeout was not enforced in flush()
• librdkafka++: just link C library, not also its dependencies
• rdkafka_performance: fix multi -p partitions allocation error and memory leak
• Fix ApiVersionRequest timeout (10s)
• rd_kafka_q_concat: fix erroneous unlock() on enq failure
• C++: Added TopicPartition::create(, ..offset)
• C++: Fix undefined references to static const ints (#961, #962, Filipp (@bbfgelman1))
• Dont start SASL for the internal broker (#968)
• Proper suppression of connect failure logs (#847)
• Proper ATOMIC_OPn (#708)
• fix: use out of scope variable (#974, HeChuan (@HeChuanXUPT))
• INTRODUCTION: fix wrong batch.num.messages default value (#924)
• configure: check for library headers too (fixes OpenSSL failures on OSX)
• mklove: lib_check now attempts compilation after pkg-config passed (for false positive of SSL on OSX)
• mklove: fix ./configure --reconfigure
• stats: fix brokers.toppars key duplication
• consumer: Correctly set timestamp for compressed messages (#858)
• Dont attempt SASL auth with broker-unsupported mechs (issue #907)
• consumer_poll() without a valid group.id would return a broken message_t
• Fix duplicate desired_link calls (crash or memory leak)
• Fix for async offset commit issue on AIX (#852, asharma339)
• _builtin.. replacements for native toolchains on sun and aix (#865, asharma339)
• Add snappy function prefix to avoid symbol collision with google snappy (#880, Li Jian (@hustlijian))
• toppars: Dont mix up UNKNOWN / DESIRED flags
• Use compiler specific atomic ops for sun studio compiler (#869, asharma339)
• fix bug rd_kafka_timer_t microsecond interval overflow. (#870, peter (@lvxv))
• Set HeartbeatRequest timeout to session.timeout.ms
• Allow max one HeartbeatRequest in transit
• DescribeGroups: Avoid zero-byte allocations (for AIX)
• fix func rd_kafka_new declare use_conf but not use cause conf->socket_cb segmentfault (#863) (wadee_liang (@wadee))

0.9.2

Additions

• Event interface (see rd_kafka_event_type_t)
• VerifiableConsumer and VerifiableProducer clients for running official kafkatests
• Added support for LZ4 compression (KIP-57)
• Added rd_kafka_commit_queue()
• Added rd_kafka_queue_length()
• Added rd_kafka_flush()
• Added IO FD-based queue triggering for including librdkafka in IO-based poll loops (issue #749)
• Added enable.partition.eof to allow turning off PARTITION_EOF
• auto.offset.reset: added aliases: beginning and end
• Added experimental blocking produce call (.._F_BLOCK flag to produce())

Enhancements

• rd_kafka_version_str(): return a git-aware version string
• Retry metadata requests if returned response is empty
• Expedite MetadataRequests in front of other requests
• Improved ...metadata() selection of broker to query
• Added ERR__TIMED_OUT_QUEUE to differentiate between in-transit and in-queue timeouts
• rdkafka_performance: added -N to disable delivery reports
• rdkafka_performance: added -G for high-level KafkaConsumer
• Regex support on Win32: added fallback regex implementation (issue #552)
• Producer: Allow unlimited number of messages per batch/MessageSet (bump batch.num.messages to 10000)
• Producer: Copy small message payloads to transmit buffer (issue #788)
• .._conf_set(): revert to default value (if any) when passed value==NULL.
• Redesigned toppar op version barriers to be more robust
• Automatically enable api.version.request for broker.version.fallback>=0.10
• Expedite termination by broadcasting state change on destroy
• Improve ALL_BROKERS_DOWN event generation
• Fall back to mutex-protected variables when 32 or 64 bit atomics are not available (#693)
• Disable SSLv3

Consumer fixes

• Fix crash in committed() with short timeouts (issue #827)
• Dont attempt offset commits for empty non-null list
• cgrp: clear stored member_id on UNKNOWN_MEMBER_ID errors
• handle_JoinGroup: dont propagate UnknownMemberId to app
• cgrp term: force final unassign call even if waiting for rebalance
• Dont send Heartbeats when calling rebalance_cb(REVOKE)
• cgrp: ignore outdated Join & Sync responses when state has transitioned
• Dont propagate error for empty commits when internal commits are retried
• Stop offset_query_tmr when stopping fetcher
• OffsetCommit&OffsetFetch request: properly coalesce partitions for the same topic (issue #785)
• C++: added Consumer.seek() (#699)
• Fix lock ordering issue in fetcher
• Update subscription on topic changes (issues #802, #644, #576, #461)
• Fixed a number of consumer (group) corner-cases
• Resume GroupCoord queries on all errors (#768)
• Error out on unknown partition.assignment.strategy:s
• Maintain cgrp version barrier for relevant internal request ops (fixes #761)
• Dynamically increase the max fetch size when needed (#597, by takano-akio)
• Proper timeout calculation for consume(), consumer_poll(), etc
• Properly handle empty assignments in SyncGroupResponse (#603)
• Fixed a number of hang-on-termination issues
• Consumers could stall in offset-wait state on leader change for up to socket.timeout.ms * retries (#843)

Producer fixes

• Send uncompressed if compression messageset is larger than uncompressed (issue #781)
• Proper ProducerResponse error checking before retries (issue #777)
• Time out messages on toppar xmit msgq too (fixes #776, #672)
• Dont create cgrp for producers if group.id is configured (issue #691)
• Fixed a number of hang-on-termination issues

General fixes

• Bring down the connection regardless of socket.max.fails if in-flight requests time out (#756)
• Count partially sent buffers as in-flight when timeout scanning (issue #756)
• Dont insert requests in front of partly sent SSL requests
• Reset partially sent buffer's offsets to 0 on disconnect (issue #756, #792)
• Dont prepend FLASH messages before partially sent buffers (possibly #756)
• Time out broker retry bufs (issue #776)
• Count topic_t app refcounts separately (issues #345, #821)
• Clean up OpenSSL global locks when all librdkafka instances are gone (issue #815)
• Fixed SSL broker_fail() arguments (issue #796)
• Improved purging of partition op & message queues to speed up termination
• Proper thread creation error propagation (issue #712)
• Forwarded queue's content was not properly moved to dest queue on fwd_set()
• Avoid endless loop when reconnect.backoff.jitter.ms is low (#779)
• Fix double free of topic.blacklist (issue #778)
• Fix SASL PLAIN auth (broken in 0.9.1)
• rdkafka_performance: could exit before all msgs sent when host was slow
• query_watermark_offsets() workaround for KAFKA-1588 (issue #700)
• Proper shrinking of rktparlist (issue #739 reported by g4bino (@diazdiego86))
• query_watermark: Trigger leader query if leader not known (closes #694, fix by Johan (@JohanAR))
• Broadcast broker state changes (et.al) to expedite waiter wakeups (closes #695)
• Properly handle empty sasl.kerberos.kinit.cmd (seen in #688)
• rd_kafka_new: delete conf only on success (#836 by Julien Lecomte (@julien-lecomte))

0.9.1

New major features

• Broker-based balanced High-Level KafkaConsumer (broker version >= 0.9)
• SASL support (client authentication) - requires Cyrus libsasl
• SSL support (client authentication, transport encryption) - requires OpenSSL
• Native Win32 support (Visual Studio)
• Broker-based quota support (KIP-13)
• Relative offsets support (KIP-31)
• Message timestamp support (KIP-32)

New enhancements

• Broker API version query support for brokers >=0.10.0 with fallbacks for older brokers. (KIP-35)
  • api.version.request - enable/disable ApiVersionRequests
  • api.version.fallback.ms - how long to use the fallback version after ApiVersionRequest failure
  • broker.version.fallback - fallback version to adapt API usage to.
  • More info: https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility
• Added broker connection max in-flight enforcement: max.in.flight.requests.per.connection
• Added config aliases "retries", "acks" (#157)
• request.required.acks/acks: added support for "all" (-1)
• Flag config properties now support explicit +addx,-dely and "all"
• Added rd_kafka_last_error() alternative to errno for legacy APIs (#571)
• Added rd_kafka_get_err_descs() to retrieve full list of error codes.
• Added rd_kafka_partitions_pause/_resume()
  • Consumer: flush local fetchq for partition(s) and stop fetching
  • Producer: dont send any ProduceRequests to broker for partition(s)
    This is also used when a rebalance_cb is registered between the time of
    rebalancing and until the application has called assign() to synchronized state.
    Prevents app from receiving messages after a rebalance but before assign()
• Keep track of app_offset: last message+1 provided to application (issue #106). This is the "next offset" for the application.
• Added log.connection.close to turn off logging of broker-triggered disconnects (issue #437)
• Added rd_kafka_mem_free()
• Formal C and C++ API docs (doxygen)
• Added RdKafka::Message::topic_name().
  Allows retreiving message topic (name) for auto consumed topics where we dont have a local Topic object
• Add C++ Metadata API (Eugenio Pérez (@eugpermar) and @incessantier)
• Changed topic.metadata.refresh.interval.ms to 5m, must be lower than server idle timeout
• Alias some new java client properties (bootstrap.servers, enable.auto.commit)
• Added topic-level "compression.codec" configuration
• Added "builtin.features" property to allow applications to query for built-in features (e.g., gzip, snappy, zlib)
• Add Consumer::OffsetTail(). (Brian Wellington (@bwelling))
• Removed enforce.isr.cnt - should be handled by broker now and request.required.acks=-1
• Added rd_kafka_topic_opaque()
• Added rd_kafka_*_conf_get() for querying conf property values on a conf object
• New high-level balanced KafkaConsumer (requires broker version >= 0.9)
• Add consumer queue support for C++. (Bob Halley (@rthalley) and Samuele Maci (@macisamuele))
• Fixed alignment errors on Sparc, et.al.
• Increased topic.metadata.refresh.interval.ms to the official default 10 minutes (#363)
• Added config property 'socket.blocking.max.ms' to control max socket poll time
• Added lazy producer message offset reporting
  When 'produce.offset.report' is off, produced offsets will still be reported or the last message in each produced batch.
  This means an application can still keep track of offsets (though a bit more roughly) without the cost of per-message offset reports.
• Added rd_kafka_yield() to break out of callback loops
• Propagate UNKNOWN_TOPIC and UNKNOWN_PARTITION to consumers (issue #348)
• Handle broker name and nodeid updates (issue #343)
  This also migrates bootstrap brokers to proper brokers if they can be exactly matched by name.
• Added consume.callback.max.messages (issue #248)
• Added C++ Consumer::consume_callback as front-end of rd_kafka_consume_callback (kodekarl)
• Add a built in consistent hashing partitioner (Michael Koziarski (@NZKoz))
• Add ability to specify a CRL for checking remote certificates (Vincent Bernat (@vincentbernat))

Fixes

• Preserve configured partition.assignment.strategy order
• Final offset commits on consumer close() are now limited to session.timeout.ms
• rd_kafka_log_print should print to stderr, not stdout
• C++ unassign() assigned an empty list, which is not an unassign..
• Dont fail/retry unsent requests on broker failure, leave them in queue.
• Portability fixes (FreeBSD, Win32, MUSL libc, SunOS, AIX)
• C++: added virtual destructors to all classes that have a virtual function (danny794)
• Consumer: fix race condition with start(logical_offset) quickly followed by seek(abs_offset)
• Maintain per-partition op-version in Fetch requests (issue #586)
• Richer CONFIGURATION.md format (supported values, value type)
• Fixed a number of possible hang-on-termination scenarios.
• Improved handling of failed broker requests (exposed by #529)
• Fixed cases where a timeout_ms of 0 would serve precisely timed events forever
• Fixed buffer retry counting
• Dont rely on partition leader broker to be set when querying metadata (issue #518)
• Ignore intermittent Topic Metadata failures while brokers are rebalancing. (issue #513)
• rd_kafka_topic_partition_available() returned 1 for leader-less partitions (issue #513). Regression since 0.8.6
• Backoff sleep on broker UP -> DOWN state transition was never called. This could trigger busy-looping when connecting to a broken broker.
• Added configurable reconnect delay with jitter: reconnect.backoff.jitter.ms (issue #403)
• Snappy endian swapping for older glibc 2.9 (issue #505, #407)
• Fix possible crash on buf retries when broker is unavailable
• Consumer errors are now rkmessage-based rather than error_cb
• Added RD_KAFKA_OFFSET_INVALID to indicate invalid offsets in various APIs
• Minimize default address caching time to 1s (issue #494)
• Added rd_kafka_errno() (issue #489)
• Proper generation and installing of pkg-config .pc files for both C and C++ libraries
• Report unsupported compression codecs (on consume) as CONSUMER_ERR ops
• RdKafka::Conf->set "socket_cb" and "open_cb" checked for wrong Conf type
• Deprecate rd_kafka_set_logger() in favour of rd_kafka_conf_set_log_cb()
• Dont reset partition count to zero on metadata topic error
  A metadata partition count of 0 is returned by the broker with errors such as "No leader broker" for a topic, since these errors are
  typically intermittent we hold on to the existing partition count.
• Emit ALL_BROKERS_DOWN if none of the initial bootstrap brokers could be added.
• Consumer offset fixes (#352, #276, #227)
• Dont loose messages when partition goes away (and there is no UA partition)
  Previously any messages in a partition's queue would be silently lost if a partition went away (and there was no UA partition).
  These messages are now properly delivery reported with UNKNOWN_PARTITION.
• Provide rkmessage->rkt topic in dr_msg_cb (#333)
• Fixed wait in rd_kafka_consume_callback_queue().
  It returned immediately if no messages were available (timeout_ms was not used).
• Fix occasional invalid json on statistics buffers (Alfred Landrum (@alfred-landrum))
• dr_msg_cb: rkmessage->partition is now properly set (issue #233)
• Renamed position() -> committed() and added position() that returns next offset. According to the Java client.
• Workaround Java client and broker Offset metadata parsing bugs (issue #577)
• Added rd_kafka_query_watermark_offsets() and rd_kafka_get_watermark_offsets() (issue #503, #530)
• C++: Added commitSync and commitAsync APIs taking a list of offsets (issue #528)
• Added enable.auto.offset.store (issue #528)
• sasl.kerberos.kinit.cmd now takes a command+args string template (issue #635)
• Send consumer group user_data as empty instead of NULL to workarund broker 0.9.0.0 bug (issue #660)