v1.16-4.10/outputs Update all dependencies

[renovate]

This PR contains the following updates:

Package	Change	Age	Confidence
fluent-plugin-datadog (source, changelog)	'0.14.4' → '0.15.0'
fluent-plugin-grafana-loki	'1.2.20' → '1.3.0'
fluent-plugin-kafka	'0.19.4' → '0.19.6'
fluent-plugin-kinesis	'3.5.0' → '3.7.0'
rdkafka	'0.21.0' → '0.26.0'

---

Release Notes

DataDog/fluent-plugin-datadog (fluent-plugin-datadog)

v0.15.0

Compare Source

• Provide a configuration option to delete kubernetes and docker attributes from the log after the relevant information has been extracted into tags #​78 by [@​sambart19].
• Fix launch.json and update readme #​76
• Add custom headers support #​69

grafana/loki (fluent-plugin-grafana-loki)

v1.3.0

What's New??

With 1.3.0 we are excited to announce several improvements focusing on performance!

First and most significant is the Query Frontend:

• 1442 cyriltovena: Loki Query Frontend

The query frontend allows for sharding queries by time and dispatching them in parallel to multiple queriers, giving true horizontal scaling ability for queries. Take a look at the jsonnet changes to see how we are deploying this in our production setup. Keep an eye out for a blog post with more information on how the frontend works and more information on this exciting new feature.

In our quest to improve query performance, we discovered that gzip, while good for compression ratio, is not the best for speed. So we introduced the ability to select from several different compression algorithms:

• 1411 cyriltovena: Adds configurable compression algorithms for chunks

We are currently testing out LZ4 and snappy, LZ4 seemed like a good fit however we found that it didn't always compress the same data to the same output which was causing some troubles for another important improvement:

• 1438 pstibrany: pkg/ingester: added sync period flags

Extending on the work done by @​bboreham on Cortex, @​pstibrany added a few new flags and code to synchronize chunks between ingesters, which reduces the number of chunks persisted to object stores and therefore also reduces the number of chunks loaded on queries and the amount of de-duplication work which needs to be done.

As mentioned above, LZ4 was in some cases compressing the same data with a different result which was interfering with this change, we are still investigating the cause of this issue (It may be in how we implemented something, or may be in the compression code itself). For now we have switched to snappy which has seen a reduction in data written to the object store from almost 3x the source data (with a replication factor of 3) to about 1.5x, saving a lot of duplicated log storage!

Another valuable change related to chunks:

• 1406 slim-bean: allow configuring a target chunk size in compressed bytes

With this change you can set a chunk_target_size and Loki will attempt to fill a chunk to approx that size before flushing (previously a chunk size was a hard coded 10 blocks where the default block size is 262144 bytes). Larger chunks are beneficial for a few reasons, mainly on reducing API calls to your object store when performing queries, but also in reducing overhead in a few places, especially when processing very high volume log streams.

Another big improvement is the introduction of accurate rate limiting when running microservices:

• 1486 pracucci: Add ingestion rate global limit support

Previously the rate limit was applied at each distributor, however with traffic split over many distributors the limit would need to be adjusted accordingly. This meant that scaling up distributors required changing the limit. Now this information is communicated between distributors such that the limit should be applied accurately regardless of the number of distributors.

And last but not least on the notable changes list is a new feature for Promtail:

• 1275 bastjan: pkg/promtail: IETF Syslog (RFC5424) Support

With this change Promtail can receive syslogs via TCP! Thanks to @​bastjan for all the hard work on this submission!

Important things to note

• 1519 Changes a core behavior in Loki regarding logs with duplicate content AND duplicate timestamps, previously Loki would store logs with duplicate timestamps and content, moving forward logs with duplicate content AND timestamps will be silently ignored. Mainly this change is to prevent duplicates that appear when a batch is retried (the first entry in the list would be inserted again, now it will be ignored). Logs with the same timestamp and different content will still be accepted.
• 1486 Deprecated -distributor.limiter-reload-period flag / distributor's limiter_reload_period config option.

All Changes

Once again we can't thank our community and contributors enough for the significant work that everyone is adding to Loki, the entire list of changes is long!!

Loki

• 1526 codesome: Support for aggregation
• 1522 cyriltovena: Adds support for the old query string regexp in the frontend.
• 1519 rfratto: pkg/chunkenc: ignore duplicate lines pushed to a stream
• 1511 sandlis: querier: fix panic in tailer when max tail duration exceeds
• 1499 slim-bean: Fix a panic in chunk prefetch
• 1495 slim-bean: Prefetch chunks while processing
• 1496 cyriltovena: Add duplicates info and remove timing informations.
• 1490 owen-d: Fix/deadlock frontend queue
• 1489 owen-d: unifies reverse iterators
• 1488 cyriltovena: Fixes response json encoding and add regression tests.
• 1486 pracucci: Add ingestion rate global limit support* 1493 pracucci: Added max streams per user global limit
• 1480 cyriltovena: Close iterator properly and check nil before releasing buffers.
• 1473 rfratto: pkg/querier: don't query all ingesters
• 1470 cyriltovena: Validates limit parameter.
• 1448 cyriltovena: Improving storage benchmark
• 1445 cyriltovena: Add decompression tracing instrumentation.
• 1442 cyriltovena: Loki Query Frontend
• 1438 pstibrany: pkg/ingester: added sync period flags
• 1433 zendern: Using strict parsing for yaml configs
• 1425 pstibrany: pkg/ingester: Added possibility to disable transfers.
• 1423 pstibrany: pkg/chunkenc: Fix BenchmarkRead to focus on reading chunks, not converting bytes to string
• 1421 pstibrany: pkg/chunkenc: change default LZ4 buffer size to 64k.
• 1420 cyriltovena: Sets the chunk encoding correctly when creating chunk from bytes.
• 1419 owen-d: Enables Series API in loki
• 1413 pstibrany: RangeQuery benchmark optimizations
• 1411 cyriltovena: Adds configurable compression algorithms for chunks
• 1409 slim-bean: change the chunk size histogram to allow for bigger buckets
• 1408 slim-bean: forgot to register the new metric for counting blocks per chunk
• 1406 slim-bean: allow configuring a target chunk size in compressed bytes
• 1405 pstibrany: Convert string to bytes once only when doing string filtering.
• 1396 pstibrany: pkg/cfg: print help only when requested, and print it on stdout
• 1383 beornf: Read websocket close in tail handler
• 1071 rfratto: pkg/ingester: limit total number of errors a stream can return on push
• 1545 joe-elliott: Critical n => m conversions
• 1541 owen-d: legacy endpoint 400s metric queries

Promtail

• 1515 slim-bean: Promtail: Improve position and size metrics
• 1485 p37ruh4: Fileglob parsing fixes
• 1472 owen-d: positions.ignore-corruptions
• 1453 chancez: pkg/promtail: Initialize counters to 0 when creating client
• 1436 rfratto: promtail: add support for passing through journal entries as JSON
• 1426 wphan: Support microsecond timestamp format
• 1416 pstibrany: pkg/promtail/client: missing URL in client returns error
• 1275 bastjan: pkg/promtail: IETF Syslog (RFC5424) Support

Fluent Bit

• 1455 JensErat: fluent-bit-plugin: re-enable failing JSON marshaller tests; pass error instead of logging and ignoring
• 1294 JensErat: fluent-bit: multi-instance support
• 1514 shane-axiom: fluent-plugin-grafana-loki: Add fluentd_thread label when flush_thread_count > 1

Fluentd

• 1500 cyriltovena: Bump fluentd plugin to 1.2.6.
• 1475 Horkyze: fluentd-plugin: call gsub for strings only

Docker Driver

• 1414 cyriltovena: Adds tenant-id for docker driver.

Logcli

• 1492 sandlis: logcli: replaced GRAFANA_with LOKI_ in logcli env vars, set default server url for logcli to localhost

Helm

• 1534 olivierboudet: helm : fix fluent-bit parser configuration syntax
• 1506 terjesannum: helm: add podsecuritypolicy for fluent-bit
• 1431 eugene100: Helm: fix issue with config.clients
• 1430 olivierboudet: helm : allow to define custom parsers to use with fluentbit.io/parser annotation
• 1418 evalsocket: Helm chart url added in helm.md
• 1336 terjesannum: helm: support adding init containers to the loki pod
• 1530 WeiBanjo: Allow extra command line args for external labels like hostname

Jsonnet

• 1518 benjaminhuo: Fix error 'Field does not exist: jaeger_mixin' in tk show
• 1501 anarcher: jsonnet: fix common/defaultPorts parameters
• 1497 cyriltovena: Update Loki mixin to include frontend QPS and latency.
• 1478 cyriltovena: Fixes the typo in the result cache config of the Loki ksonnet lib.
• 1543 sh0rez: fix(ksonnet): use apps/v1

Docs

• 1531 fitzoh: Documentation: Add note on using Loki with Amazon ECS
• 1521 rfratto: docs: Document timestamp ordering rules
• 1516 rfratto: Link to release docs in README.md, not master docs
• 1508 cyriltovena: Fixes bad json in Loki API documentation.
• 1505 sandlis: doc: fix sample yaml in docs for installing promtail to k8s
• 1481 terjesannum: docs: fix broken promtail link
• 1474 Eraac: : information about max_look_back_period
• 1471 cyriltovena: Update README.md
• 1466 Eraac: : Update IAM requirement
• 1441 vtereso: : README spelling fix
• 1437 daixiang0: fix all misspell
• 1432 joe-elliott: Removed unsupported encodings from docs
• 1399 vishesh92: Docs: Add configuration docs for redis
• 1394 chancez: Documentation: Fix example AWS storage configuration
• 1227 daixiang0: Add docker install doc
• 1560 robshep: Promtail Docs: Update output.md
• 1546 mattmendick: Removing third-party link
• 1539 j18e: docs: fix syntax error in pipeline example

Build

• 1494 pracucci: Fixed TOUCH_PROTOS in all DroneCI pipelines
• 1479 owen-d: TOUCH_PROTOS build arg for dockerfile
• 1476 owen-d: initiates docker daemon for circle windows builds
• 1469 rfratto: Makefile: re-enable journal scraping on ARM

New Members

• 1415 cyriltovena: Add Joe as member of the team.

fluent/fluent-plugin-kafka (fluent-plugin-kafka)

v0.19.6

Compare Source

v0.19.5

Compare Source

awslabs/aws-fluent-plugin-kinesis (fluent-plugin-kinesis)

v3.7.0

• Feature - Add use_fips_endpoint configuration parameter for FIPS-compliant endpoints : #​227
• Feature - Add aggregated_record_size configuration parameter for kinesis_firehose to reduce Firehose 5 KB per-record billing roundup : #​193
• Bug fix - Fix binary data appearing in error log when compression is enabled : #​231
• Bug fix - Suppress ProcessCredentials insecure string warning from newer aws-sdk-core
• Enhancement - Add fluent-package v6 LTS and Fluentd 1.19.2 with Ruby 3.4.8 to test cases
• Enhancement - Fix typos and update links in README

v3.6.1

• Dependency - Update gem dependency to Ruby 2.7.1+

v3.6.0

• Enhancement - Add fluent-package 5.2 and Fluentd 1.19.0 with Ruby 3.4.5 to test cases
• Dependency - Update google-protobuf dependency to v4

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • Between 12:00 AM and 03:59 AM (* 0-3 * * *)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.