Cortex 1.15.2

This release includes Go runtime upgrade to 1.20.4 to address critical CVE.

• [ENHANCEMENT] Update Go version to 1.20.4. #5299

Cortex 1.15.1

This release includes:

• [CHANGE] Alertmanager: Validating new fields on the PagerDuty AM config. #5290
• [BUGFIX] Querier: Convert gRPC ResourceExhausted status code from store gateway to 422 limit error. #5286

Cortex 1.15.0

This release contains 177 contributions from 24 contributors. We also have 13 new contributors. Thank you all for the contribution!

Some notable changes release are:

• Out of order samples ingestion
• MultiKey KV ring for DynamoDB
• Snappy-block gRPC compression
• Redis as index cache and caching bucket backend
• ARM images support
• Thanos PromQL engine support
• Multiples performance improvements and bug fixes

Cortex

• [CHANGE] Storage: Make Max exemplars config per tenant instead of global configuration. #5080 #5122
• [CHANGE] Alertmanager: Local file disclosure vulnerability in OpsGenie configuration has been fixed. #5045
• [CHANGE] Rename oltp_endpoint to otlp_endpoint to match opentelemetry spec and lib name. #5068
• [CHANGE] Distributor/Ingester: Log warn level on push requests when they have status code 4xx. Do not log if status is 429. #5103
• [CHANGE] Tracing: Use the default OTEL trace sampler when -tracing.otel.exporter-type is set to awsxray. #5141
• [CHANGE] Ingester partial error log line to debug level. #5192
• [CHANGE] Change HTTP status code from 503/422 to 499 if a request is canceled. #5220
• [CHANGE] Store gateways summary metrics have been converted to histograms cortex_bucket_store_series_blocks_queried, cortex_bucket_store_series_data_fetched, cortex_bucket_store_series_data_size_touched_bytes, cortex_bucket_store_series_data_size_fetched_bytes, cortex_bucket_store_series_data_touched, cortex_bucket_store_series_result_series #5239
• [FEATURE] Querier/Query Frontend: support Prometheus /api/v1/status/buildinfo API. #4978
• [FEATURE] Ingester: Add active series to all_user_stats page. #4972
• [FEATURE] Ingester: Added -blocks-storage.tsdb.head-chunks-write-queue-size allowing to configure the size of the in-memory queue used before flushing chunks to the disk . #5000
• [FEATURE] Query Frontend: Log query params in query frontend even if error happens. #5005
• [FEATURE] Ingester: Enable snapshotting of In-memory TSDB on disk during shutdown via -blocks-storage.tsdb.memory-snapshot-on-shutdown. #5011
• [FEATURE] Query Frontend/Scheduler: Add a new counter metric cortex_request_queue_requests_total for total requests going to queue. #5030
• [FEATURE] Build ARM docker images. #5041
• [FEATURE] Query-frontend/Querier: Create spans to measure time to merge promql responses. #5041
• [FEATURE] Querier/Ruler: Support the new thanos promql engine. This is an experimental feature and might change in the future. #5093
• [FEATURE] Added zstd as an option for grpc compression #5092
• [FEATURE] Ring: Add new kv store option dynamodb. #5026
• [FEATURE] Cache: Support redis as backend for caching bucket and index cache. #5057
• [FEATURE] Querier/Store-Gateway: Added -blocks-storage.bucket-store.ignore-blocks-within allowing to filter out the recently created blocks from being synced by queriers and store-gateways. #5166
• [FEATURE] AlertManager/Ruler: Added support for keep_firing_for on alerting rulers.
• [FEATURE] Alertmanager: Add support for time_intervals. #5102
• [FEATURE] Added snappy-block as an option for grpc compression #5215
• [FEATURE] Enable experimental out-of-order samples support. Added 2 new configs ingester.out_of_order_time_window and blocks-storage.tsdb.out_of_order_cap_max. #4964
• [ENHANCEMENT] Querier: limit series query to only ingesters if start param is not specified. #4976
• [ENHANCEMENT] Query-frontend/scheduler: add a new limit frontend.max-outstanding-requests-per-tenant for configuring queue size per tenant. Started deprecating two flags -query-scheduler.max-outstanding-requests-per-tenant and -querier.max-outstanding-requests-per-tenant, and change their value default to 0. Now if both the old flag and new flag are specified, the old flag's queue size will be picked. #4991
• [ENHANCEMENT] Query-tee: Add /api/v1/query_exemplars API endpoint support. #5010
• [ENHANCEMENT] Let blocks_cleaner delete blocks concurrently(default 16 goroutines). #5028
• [ENHANCEMENT] Query Frontend/Query Scheduler: Increase upper bound to 60s for queue duration histogram metric. #5029
• [ENHANCEMENT] Query Frontend: Log Vertical sharding information when query_stats_enabled is enabled. #5037
• [ENHANCEMENT] Ingester: The metadata APIs should honour querier.query-ingesters-within when querier.query-store-for-labels-enabled is true. #5027
• [ENHANCEMENT] Query Frontend: Skip instant query roundtripper if sharding is not applicable. #5062
• [ENHANCEMENT] Push reduce one hash operation of Labels. #4945 #5114
• [ENHANCEMENT] Alertmanager: Added -alertmanager.enabled-tenants and -alertmanager.disabled-tenants to explicitly enable or disable alertmanager for specific tenants. #5116
• [ENHANCEMENT] Upgraded Docker base images to alpine:3.17. #5132
• [ENHANCEMENT] Add retry logic to S3 bucket client. #5135
• [ENHANCEMENT] Update Go version to 1.20.1. #5159
• [ENHANCEMENT] Distributor: Reuse byte slices when serializing requests from distributors to ingesters. #5193
• [ENHANCEMENT] Query Frontend: Add number of chunks and samples fetched in query stats. #5198
• [ENHANCEMENT] Implement grpc.Compressor.DecompressedSize for snappy to optimize memory allocations. #5213
• [ENHANCEMENT] Querier: Batch Iterator optimization to prevent transversing it multiple times query ranges steps does not overlap. #5237
• [BUGFIX] Updated golang.org/x/net dependency to fix CVE-2022-27664. #5008
• [BUGFIX] Fix panic when otel and xray tracing is enabled. #5044
• [BUGFIX] Fixed no compact block got grouped in shuffle sharding grouper. #5055
• [BUGFIX] Fixed ingesters with less tokens stuck in LEAVING. #5061
• [BUGFIX] Tracing: Fix missing object storage span instrumentation. #5074
• [BUGFIX] Ingester: Fix Ingesters returning empty response for metadata APIs. #5081
• [BUGFIX] Ingester: Fix panic when querying metadata from blocks that are being deleted. #5119
• [BUGFIX] Ring: Fix case when dynamodb kv reaches the limit of 25 actions per batch call. #5136
• [BUGFIX] Query-frontend: Fix shardable instant queries do not produce sorted results for sort, sort_desc, topk, bottomk functions. #5148, #5170
• [BUGFIX] Querier: Fix /api/v1/series returning 5XX instead of 4XX when limits are hit. #5169
• [BUGFIX] Compactor: Fix issue that shuffle sharding planner return error if block is under visit by other compactor. #5188
• [BUGFIX] Fix S3 BucketWithRetries upload empty content issue #5217
• [BUGFIX] Query Frontend: Disable absent, absent_over_time and scalar for vertical sharding. #5221
• [BUGFIX] Catch context error in the s3 bucket client. #5240
• [BUGFIX] Fix query frontend remote read empty body. #5257
• [BUGFIX] Fix query frontend incorrect error response format at SplitByQuery middleware. #5260

Cortex 1.15.0-rc.2

Over v1.15.0-rc.1 to include two bug fixes.

• [BUGFIX] Fix query frontend remote read empty body. #5257
• [BUGFIX] Fix query frontend incorrect error response format at SplitByQuery middleware. #5260

Cortex 1.15.0-rc.1

Over v1.15.0-rc.0 to include some bug fixes and performance improvements.

• [CHANGE] Store gateways summary metrics have been converted to histograms cortex_bucket_store_series_blocks_queried, cortex_bucket_store_series_data_fetched, cortex_bucket_store_series_data_size_touched_bytes, cortex_bucket_store_series_data_size_fetched_bytes, cortex_bucket_store_series_data_touched, cortex_bucket_store_series_result_series #5239
• [ENHANCEMENT] Querier: Batch Iterator optimization to prevent transversing it multiple times query ranges steps does not overlap. #5237
• [BUGFIX] Catch context error in the s3 bucket client. #5240

Cortex 1.15.0-rc.0

This release contains 172 contributions from 24 contributors. We also have 13 new contributors. Thank you all for the contribution!

Some notable changes release are:

• Out of order samples ingestion
• MultiKey KV ring for DynamoDB
• Snappy-block gRPC compression
• Redis as index cache and caching bucket backend
• Arm images support
• Thanos PromQL engine support
• Multiples performance improvements and bug fixes

Cortex

• [CHANGE] Storage: Make Max exemplars config per tenant instead of global configuration. #5080 #5122
• [CHANGE] Alertmanager: Local file disclosure vulnerability in OpsGenie configuration has been fixed. #5045
• [CHANGE] Rename oltp_endpoint to otlp_endpoint to match opentelemetry spec and lib name. #5068
• [CHANGE] Distributor/Ingester: Log warn level on push requests when they have status code 4xx. Do not log if status is 429. #5103
• [CHANGE] Tracing: Use the default OTEL trace sampler when -tracing.otel.exporter-type is set to awsxray. #5141
• [CHANGE] Ingester partial error log line to debug level. #5192
• [CHANGE] Change HTTP status code from 503/422 to 499 if a request is canceled. #5220
• [FEATURE] Querier/Query Frontend: support Prometheus /api/v1/status/buildinfo API. #4978
• [FEATURE] Ingester: Add active series to all_user_stats page. #4972
• [FEATURE] Ingester: Added -blocks-storage.tsdb.head-chunks-write-queue-size allowing to configure the size of the in-memory queue used before flushing chunks to the disk . #5000
• [FEATURE] Query Frontend: Log query params in query frontend even if error happens. #5005
• [FEATURE] Ingester: Enable snapshotting of In-memory TSDB on disk during shutdown via -blocks-storage.tsdb.memory-snapshot-on-shutdown. #5011
• [FEATURE] Query Frontend/Scheduler: Add a new counter metric cortex_request_queue_requests_total for total requests going to queue. #5030
• [FEATURE] Build ARM docker images. #5041
• [FEATURE] Query-frontend/Querier: Create spans to measure time to merge promql responses. #5041
• [FEATURE] Querier/Ruler: Support the new thanos promql engine. This is an experimental feature and might change in the future. #5093
• [FEATURE] Added zstd as an option for grpc compression #5092
• [FEATURE] Ring: Add new kv store option dynamodb. #5026
• [FEATURE] Cache: Support redis as backend for caching bucket and index cache. #5057
• [FEATURE] Querier/Store-Gateway: Added -blocks-storage.bucket-store.ignore-blocks-within allowing to filter out the recently created blocks from being synced by queriers and store-gateways. #5166
• [FEATURE] AlertManager/Ruler: Added support for keep_firing_for on alerting rulers.
• [FEATURE] Alertmanager: Add support for time_intervals. #5102
• [FEATURE] Added snappy-block as an option for grpc compression #5215
• [FEATURE] Enable experimental out-of-order samples support. Added 2 new configs ingester.out_of_order_time_window and blocks-storage.tsdb.out_of_order_cap_max. #4964
• [ENHANCEMENT] Querier: limit series query to only ingesters if start param is not specified. #4976
• [ENHANCEMENT] Query-frontend/scheduler: add a new limit frontend.max-outstanding-requests-per-tenant for configuring queue size per tenant. Started deprecating two flags -query-scheduler.max-outstanding-requests-per-tenant and -querier.max-outstanding-requests-per-tenant, and change their value default to 0. Now if both the old flag and new flag are specified, the old flag's queue size will be picked. #4991
• [ENHANCEMENT] Query-tee: Add /api/v1/query_exemplars API endpoint support. #5010
• [ENHANCEMENT] Let blocks_cleaner delete blocks concurrently(default 16 goroutines). #5028
• [ENHANCEMENT] Query Frontend/Query Scheduler: Increase upper bound to 60s for queue duration histogram metric. #5029
• [ENHANCEMENT] Query Frontend: Log Vertical sharding information when query_stats_enabled is enabled. #5037
• [ENHANCEMENT] Ingester: The metadata APIs should honour querier.query-ingesters-within when querier.query-store-for-labels-enabled is true. #5027
• [ENHANCEMENT] Query Frontend: Skip instant query roundtripper if sharding is not applicable. #5062
• [ENHANCEMENT] Push reduce one hash operation of Labels. #4945 #5114
• [ENHANCEMENT] Alertmanager: Added -alertmanager.enabled-tenants and -alertmanager.disabled-tenants to explicitly enable or disable alertmanager for specific tenants. #5116
• [ENHANCEMENT] Upgraded Docker base images to alpine:3.17. #5132
• [ENHANCEMENT] Add retry logic to S3 bucket client. #5135
• [ENHANCEMENT] Update Go version to 1.20.1. #5159
• [ENHANCEMENT] Distributor: Reuse byte slices when serializing requests from distributors to ingesters. #5193
• [ENHANCEMENT] Query Frontend: Add number of chunks and samples fetched in query stats. #5198
• [ENHANCEMENT] Implement grpc.Compressor.DecompressedSize for snappy to optimize memory allocations. #5213
• [BUGFIX] Updated golang.org/x/net dependency to fix CVE-2022-27664. #5008
• [BUGFIX] Fix panic when otel and xray tracing is enabled. #5044
• [BUGFIX] Fixed no compact block got grouped in shuffle sharding grouper. #5055
• [BUGFIX] Fixed ingesters with less tokens stuck in LEAVING. #5061
• [BUGFIX] Tracing: Fix missing object storage span instrumentation. #5074
• [BUGFIX] Ingester: Fix Ingesters returning empty response for metadata APIs. #5081
• [BUGFIX] Ingester: Fix panic when querying metadata from blocks that are being deleted. #5119
• [BUGFIX] Ring: Fix case when dynamodb kv reaches the limit of 25 actions per batch call. #5136
• [BUGFIX] Query-frontend: Fix shardable instant queries do not produce sorted results for sort, sort_desc, topk, bottomk functions. #5148, #5170
• [BUGFIX] Querier: Fix /api/v1/series returning 5XX instead of 4XX when limits are hit. #5169
• [BUGFIX] Compactor: Fix issue that shuffle sharding planner return error if block is under visit by other compactor. #5188
• [BUGFIX] Fix S3 BucketWithRetries upload empty content issue #5217
• [BUGFIX] Query Frontend: Disable absent, absent_over_time and scalar for vertical sharding. #5221

Cortex 1.14.1

This release includes:

• [CHANGE] Alertmanager: Local file disclosure vulnerability in OpsGenie configuration has been fixed. #5045
  • CVE-2022-23536
• [BUGFIX] Fix panic when otel and xray tracing is enabled. #5044
• [FEATURE] Build ARM docker images. #5041

Docker Images:

• quay.io/cortexproject/cortex:v1.14.1
• cortexproject/cortex:v1.14.1

Cortex 1.13.2

This release includes:

• [CHANGE] Alertmanager: Local file disclosure vulnerability in OpsGenie configuration has been fixed. #5045
  • CVE-2022-23536

Docker Images:

• quay.io/cortexproject/cortex:v1.13.2
• cortexproject/cortex:v1.13.2

Cortex 1.14.0

This release contains 115 contributions from 28 contributors. Thank you!

Some notable changes release are:

• Remove support for chunks storage
• Experimental support for vertical query sharding
• Enable PromQL @ modifier with negative offset always
• Added configurations for Azure MSI in blocks-storage
• New limits (Querier/QueryFrontend)
• OpenTelemetry Bridge for Tracing
• Multiples performance improvements and bug fixes

Cortex

• [CHANGE] Remove support for chunks storage entirely. If you are using chunks storage on a previous version, you must migrate your data on version 1.12 or earlier. Before upgrading to this release, you should also remove any deprecated chunks-related configuration, as this release will no longer accept that. The following flags are gone:
  • -dynamodb.*
  • -metrics.*
  • -s3.*
  • -azure.*
  • -bigtable.*
  • -gcs.*
  • -cassandra.*
  • -boltdb.*
  • -local.*
  • some -ingester flags:
    • -ingester.wal-enabled
    • -ingester.checkpoint-enabled
    • -ingester.recover-from-wal
    • -ingester.wal-dir
    • -ingester.checkpoint-duration
    • -ingester.flush-on-shutdown-with-wal-enabled
    • -ingester.max-transfer-retries
    • -ingester.max-samples-per-query
    • -ingester.min-chunk-length
    • -ingester.flush-period
    • -ingester.retain-period
    • -ingester.max-chunk-idle
    • -ingester.max-stale-chunk-idle
    • -ingester.flush-op-timeout
    • -ingester.max-chunk-age
    • -ingester.chunk-age-jitter
    • -ingester.concurrent-flushes
    • -ingester.spread-flushes
    • -store.* except -store.engine and -store.max-query-length
    • -store.query-chunk-limit was deprecated and replaced by -querier.max-fetched-chunks-per-query
  • -deletes.*
  • -grpc-store.*
  • -flusher.wal-dir, -flusher.concurrent-flushes, -flusher.flush-op-timeout
• [CHANGE] Remove support for alertmanager and ruler legacy store configuration. Before upgrading, you need to convert your configuration to use the alertmanager-storage and ruler-storage configuration on the version that you're already running, then upgrade.
• [CHANGE] Disables TSDB isolation. #4825
• [CHANGE] Drops support Prometheus 1.x rule format on configdb. #4826
• [CHANGE] Removes -ingester.stream-chunks-when-using-blocks experimental flag and stream chunks by default when querier.ingester-streaming is enabled. #4864
• [CHANGE] Compactor: Added cortex_compactor_runs_interrupted_total to separate compaction interruptions from failures
• [CHANGE] Enable PromQL @ modifier, negative offset always. #4927
• [CHANGE] Store-gateway: Add user label to cortex_bucket_store_blocks_loaded metric. #4918
• [CHANGE] AlertManager: include status label in cortex_alertmanager_alerts_received_total. #4907
• [FEATURE] Compactor: Added -compactor.block-files-concurrency allowing to configure number of go routines for download/upload block files during compaction. #4784
• [FEATURE] Compactor: Added -compactor.blocks-fetch-concurrency allowing to configure number of go routines for blocks during compaction. #4787
• [FEATURE] Compactor: Added configurations for Azure MSI in blocks-storage, ruler-storage and alertmanager-storage. #4818
• [FEATURE] Ruler: Add support to pass custom implementations of queryable and pusher. #4782
• [FEATURE] Create OpenTelemetry Bridge for Tracing. Now cortex can send traces to multiple destinations using OTEL Collectors. #4834
• [FEATURE] Added -api.http-request-headers-to-log allowing for the addition of HTTP Headers to logs #4803
• [FEATURE] Distributor: Added a new limit -validation.max-labels-size-bytes allowing to limit the combined size of labels for each timeseries. #4848
• [FEATURE] Storage/Bucket: Added -*.s3.bucket-lookup-type allowing to configure the s3 bucket lookup type. #4794
• [FEATURE] QueryFrontend: Implement experimental vertical sharding at query frontend for range/instant queries. #4863
• [FEATURE] QueryFrontend: Support vertical sharding for subqueries. #4955
• [FEATURE] Querier: Added a new limit -querier.max-fetched-data-bytes-per-query allowing to limit the maximum size of all data in bytes that a query can fetch from each ingester and storage. #4854
• [FEATURE] Added 2 flags -alertmanager.alertmanager-client.grpc-compression and -querier.store-gateway-client.grpc-compression to configure compression methods for grpc clients. #4889
• [ENHANCEMENT] AlertManager: Retrying AlertManager Get Requests (Get Alertmanager status, Get Alertmanager Receivers) on next replica on error #4840
• [ENHANCEMENT] Querier/Ruler: Retry store-gateway in case of unexpected failure, instead of failing the query. #4532 #4839
• [ENHANCEMENT] Ring: DoBatch prioritize 4xx errors when failing. #4783
• [ENHANCEMENT] Cortex now built with Go 1.18. #4829
• [ENHANCEMENT] Ingester: Prevent ingesters to become unhealthy during wall replay. #4847
• [ENHANCEMENT] Compactor: Introduced visit marker file for blocks so blocks are under compaction will not be picked up by another compactor. #4805
• [ENHANCEMENT] Distributor: Add label name to labelValueTooLongError. #4855
• [ENHANCEMENT] Enhance traces with hostname information. #4898
• [ENHANCEMENT] Improve the documentation around limits. #4905
• [ENHANCEMENT] Distributor: cache user overrides to reduce lock contention. #4904
• [BUGFIX] Storage/Bucket: Enable AWS SDK for go authentication for s3 to fix IMDSv1 authentication. #4897
• [BUGFIX] Memberlist: Add join with no retrying when starting service. #4804
• [BUGFIX] Ruler: Fix /ruler/rule_groups returns YAML with extra fields. #4767
• [BUGFIX] Respecting -tracing.otel.sample-ratio configuration when enabling OpenTelemetry tracing with X-ray. #4862
• [BUGFIX] QueryFrontend: fixed query_range requests when query has start equals to end. #4877
• [BUGFIX] AlertManager: fixed issue introduced by #4495 where templates files were being deleted when using alertmanager local store. #4890
• [BUGFIX] Ingester: fixed incorrect logging at the start of ingester block shipping logic. #4934
• [BUGFIX] Storage/Bucket: fixed global mark missing on deletion. #4949
• [BUGFIX] QueryFrontend/Querier: fixed regression added by #4863 where we stopped compressing the response between querier and query frontend. #4960
• [BUGFIX] QueryFrontend/Querier: fixed fix response error to be ungzipped when status code is not 2xx. #4975

Cortex 1.14.0-rc.1

Over v1.14.0-rc.0 to include the bugfix where query responses with status code other than 2xx were not being ungzipped.

• [BUGFIX] QueryFrontend/Querier: fixed fix response error to be ungzipped when status code is not 2xx. #4975