From e594e2ba0a439e9d2d466895ca4623ea1cbd947b Mon Sep 17 00:00:00 2001 From: Charlie Le Date: Sat, 26 Jul 2025 09:45:20 -0700 Subject: [PATCH 1/2] docs: fix typos in guides Signed-off-by: Charlie Le --- docs/guides/capacity-planning.md | 9 ++++----- docs/guides/encryption-at-rest.md | 5 ++--- docs/guides/ha-pair-handling.md | 6 +++--- docs/guides/tracing.md | 6 +++--- 4 files changed, 12 insertions(+), 14 deletions(-) diff --git a/docs/guides/capacity-planning.md b/docs/guides/capacity-planning.md index c7c5a2b5ed7..eb094c3d554 100644 --- a/docs/guides/capacity-planning.md +++ b/docs/guides/capacity-planning.md @@ -28,7 +28,7 @@ Some key parameters are: Prometheus](https://www.robustperception.io/using-tsdb-analyze-to-investigate-churn-and-cardinality). 4. How compressible the time-series data are. If a metric stays at the same value constantly, then Cortex can compress it very well, so - 12 hours of data sampled every 15 seconds would be around 2KB. On + 12 hours of data sampled every 15 seconds would be around 2KB. On the other hand, if the value jumps around a lot, it might take 10KB. There are not currently any tools available to analyse this. 5. How long you want to retain data for, e.g. 1 month or 2 years. @@ -47,21 +47,20 @@ Now, some rules of thumb: replication factor. This is with the default of 12-hour chunks - RAM required will reduce if you set `-ingester.max-chunk-age` lower (trading off more back-end database I/O). - There are some additional considerations for planning for ingester memory usage. + There are some additional considerations for planning for ingester memory usage: 1. Memory increases during write-ahead log (WAL) replay, [See Prometheus issue #6934](https://github.com/prometheus/prometheus/issues/6934#issuecomment-726039115). If you do not have enough memory for WAL replay, the ingester will not be able to restart successfully without intervention. 2. Memory temporarily increases during resharding since timeseries are temporarily on both the new and old ingesters. This means you should scale up the number of ingesters before memory utilization is too high, otherwise you will not have the headroom to account for the temporary increase. 2. Each million series (including churn) consumes 15GB of chunk storage and 4GB of index, per day (so multiply by the retention period). -3. The distributors’ CPU utilization depends on the specific Cortex cluster +3. The distributors' CPU utilization depends on the specific Cortex cluster setup, while they don't need much RAM. Typically, distributors are capable of processing between 20,000 and 100,000 samples/sec with 1 CPU core. It's also highly recommended to configure Prometheus `max_samples_per_send` to 1,000 - samples, in order to reduce the distributors’ CPU utilization given the same + samples, in order to reduce the distributors' CPU utilization given the same total samples/sec throughput. If you turn on compression between distributors and ingesters (for example, to save on inter-zone bandwidth charges at AWS/GCP), they will use significantly more CPU (approx. 100% more for distributor and 50% more for ingester). - diff --git a/docs/guides/encryption-at-rest.md b/docs/guides/encryption-at-rest.md index d713c821fcd..7139495250e 100644 --- a/docs/guides/encryption-at-rest.md +++ b/docs/guides/encryption-at-rest.md @@ -40,11 +40,11 @@ sse: ### Ruler -The ruler S3 server-side encryption can be configured similarly to the blocks storage. The per-tenant overrides are supported when using the storage backend configurable the `-ruler-storage.` flag prefix (or their respective YAML config options). +The ruler S3 server-side encryption can be configured similarly to the blocks storage. The per-tenant overrides are supported when using the storage backend configurable with the `-ruler-storage.` flag prefix (or their respective YAML config options). ### Alertmanager -The alertmanager S3 server-side encryption can be configured similarly to the blocks storage. The per-tenant overrides are supported when using the storage backend configurable the `-alertmanager-storage.` flag prefix (or their respective YAML config options). +The alertmanager S3 server-side encryption can be configured similarly to the blocks storage. The per-tenant overrides are supported when using the storage backend configurable with the `-alertmanager-storage.` flag prefix (or their respective YAML config options). ### Per-tenant config overrides @@ -61,4 +61,3 @@ The following settings can be overridden for each tenant: ## Other storages Other storage backends may support encryption at rest, configuring it directly at the storage level. - diff --git a/docs/guides/ha-pair-handling.md b/docs/guides/ha-pair-handling.md index 058d0d9f0c0..fd1a389b4a6 100644 --- a/docs/guides/ha-pair-handling.md +++ b/docs/guides/ha-pair-handling.md @@ -7,13 +7,13 @@ slug: ha-pair-handling ## Context -You can have more than a single Prometheus monitoring and ingesting the same metrics for redundancy. Cortex already does replication for redundancy, and it doesn't make sense to ingest the same data twice. So in Cortex, we made sure we can dedupe the data we receive from HA Pairs of Prometheus. We do this via the following: +You can have more than a single Prometheus instance monitoring and ingesting the same metrics for redundancy. Cortex already does replication for redundancy, and it doesn't make sense to ingest the same data twice. So in Cortex, we made sure we can dedupe the data we receive from HA Pairs of Prometheus. We do this via the following: -Assume that there are two teams, each running their own Prometheus, monitoring different services. Let's call the Prometheus T1 and T2. Now, if the teams are running HA pairs, let's call the individual Prometheus, T1.a, T1.b, and T2.a and T2.b. +Assume that there are two teams, each running their own Prometheus, monitoring different services. Let's call the Prometheus instances T1 and T2. Now, if the teams are running HA pairs, let's call the individual Prometheus instances T1.a, T1.b, and T2.a and T2.b. In Cortex, we make sure we only ingest from one of T1.a and T1.b, and only from one of T2.a and T2.b. We do this by electing a leader replica for each cluster of Prometheus. For example, in the case of T1, let it be T1.a. As long as T1.a is the leader, we drop the samples sent by T1.b. And if Cortex sees no new samples from T1.a for a short period (30s by default), it'll switch the leader to be T1.b. -This means if T1.a goes down for a few minutes, Cortex's HA sample handling will have switched and elected T1.b as the leader. This failover timeout is what enables us to only accept samples from a single replica at a time, but ensure we don't drop too much data in case of issues. Note that with the default scrape period of 15s, and the default timeouts in Cortex, in most cases, you'll only lose a single scrape of data in the case of a leader election failover. For any rate queries, the rate window should be at least 4x the scrape period to account for any of these failover scenarios, for example, with the default scrape period of 15s, then you should calculate rates over at least 1m periods. +This means if T1.a goes down for a few minutes, Cortex's HA sample handling will have switched and elected T1.b as the leader. This failover timeout is what enables us to only accept samples from a single replica at a time, but ensure we don't drop too much data in case of issues. Note that with the default scrape period of 15s, and the default timeouts in Cortex, in most cases, you'll only lose a single scrape of data in the case of a leader election failover. For any rate queries, the rate window should be at least 4x the scrape period to account for any of these failover scenarios. For example, with the default scrape period of 15s, then you should calculate rates over at least 1m periods. Now we do the same leader election process for T2. diff --git a/docs/guides/tracing.md b/docs/guides/tracing.md index ead9291e34e..27f89f578fa 100644 --- a/docs/guides/tracing.md +++ b/docs/guides/tracing.md @@ -15,7 +15,7 @@ Cortex in production. In order to send traces, you will need to set up a Jaeger deployment. A deployment includes either the Jaeger all-in-one binary or else a distributed -system of agents, collectors, and queriers. If running on Kubernetes, [Jaeger +system of agents, collectors, and queriers. If running on Kubernetes, [Jaeger Kubernetes](https://github.com/jaegertracing/jaeger-kubernetes) is an excellent resource. @@ -49,8 +49,8 @@ even if you plan to use the default values. In order to send traces, you will need to set up an OpenTelemetry Collector. The collector will be able to send traces to multiple destinations such as [AWS X-Ray](https://aws-otel.github.io/docs/getting-started/x-ray), [Google Cloud](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudexporter), -[Dash0](https://www.dash0.com/hub/integrations/int_opentelemetry-collector/overview) -[DataDog](https://docs.datadoghq.com/tracing/trace_collection/open_standards/otel_collector_datadog_exporter/) and +[Dash0](https://www.dash0.com/hub/integrations/int_opentelemetry-collector/overview), +[DataDog](https://docs.datadoghq.com/tracing/trace_collection/open_standards/otel_collector_datadog_exporter/), and [others](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter). OpenTelemetry Collector provides a [helm chart](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector/examples/deployment-otlp-traces) to set up the environment. From 097d52a8c841b4bbd9e6800786275b5011dd6d8d Mon Sep 17 00:00:00 2001 From: Charlie Le Date: Sat, 26 Jul 2025 09:49:28 -0700 Subject: [PATCH 2/2] docs: fix typos in production-tips.md and how-to-upgrade-golang-version.md Signed-off-by: Charlie Le --- docs/blocks-storage/production-tips.md | 24 +++++++++---------- .../how-to-upgrade-golang-version.md | 6 ++--- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/blocks-storage/production-tips.md b/docs/blocks-storage/production-tips.md index ff400e4fd96..4ae29a0c118 100644 --- a/docs/blocks-storage/production-tips.md +++ b/docs/blocks-storage/production-tips.md @@ -5,17 +5,17 @@ weight: 4 slug: production-tips --- -This page shares some tips and things to take in consideration when setting up a production Cortex cluster based on the blocks storage. +This page shares some tips and things to take into consideration when setting up a production Cortex cluster based on the blocks storage. ## Ingester ### Ensure a high number of max open file descriptors -The ingester stores received series into per-tenant TSDB blocks. Both TSDB WAL, head and compacted blocks are composed by a relatively large number of files which gets loaded via mmap. This means that the ingester keeps file descriptors open for TSDB WAL segments, chunk files and compacted blocks which haven't reached the retention period yet. +The ingester stores received series into per-tenant TSDB blocks. Both TSDB WAL, head and compacted blocks are composed of a relatively large number of files which get loaded via mmap. This means that the ingester keeps file descriptors open for TSDB WAL segments, chunk files and compacted blocks which haven't reached the retention period yet. -If your Cortex cluster has many tenants or ingester is running with a long `-blocks-storage.tsdb.retention-period`, the ingester may hit the **`file-max` ulimit** (maximum number of open file descriptions by a process); in such case, we recommend increasing the limit on your system or enabling [shuffle sharding](../guides/shuffle-sharding.md). +If your Cortex cluster has many tenants or the ingester is running with a long `-blocks-storage.tsdb.retention-period`, the ingester may hit the **`file-max` ulimit** (maximum number of open file descriptions by a process); in such case, we recommend increasing the limit on your system or enabling [shuffle sharding](../guides/shuffle-sharding.md). -The rule of thumb is that a production system shouldn't have the `file-max` ulimit below `65536`, but higher values are recommended (eg. `1048576`). +The rule of thumb is that a production system shouldn't have the `file-max` ulimit below `65536`, but higher values are recommended (e.g. `1048576`). ### Ingester disk space @@ -25,13 +25,13 @@ We typically configure ingesters to retain these blocks for longer, to allow tim If you configure ingesters with `-blocks-storage.tsdb.retention-period=24h`, a rule of thumb for disk space required is to take the number of timeseries after replication and multiply by 30KB. -For example, if you have 20M active series replicated 3 ways, this gives approx 1.7TB. Divide by the number of ingesters and allow some margin for growth, e.g. if you have 20 ingesters then 100GB each should work, or 150GB each to be more comfortable. +For example, if you have 20M active series replicated 3 ways, this gives approx 1.7TB. Divide by the number of ingesters and allow some margin for growth, e.g. if you have 20 ingesters then 100GB each should work, or 150GB each to be more comfortable. ## Querier ### Ensure caching is enabled -The querier relies on caching to reduce the number API calls to the storage bucket. Ensure [caching](./querier.md#caching) is properly configured and [properly scaled](#ensure-memcached-is-properly-scaled). +The querier relies on caching to reduce the number of API calls to the storage bucket. Ensure [caching](./querier.md#caching) is properly configured and [properly scaled](#ensure-memcached-is-properly-scaled). ### Ensure bucket index is enabled @@ -41,8 +41,8 @@ The bucket index reduces the number of API calls to the storage bucket and, when When running Cortex blocks storage cluster at scale, querying non compacted blocks may be inefficient for two reasons: -1. Non compacted blocks contain duplicated samples (as effect of the ingested samples replication) -2. Overhead introduced querying many small indexes +1. Non compacted blocks contain duplicated samples (as an effect of the ingested samples replication) +2. Overhead introduced by querying many small indexes Because of this, we would suggest to avoid querying non compacted blocks. In order to do it, you should: @@ -56,7 +56,7 @@ Because of this, we would suggest to avoid querying non compacted blocks. In ord The `-querier.query-store-after` should be set to a duration large enough to give compactor enough time to compact newly uploaded blocks, and queriers and store-gateways to discover and sync newly compacted blocks. -The following diagram shows all the timings involved in the estimation. This diagram should be used only as a template and you're expected to tweak the assumptions based on real measurements in your Cortex cluster. In this example, the following assumptions have been done: +The following diagram shows all the timings involved in the estimation. This diagram should be used only as a template and you're expected to tweak the assumptions based on real measurements in your Cortex cluster. In this example, the following assumptions have been made: - An ingester takes up to 30 minutes to upload a block to the storage - The compactor takes up to 3 hours to compact 2h blocks shipped from all ingesters @@ -79,9 +79,9 @@ The bucket index reduces the number of API calls to the storage bucket and the s ### Ensure a high number of max open file descriptors -The store-gateway stores each block’s index-header on the local disk and loads it via mmap. This means that the store-gateway keeps a file descriptor open for each loaded block. If your Cortex cluster has many blocks in the bucket, the store-gateway may hit the **`file-max` ulimit** (maximum number of open file descriptions by a process); in such case, we recommend increasing the limit on your system or running more store-gateway instances with blocks sharding enabled. +The store-gateway stores each block's index-header on the local disk and loads it via mmap. This means that the store-gateway keeps a file descriptor open for each loaded block. If your Cortex cluster has many blocks in the bucket, the store-gateway may hit the **`file-max` ulimit** (maximum number of open file descriptions by a process); in such case, we recommend increasing the limit on your system or running more store-gateway instances with blocks sharding enabled. -The rule of thumb is that a production system shouldn't have the `file-max` ulimit below `65536`, but higher values are recommended (eg. `1048576`). +The rule of thumb is that a production system shouldn't have the `file-max` ulimit below `65536`, but higher values are recommended (e.g. `1048576`). ## Compactor @@ -101,7 +101,7 @@ We also recommend to run a different memcached cluster for each cache type (meta ### Ensure Alertmanager networking is hardened -If the Alertmanager API is enabled, users with access to Cortex can autonomously configure the Alertmanager, including receiver integrations that allow to issue network requests to the configured URL (eg. webhook). If the Alertmanager network is not hardened, Cortex users may have the ability to issue network requests to any network endpoint including services running in the local network accessible by the Alertmanager itself. +If the Alertmanager API is enabled, users with access to Cortex can autonomously configure the Alertmanager, including receiver integrations that allow to issue network requests to the configured URL (e.g. webhook). If the Alertmanager network is not hardened, Cortex users may have the ability to issue network requests to any network endpoint including services running in the local network accessible by the Alertmanager itself. Despite hardening the system is out of the scope of Cortex, Cortex provides a basic built-in firewall to block connections created by Alertmanager receiver integrations: diff --git a/docs/contributing/how-to-upgrade-golang-version.md b/docs/contributing/how-to-upgrade-golang-version.md index bf7513d08c7..398416ba204 100644 --- a/docs/contributing/how-to-upgrade-golang-version.md +++ b/docs/contributing/how-to-upgrade-golang-version.md @@ -5,10 +5,10 @@ weight: 4 slug: how-to-upgrade-golang-version --- -To upgrade the Go's runtime version: +To upgrade the Go runtime version: -1. Upgrade build image version with golang version as describe [here](./how-to-update-the-build-image.md) +1. Upgrade build image version with golang version as described [here](./how-to-update-the-build-image.md) -If the minimum support Go's language version should be upgraded as well: +If the minimum supported Go language version should be upgraded as well: 1. Upgrade `go` version in `go.mod`