Skip to content

docs: fix typos in guides #6912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions docs/blocks-storage/production-tips.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,17 @@ weight: 4
slug: production-tips
---

This page shares some tips and things to take in consideration when setting up a production Cortex cluster based on the blocks storage.
This page shares some tips and things to take into consideration when setting up a production Cortex cluster based on the blocks storage.

## Ingester

### Ensure a high number of max open file descriptors

The ingester stores received series into per-tenant TSDB blocks. Both TSDB WAL, head and compacted blocks are composed by a relatively large number of files which gets loaded via mmap. This means that the ingester keeps file descriptors open for TSDB WAL segments, chunk files and compacted blocks which haven't reached the retention period yet.
The ingester stores received series into per-tenant TSDB blocks. Both TSDB WAL, head and compacted blocks are composed of a relatively large number of files which get loaded via mmap. This means that the ingester keeps file descriptors open for TSDB WAL segments, chunk files and compacted blocks which haven't reached the retention period yet.

If your Cortex cluster has many tenants or ingester is running with a long `-blocks-storage.tsdb.retention-period`, the ingester may hit the **`file-max` ulimit** (maximum number of open file descriptions by a process); in such case, we recommend increasing the limit on your system or enabling [shuffle sharding](../guides/shuffle-sharding.md).
If your Cortex cluster has many tenants or the ingester is running with a long `-blocks-storage.tsdb.retention-period`, the ingester may hit the **`file-max` ulimit** (maximum number of open file descriptions by a process); in such case, we recommend increasing the limit on your system or enabling [shuffle sharding](../guides/shuffle-sharding.md).

The rule of thumb is that a production system shouldn't have the `file-max` ulimit below `65536`, but higher values are recommended (eg. `1048576`).
The rule of thumb is that a production system shouldn't have the `file-max` ulimit below `65536`, but higher values are recommended (e.g. `1048576`).

### Ingester disk space

Expand All @@ -25,13 +25,13 @@ We typically configure ingesters to retain these blocks for longer, to allow tim

If you configure ingesters with `-blocks-storage.tsdb.retention-period=24h`, a rule of thumb for disk space required is to take the number of timeseries after replication and multiply by 30KB.

For example, if you have 20M active series replicated 3 ways, this gives approx 1.7TB. Divide by the number of ingesters and allow some margin for growth, e.g. if you have 20 ingesters then 100GB each should work, or 150GB each to be more comfortable.
For example, if you have 20M active series replicated 3 ways, this gives approx 1.7TB. Divide by the number of ingesters and allow some margin for growth, e.g. if you have 20 ingesters then 100GB each should work, or 150GB each to be more comfortable.

## Querier

### Ensure caching is enabled

The querier relies on caching to reduce the number API calls to the storage bucket. Ensure [caching](./querier.md#caching) is properly configured and [properly scaled](#ensure-memcached-is-properly-scaled).
The querier relies on caching to reduce the number of API calls to the storage bucket. Ensure [caching](./querier.md#caching) is properly configured and [properly scaled](#ensure-memcached-is-properly-scaled).

### Ensure bucket index is enabled

Expand All @@ -41,8 +41,8 @@ The bucket index reduces the number of API calls to the storage bucket and, when

When running Cortex blocks storage cluster at scale, querying non compacted blocks may be inefficient for two reasons:

1. Non compacted blocks contain duplicated samples (as effect of the ingested samples replication)
2. Overhead introduced querying many small indexes
1. Non compacted blocks contain duplicated samples (as an effect of the ingested samples replication)
2. Overhead introduced by querying many small indexes

Because of this, we would suggest to avoid querying non compacted blocks. In order to do it, you should:

Expand All @@ -56,7 +56,7 @@ Because of this, we would suggest to avoid querying non compacted blocks. In ord

The `-querier.query-store-after` should be set to a duration large enough to give compactor enough time to compact newly uploaded blocks, and queriers and store-gateways to discover and sync newly compacted blocks.

The following diagram shows all the timings involved in the estimation. This diagram should be used only as a template and you're expected to tweak the assumptions based on real measurements in your Cortex cluster. In this example, the following assumptions have been done:
The following diagram shows all the timings involved in the estimation. This diagram should be used only as a template and you're expected to tweak the assumptions based on real measurements in your Cortex cluster. In this example, the following assumptions have been made:

- An ingester takes up to 30 minutes to upload a block to the storage
- The compactor takes up to 3 hours to compact 2h blocks shipped from all ingesters
Expand All @@ -79,9 +79,9 @@ The bucket index reduces the number of API calls to the storage bucket and the s

### Ensure a high number of max open file descriptors

The store-gateway stores each blocks index-header on the local disk and loads it via mmap. This means that the store-gateway keeps a file descriptor open for each loaded block. If your Cortex cluster has many blocks in the bucket, the store-gateway may hit the **`file-max` ulimit** (maximum number of open file descriptions by a process); in such case, we recommend increasing the limit on your system or running more store-gateway instances with blocks sharding enabled.
The store-gateway stores each block's index-header on the local disk and loads it via mmap. This means that the store-gateway keeps a file descriptor open for each loaded block. If your Cortex cluster has many blocks in the bucket, the store-gateway may hit the **`file-max` ulimit** (maximum number of open file descriptions by a process); in such case, we recommend increasing the limit on your system or running more store-gateway instances with blocks sharding enabled.

The rule of thumb is that a production system shouldn't have the `file-max` ulimit below `65536`, but higher values are recommended (eg. `1048576`).
The rule of thumb is that a production system shouldn't have the `file-max` ulimit below `65536`, but higher values are recommended (e.g. `1048576`).

## Compactor

Expand All @@ -101,7 +101,7 @@ We also recommend to run a different memcached cluster for each cache type (meta

### Ensure Alertmanager networking is hardened

If the Alertmanager API is enabled, users with access to Cortex can autonomously configure the Alertmanager, including receiver integrations that allow to issue network requests to the configured URL (eg. webhook). If the Alertmanager network is not hardened, Cortex users may have the ability to issue network requests to any network endpoint including services running in the local network accessible by the Alertmanager itself.
If the Alertmanager API is enabled, users with access to Cortex can autonomously configure the Alertmanager, including receiver integrations that allow to issue network requests to the configured URL (e.g. webhook). If the Alertmanager network is not hardened, Cortex users may have the ability to issue network requests to any network endpoint including services running in the local network accessible by the Alertmanager itself.

Despite hardening the system is out of the scope of Cortex, Cortex provides a basic built-in firewall to block connections created by Alertmanager receiver integrations:

Expand Down
6 changes: 3 additions & 3 deletions docs/contributing/how-to-upgrade-golang-version.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@ weight: 4
slug: how-to-upgrade-golang-version
---

To upgrade the Go's runtime version:
To upgrade the Go runtime version:

1. Upgrade build image version with golang version as describe [here](./how-to-update-the-build-image.md)
1. Upgrade build image version with golang version as described [here](./how-to-update-the-build-image.md)

If the minimum support Go's language version should be upgraded as well:
If the minimum supported Go language version should be upgraded as well:

1. Upgrade `go` version in `go.mod`
9 changes: 4 additions & 5 deletions docs/guides/capacity-planning.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Some key parameters are:
Prometheus](https://www.robustperception.io/using-tsdb-analyze-to-investigate-churn-and-cardinality).
4. How compressible the time-series data are. If a metric stays at
the same value constantly, then Cortex can compress it very well, so
12 hours of data sampled every 15 seconds would be around 2KB. On
12 hours of data sampled every 15 seconds would be around 2KB. On
the other hand, if the value jumps around a lot, it might take 10KB.
There are not currently any tools available to analyse this.
5. How long you want to retain data for, e.g. 1 month or 2 years.
Expand All @@ -47,21 +47,20 @@ Now, some rules of thumb:
replication factor. This is with the default of 12-hour chunks - RAM
required will reduce if you set `-ingester.max-chunk-age` lower
(trading off more back-end database I/O).
There are some additional considerations for planning for ingester memory usage.
There are some additional considerations for planning for ingester memory usage:
1. Memory increases during write-ahead log (WAL) replay, [See Prometheus issue #6934](https://github.com/prometheus/prometheus/issues/6934#issuecomment-726039115). If you do not have enough memory for WAL replay, the ingester will not be able to restart successfully without intervention.
2. Memory temporarily increases during resharding since timeseries are temporarily on both the new and old ingesters. This means you should scale up the number of ingesters before memory utilization is too high, otherwise you will not have the headroom to account for the temporary increase.
2. Each million series (including churn) consumes 15GB of chunk
storage and 4GB of index, per day (so multiply by the retention
period).
3. The distributors CPU utilization depends on the specific Cortex cluster
3. The distributors' CPU utilization depends on the specific Cortex cluster
setup, while they don't need much RAM. Typically, distributors are capable
of processing between 20,000 and 100,000 samples/sec with 1 CPU core. It's also
highly recommended to configure Prometheus `max_samples_per_send` to 1,000
samples, in order to reduce the distributors CPU utilization given the same
samples, in order to reduce the distributors' CPU utilization given the same
total samples/sec throughput.

If you turn on compression between distributors and ingesters (for
example, to save on inter-zone bandwidth charges at AWS/GCP), they will use
significantly more CPU (approx. 100% more for distributor and 50% more
for ingester).

5 changes: 2 additions & 3 deletions docs/guides/encryption-at-rest.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,11 @@ sse:

### Ruler

The ruler S3 server-side encryption can be configured similarly to the blocks storage. The per-tenant overrides are supported when using the storage backend configurable the `-ruler-storage.` flag prefix (or their respective YAML config options).
The ruler S3 server-side encryption can be configured similarly to the blocks storage. The per-tenant overrides are supported when using the storage backend configurable with the `-ruler-storage.` flag prefix (or their respective YAML config options).

### Alertmanager

The alertmanager S3 server-side encryption can be configured similarly to the blocks storage. The per-tenant overrides are supported when using the storage backend configurable the `-alertmanager-storage.` flag prefix (or their respective YAML config options).
The alertmanager S3 server-side encryption can be configured similarly to the blocks storage. The per-tenant overrides are supported when using the storage backend configurable with the `-alertmanager-storage.` flag prefix (or their respective YAML config options).

### Per-tenant config overrides

Expand All @@ -61,4 +61,3 @@ The following settings can be overridden for each tenant:
## Other storages

Other storage backends may support encryption at rest, configuring it directly at the storage level.

6 changes: 3 additions & 3 deletions docs/guides/ha-pair-handling.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ slug: ha-pair-handling

## Context

You can have more than a single Prometheus monitoring and ingesting the same metrics for redundancy. Cortex already does replication for redundancy, and it doesn't make sense to ingest the same data twice. So in Cortex, we made sure we can dedupe the data we receive from HA Pairs of Prometheus. We do this via the following:
You can have more than a single Prometheus instance monitoring and ingesting the same metrics for redundancy. Cortex already does replication for redundancy, and it doesn't make sense to ingest the same data twice. So in Cortex, we made sure we can dedupe the data we receive from HA Pairs of Prometheus. We do this via the following:

Assume that there are two teams, each running their own Prometheus, monitoring different services. Let's call the Prometheus T1 and T2. Now, if the teams are running HA pairs, let's call the individual Prometheus, T1.a, T1.b, and T2.a and T2.b.
Assume that there are two teams, each running their own Prometheus, monitoring different services. Let's call the Prometheus instances T1 and T2. Now, if the teams are running HA pairs, let's call the individual Prometheus instances T1.a, T1.b, and T2.a and T2.b.

In Cortex, we make sure we only ingest from one of T1.a and T1.b, and only from one of T2.a and T2.b. We do this by electing a leader replica for each cluster of Prometheus. For example, in the case of T1, let it be T1.a. As long as T1.a is the leader, we drop the samples sent by T1.b. And if Cortex sees no new samples from T1.a for a short period (30s by default), it'll switch the leader to be T1.b.

This means if T1.a goes down for a few minutes, Cortex's HA sample handling will have switched and elected T1.b as the leader. This failover timeout is what enables us to only accept samples from a single replica at a time, but ensure we don't drop too much data in case of issues. Note that with the default scrape period of 15s, and the default timeouts in Cortex, in most cases, you'll only lose a single scrape of data in the case of a leader election failover. For any rate queries, the rate window should be at least 4x the scrape period to account for any of these failover scenarios, for example, with the default scrape period of 15s, then you should calculate rates over at least 1m periods.
This means if T1.a goes down for a few minutes, Cortex's HA sample handling will have switched and elected T1.b as the leader. This failover timeout is what enables us to only accept samples from a single replica at a time, but ensure we don't drop too much data in case of issues. Note that with the default scrape period of 15s, and the default timeouts in Cortex, in most cases, you'll only lose a single scrape of data in the case of a leader election failover. For any rate queries, the rate window should be at least 4x the scrape period to account for any of these failover scenarios. For example, with the default scrape period of 15s, then you should calculate rates over at least 1m periods.

Now we do the same leader election process for T2.

Expand Down
6 changes: 3 additions & 3 deletions docs/guides/tracing.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Cortex in production.

In order to send traces, you will need to set up a Jaeger deployment. A
deployment includes either the Jaeger all-in-one binary or else a distributed
system of agents, collectors, and queriers. If running on Kubernetes, [Jaeger
system of agents, collectors, and queriers. If running on Kubernetes, [Jaeger
Kubernetes](https://github.com/jaegertracing/jaeger-kubernetes) is an excellent
resource.

Expand Down Expand Up @@ -49,8 +49,8 @@ even if you plan to use the default values.
In order to send traces, you will need to set up an OpenTelemetry Collector. The collector will be able to send traces to
multiple destinations such as [AWS X-Ray](https://aws-otel.github.io/docs/getting-started/x-ray),
[Google Cloud](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudexporter),
[Dash0](https://www.dash0.com/hub/integrations/int_opentelemetry-collector/overview)
[DataDog](https://docs.datadoghq.com/tracing/trace_collection/open_standards/otel_collector_datadog_exporter/) and
[Dash0](https://www.dash0.com/hub/integrations/int_opentelemetry-collector/overview),
[DataDog](https://docs.datadoghq.com/tracing/trace_collection/open_standards/otel_collector_datadog_exporter/), and
[others](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter). OpenTelemetry Collector
provides a [helm chart](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector/examples/deployment-otlp-traces)
to set up the environment.
Expand Down
Loading