From 209efac134a92691db4de3ed43adebeb1919d030 Mon Sep 17 00:00:00 2001 From: Alvar Penning Date: Fri, 25 Apr 2025 13:19:32 +0200 Subject: [PATCH 1/3] doc/01-About.md: Big Picture Inspired by the "Big Picture" section in the Icinga Notifications documentation[^0], such a section was introduced for Icinga DB as well. It aims to describe the essential components in an Icinga DB setup along the architecture diagram for novice users. This hopefully helps new users to understand what Icinga DB actually is and how it interconnects. [^0]: https://icinga.com/docs/icinga-notifications/latest/#big-picture --- doc/01-About.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/doc/01-About.md b/doc/01-About.md index 056da3ff2..c880ac00c 100644 --- a/doc/01-About.md +++ b/doc/01-About.md @@ -12,8 +12,28 @@ visualizing monitoring data in the Icinga ecosystem, consisting of: [Icinga DB Web](https://icinga.com/docs/icinga-db-web) module enabled, which connects to both Redis® and the database to display and work with the most up-to-date data +## Big Picture + ![Icinga DB Architecture](images/icingadb-architecture.png) +Icinga DB consists of several components in an Icinga setup. +This section tries to help understanding how these components relate, following the architecture diagram shown above. + +First things first, Icinga DB is not a database itself, but consumes and passes data from Icinga 2 to be displayed in Icinga DB Web and persisted in a relational database. + +Let's start with Icinga 2. +With the Icinga DB feature enabled, Icinga 2 synchronizes its state to a Redis® server that can be queried by both the Icinga DB daemon and Icinga DB Web. + +The Icinga DB daemon reads all the information from Redis®, transforms it, and finally inserts it into a relational database such as MySQL, MariaDB, or PostgreSQL. +Doing so removes load from Icinga 2 and lets Icinga DB do the more time-consuming database operations in bulk. +In addition, the Icinga DB daemon does some bookkeeping, such as removing old history entries if a retention is configured. + +To display information in Icinga Web 2, the Icinga DB Web module fetches the latest service and host state information from Redis®. +Additional monitoring data and history information is retrieved from the relational database. +Icinga DB Web also connects to the Icinga 2 API with its Command Transport to acknowledge problems, trigger check executions, and so on. + +These are the components of Icinga DB embedded into an Icinga setup with Icinga 2 and Icinga Web 2. + ## Installation To install Icinga DB see [Installation](02-Installation.md). From be56bc12953d53f5b9ea53fec4b76ecb9a4d30eb Mon Sep 17 00:00:00 2001 From: Alvar Penning Date: Fri, 25 Apr 2025 13:28:14 +0200 Subject: [PATCH 2/3] doc/03-Configuration.md: Mention forced log events The logging was slightly modified in #920, resulting in a few log messages being shown regardless of the user configured log level. Since this may not be the expected behavior, it is now mentioned in the docs. --- doc/03-Configuration.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/03-Configuration.md b/doc/03-Configuration.md index 2cd7900b3..cccfd2233 100644 --- a/doc/03-Configuration.md +++ b/doc/03-Configuration.md @@ -99,6 +99,11 @@ For environment variables, each option is prefixed with `ICINGADB_LOGGING_`. | interval | **Optional.** Interval for periodic logging defined as [duration string](#duration-string). Defaults to `"20s"`. | | options | **Optional.** Map of component name to logging level in order to set a different logging level for each component instead of the default one. See [logging components](#logging-components) for details. | +!!! info + + There are a few log messages that are logged regardless of the configured log level. + For example, the startup message is always shown. + ### Logging Components The independent components of Icinga DB produce log entries. From d8f42a3fe8220573b3b7460bbfcbd666f168541f Mon Sep 17 00:00:00 2001 From: Alvar Penning Date: Fri, 25 Apr 2025 14:11:32 +0200 Subject: [PATCH 3/3] doc/07-Operations.md: New Operations section The newly introduced Operations section is a first attempt at an operational Icinga DB documentation. It covers - essential Icinga DB monitoring, - backups and corner cases for MySQL/MariaDB, - MySQL/MariaDB configuration options, including AWS RDS and Galera, - and memory overcommitment for Redis. As such a section can never be completed, this documentation is a target for continuous improvement. Fixes #745. --- doc/07-Operations.md | 98 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 doc/07-Operations.md diff --git a/doc/07-Operations.md b/doc/07-Operations.md new file mode 100644 index 000000000..fcbaed5aa --- /dev/null +++ b/doc/07-Operations.md @@ -0,0 +1,98 @@ +# Operations + +This section is a loose collection of various topics and external references for running Icinga DB on a day-to-day basis. +It covers topics such as self-monitoring, backups, and specifics of third-party components. + +## Monitor Icinga DB + +It is strongly recommended to monitor the monitoring. + +There is a built-in [`icingadb` check command](https://icinga.com/docs/icinga-2/latest/doc/10-icinga-template-library/#icingadb) in the Icinga 2 ITL. +It covers several potential errors, including operations that take too long or invalid high availability scenarios. +Even if the Icinga DB has crashed, checks will still run and Icinga 2 would generate notifications. + +In addition, both the Redis® and the relational database should be monitored. +There are predefined check commands in the ITL to choose from. + +- [`redis`](https://icinga.com/docs/icinga-2/latest/doc/10-icinga-template-library/#redis) +- [`mysql`](https://icinga.com/docs/icinga-2/latest/doc/10-icinga-template-library/#mysql) +- [`mysql_health`](https://icinga.com/docs/icinga-2/latest/doc/10-icinga-template-library/#mysql_health) +- [`postgres`](https://icinga.com/docs/icinga-2/latest/doc/10-icinga-template-library/#postgres) + +A simpler approach would be to check if the processes are running, e.g., +with [`proc`](https://icinga.com/docs/icinga-2/latest/doc/10-icinga-template-library/#procs) or +[`systemd`](https://icinga.com/docs/icinga-2/latest/doc/10-icinga-template-library/#systemd). + +## Backups + +There are only two things to back up in Icinga DB. + +1. The configuration file in `/etc/icingadb` and +2. the relational database, using `mysqldump`, `mariadb-dump` or `pg_dump`. + +!!! warning + + When creating a database dump for MySQL or MariaDB with `mysqldump` or `mariadb-dump`, + use the [`--single-transaction` command line argument flag](https://dev.mysql.com/doc/refman/8.4/en/mysqldump.html#option_mysqldump_single-transaction) + to not lock the whole database while the backup is running. + +## Third-Party Configuration + +Icinga DB relies on external components to work. +The following collection is based on experience. +It is a target for continuous improvement. + +### MySQL and MariaDB + +#### `max_allow_packets` + +The `max_allow_packets` system variable limits the size of messages between MySQL/MariaDB servers and clients. +More information is available in +[MySQL's "Replication and max_allowed_packet" documentation section](https://dev.mysql.com/doc/refman/8.4/en/replication-features-max-allowed-packet.html), +[MySQL's variable documentation](https://dev.mysql.com/doc/refman/8.4/en/server-system-variables.html#sysvar_max_allowed_packet) and +[MariaDB's variable documentation](https://mariadb.com/kb/en/server-system-variables/#max_allowed_packet). + +The database configuration should have `max_allow_packets` set to at least `64M`. + +#### Amazon RDS for MySQL + +When importing the MySQL schema into Amazon RDS for MySQL, the following may occur. + +``` +Error 1419: You do not have the SUPER privilege and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable) +``` + +This error can be mitigated by creating and modifying a custom DB parameter group as described in the related [AWS Knowledge Center article](https://repost.aws/knowledge-center/rds-mysql-functions). + +#### Galera Cluster + +Starting with Icinga DB version 1.2.0, Galera support has been added to the Icinga DB daemon. +Its specific database configuration is described in the [Galera configuration section](03-Configuration.md#galera-cluster). + +As mentioned in [MariaDB's known Galera cluster limitations](https://mariadb.com/kb/en/mariadb-galera-cluster-known-limitations/), +transactions are limited in both amount of rows (128K) and size (2GiB). +A busy Icinga setup can cause Icinga DB to create transactions that exceed these limits with the default configuration. + +If you get an error like `Error 1105 (HY000): Maximum writeset size exceeded` +and your Galera node logs something like `WSREP: transaction size limit (2147483647) exceeded`, +decrease the values of `max_placeholders_per_statement` and `max_rows_per_transaction` in Icinga DB's +[Database Options](https://icinga.com/docs/icinga-db/latest/doc/03-Configuration/#database-options). + +### Redis® + +On Linux, enable [memory overcommitting](https://www.kernel.org/doc/Documentation/vm/overcommit-accounting). + +```shell +sysctl vm.overcommit_memory=1 +``` + +To persist this setting across reboots, add the following line to [`sysctl.conf(5)`](https://man7.org/linux/man-pages/man5/sysctl.conf.5.html). +If your distribution uses systemd, a configuration file under `/etc/sysctl.d/` is required, as described by +[`systemd-sysctl.service(8)`](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysctl.service.html) and +[`sysctl.d(5)`](https://man7.org/linux/man-pages/man5/sysctl.d.5.html). + +``` +vm.overcommit_memory = 1 +``` + +In addition, the official [Redis® administration documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/admin/) is quite useful.