openobserve · DebashisBorgohainO2 · Oct 2, 2025 · Oct 3, 2025 · Oct 3, 2025 · Oct 3, 2025
diff --git a/.github/workflows/deploy-docs-staging.yaml b/.github/workflows/deploy-docs-staging.yaml
@@ -49,8 +49,8 @@ jobs:
 
     - name: Deploy to S3
       run: |
-        aws s3 sync ./site s3://openobserve-website-staging/docs --exclude=".git/*"
+        aws s3 sync ./site s3://openobserve-website-staging/docs --exclude=".git/*" --delete
 
     - name: Invalidate CloudFront cache
       run: |
-        aws cloudfront create-invalidation --distribution-id E2GZJM0TJIDFRM --paths "/docs/*"
+        aws cloudfront create-invalidation --distribution-id E2GZJM0TJIDFRM --paths "/docs/*"
diff --git a/docs/images/config-remote-destination-header.png b/docs/images/config-remote-destination-header.png
diff --git a/docs/images/config-remote-destination-headers.png b/docs/images/config-remote-destination-headers.png
diff --git a/docs/images/config-remote-destination-method.png b/docs/images/config-remote-destination-method.png
diff --git a/docs/images/config-remote-destination-output-format.png b/docs/images/config-remote-destination-output-format.png
diff --git a/docs/images/config-remote-destination.png b/docs/images/config-remote-destination.png
diff --git a/docs/images/current-cluster-query-result.png b/docs/images/current-cluster-query-result.png
diff --git a/docs/images/current-cluster-query.png b/docs/images/current-cluster-query.png
diff --git a/docs/images/federated-search-multi-select.png b/docs/images/federated-search-multi-select.png
diff --git a/docs/images/federated-search-result.png b/docs/images/federated-search-result.png
diff --git a/docs/images/federated-search.png b/docs/images/federated-search.png
diff --git a/docs/images/remote-destination-config-from-management.png b/docs/images/remote-destination-config-from-management.png
diff --git a/docs/images/remote-destination-config-from-pipeline-editor.png b/docs/images/remote-destination-config-from-pipeline-editor.png
diff --git a/docs/images/remote-destination-from-pipeline-editor.png b/docs/images/remote-destination-from-pipeline-editor.png
diff --git a/docs/images/sql-reference/not-str-match-with-and.png b/docs/images/sql-reference/not-str-match-with-and.png
diff --git a/docs/images/sql-reference/not-str-match-with-or.png b/docs/images/sql-reference/not-str-match-with-or.png
diff --git a/docs/images/sql-reference/not-str-match.png b/docs/images/sql-reference/not-str-match.png
diff --git a/docs/images/use-pipeline-destination.png b/docs/images/use-pipeline-destination.png
diff --git a/docs/operator-guide/.pages b/docs/operator-guide/.pages
@@ -7,3 +7,4 @@ nav:
     - Etcd maintenance: etcd.md
     - Etcd restore: etcd_restore.md
     - Nginx proxy: nginx_proxy.md
+    - Configuration Management: config-management.md
diff --git a/docs/operator-guide/config-management.md b/docs/operator-guide/config-management.md
@@ -0,0 +1,113 @@
+---
+title: Custom Configuration File and Dynamic Reloading in OpenObserve
+description: Learn how to use custom config paths and dynamic config reloading in OpenObserve to apply changes without restarts.
+---
+This guide explains how to use custom configuration file locations and
+dynamic configuration reloading in OpenObserve to manage deployments
+without system restarts.
+
+## Overview
+Earlier versions of OpenObserve only read the **.env** file from the current
+directory with no way to specify a different path. When running
+OpenObserve in different deployment modes (Kubernetes, virtual machines,
+and systemd services) existing frameworks require config files in
+specific locations. <br><br>
+Additionally, when configuration changes are needed, you need to restart
+the whole cluster or the system, which is expensive. They cause
+downtime, drop capacity and bandwidth, and you do not want to restart
+your whole cluster just to change a simple configuration like increasing
+an interval by 2 seconds.
+<br><br>
+OpenObserve now provides a custom config file CLI argument (`-c` or
+`--config`) to specify config files at any location, and a Config Watcher
+that monitors the config file and automatically reloads specific
+configurations every 30 seconds without requiring a restart.
+
+!!! note "Who should use this"
+    These features are for you if you:
+
+    -   Deploy OpenObserve on virtual machines (VMs)
+    -   Run OpenObserve as systemd services or system daemons
+    -   Use custom deployment frameworks without container orchestration tools like Kubernetes
+    -   Need to avoid expensive restarts that cause downtime
+
+!!! note "Who does NOT need this"
+    You do not need these features if you:
+
+    -   Deploy OpenObserve in Kubernetes clusters
+    -   Use container orchestration platforms
+
+    **Why Kubernetes users do not need this**
+    <br>
+    In Kubernetes deployments:
+
+    -   Configurations are managed through ConfigMaps, Secrets, and environment variables in YAML manifests
+    -   Any configuration change automatically triggers pod restarts and rollouts
+    -   Kubernetes handles configuration updates through its native mechanisms
+    -   The .env file pattern is not used in containerized deployments
+
+## Custom config file CLI argument
+
+### What it does
+It adds a CLI argument to the OpenObserve binary that allows you to pass
+a path to a config file, adding flexibility to load environment
+variables from different files. Earlier versions strictly looked for the
+.env file in the current directory.
+
+### How to use
+
+```bash linenums="1"
+# Short form
+./openobserve -c /path/to/your/config/file
+
+```
+
+```bash linenums="1"
+# Long form
+./openobserve --config /path/to/your/config/file
+```
+
+## Dynamic configuration reload
+
+### What it does
+It adds a job to watch the config file. The watcher monitors the config
+file and reloads the configuration if any changes are detected, allowing
+configuration updates without restarting OpenObserve.
+
+### How it works
+
+-   Runs every 30 seconds by default (configurable via `ZO_CONFIG_WATCHER_INTERVAL` environment variable)
+-   Watches the active config file (.env by default, or custom file if specified using `-c` flag)
+-   Detects changes to configuration values
+-   Automatically reloads changes for reloadable configurations only
+-   If no config file exists, the watcher remains inactive
+
+### Workflow example
+
+1.  OpenObserve is running with a config file
+2.  Edit the config file and change a reloadable configuration (for example, stream stats interval from 2 to 5 seconds)
+3.  Wait up to 30 seconds
+4.  Change is applied automatically without any restart and downtime
+
+### Limitation: Only specific configs are reloadable
+
+Any configuration not in the list below requires a full OpenObserve
+restart to take effect.
+
+### Reloadable environment variables
+| Category | Environment Variables |
+|----------|----------------------|
+| **Data Processing Intervals** | - ZO_FILE_PUSH_INTERVAL<br>- ZO_MEM_PERSIST_INTERVAL<br>- ZO_COMPACT_INTERVAL<br>- ZO_COMPACT_OLD_DATA_INTERVAL<br>- ZO_COMPACT_SYNC_TO_DB_INTERVAL<br>- ZO_COMPACT_JOB_RUN_TIMEOUT<br>- ZO_COMPACT_JOB_CLEAN_WAIT_TIME<br>- ZO_COMPACT_PENDING_JOBS_METRIC_INTERVAL |
+| **Cache Management** | - ZO_MEMORY_CACHE_GC_INTERVAL<br>- ZO_MEMORY_CACHE_MAX_SIZE<br>- ZO_MEMORY_CACHE_DATAFUSION_MAX_SIZE<br>- ZO_DISK_CACHE_GC_INTERVAL<br>- ZO_DISK_CACHE_MAX_SIZE<br>- ZO_DISK_CACHE_SKIP_SIZE<br>- ZO_DISK_CACHE_RELEASE_SIZE<br>- ZO_DISK_CACHE_GC_SIZE<br>- ZO_S3_SYNC_TO_CACHE_INTERVAL |
+| **Metrics and Monitoring** | - ZO_METRICS_LEADER_PUSH_INTERVAL<br>- ZO_METRICS_LEADER_ELECTION_INTERVAL<br>- ZO_CALCULATE_STATS_INTERVAL<br>- ZO_CALCULATE_STATS_STEP_LIMIT<br>- ZO_TELEMETRY_HEARTBEAT |
+| **Scheduling** | - ZO_ALERT_SCHEDULE_INTERVAL<br>- ZO_DERIVED_STREAM_SCHEDULE_INTERVAL<br>- ZO_SCHEDULER_CLEAN_INTERVAL<br>- ZO_SCHEDULER_WATCH_INTERVAL<br>- ZO_SEARCH_JOB_SCHEDULE_INTERVAL<br>- ZO_SEARCH_JOB_SCHEDULER_INTERVAL<br>- ZO_DISTINCT_VALUES_INTERVAL |
+| **Job and Process Timeouts** | - ZO_SEARCH_JOB_TIMEOUT<br>- ZO_SEARCH_JOB_RUN_TIMEOUT<br>- ZO_SEARCH_JOB_RETENTION<br>- ZO_SEARCH_JOB_DELETE_INTERVAL<br>- ZO_HEALTH_CHECK_TIMEOUT |
+| **Memory and Buffer Sizes** | - ZO_MEMORY_CACHE_SKIP_SIZE<br>- ZO_MEMORY_CACHE_RELEASE_SIZE<br>- ZO_MEMORY_CACHE_GC_SIZE<br>- ZO_MEM_TABLE_MAX_SIZE |
+| **File Sizes** | - ZO_MAX_FILE_SIZE_ON_DISK<br>- ZO_MAX_FILE_SIZE_IN_MEMORY<br>- ZO_MAX_FILE_RETENTION_TIME<br>- ZO_COMPACT_MAX_FILE_SIZE |
+| **Network and Protocol Sizes** | - ZO_STREAMING_RESPONSE_CHUNK_SIZE_MB |
+| **Batch Sizes** | - ZO_COMPACT_BATCH_SIZE<br>- ZO_EVENTS_BATCH_SIZE |
+| **Pipeline Configuration** | - ZO_PIPELINE_OFFSET_FLUSH_INTERVAL<br>- ZO_PIPELINE_REMOTE_REQUEST_TIMEOUT<br>- ZO_PIPELINE_WAL_SIZE_LIMIT<br>- ZO_PIPELINE_BATCH_SIZE<br>- ZO_PIPELINE_BATCH_TIMEOUT_MS<br>- ZO_PIPELINE_BATCH_SIZE_BYTES<br>- ZO_PIPELINE_BATCH_RETRY_INITIAL_DELAY_MS<br>- ZO_PIPELINE_BATCH_RETRY_MAX_DELAY_MS<br>- ZO_PIPELINE_MAX_FILE_SIZE_ON_DISK_MB |
+| **WAL and Buffer Configurations** | - ZO_WAL_WRITE_BUFFER_SIZE<br>- ZO_WAL_WRITE_QUEUE_SIZE |
+| **Search Group Configuration** | - O2_SEARCH_GROUP_BASE_SECS<br>- O2_SEARCH_GROUP_BASE_SPEED<br>- O2_SEARCH_GROUP_LONG_MAX_CPU<br>- O2_SEARCH_GROUP_SHORT_MAX_CPU<br>- O2_SEARCH_GROUP_LONG_MAX_CONCURRENCY<br>- O2_SEARCH_GROUP_SHORT_MAX_CONCURRENCY<br>- O2_SEARCH_GROUP_LONG_MAX_MEMORY<br>- O2_SEARCH_GROUP_SHORT_MAX_MEMORY<br>- O2_SEARCH_GROUP_USER_SHORT_MAX_CONCURRENCY<br>- O2_SEARCH_GROUP_USER_LONG_MAX_CONCURRENCY |
+| **AI Configuration** | - O2_AI_ENABLED<br>- O2_AI_API_URL<br>- O2_AI_MODEL<br>- O2_AI_PROVIDER<br>- O2_AI_API_KEY |
+| **Other Configurations** | - ZO_ROUTE_MAX_CONNECTIONS<br>- ZO_ENRICHMENT_TABLE_MERGE_INTERVAL<br>- ZO_DOWNSAMPLING_DOWNSAMPLING_INTERVAL<br>- ZO_DATAFUSION_FILE_STAT_CACHE_MAX_ENTRIES<br>- ZO_SCHEMA_MAX_FIELDS_TO_ENABLE_UDS |
diff --git a/docs/operator-guide/etcd_restore.md b/docs/operator-guide/etcd_restore.md
@@ -3,7 +3,10 @@ description: >-
   Restore a broken etcd cluster in OpenObserve by restarting pods, resetting
   data, and rejoining members using CLI and updated Helm configs.
 ---
-# Etcd Cluster Restore (Deprecated)
+# Etcd Cluster Restore (Removed)
+
+!!! warning "Removal notice"
+    Etcd support has been removed. Use NATS instead.
 
 Many users ran into the case only one of the 3 pods of etcd cluster can works. The other 2 pods always restart and can't back to work.
 

diff --git a/docs/sql-functions/full-text-search.md b/docs/sql-functions/full-text-search.md
@@ -25,6 +25,34 @@ This query filters logs from the `default` stream where the `k8s_pod_name` field
 <br>
 ![str_match](../images/sql-reference/str-match.png)
 
+---
+
+### `not str_match`
+**Syntax**: `not str_match(field, 'value')` <br>
+**Description**:<br>
+
+- Filters logs where the specified field does NOT contain the exact string value.
+- The match is case-sensitive.
+- Only logs that do not include the exact characters and casing specified will be returned.
+- Can be combined with other conditions using AND/OR operators.
+
+**Example**: <br>
+```sql
+SELECT * FROM "default" WHERE NOT str_match(k8s_app_instance, 'dev2')
+```
+![not str_match](../images/sql-reference/not-str-match.png)
+
+**Combining multiple NOT conditions with AND:**
+```sql
+SELECT * FROM "default" WHERE (NOT str_match(k8s_app_instance, 'dev2')) AND (NOT str_match(k8s_cluster, 'dev2'))
+```
+![not str_match with AND operator](../images/sql-reference/not-str-match-with-and.png)
+
+**Combining NOT conditions with OR:**
+```sql
+SELECT * FROM "default" WHERE NOT ((str_match(k8s_app_instance, 'dev2') OR str_match(k8s_cluster, 'dev2')))
+```
+![not str_match with OR operator](../images/sql-reference/not-str-match-with-or.png)
 
 ---
 ### `str_match_ignore_case`
@@ -46,7 +74,7 @@ SELECT * FROM "default" WHERE str_match_ignore_case(k8s_pod_name, 'MAIN-OPENOBSE
 This query filters logs from the `default` stream where the `k8s_pod_name` field contains any casing variation of `main-openobserve-ingester-1`, such as `MAIN-OPENOBSERVE-INGESTER-1`, `Main-OpenObserve-Ingester-1`, or `main-openobserve-ingester-1`.
 <br>
 ![str_match_ignore_case](../images/sql-reference/str-ignore-case.png)
-
+<br>
 ---
 
 ### `match_all`
@@ -69,6 +97,53 @@ This query returns all logs in the `default` stream where the keyword `openobser
 <br>
 ![match_all](../images/sql-reference/match-all.png)
 
+
+**More pattern support**
+The `match_all` function also supports the following patterns for flexible searching:
+
+- **Prefix search**: Matches keywords that start with the specified prefix:
+```sql
+SELECT * FROM "default" WHERE match_all('ab*')
+```
+- **Postfix search**: Matches keywords that end with the specified suffix:
+```sql
+SELECT * FROM "default" WHERE match_all('*ab')
+```
+- **Contains search**: Matches keywords that contain the substring anywhere:
+```sql
+SELECT * FROM "default" WHERE match_all('*ab*')
+```
+- **Phrase prefix search**: Matches keywords where the last term uses prefix matching:
+```sql
+SELECT * FROM "default" WHERE match_all('key1 key2*')
+```
+### `not match_all`
+**Syntax**: `not match_all('value')`
+**Description**: <br>
+
+- Filters logs by excluding records where the keyword appears in any field that has the Index Type set to Full Text Search in the stream settings. 
+- This function is case-insensitive and excludes matches regardless of the keyword's casing.
+- **Important**: Only searches fields configured as Full Text Search fields. Other fields in the record are not evaluated.
+- Provides significant performance improvements when used with indexed fields.
+
+**Example**:
+```sql
+SELECT * FROM "default" WHERE NOT match_all('foo')
+```
+This query returns all logs in the `default` stream where the keyword `foo` does NOT appear in any of the full-text indexed fields. Fields not configured for full-text search are ignored.
+
+**Combining NOT match_all with NOT str_match**:
+```sql
+SELECT * FROM "default" WHERE (NOT str_match(f1, 'bar')) AND (NOT match_all('foo'))
+```
+This query returns logs where field `f1` does NOT contain `bar` AND no full-text indexed field contains `foo`. In other words, it excludes records that match either condition.
+
+**Using NOT with OR conditions**:
+```sql
+SELECT * FROM "default" WHERE NOT (str_match(f1, 'bar') OR match_all('foo'))
+```
+This query returns logs where BOTH conditions are false: field `f1` does NOT contain `bar` AND no full-text indexed field contains `foo`. In other words, it excludes records that match either condition.
+
 ---
 
 ### `re_match`