Initial Version of nginx guide #4588
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,285 @@ | ||
| --- | ||
| slug: /use-cases/observability/clickstack/integrations/nginx | ||
| title: 'Monitoring Nginx with ClickStack' | ||
| sidebar_label: 'Nginx' | ||
| pagination_prev: null | ||
| pagination_next: null | ||
| description: 'Monitoring Nginx with ClickStack' | ||
| doc_type: 'guide' | ||
| --- | ||
|
|
||
| import Image from '@theme/IdealImage'; | ||
| import import_dashboard from '@site/static/images/clickstack/import-dashboard.png'; | ||
| import finish_import from '@site/static/images/clickstack/finish-import.png'; | ||
| import example_dashboard from '@site/static/images/clickstack/example-dashboard.png'; | ||
|
|
||
| # Monitoring Nginx with ClickStack {#nginx-clickstack} | ||
|
|
||
| ::::note[TLDR] | ||
| This guide shows you how to monitor nginx with ClickStack by configuring the OpenTelemetry collector to ingest nginx access logs. You'll learn how to: | ||
|
|
||
| - Configure nginx to output JSON-formatted logs | ||
| - Create a custom OTel collector configuration for log ingestion | ||
| - Deploy ClickStack with your custom configuration | ||
| - Use a pre-built dashboard to visualize nginx metrics (requests, errors, latency) | ||
|
|
||
| A demo dataset with 10,000 sample logs is provided to test the integration before connecting your production nginx instances. | ||
|
|
||
| Time Required: 5-10 minutes. | ||
| :::: | ||
|
|
||
| ## Prerequisites {#prerequisites} | ||
| - ClickStack instance running | ||
| - Existing nginx installation | ||
| - Access to modify nginx configuration files | ||
|
|
||
| ## Integration with existing nginx {#existing-nginx} | ||
|
|
||
| This section covers configuring your existing nginx installation to send logs to ClickStack by modifying the ClickStack OTel collector configuration. | ||
|
|
||
| <VerticalStepper> | ||
|
|
||
| ## Configure nginx log format {#configure-nginx} | ||
| First, configure nginx to output logs in JSON format for easier parsing. Add this log format definition to your nginx.conf: | ||
|
|
||
| ```json | ||
| http { | ||
| log_format json_combined escape=json | ||
| '{' | ||
| '"time_local":"$time_local",' | ||
| '"remote_addr":"$remote_addr",' | ||
| '"request_method":"$request_method",' | ||
| '"request_uri":"$request_uri",' | ||
| '"status":$status,' | ||
| '"body_bytes_sent":$body_bytes_sent,' | ||
| '"request_time":$request_time,' | ||
| '"upstream_response_time":"$upstream_response_time",' | ||
| '"http_referer":"$http_referer",' | ||
| '"http_user_agent":"$http_user_agent"' | ||
| '}'; | ||
|
|
||
| access_log /var/log/nginx/access.log json_combined; | ||
| error_log /var/log/nginx/error.log warn; | ||
| } | ||
| ``` | ||
|
|
||
| After making this change, reload nginx. | ||
|
|
||
| ## Create custom otel collector configuration {#custom-otel} | ||
|
|
||
| ClickStack allows you to extend the base OpenTelemetry Collector configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP. | ||
|
|
||
| Create a file named nginx-monitoring.yaml with the following configuration: | ||
|
|
||
| ```yaml | ||
| receivers: | ||
| filelog: | ||
| include: | ||
| - /var/log/nginx/access.log | ||
| - /var/log/nginx/error.log | ||
| start_at: end | ||
| operators: | ||
| - type: json_parser | ||
| parse_from: body | ||
| parse_to: attributes | ||
| - type: time_parser | ||
| parse_from: attributes.time_local | ||
| layout: '%d/%b/%Y:%H:%M:%S %z' | ||
| - type: add | ||
| field: attributes.source | ||
| value: "nginx" | ||
|
|
||
| service: | ||
| pipelines: | ||
| logs/nginx: | ||
| receivers: [filelog] | ||
| processors: | ||
| - memory_limiter | ||
| - transform | ||
| - batch | ||
| exporters: | ||
| - clickhouse | ||
| ``` | ||
|
|
||
| This configuration: | ||
| - Reads nginx logs from their standard locations | ||
| - Parses JSON log entries | ||
| - Extracts and preserves the original log timestamps | ||
| - Adds source: nginx attribute for filtering in HyperDX | ||
| - Routes logs to the ClickHouse exporter via a dedicated pipeline | ||
|
|
||
| ::::note | ||
| - You only define new receivers and pipelines in the custom config | ||
| - The processors (memory_limiter, transform, batch) and exporters (clickhouse) are already defined in the base ClickStack configuration - you just reference them by name | ||
| - The time_parser operator extracts timestamps from nginx's time_local field to preserve original log timing | ||
| - The pipelines route data from your receivers to the ClickHouse exporter via the existing processors | ||
| :::: | ||
|
|
||
| ## Configure ClickStack to load custom configuration {#load-custom} | ||
|
|
||
| To enable custom collector configuration in your existing ClickStack deployment, you must: | ||
|
|
||
| 1. Mount the custom config file at /etc/otelcol-contrib/custom.config.yaml | ||
| 2. Set the environment variable CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml | ||
| 3. Mount your nginx log directories so the collector can read them | ||
|
|
||
| Update your ClickStack deployment configuration to include these settings, this example uses docker compose. | ||
|
|
||
| ```yaml | ||
| services: | ||
| clickstack: | ||
| # ... existing configuration ... | ||
| environment: | ||
| - CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml | ||
| # ... other environment variables ... | ||
| volumes: | ||
| - ./nginx-monitoring.yaml:/etc/otelcol-contrib/custom.config.yaml:ro | ||
| - /var/log/nginx:/var/log/nginx:ro | ||
| # ... other volumes ... | ||
| ``` | ||
|
There was a problem hiding this comment. I think we should also provide a docker command example of doing this i.e. for the all in one image |
||
|
|
||
| ::::note | ||
| Ensure the ClickStack collector has appropriate permissions to read the nginx log files. In production, use read-only mounts (:ro) and follow the principle of least privilege. | ||
| :::: | ||
|
|
||
| ## Verifying Logs in ClickStack {#verifying-logs} | ||
| Once configured, log into HyperDX and verify logs are flowing: | ||
|
|
||
| 1. Navigate to the Logs view | ||
| 2. Filter by source:nginx to see only nginx logs | ||
| 3. Verify you see JSON-parsed log entries with fields like request_method, request_uri, status, etc. | ||
| </VerticalStepper> | ||
|
|
||
|
|
||
| ## Troubleshooting {#troubleshooting} | ||
|
|
||
| ### Custom config not loading {#troubleshooting-not-loading} | ||
|
|
||
| - Verify the environment variable CUSTOM_OTELCOL_CONFIG_FILE is set correctly | ||
|
There was a problem hiding this comment. add simple commands to verify these |
||
| - Check that the custom config file is mounted at /etc/otelcol-contrib/custom.config.yaml | ||
| - Verify the file is mounted as a file, not a directory: docker exec `<container>` ls -la /etc/otelcol-contrib/ | ||
|
|
||
| ### No logs appearing in HyperDX {#no-logs} | ||
|
|
||
| - Ensure nginx is writing JSON logs: tail -f /var/log/nginx/access.log | ||
| - Check the collector can read the logs: docker exec `<container>` cat /var/log/nginx/access.log | ||
| - Verify the effective config includes your filelog receiver: docker exec `<container>` cat /etc/otel/supervisor-data/effective.yaml | grep filelog | ||
| - Check for errors in the collector logs: docker exec `<container>` cat /etc/otel/supervisor-data/agent.log | ||
|
|
||
| ## Demo dataset {#demo-dataset} | ||
|
|
||
| For users who want to test the nginx integration before configuring their production systems, we provide a sample dataset of pre-generated nginx access logs with realistic traffic patterns. | ||
|
|
||
| <VerticalStepper> | ||
|
|
||
| ## Using the Sample Dataset {#using-data} | ||
|
|
||
| 1. [Download](../../../../../static/examples/nginx-sample-logs.json) and place the sample file in `/tmp/nginx-demo/access.log` | ||
|
There was a problem hiding this comment. I dont think we need to be prescriptive about the location i.e. can't it just be in the folder we launch from? |
||
|
|
||
| 2. **Create a test collector config** (`nginx-demo.yaml`): | ||
|
|
||
| ```yaml | ||
| receivers: | ||
| filelog: | ||
| include: | ||
| - /tmp/nginx-demo/access.log | ||
| start_at: beginning # Read from beginning for demo data | ||
| operators: | ||
| - type: json_parser | ||
| parse_from: body | ||
| parse_to: attributes | ||
| - type: time_parser | ||
| parse_from: attributes.time_local | ||
| layout: '%d/%b/%Y:%H:%M:%S %z' | ||
| - type: add | ||
| field: attributes.source | ||
| value: "nginx-demo" | ||
|
|
||
| service: | ||
| pipelines: | ||
| logs/nginx-demo: | ||
| receivers: [filelog] | ||
| processors: | ||
| - memory_limiter | ||
| - transform | ||
| - batch | ||
| exporters: | ||
| - clickhouse | ||
| ``` | ||
|
|
||
| 3. **Run ClickStack with the demo config:** | ||
|
|
||
| ```bash | ||
| docker run --name clickstack-demo \ | ||
| -p 8080:8080 -p 4317:4317 -p 4318:4318 \ | ||
| -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \ | ||
| -v "$(pwd)/nginx-demo.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \ | ||
| -v /tmp/nginx-demo:/tmp/nginx-demo:ro \ | ||
| docker.hyperdx.io/hyperdx/hyperdx-all-in-one:latest | ||
|
There was a problem hiding this comment. this wont work surely? we need to mount in the log file as well |
||
| ``` | ||
|
|
||
| 4. **Verify in HyperDX:** | ||
|
|
||
| - Navigate to the Logs view | ||
| - Filter by `source:nginx-demo` | ||
| - Set time range to last 24 hours | ||
| - You should see ~10,000 log entries | ||
|
|
||
|
|
||
| You can query the data to verify patterns: | ||
|
|
||
| ```shell | ||
|
|
||
| docker exec clickstack-demo clickhouse-client --query " | ||
| SELECT | ||
| toHour(Timestamp) as hour, | ||
| count() as total_logs, | ||
| countIf(toInt32(LogAttributes['status']) >= 500) as server_errors, | ||
| countIf(toInt32(LogAttributes['status']) >= 400 AND toInt32(LogAttributes['status']) < 500) as client_errors, | ||
| round(avg(toFloat64OrNull(LogAttributes['request_time'])), 3) as avg_response_time | ||
| FROM default.otel_logs | ||
| WHERE LogAttributes['source'] = 'nginx-demo' | ||
| GROUP BY hour | ||
| ORDER BY hour | ||
| " | ||
| ``` | ||
|
|
||
| ::::note | ||
| The demo dataset uses dynamic timestamps (last 24 hours from generation). The traffic patterns are intentionally dramatic to make visualizations clear and obvious in dashboards. | ||
| :::: | ||
| </VerticalStepper> | ||
|
|
||
| ## Dashboards and visualization {#dashboards} | ||
|
|
||
| To help you get started monitoring nginx with ClickStack, we provide a pre-built dashboard with essential nginx metrics and visualizations. | ||
|
There was a problem hiding this comment. essential visualizations for nginx logs |
||
|
|
||
| ### Import Pre-built Dashboard {#import-dashboard} | ||
| [Download](../../../../../static/examples/example-dashboard.json) the dashboard configuration. | ||
|
|
||
| 1. Open HyperDX and navigate to the Dashboards section. | ||
| 2. Click "Import Dashboard" in the upper right corner under the elipses. | ||
|
|
||
| <Image img={import_dashboard} alt="Import Dashboard"/> | ||
|
|
||
| 3. Upload the Example Dashboard.json file and click finish import. | ||
|
|
||
| <Image img={finish_import} alt="Finish Import"/> | ||
|
|
||
| 4. The dashboard will be created with all visualizations pre-configured. | ||
|
|
||
| <Image img={example_dashboard} alt="Example Dashboard"/> | ||
|
|
||
| ### Customizing the Dashboard {#customizing} | ||
|
|
||
| The dashboard can be customized to fit your specific needs: | ||
| - Filter by specific endpoints, methods, or other log attributes | ||
| - Change time buckets for different zoom levels | ||
| - Create additional charts for metrics like: | ||
| - Top requested endpoints | ||
| - Geographic distribution (if using IP geolocation) | ||
| - User agent analysis | ||
| - Bytes sent/received trends | ||
|
|
||
| ## Next Steps {#next-steps} | ||
| If you want to explore further, here are some next steps to experiment with your dashboard | ||
|
|
||
| - Set up alerts for critical metrics (error rates, latency thresholds) | ||
| - Create additional dashboards for specific use cases (API monitoring, security events) | ||
| - Correlate with other data sources by adding traces and metrics to the same dashboard | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"version":"0.1.0","name":"Example Dashboard","tiles":[{"id":"tp56x","x":0,"y":0,"w":8,"h":10,"config":{"name":"Requests over time","source":"Logs","displayType":"line","granularity":"auto","select":[{"aggFn":"count","aggCondition":"","aggConditionLanguage":"lucene","valueExpression":""}],"where":"","whereLanguage":"lucene","groupBy":""}},{"id":"yrkd9","x":8,"y":0,"w":8,"h":10,"config":{"name":"Errors Over Time","source":"Logs","displayType":"stacked_bar","granularity":"auto","select":[{"aggFn":"count","aggCondition":"toInt32(LogAttributes['status']) >= 400","aggConditionLanguage":"sql","valueExpression":""}],"where":"","whereLanguage":"lucene","groupBy":"LogAttributes['status']"}},{"id":"pha6m","x":8,"y":10,"w":8,"h":10,"config":{"name":"Request times (90,95,99 percentile)","source":"Logs","displayType":"line","granularity":"5 minute","select":[{"aggFn":"quantile","aggCondition":"","aggConditionLanguage":"lucene","valueExpression":"LogAttributes['request_time']"},{"aggFn":"quantile","aggCondition":"","aggConditionLanguage":"lucene","valueExpression":"LogAttributes['request_time']"},{"aggFn":"quantile","aggCondition":"","aggConditionLanguage":"lucene","valueExpression":"LogAttributes['request_time']"}],"where":"","whereLanguage":"lucene"}},{"id":"bwkhq","x":16,"y":10,"w":8,"h":10,"config":{"name":"Status code counts","source":"Logs","displayType":"table","granularity":"auto","select":[{"aggFn":"count","aggCondition":"","aggConditionLanguage":"lucene","valueExpression":""}],"where":"","whereLanguage":"lucene","groupBy":"LogAttributes['status']"}},{"id":"17fl0x","x":16,"y":0,"w":8,"h":10,"config":{"name":"Status codes over time","source":"Logs","displayType":"line","granularity":"auto","select":[{"aggFn":"count","aggCondition":"","aggConditionLanguage":"lucene","valueExpression":""}],"where":"","whereLanguage":"lucene","groupBy":"LogAttributes['status']"}},{"id":"9eb6o","x":0,"y":10,"w":8,"h":10,"config":{"name":"Average upstream response time","source":"Logs","displayType":"line","granularity":"auto","select":[{"aggFn":"avg","aggCondition":"","aggConditionLanguage":"lucene","valueExpression":"LogAttributes['upstream_response_time']"}],"where":"","whereLanguage":"lucene"}}],"filters":[]} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indicate where this is often found