TT-15462 - Track 404s with Pump: feat: send 404 requests to analytics/Pump when track_404_logs enabled

[sedkis]

When track_404_logs and enable_analytics are both true, 404 responses
now create analytics records that flow to Tyk Pump and appear in
Prometheus metrics. This provides visibility into unmatched requests
for debugging API loading issues and monitoring traffic patterns.

• Add Gateway reference to proxyMux for analytics access
• Create analytics records in handle404() with 404 status and tags
• Respect existing analytics controls (enable_analytics, ignored_ips)
• Add comprehensive test coverage for 404 analytics recording
• Maintain backwards compatibility with existing configurations

Ticket Details

TT-15462

Status	In Dev
Summary	Generate traffic logs for requests generating HTTP 404

Generated at: 2025-11-11 20:02:53

[github-actions]

API Changes

no api changes detected

[probelabs]

🔍 Code Analysis Results

This PR introduces a feature to send analytics events for HTTP 404 (Not Found) responses. When both track_404_logs and enable_analytics are enabled in the gateway configuration, requests to non-existent paths will generate an analytics record. This provides visibility into unmatched requests, aiding in debugging API configurations and monitoring traffic patterns. The implementation respects existing analytics controls, such as the ignored_ips list, and includes comprehensive tests to ensure correctness.

Files Changed Analysis

• gateway/api_loader.go: A single line was added to pass the Gateway instance to the proxyMux. This is a crucial dependency injection that gives the 404 handler access to the gateway's analytics system.
• gateway/proxy_muxer.go: This file contains the core logic. The proxyMux struct was updated to hold a reference to the Gateway. The handle404 function was extended to check the relevant configuration flags and, if enabled, construct and send an analytics.AnalyticsRecord for the 404 event. The record is tagged with "404" and "path-not-found".
• gateway/proxy_muxer_test.go: A new test suite, TestHandle404WithAnalytics, was added. It includes three distinct scenarios: verifying that no record is created when 404 tracking is disabled, confirming a record is created when enabled, and ensuring requests from ignored IP addresses are correctly skipped.

Architecture & Impact Assessment

• What this PR accomplishes: It enhances observability by capturing unmatched requests (404s) in the analytics pipeline. Previously, these requests were only logged to the console, making them difficult to aggregate and analyze. Now, they can be processed by Tyk Pump and sent to monitoring systems like Prometheus or Elasticsearch.

• Key technical changes introduced:

  1. The proxyMux router now holds a reference to the Gateway instance, allowing it to access the analytics processor.
  2. The handle404 function is updated to conditionally create and dispatch an AnalyticsRecord for 404 responses.
  3. The feature is gated by the existing track_404_logs and enable_analytics flags, ensuring backward compatibility and opt-in behavior.

• Affected system components:

  • Tyk Gateway: The proxyMux component, responsible for handling unmatched routes, is directly modified.
  • Tyk Pump & Analytics Pipeline: This change introduces a new type of analytics event. Downstream systems will receive records for 404s, which lack API-specific metadata (like api_id or org_id).

• Flow Diagram:

  sequenceDiagram
      participant Client
      participant Gateway (proxyMux)
      participant Analytics System
  
      Client->>Gateway (proxyMux): GET /unmatched/path
      Gateway (proxyMux)->>Gateway (proxyMux): No route found, trigger handle404
      alt track_404_logs AND enable_analytics are true
          Gateway (proxyMux)->>Gateway (proxyMux): Log error to console
          Gateway (proxyMux)->>Gateway (proxyMux): Create AnalyticsRecord for 404
          Gateway (proxyMux)->>Analytics System: RecordHit(record)
      end
      Gateway (proxyMux)-->>Client: HTTP 404 Not Found
  

  Loading

Scope Discovery & Context Expansion

• The changes are well-contained within the gateway module, primarily affecting proxy_muxer.go and its initialization in api_loader.go.
• The primary impact extends to the analytics data pipeline. The introduction of API-less analytics records means that downstream consumers (e.g., Tyk Pump, custom dashboards) must be prepared to handle records where fields like APIID, APIName, and OrgID are empty. This could affect filtering, aggregation, and reporting if not accounted for.
• The logic for creating the AnalyticsRecord in handle404 is implemented manually. This duplicates the pattern of record creation found elsewhere in the gateway for successful requests and other errors. This introduces a potential maintenance issue, as future changes to the AnalyticsRecord struct or its population logic would need to be replicated here. A centralized factory or helper function for creating analytics records could mitigate this.

Metadata

• Review Effort: 2 / 5
• Primary Label: feature

---

Powered by Visor from Probelabs

Last updated: 2025-11-11T20:07:02.235Z | Triggered by: synchronize | Commit: bb111a5

💡 TIP: You can chat with Visor using /visor ask <your question>

[probelabs]

🔍 Code Analysis Results

Security Issues (3)

Severity	Location	Issue
🟢 Info	gateway/proxy_muxer.go:218	The client IP address, used for the `ignored_ips` check, is determined using `request.RealIP`. This function trusts HTTP headers like `X-Real-IP` and `X-Forwarded-For`. If Tyk is deployed behind a reverse proxy or load balancer that does not properly sanitize these headers, a client could spoof their IP address. This would allow them to bypass the `ignored_ips` filter, causing their requests to be logged to analytics when they should be ignored. 💡 Suggestion Add documentation to highlight that for the `ignored_ips` feature to be effective for 404 logging, the networking infrastructure in front of Tyk must be configured to correctly set or strip `X-Forwarded-For` and `X-Real-IP` headers.
🟡 Warning	gateway/proxy_muxer.go:230	The analytics record for 404 requests includes the raw URL query string (`r.URL.RawQuery`). If sensitive information such as API keys, session tokens, or personally identifiable information (PII) is passed in the URL, it will be persisted in the analytics store. This could lead to sensitive data exposure in logs and analytics systems. 💡 Suggestion Implement a scrubbing mechanism to redact sensitive query parameters from the `RawRequest` field before recording the analytics data. This could be based on a configurable blocklist of parameter names (e.g., 'token', 'password', 'apikey').
🟡 Warning	gateway/proxy_muxer.go:215-269	This feature introduces analytics recording for every 404 request when `track_404_logs` is enabled. Since these requests do not match any defined API, they are not subject to API-level rate limiting or quota management. A malicious actor could launch a high-volume denial-of-service attack by sending requests to random, non-existent endpoints, potentially overwhelming the gateway's analytics buffer, Redis, and the Tyk Pump infrastructure. 💡 Suggestion Consider introducing a global rate limit specifically for logging 404 errors to prevent the analytics system from being flooded by unauthenticated traffic.

Architecture Issues (2)

Severity	Location	Issue
🟠 Error	gateway/proxy_muxer.go:232-264	The manual creation of `analytics.AnalyticsRecord` duplicates a significant amount of logic for populating the record from an `http.Request`, which is also present in `gateway/handler_success.go` in the `RecordHit` method. This large block of code represents a missed opportunity for abstraction and reuse, leading to maintainability issues. 💡 Suggestion To improve reusability and reduce code duplication, create a helper function that constructs a base `AnalyticsRecord` from an `*http.Request`. This function would populate common fields (e.g., `Method`, `Host`, `Path`, `UserAgent`, `TimeStamp`). Both `handle404` and `SuccessHandler.RecordHit` could then call this helper and add their context-specific data (e.g., `ResponseCode`, `Tags`, `APIID`). This would centralize the record creation logic, making it more maintainable and consistent.
🟡 Warning	gateway/api_loader.go:1068	The `proxyMux` now holds a reference to the entire `Gateway` object. This increases coupling between the routing component (`proxyMux`) and the main gateway controller, granting it access to more state than necessary for its task of recording 404 analytics. 💡 Suggestion For a more loosely coupled design, consider passing a more focused dependency to `proxyMux` instead of the entire `Gateway` object. An interface exposing only the necessary methods (e.g., `RecordHit`, `GetAnalyticsConfig`) would follow the Principle of Least Privilege and reduce coupling, making `proxyMux` easier to test and reason about in isolation.

Performance Issues (1)

Severity	Location	Issue
🟡 Warning	gateway/proxy_muxer.go:221	The `StoreAnalytics` function, which is called for every 404 request when this feature is enabled, performs a linear scan over the `AnalyticsConfig.IgnoredIPs` slice. This operation has a time complexity of O(n), where n is the number of IPs in the ignore list. For configurations with a large number of ignored IPs, this can introduce latency and CPU overhead on a potentially hot path (handling unmatched requests). 💡 Suggestion To optimize the IP lookup, convert the `IgnoredIPs` slice into a map (e.g., `map[string]struct{}`) when the gateway configuration is loaded. This will change the lookup complexity to O(1), providing a significant performance improvement for users with large IP ignore lists.

✅ Quality Check Passed

No quality issues found – changes LGTM.

✅ Dependency Check Passed

No dependency issues found – changes LGTM.

Connectivity Issues (1)

Severity	Location	Issue
🟡 Warning	gateway/proxy_muxer.go:269	A high volume of 404 requests will trigger a call to `m.gw.Analytics.RecordHit` for each request. This function sends data to a buffered channel which, if full, will block the request-handling goroutine. This can lead to performance degradation and resource exhaustion under heavy load or a DoS attack, as the analytics subsystem applies backpressure to the critical request path. This also increases the load on the underlying analytics storage, typically Redis. 💡 Suggestion To ensure gateway stability, the call to send analytics data should be non-blocking. Modify the underlying `RecordHit` function to use a `select` statement with a `default` case. This will drop the analytic record if the channel is full, preventing the analytics pipeline from blocking request handlers during high-traffic scenarios.

---

Powered by Visor from Probelabs

Last updated: 2025-11-11T20:07:03.691Z | Triggered by: synchronize | Commit: bb111a5

💡 TIP: You can chat with Visor using /visor ask <your question>

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
98.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud