feat: Add Lighthouse recommendations and caching

[Chan9390]

Context

Lighthouse provides a chat interface to interact with findings. This PR provides recommendations and insights for cloud security findings of most recent scans.

Description

This PR adds the following:

• Create a summary of recent completed scans
• Generate recommendation on the next steps to improve cloud security
• Cache scan summary, recommendation (and recommendation banner heading) and answers to suggested questions in Valkey

With this PR, NextJS directly integrates with Valkey. It stores data with tenant key in it. All valkey keys have the following format: _lighthouse:<TENANTUUID>:<KEYNAME> (ex: _lighthouse:70bd85f3-6446-4bb5-bd68-ad52264fcc87:processed_scan_ids)

Different keys and their purpose:

• processed_scan_ids - Most recent scans IDs that were processed
• scan-summary - Contains scan summary of scans completed in last 24 hours. It highlights newly detected issues in the last scans ordered by severity. It also list top 10 issues based on severity.
• recommendations - Short title that will be displayed in banner
• cached-messages:recommendation - The long description & explanation showed when user clicks on banner
• cached-messages:question_1 - Answer to suggested question 1
• cached-messages:question_2 - Answer to suggested question 2
• cached-messages:question_3 - Answer to suggested question 3
• cached-messages:question_4 - Answer to suggested question 4

graph TD
    A[Start: User refreshes overview page] --> B{Is Lighthouse configured?}
    
    B -->|No| C["Print banner: Enable Lighthouse to Secure Your Cloud with AI Insights"]
    B -->|Yes| D{Are there completed scans in last 24 hours?}
    
    D -->|No| E{Does recommendations key exist in valkey?}
    D -->|Yes| F{Are scan IDs present in processed_scan_ids key?}
    
    E -->|No| G[No banner displayed]
    E -->|Yes| H[Display cached recommendation banner]
    
    F -->|Yes| H
    F -->|No| I["Print banner: Lighthouse is reviewing your findings for insights"]
    
    I --> J[Start async background processing]
    
    J --> K["Create summary:<br/>- Newly detected failed findings in last 24h<br/>- Top 10 failed findings by severity"]
    
    K --> L[Generate single recommendation for banner]
    
    L --> M[Store recommendation in <b>recommendations</b> key]
    
    M --> N[Store full recommendation in <b>cached-messages:recommendation</b> key]
    
    N --> O[Generate responses to suggested questions and cache in valkey]
    
    O --> P[Add scan IDs to <b>processed_scan_ids</b> key]
    
    P --> H
    
    C --> Q[End]
    G --> Q
    H --> Q

Loading

How to test

Create a new instance of Prowler (without Lighthouse configuration). You should find a banner as follows:

Without adding any providers, configure valid lighthouse keys. The dashboard shouldn't have any banner (as there's no completed scan in last 24 hours)

Add a provider and then initiate a scan. The dashboard would have the following banner:

Once the processing is done, you would have a banner with recommendation from your environment (the recommendation would differ based on your findings):

When you click on the banner, it will take you to a chat interface with detailed information about the recommendation:

You can then continue interacting with the recommendation:

Checklist

• Are there new checks included in this PR? No
  • If so, do we need to update permissions for the provider? Please review this carefully.
• Review if the code is being covered by tests.
• Review if code is being documented following this specification https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings
• Review if backport is needed.
• Review if is needed to change the Readme.md
• Ensure new entries are added to CHANGELOG.md, if applicable.

API

• Verify if API specs need to be regenerated.
• Check if version updates are required (e.g., specs, Poetry, etc.).
• Ensure new entries are added to CHANGELOG.md, if applicable.

License

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[github-actions]

⚠️ Changes detected in the following folders without a corresponding update to the CHANGELOG.md:

• ui

Please add an entry to the corresponding CHANGELOG.md file to maintain a clear history of changes.

[alejandrobailo]

Please @Chan9390 , can you add the CHANGELOG.md and resolve the conflicts with master