Skip to content

Add filtering out of documents with error.message from latest indexes #15722

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Add filtering out of documents with error.message from latest indexes #15722

wants to merge 2 commits into from
+142 −26

Conversation

@alexreal1314
Copy link
Contributor

@alexreal1314 alexreal1314 commented Oct 22, 2025 •
edited
Loading

Proposed commit message

Purpose of this PR is to filter out documents who contain error.message in the source indexes for all supported 3p integration and our native integration. This is in order to improve UI experience by filtering them out from the cdr workflows.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

  1. install one of the changed integrations - for example with wiz
  2. toggle Collect Wiz logs via API
  3. type in the wrong client id or client secret
  4. add an agent and let the integration ingest data
  5. vulnerability source index logs-wiz.vulnerability-default should contain a document with error.message field but no documents should be in the dest index security_solution-wiz.vulnerability_latest-v2.

Screenshots

source index - wiz:
image

dest index - wiz:
image

@alexreal1314 alexreal1314 force-pushed the 11031-latest-index-error-filtering branch from 005bfac to 4f4e661 Compare October 22, 2025 15:41
@alexreal1314 alexreal1314 changed the title add filtering out of documents with error.message from latest misconf… Add filtering out of documents with error.message from latest indexes Oct 22, 2025
@andrewkroh andrewkroh added Integration:google_scc Google Security Command Center Integration:wiz Wiz Integration:cloud_security_posture Security Posture Management Integration:rapid7_insightvm Rapid7 InsightVM Integration:m365_defender Microsoft Defender XDR Integration:qualys_vmdr Qualys VMDR Integration:aws AWS Integration:tenable_io Tenable Vulnerability Management Integration:microsoft_defender_endpoint Microsoft Defender for Endpoint labels Oct 22, 2025
@alexreal1314 alexreal1314 force-pushed the 11031-latest-index-error-filtering branch from 4f4e661 to 7bc623c Compare October 22, 2025 20:46
@alexreal1314 alexreal1314 added the Integration:microsoft_defender_cloud Microsoft Defender for Cloud label Oct 22, 2025
@elastic-vault-github-plugin-prod
Copy link

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@alexreal1314 alexreal1314 marked this pull request as ready for review October 23, 2025 06:01
@alexreal1314 alexreal1314 requested review from a team as code owners October 23, 2025 06:01
@alexreal1314 alexreal1314 self-assigned this Oct 23, 2025
@efd6
Copy link
Contributor

efd6 commented Oct 23, 2025

Is there an issue for this explaining why it's necessary?

@alexreal1314 alexreal1314 marked this pull request as draft October 23, 2025 06:45
@alexreal1314 alexreal1314 force-pushed the 11031-latest-index-error-filtering branch from 7bc623c to 256f44a Compare October 23, 2025 07:35
@alexreal1314
add filtering out of documents with error.message from latest misconf…
a672def 
…iguration and vulnerability index

change is added to all supported native and 3p integrations
@alexreal1314 alexreal1314 force-pushed the 11031-latest-index-error-filtering branch from 256f44a to a672def Compare October 23, 2025 07:57
@alexreal1314
Copy link
Contributor Author

Is there an issue for this explaining why it's necessary?

This PR closes this issue, the intention it to maintain clean latest indexes and prevent documents with errors reaching them.

@alexreal1314 alexreal1314 force-pushed the 11031-latest-index-error-filtering branch from 5fdc011 to b65571b Compare October 23, 2025 08:51
@alexreal1314 alexreal1314 added Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Cloud Security Cloud Security team [elastic/cloud-security-posture] labels Oct 23, 2025
@elasticmachine
Copy link

💚 Build Succeeded

History

cc @alexreal1314

@efd6
Copy link
Contributor

efd6 commented Oct 23, 2025

I think this should either be conditional on user configuration or done dynamically rather than my mutating the index.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@alexreal1314 alexreal1314

Labels

Integration:aws AWS Integration:cloud_security_posture Security Posture Management Integration:google_scc Google Security Command Center Integration:m365_defender Microsoft Defender XDR Integration:microsoft_defender_cloud Microsoft Defender for Cloud Integration:microsoft_defender_endpoint Microsoft Defender for Endpoint Integration:qualys_vmdr Qualys VMDR Integration:rapid7_insightvm Rapid7 InsightVM Integration:tenable_io Tenable Vulnerability Management Integration:wiz Wiz Team:Cloud Security Cloud Security team [elastic/cloud-security-posture] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@alexreal1314 @efd6 @elasticmachine @andrewkroh