Document existing practices for responding to AI-Slop

[taladrane]

Draft and review a comprehensive existing practices used by open source maintainers to respond effectively to AI-generated low-quality vulnerability reports, aka "AI-slop." Tasks include:

• Aggregate relevant examples and pain points from existing projects
• Summarize lessons learned and challenges from vulnerability reporting platforms
• Guidance for reducing personal attacks and providing consistent project responses
• Invite maintainers to share experience and feedback during document drafting

Google Doc: https://docs.google.com/document/d/1csseaiMVQeILSPjx3BvpCBH88PifgPf_ebXVKD5DIOs/edit?tab=t.0

Deliverable: A summary of research of existing practices to serve as a basis for the guide of recommendations and best practices

[oej]

In the WG meeting we suggested that OpenSSF (this wg and the AI/ML wg at least) invites to an open meeting to get feedback on the current state from security teams in OS projects and other maintainers.

[ppkarwasz]

Log4j Bug Bounty Program (YesWeHack)

Within the Log4j Bug Bounty Program, we have observed 60+ clear examples of AI-slop. While the platform does not allow us to publicly share these reports, we may be able to provide anonymized examples after internal discussion.

Our primary pain points are:

• Lack of effective triage. Triagers are understaffed and overwhelmed. When triage does occur, it often happens only after several weeks. We review all reports within 72 hours (usually the same day).
• “Technically true, but not a vulnerability”. Many reports describe real, non-hallucinated behavior. However, the vast majority do not meet the definition of a vulnerability under our threat model.

Each such report typically requires 3–4 hours of work, because we do not simply reply with “This is not a vulnerability.” Instead, we aim to provide a reasoned and educational response:

• Cite relevant sections of our threat model
• Look for comparable CVEs or industry precedent
• Review the response internally before posting
• Personally I run the final text through an LLM to remove frustration or unintended aggressive tone

One interesting observation: despite the high volume, we rarely receive overtly hostile or abusive follow-ups. Many reports appear to be fully AI-generated, including overly apologetic language (“I am very sorry, but…”), which likely dampens personal attacks. In at least one case, the LLM concluded itself that the report did not constitute a vulnerability, but nevertheless the user posted the comment.

We would be happy to contribute anonymized examples and lessons learned to a best-practices document.

[ctcpip]

dupe of #178 ?