[CrowdStrike]: Vulnerability Data Stream Fails on Pagination Due to Expired 'after' Token

[andrewkroh]

Integration Name

CrowdStrike [crowdstrike]

Dataset Name

crowdstrike.vulnerability

Integration Version

1.77.0

Agent Version

8.18.3

Agent Output Type

elasticsearch

Elasticsearch Version

8.18.3

OS Version and Architecture

Agentless

Software/API Version

No response

Error Message

{
  "meta": {
    "query_time": 0.065321854,
    "pagination": {
      "limit": 400,
      "total": 0,
      "after": ""
    },
    "powered_by": "spapi",
    "trace_id": "<uuid>"
  },
  "resources": [],
  "errors": [
    {
      "code": 404,
      "message": "Search context expired, 'after' key no longer valid"
    }
  ]
}

Event Original

No response

What did you do?

The CrowdStrike vulnerability data stream is configured to collect vulnerabilities.

What did you see?

The integration fails during pagination with the error "Search context expired, 'after' key no longer valid". This happens when the time between paginated requests to the /spotlight/queries/vulnerabilities/v1 endpoint exceeds 120 seconds.

IIUC, the current implementation fetches a list of vulnerability IDs and then fetches the full details for each vulnerability, which can take longer than the token's lifetime.

What did you expect to see?

No errors.

Anything else?

The Crowdstrike documentation states this about the returned after token:

Token used with the limit parameter to manage pagination of results. On your first request, don't provide an after token. On subsequent requests, provide the after token from the previous response to continue from that place in the results.
Tokens expire 120 seconds after a call is made.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[NateUT99]

I have seen the same issue here and I think the integration can be updated as I think it is doing an extra step that is not necessary. Instead of querying the "/spotlight/queries/vulnerabilities/v1" endpoint to get a list of 400 IDs, formatting them properly, and then sending them to "/spotlight/entities/vulnerabilities/v2", you can just send a request to the "/spotlight/combined/vulnerabilities/v1" endpoint and receive up to 5000 full records at a time (supports FQL and pagination).

We have noticed "vulnerability drift" between CrowdStrike and what we had ingested into Elastic, and found instances where documents seemed to be getting "lost". I built a workflow in Torq that uses the combined endpoint to gather the data and submit it via the Bulk API that has been working great so far. More than happy to share any other data that might be useful.

Some additiional thoughts on this integration overall...
I will say it would be nice to be able to also be able to choose one or more (or none) facets that you want the data to return (host_info, remediation, cve, evaluation_logic) as well as bring in the "remediation" and "evaluation_logic" data seperately so there is not so much duplication. These details can be gathered using the "/spotlight/entities/remediations/v2" and "/spotlight/combined/evaluation-logic/v1" respectively. This would better align to CrowdStrike's best practice recommendation for extracting vulnerability data. I have an open request with them to add a simliar endpoint so CVE data can be pulled seperately (and joined afterwards using the ids provided in the vulnerability document.

Host_info is already available in the crowdstrike.host dataset and can be joined using the aid, so that may be redudant data as well.

[NateUT99]

I have seen the same issue here and I think the integration can be updated as I think it is doing an extra step that is not necessary. Instead of querying the "/spotlight/queries/vulnerabilities/v1" endpoint to get a list of 400 IDs, formatting them properly, and then sending them to "/spotlight/entities/vulnerabilities/v2", you can just send a request to the "/spotlight/combined/vulnerabilities/v1" endpoint and receive up to 5000 full records at a time (supports FQL and pagination).

We have noticed "vulnerability drift" between CrowdStrike and what we had ingested into Elastic, and found instances where documents seemed to be getting "lost". I built a workflow in Torq that uses the combined endpoint to gather the data and submit it via the Bulk API that has been working great so far. More than happy to share any other data that might be useful.

Some additiional thoughts on this integration overall... I will say it would be nice to be able to also be able to choose one or more (or none) facets that you want the data to return (host_info, remediation, cve, evaluation_logic) as well as bring in the "remediation" and "evaluation_logic" data seperately so there is not so much duplication. These details can be gathered using the "/spotlight/entities/remediations/v2" and "/spotlight/combined/evaluation-logic/v1" respectively. This would better align to CrowdStrike's best practice recommendation for extracting vulnerability data. I have an open request with them to add a simliar endpoint so CVE data can be pulled seperately (and joined afterwards using the ids provided in the vulnerability document.

Host_info is already available in the crowdstrike.host dataset and can be joined using the aid, so that may be redudant data as well.

@navnit-elastic

Just wanted to provide a quick update on some of my comments. I had a conversation with CrowdStrike about the use of the "/spotlight/combined/evaluation-logic/v1" endpoint and why it only supported 400 records at a time. They stated that they no longer recommend using this endpoint, and to just pull that data as part of the vulnerability record via the /spotlight/combined/vulnerabilities/v1 endpoint if it is important.

I have also noticed that there are some additional fields present in the vulnerability host_info that are not part of the "main" host dataset in CrowdStrike (like asset_criticality, asset_roles, etc).

[navnit-elastic]

@NateUT99, Thanks for the update. The details you provided are very useful.
It would be good to migrate to the "/spotlight/combined/vulnerabilities/v1" endpoint with the option to choose which facets to return. However, I am not sure about retrieving "remediation" and "evaluation_logic" data separately, as that could be confusing—users will have multiple choices to fetch those.

Update:
The combined endpoint does not capture changes to remediation details. Hence it would make sense to bring remediation data separately.

Difference between API response: "/spotlight/entities/vulnerabilities/v2" -> "/spotlight/combined/vulnerabilities/v1"

@@ -8,13 +8,16 @@
   ],
   "meta": {
     "pagination": {
-      "expires_at": 0,
+      "after": "string",
       "limit": 0,
-      "offset": "string",
       "total": 0
     },
     "powered_by": "string",
     "query_time": 0,
+    "quota": {
+      "total": 0,
+      "used": 0
+    },
     "trace_id": "string"
   },
   "resources": [
@@ -174,16 +177,6 @@
         ]
       },
       "id": "string",
-      "network_scan": [
-        {
-          "network_scan_info": [
-            {
-              "scan_id": "string",
-              "scan_time": "string"
-            }
-          ]
-        }
-      ],
       "remediation": {
         "entities": [
           {

[navnit-elastic]

The linked PR introduces /spotlight/combined/vulnerabilities/v1 endpoint to fetch vulnerability data. This will reduce the chances of page token being expired. Added troubleshooting note for this failure with the possible resolution.