V2 Consideration: Matching Arrays with sigma

[humpalum]

Matching Arrays with sigma

Problem

Sigma currently lacks the capability to match values within arrays in log events. This limitation restricts its effectiveness in matching complex log data that includes arrays. For example:

{
    "message": "Some Message",
    "process": "Some Process",
    "connections": [
        "123.1.1.1",
        "123.1.2.2",
        "123.3.3.3"
    ]
}

It is not possible to match a value in that array for example with:

connection: "123.1.1.1"

Proposed Solution

Enhance Sigma with key modifiers for array matching, such as |arrayAny, |arrayAll, |arrayOne, and |arrayNone. This would allow for more flexible and powerful matching scenarios.

connections|arrayAny: 123.1.1.1

This would return true if any element in the connections array matches "123.1.1.1".

connections|arrayAll|startswith: 123

This would return true if all elements in the array start with "123".

--> Inspired by https://expr-lang.org/docs/Language-Definition#array-functions

Further Enhancement

Even further we could allow matching of nested keys on arrays of objects:

{
    "message": "Some Message",
    "process": "Some Process",
    "connections": [
        {"protocol":"TCP","ip":"123.1.1.1"},
        {"protocol":"TCP","ip":"123.1.2.2"},
        {"protocol":"UDP","ip":"123.3.3.3"}
    ]
}

Allowing to "chain" multiple keys would allow to match on array of objects. Taking above example, we could write detections like this:

# Check if there is one UDP connection beginning with 123
connections|arrayAny:
    protocol: "UDP"
    ip|startswith: "123"
# --> Would return true

or

# Check if all connections beginning with 123 are using TCP
connections|arrayAll:
    protocol: "TCP"
    ip|startswith: "123"
# --> Would return false

I can see how this is a big change of current sigma philosophy and I am not sure how implementable this is in backends but taking elasticsearch as an example, it would work on nested objects: Nested Fields

[norbert791]

I would like to bump this topic and add a few things from myself.

Rationale

I believe we have a good reason to add array support to sigma. Many native log sources use arrays - including (but definitely not limited to):

• K8s audit logs
• Cloudtrail
• Azure Activity logs
• GCP Activity logs
• Okta

It's worth to note that only Azure would not benefit from the Further Enhancement feature (proposed by @humpalum) since it does not appear to use JSON objects inside arrays. Additionally, note that sigma does already have rules for most of the log sources from the list above.

Additional observations

I am not sure how sigma handles nested fields at the moment. We do have some rules that use . to access fields that are nested in original log sources.

Proposed solution

The proposed solution roughly aligns with what @humpalum suggests, but I propose different syntax (for the sake of discussion).

The format below is somewhat inspired by keywords search

Let us consider the example by @humpalum once more:

{
    "message": "Some Message",
    "process": "Some Process",
    "connections": [
        {"protocol":"TCP","ip":"123.1.1.1"},
        {"protocol":"TCP","ip":"123.1.2.2"},
        {"protocol":"UDP","ip":"123.3.3.3"}
    ]
}

A rule that matches events with such schema if it contains any TCP connection with IP address "123.1.1.1" would look like this:

selection:
  matchNode:
    connections[any].protocol: "TCP"
    connections[any].ip: "123.1.1.1"

which should be interpreted as a 'flattened' version of @humpalum's syntax. Consequently, the rule below

selection:
  matchNode:
    connections[any].protocol: "TCP"
    connections[all].ip: "123.1.1.1"

would mean **match events where any connection is a TCP, but all of them have IP "123.1.1.1".

This syntax can be nested easily. Let's consider another example:

{
   "configName": "Some Name",
   "rules": [
       {"type":"allow", "ip":["123.1.1.1", "123.1.1.2", "123.1.1.3"]},
       {"type":"allow", "ip":["123.1.2.2"]},
       {"type":"block","ip":["124.3.3.3"]}
   ]
}

And a rule

selection:
  matchNode:
    rules[any].type: "allow"
    rules[any].ip[all]|startswith: "123.1.1"
    rules[all].ip[all]|startswith: "123"

This time, the rule matches any events where all rule ips start with "123". Additionally there must be at least one rule object R which satisfies:

• R.type == "allow"
• For each s in R.ip array: s starts with "123.1.1"

Pros and Cons

I believe this syntax might be a bit more consistent with current sigma format. It would also not increase the depth of rules much. It also allows fine grained control over how nested logs should be matched. On the flip side it might be less straightforward and more verbose than @humpalum's solution.

Regardless of the format, I would really embrace any official support for nested formats.