Proposal: new aggregate functions in correlation rules

[Res260]

Context

Right now, correlations searches only support "COUNT" as an aggregation function

Proposal

There exists many more, like "SUM", "AVERAGE", "MAX", "MIN", "MEDIAN", "MODE", "PERCENTILE", "RANGE" and "STANDARD DEVIATION".

This sounds like a lot, but for most backends, the resulting rule would be the same for all of these, just the aggregate functions that changes.

What this would allow

This would allow to create Sigma Correlations rules like those:

• At least 1000MB was sent to a website in the span of 1 day by a single user.
• A computer spawned more processes in a day than 99% of the other computers.
• More than half of the exit codes of the IIS process were non-zero.

etc.

Proposed implementation

Here is my proposed implementation for the rule "At least 1000MB was sent to a website in the span of 1 day by a single user."

title: At least 1000MB was sent to a website in the span of 1 day by a single user.
id: c167d0df-442a-4410-9b48-0a384b679116
status: stable
level: medium
correlation:
  type: sum  # <-- Could be any choice of sum, average, max, min, median, mode, percentile, range, standard_deviation
  rules: 
    - request_to_website
  group-by:
    - domainName
    - Username
  timespan: 24h
  condition:
    gte: 1000000000
    field: BytesOut
---
title: Single request to a website
name: request_to_website
id: 24b39749-f29a-4bfe-a852-21a3327f8c5a
logsource:
  category: Web Proxy
detection:
  selection: 
    Event: 'Request'
  condition: selection

This requires very little change to the specification to allow such rules. We simply allow these new types.

Here is the "A computer spawned more processes in a day than 99% of the other computers." example:

title: A computer spawned more processes in a day than 99% of the other computers.
id: c167d0df-442a-4410-9b48-0a384b679116
status: stable
level: medium
correlation:
  type: percentile
  rules: 
    - process_creation
  group-by:
    - ComputerName
  timespan: 24h
  condition:
    gte: 99
---
title: A single process creation
name: process_creation
id: 24b39749-f29a-4bfe-a852-21a3327f8c5a
logsource:
  category: process_creation
detection:
  selection: 
    Event: 'process creation'
  condition: selection

title: More than half of the exit codes of the IIS process were non-zero.
id: c167d0df-442a-4410-9b48-0a384b679116
status: stable
level: medium
correlation:
  type: median
  rules: 
    - end_of_process
  group-by:
    - ComputerName
  timespan: 24h
  condition:
    gt: 0
    field: ExitCode
---
title: A single end of process 
name: end_of_process
id: 24b39749-f29a-4bfe-a852-21a3327f8c5a
logsource:
  category: end_of_process
detection:
  selection: 
    Event: 'end of process'
  condition: selection

Conclusion

Adding these to the Sigma Correlations Specification would allow many very interesting detections to be developed. At my organization, we strive to achieve 100% of our rules to be expressed in Sigma, and this would be a big step in this direction.

[sinnwise]

Would be interesting to know which SIEMs support this, or if it's only going to get support in the main 3, (Splunk, Elastic, Grafana).

[Res260]

Aggregate functions are a common feature in SIEMs. Here is a quick rundown:

• Splunk: https://docs.splunk.com/Documentation/SCS/current/SearchReference/Aggregatefunctions
• Elastic: https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-functions-operators.html#esql-functions (section Aggregate Functions)
• Grafana: https://grafana.com/docs/grafana-cloud/testing/k6/reference/query-types/metric-query-aggregation-methods/
• QRadar: https://www.ibm.com/docs/en/qsip/7.4?topic=language-aql-data-aggregation-functions
• Sentinel/KQL: https://learn.microsoft.com/en-us/kusto/query/aggregation-functions?view=microsoft-fabric#statistical-functions
• Logscale: https://library.humio.com/data-analysis/functions-aggregate.html
• Anything that uses SQL: https://www.w3schools.com/sql/sql_aggregate_functions.asp
• Datadog: https://docs.datadoghq.com/dashboards/querying/#aggregate-and-rollup
• FortiSIEM: https://help.fortinet.com/fsiem/7-1-3/Online-Help/HTML5_Help/appendix-functions-aggregate.htm
• Sumo Logic: https://help.sumologic.com/docs/search/search-query-language/group-aggregate-operators/

[thomaspatzke]

I like it! And from implementation point of view it should be only the introduction of some new types and query templates, but a big thing to achieve a lot.

[Res260]

If there is no opposition to this proposal, I could get started implementing it.

[Res260]

Paging other common contributors
@nasbench @frack113

[frack113]

I have work on baseline some time ago https://github.com/frack113/sigma-specification/blob/new-correlation/specification/sigma-correlation-rules-specification.md#base-line-baseline
It's a Very Very Works In Progress 😄

I'm more on somethink like:

type: metric
condition:
  operation: "SUM", "AVERAGE", "MAX", "MIN", "MEDIAN", "MODE", "PERCENTILE", "RANGE" and "STANDARD DEVIATION" (lowercase )

my baseline-type can be change to "operation ?" too be more consistent in specifications.

[frack113]

My thought is that I'd rather have a new rule family with a subtype in the condition than lots of new ones.

Statistical Metric (metric/stats ?)

The aim is to find a deviance from a fixe math limit .......

Requires:
rules
group-by
timespan

In the condition section
operation
math operator (gt;lt,...)

operation

sum

average

I not sure for STANDARD DEVIATION but all the other are basic (I just need to find my olds statistics courses 😄 )

[Res260]

Let's see what the others think. However, in your proposal, there seems to be missing a key. It doesn't seem possible to group by source IP, then SUM the bytesOut with only rules group-by timespan operation and math operator.

[frack113]

It's only an example that needs to be worked on.
requires is the minimal that I think about.
In sun you need a field with a numeric value.
In medium need a math operator
And so on

[frack113]

@Res260 I start to write a draft https://github.com/frack113/sigma-specification/blob/Metric-Aggregate/specification/sigma-correlation-rules-specification.md

[Res260]

Cool! I'll fork and contribute to it.
I don't really like the term metric, I'll try to think of something else a little bit more precise.

[nasbench]

Added to the specs in part - See https://github.com/SigmaHQ/sigma-specification/blob/main/specification/sigma-correlation-rules-specification.md