Reference Sets/Lookups Proposal

[Res260]

I'd like to discuss the possibility of adding reference sets/lookups into the Sigma specification.

What are reference sets/lookups?

It's a list external to a query that you use into a query/detection.

SIEM	What it's called
QRadar	Reference set/map/table
Splunk	Lookups
Sentinel	Lookups/watchlists
Arcsight	Active lists
Logscale	lookup

Example of a rule that would use lookups

Here is a rule that would match when someone is added to a critical AD group:

logsource:
  product: Windows logs
detection:
  selection:
    EventID: 1234  # I dont know the event id lol
    TargetGroupName|lookup: 'critical_group_list'  # actual implementation must be discussed as well
  condition: selection

Why are they helpful and needed?

1. Lookups favor re-use of content accross rules. If we take the above example, we could have a rule when someone is removed from one of the same groups and re-use the list of critical groups.
2. Lookups keep a rule clean. Without lookups, a simple rule could get very unreadable if you need to lookup a list of 20+ items
3. Lookups work for a large amount of data. Lots of backend limit the maximum length of their query for performance and other reasons, which poses problems for large (50+) reference sets.

SIEM	Character limit per query
QRadar	<5,000
Splunk	None
Sentinel	10,000
Arcsight	idk
Logscale	66,000

Don't placeholders already solve this?

Here is the relevant part for the placeholders documentation:

Placeholders can be used to select a set of elements that can be expanded during conversion. Placeholders map a an identifier to a user defined value that can be set in config files for an automatic replacement during conversion runs. Placeholders are meaningful identifiers that users can easily expand themselves.

This is close from the use case I'm describing, but not quite it either:

• Sigma does not define the behavior of placeholders and uses the "this exercise is left to the reader" approach, which means 10 people reading this will likely create 10 different implementation of the same problem.
• This vagueness means no backend actually implement this feature and one must either create their own backend or use processing pipeline to try to make it work.
• Placeholders do not address point # 3, which is IMO the most important.

This goes against the SigmaHQ philosophy of "rules should be standalone and easily sharable"

This is true. However:

• Placeholders are worse for sharability and standaloneness yet they made their way into the Sigma specification
• In my opinion, the Sigma specification has broader use and is more important than the SigmaHQ project and concerns for the SigmaHQ project should not interfere with development of the Sigma specification, which is a major building block in detection-as-code implementation in many organizations, including mine. I love SigmaHQ and the work that's being done on this repo and frequently refer to it for reference, but I feel the rigidity of the contribution rules that come with it heavily influence the Sigma specification and I think, longer-term, this might become harmful to the specification.
• Reference sets/lookups can also easily be shared and they could even be bundled in the same file as the rule if needed.

Implementation proposal

Reference sets would be separate YAML files with a simple structure such as:

name: Critical AD Groups
type: string
entries:
  - Domain Admins
  - Enterprise Admins
  - Administrateurs du domaine
  - MyCustomAdminGroup1
  - ...

Rules could then reference them with something like a |lookup modifier:

logsource:
  product: Windows logs
detection:
  selection:
    EventID: 1234  # I dont know the event id lol
    TargetGroupName|lookup: 'Critical AD Groups'
  condition: selection

More things that need discussions:

• The discovery of reference sets needs to be discussed as well, maybe an extra argument to sigma convert
• Obviously sigma-cli should not be responsible for deployment of such reference sets, but sigma convert could take into account reference sets for output types like plugins or bundled applications like some backends support.
• Probably other things I didnt think about

Personal experience

We internally have implemented reference sets for our custom QRadar backend using placeholders. It works with a custom internal specification that looks like this: TargetGroupName|expand: '%refset:Critical AD Groups%'. This theoretically respects the Sigma specification, but there is a lot of custom stuff I needed to code to make it happen and think about it myself. I feel this is a common enough use case that we should standardize it into the Sigma specification, hence my proposal.

Let me know what you think!

[thomaspatzke]

Sorry, totally missed that!

• Sigma does not define the behavior of placeholders and uses the "this exercise is left to the reader" approach, which means 10 people reading this will likely create 10 different implementation of the same problem.

The rule specification was intentionally made vague at this point to keep it agnostic against concepts used in the target systems and environments. It's indeed intended that the user or conversion tool finally decides what to do with them. Even with a target system that supports lookups there can be valid reasons that the user just wants to have the list of possible values in the query.

• This vagueness means no backend actually implement this feature and one must either create their own backend or use processing pipeline to try to make it work.

No backend actually needs to implement placeholders because processing pipelines implement various handling operations in a backend-agnostic way 😉 The placeholders are transformed into lists of values, raw query expressions in the target query language or just wildcards (if the user really wants to ignore the placeholder or as fallback) and these get then passed to the backend.

This blog post describes the details.

• Placeholders do not address point # 3, which is IMO the most important.

The query_expression_placeholders solves the lookup use case at least for query languages where this is expressed somehow with an expression, function etc, e.g. [ inputlookup ... ] in Splunk.

This goes against the SigmaHQ philosophy of "rules should be standalone and easily sharable"

Disagree on this (or better said: agree with you 😉), the detection logic might be standalone, but finally the operationalization requires processing pipelines and in case of placeholders some definition how to handle them.

• Placeholders are worse for sharability and standaloneness yet they made their way into the Sigma specification

In my opinion, placeholders are the enabler for sharing of some detections that are scoped somehow, e.g. "only on clients" etc.

• In my opinion, the Sigma specification has broader use and is more important than the SigmaHQ project and concerns for the SigmaHQ project should not interfere with development of the Sigma specification, which is a major building block in detection-as-code implementation in many organizations, including mine. I love SigmaHQ and the work that's being done on this repo and frequently refer to it for reference, but I feel the rigidity of the contribution rules that come with it heavily influence the Sigma specification and I think, longer-term, this might become harmful to the specification.

I agree, but SigmaHQ doesn't equals the Sigma rule specification. The SigmaHQ rule repository is more strict, even has an extended set of validators that the rules have to pass, but this doesn't means that certain features might not be used outside of the SigmaHQ context.

There's even a place for rules with placeholders in the SigmaHQ repository. My personal preference was to integrate them with "normal" rules. But finally, we decided it this way because many users expect all rules to convert with minimal effort and having placeholders in the rules means also that it must be defined how these are handled in processing pipelines. The conversion stops on unhandled placeholders.