lib/logstorage: add pattern_match() and pattern_match_full() filters for searching for logs by the given patterns

The patterns may contain various placeholders such as <N>, <IP4>, <TIME>, etc.
Such patterns with these placeholders are generated by `| collapse_nums prettify` pipe.

See https://docs.victoriametrics.com/victorialogs/logsql/#pattern-match-filter for details.

These filters are needed for VictoriaMetrics#518

Author: valyala

docs/victorialogs/CHANGELOG.md

@@ -18,6 +18,8 @@ according to [these docs](https://docs.victoriametrics.com/victorialogs/quicksta
 
 ## tip
 
+* FEATURE: [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/): add [pattern match filter](https://docs.victoriametrics.com/victorialogs/logsql/#pattern-match-filter) for searching logs by the given patterns such as `<DATETIME>: user_id=<N>, ip=<IP4>, trace_id=<UUID>`. These filters are needed for [#518](https://github.com/VictoriaMetrics/VictoriaLogs/issues/518).
+
 ## [v1.32.0](https://github.com/VictoriaMetrics/VictoriaLogs/releases/tag/v1.32.0)
 
 Released at 2025-09-03

docs/victorialogs/LogsQL.md

@@ -264,15 +264,18 @@ The list of LogsQL filters:
 - [Phrase filter](#phrase-filter) - matches logs with the given phrase
 - [Prefix filter](#prefix-filter) - matches logs with the given word prefix or phrase prefix
 - [Substring filter](#substring-filter) - matches logs with the given substring
+- [Pattern match filter](#pattern-match-filter) - matches logs by the given pattern
 - [Range comparison filter](#range-comparison-filter) - matches logs with field values in the provided range
 - [Empty value filter](#empty-value-filter) - matches logs without the given [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model)
 - [Any value filter](#any-value-filter) - matches logs with the given non-empty [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model)
 - [Exact filter](#exact-filter) - matches logs with the exact value for the given [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model)
 - [Exact prefix filter](#exact-prefix-filter) - matches logs starting with the given prefix for the given [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model)
 - [Multi-exact filter](#multi-exact-filter) - matches logs with one of the specified exact values for the given [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model)
 - [Subquery filter](#subquery-filter) - matches logs with [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) values matching the results of another query
-- [`contains_all` filter](#contains_any-filter) - matches logs with [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) containing all the provided [words](#word) / phrases
-- [`contains_any` filter](#contains_any-filter) - matches logs with [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) containing at least one of the provided [words](#word) / phrases
+- [`contains_all` filter](#contains_any-filter) - matches logs with [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) containing
+  all the provided [words](#word) / phrases
+- [`contains_any` filter](#contains_any-filter) - matches logs with [log field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model) containing
+  at least one of the provided [words](#word) / phrases
 - [Case-insensitive filter](#case-insensitive-filter) - matches logs with the given case-insensitive word, phrase or prefix
 - [Sequence filter](#sequence-filter) - matches logs with the given sequence of words or phrases
 - [Regexp filter](#regexp-filter) - matches logs for the given regexp
@@ -720,6 +723,41 @@ See also:
 - [Exact-filter](#exact-filter)
 - [Logical filter](#logical-filter)
 
+### Pattern match filter
+
+VictoriaLogs supports filtering logs by patterns with the `pattern_match("pattern")` filter. This filter matches logs where
+[`_msg` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#message-field) contains the given `"pattern"`.
+
+The filter can be applied to any given log field with the `log_field:pattern_match("pattern")` syntax.
+
+The `"pattern"` must contain the text to match, plus arbitrary number of the following placeholders:
+
+- `<N>` - matches any integer number. It also matches hexadecimal numbers with the length of 4 chars and longer. For example, it matches `123` and `12abcdEF`.
+  It doesn't match floating point numbers such as `123.456`. Use `<N>.<N>` pattern for matching such numbers.
+- `<UUID>` - matches any UUID such as `2edfed59-3e98-4073-bbb2-28d321ca71a7`.
+- `<IP4>` - matches IPv4 such as `123.45.67.89`. Use `<IP4>/<N>` for matching IPv4 masks.
+- `<TIME>` - matches time strings such as `10:20:30`. It also captures fractional seconds such as `10:20:30.123` and `10:20:30,123`.
+- `<DATE>` - matches date strings such as `2025-10-20` and `2025/10/20`.
+- `<DATETIME>` - matches datetime strings such as `2025-10-20T08:09:11` and `2025-10-20 08:09:11`. It also captures fractional seconds and timezones.
+
+Such patterns are generated by the [`collapse_nums` pipe](#collapse_nums-pipe).
+
+For example, the following filter matches `_msg` field with the `<arbitrary_prefix>user_id=123, ip=45.67.89.12, time=2025-10-20T23:32:12Z<aribtrary_suffix>` contents:
+
+```logsql
+pattern_match("user_id=<N>, ip=<IP4>, time=<TIMESTAMP>")
+```
+
+If you need matching the whole `_msg` field value, then use `pattern_match_full("pattern")` filter.
+
+See also:
+
+- [`collapse_nums` pipe](#collapse_nums-pipe)
+- [Sequence filter](#sequence-filter)
+- [Phrase filter](#phrase-filter)
+- [Substring filter](#substring-filter)
+- [Logical filter](#logical-filter)
+
 ### Substring filter
 
 If it is needed to find logs with some substring, then `*substring*` filter can be used. The substring can be but in quotes according to [these docs](#string-literals) if needed.
@@ -741,6 +779,7 @@ Performance tip: prefer using [word filter](#word-filter) and [phrase filter](#p
 
 See also:
 
+- [Pattern match filter](#pattern-match-filter)
 - [Word filter](#word-filter)
 - [Phrase filter](#phrase-filter)
 - [Regexp filter](#regexp-filter)
@@ -1106,6 +1145,7 @@ For example, the following query matches `event:original` field containing `(err
 
 See also:
 
+- [Pattern match filter](#pattern-match-filter)
 - [`contains_all` filter](#contains_all-filter)
 - [Word filter](#word-filter)
 - [Phrase filter](#phrase-filter)
@@ -1606,12 +1646,15 @@ when the following query is executed:
 _time:1h | collapse_nums prettify
 ```
 
+The patterns returned by `collapse_nums prettify` pipe can be used in [pattern match filter](#pattern-match-filter).
+
 `collapse_nums` can miss some numbers or can collapse unexpected numbers. In this case [conditional `collapse_nums`](#conditional-collapse_nums) can be used
 for skipping such values and pre-processing them separately with [`replace_regexp`](#replace_regexp-pipe).
 
 See also:
 
 - [conditional `collapse_nums`](#conditional-collapse_nums)
+- [pattern match filter](#pattern-match-filter)
 - [`replace`](#replace-pipe)
 - [`replace_regexp`](#replace_regexp-pipe)
 

lib/logstorage/filter_and.go

@@ -143,13 +143,13 @@ func getCommonTokensForAndFilters(filters []filter) []fieldTokens {
 		case *filterExactPrefix:
 			tokens := t.getTokens()
 			mergeFieldTokens(t.fieldName, tokens)
-		case *filterPhrase:
+		case *filterPatternMatch:
 			tokens := t.getTokens()
 			mergeFieldTokens(t.fieldName, tokens)
-		case *filterPrefix:
+		case *filterPhrase:
 			tokens := t.getTokens()
 			mergeFieldTokens(t.fieldName, tokens)
-		case *filterSubstring:
+		case *filterPrefix:
 			tokens := t.getTokens()
 			mergeFieldTokens(t.fieldName, tokens)
 		case *filterRegexp:
@@ -158,6 +158,9 @@ func getCommonTokensForAndFilters(filters []filter) []fieldTokens {
 		case *filterSequence:
 			tokens := t.getTokens()
 			mergeFieldTokens(t.fieldName, tokens)
+		case *filterSubstring:
+			tokens := t.getTokens()
+			mergeFieldTokens(t.fieldName, tokens)
 		case *filterOr:
 			bfts := t.getByFieldTokens()
 			for _, bft := range bfts {

lib/logstorage/filter_or.go

@@ -147,13 +147,13 @@ func getCommonTokensForOrFilters(filters []filter) []fieldTokens {
 		case *filterExactPrefix:
 			tokens := t.getTokens()
 			mergeFieldTokens(t.fieldName, tokens)
-		case *filterPhrase:
+		case *filterPatternMatch:
 			tokens := t.getTokens()
 			mergeFieldTokens(t.fieldName, tokens)
-		case *filterPrefix:
+		case *filterPhrase:
 			tokens := t.getTokens()
 			mergeFieldTokens(t.fieldName, tokens)
-		case *filterSubstring:
+		case *filterPrefix:
 			tokens := t.getTokens()
 			mergeFieldTokens(t.fieldName, tokens)
 		case *filterRegexp:
@@ -162,6 +162,9 @@ func getCommonTokensForOrFilters(filters []filter) []fieldTokens {
 		case *filterSequence:
 			tokens := t.getTokens()
 			mergeFieldTokens(t.fieldName, tokens)
+		case *filterSubstring:
+			tokens := t.getTokens()
+			mergeFieldTokens(t.fieldName, tokens)
 		case *filterAnd:
 			bfts := t.getByFieldTokens()
 			for _, bft := range bfts {