Skip to content

Commit 53b47fc

Browse files
srikanthpadakantiSrikanth Padakanti
authored and
sunqijun.jun
committed
Enhance terms lookup query to take a query clause (opensearch-project#18195)
Signed-off-by: Srikanth Padakanti <[email protected]> Signed-off-by: Srikanth Padakanti <[email protected]> Co-authored-by: Srikanth Padakanti <[email protected]> Signed-off-by: sunqijun.jun <[email protected]>
1 parent a84fb44 commit 53b47fc

File tree

7 files changed

+1031
-69
lines changed

7 files changed

+1031
-69
lines changed

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
55

66
## [Unreleased 3.x]
77
### Added
8+
- [Feature Request] Enhance Terms lookup query to support query clause instead of docId ([#18195](https://github.com/opensearch-project/OpenSearch/issues/18195))
89
- Add hierarchical routing processors for ingest and search pipelines ([#18826](https://github.com/opensearch-project/OpenSearch/pull/18826))
910
- Add support for Warm Indices Write Block on Flood Watermark breach ([#18375](https://github.com/opensearch-project/OpenSearch/pull/18375))
1011
- FS stats for warm nodes based on addressable space ([#18767](https://github.com/opensearch-project/OpenSearch/pull/18767))

‎rest-api-spec/src/main/resources/rest-api-spec/test/search/171_terms_lookup_query.yml‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
---
2+
"Terms Query - Lookup by Query":
3+
# --- SKIP IF VERSION IS LESS THAN 3.2.0 --- #
4+
- skip:
5+
version: " - 3.1.99"
6+
reason: All tests in this file require features added in 3.2.0 or later (terms query lookup by query).
7+
8+
# --- SETUP: CREATE INDICES AND POPULATE DATA --- #
9+
- do:
10+
indices.create:
11+
index: lookup_index
12+
body:
13+
settings:
14+
number_of_replicas: 0
15+
# Wait for index to be fully ready (avoids race in multi-node clusters)
16+
- do:
17+
cluster.health:
18+
index: lookup_index
19+
wait_for_status: green
20+
timeout: 30s
21+
- do:
22+
indices.create:
23+
index: main_index
24+
body:
25+
settings:
26+
number_of_replicas: 0
27+
- do:
28+
cluster.health:
29+
index: main_index
30+
wait_for_status: green
31+
timeout: 30s
32+
33+
# Populate lookup_index
34+
- do:
35+
bulk:
36+
refresh: true
37+
body: |
38+
{ "index": { "_index": "lookup_index", "_id": "1" } }
39+
{ "group": "g1", "followers": ["foo", "bar"], "tag": "a" }
40+
{ "index": { "_index": "lookup_index", "_id": "2" } }
41+
{ "group": "g1", "followers": ["baz"], "tag": "b" }
42+
{ "index": { "_index": "lookup_index", "_id": "3" } }
43+
{ "group": "g2", "followers": null, "tag": "c" }
44+
{ "index": { "_index": "lookup_index", "_id": "4" } }
45+
{ "group": "g1", "tag": "d" }
46+
{ "index": { "_index": "lookup_index", "_id": "5" } }
47+
{ "group": "g1", "followers": ["baz"], "tag": "e" }
48+
{ "index": { "_index": "lookup_index", "_id": "6" } }
49+
{ "group": "g3", "followers": [], "tag": "f" }
50+
- match: { errors: false }
51+
52+
# Populate main_index
53+
- do:
54+
bulk:
55+
refresh: true
56+
body: |
57+
{ "index": { "_index": "main_index", "_id": "u1" } }
58+
{ "user": "foo" }
59+
{ "index": { "_index": "main_index", "_id": "u2" } }
60+
{ "user": "bar" }
61+
{ "index": { "_index": "main_index", "_id": "u3" } }
62+
{ "user": "baz" }
63+
{ "index": { "_index": "main_index", "_id": "u4" } }
64+
{ "user": "qux" }
65+
{ "index": { "_index": "main_index", "_id": "u5" } }
66+
{ "user": "foo" }
67+
- match: { errors: false }
68+
69+
- do:
70+
cluster.health:
71+
index: lookup_index,main_index
72+
wait_for_status: green
73+
timeout: 30s
74+
75+
# --- TEST CASES --- #
76+
77+
# Match all docs in lookup_index with group=g1, use their 'followers' as terms
78+
- do:
79+
search:
80+
rest_total_hits_as_int: true
81+
index: main_index
82+
body:
83+
query:
84+
terms:
85+
user:
86+
index: lookup_index
87+
path: followers
88+
query:
89+
term:
90+
group: "g1"
91+
- match: { hits.total: 4 } # foo (u1), bar (u2), baz (u3), foo (u5)
92+
93+
# Query returns docs, but some have missing/null/empty field
94+
- do:
95+
search:
96+
rest_total_hits_as_int: true
97+
index: main_index
98+
body:
99+
query:
100+
terms:
101+
user:
102+
index: lookup_index
103+
path: followers
104+
query:
105+
terms:
106+
tag: ["a", "c", "f"]
107+
- match: { hits.total: 3 } # foo (u1), bar (u2), foo (u5)
108+
109+
# Query returns docs but field is always empty list
110+
- do:
111+
search:
112+
rest_total_hits_as_int: true
113+
index: main_index
114+
body:
115+
query:
116+
terms:
117+
user:
118+
index: lookup_index
119+
path: followers
120+
query:
121+
term:
122+
tag: "d"
123+
- match: { hits.total: 0 } # No terms found
124+
125+
# Query returns docs but field is scalar, not array (now array, see above)
126+
- do:
127+
search:
128+
rest_total_hits_as_int: true
129+
index: main_index
130+
body:
131+
query:
132+
terms:
133+
user:
134+
index: lookup_index
135+
path: followers
136+
query:
137+
term:
138+
tag: "e"
139+
- match: { hits.total: 1 } # Only user baz (from doc 5: followers=["baz"])
140+
141+
# Query returns no docs
142+
- do:
143+
search:
144+
rest_total_hits_as_int: true
145+
index: main_index
146+
body:
147+
query:
148+
terms:
149+
user:
150+
index: lookup_index
151+
path: followers
152+
query:
153+
term:
154+
tag: "zzz"
155+
- match: { hits.total: 0 }
156+
157+
# Query returns docs, some with missing field, some with lists, some with nulls
158+
- do:
159+
search:
160+
rest_total_hits_as_int: true
161+
index: main_index
162+
body:
163+
query:
164+
terms:
165+
user:
166+
index: lookup_index
167+
path: followers
168+
query:
169+
terms:
170+
tag: ["a", "b", "c", "d", "e", "f"]
171+
- match: { hits.total: 4 } # foo (u1), bar (u2), baz (u3), foo (u5)
172+
173+
# Duplicates across docs should be deduplicated (foo appears in 2 docs)
174+
- do:
175+
search:
176+
rest_total_hits_as_int: true
177+
index: main_index
178+
body:
179+
query:
180+
terms:
181+
user:
182+
index: lookup_index
183+
path: followers
184+
query:
185+
terms:
186+
tag: ["a", "b"]
187+
- match: { hits.total: 4 } # foo (u1), bar (u2), baz (u3), foo (u5)
188+
189+
# Query returns docs but none have the field specified.
190+
- do:
191+
search:
192+
rest_total_hits_as_int: true
193+
index: main_index
194+
body:
195+
query:
196+
terms:
197+
user:
198+
index: lookup_index
199+
path: not_a_field
200+
query:
201+
match_all: {}
202+
- match: { hits.total: 0 }
203+
204+
# Optional: sanity check for main_index docs
205+
- do:
206+
search:
207+
rest_total_hits_as_int: true
208+
index: main_index
209+
body:
210+
query:
211+
match_all: {}
212+
- match: { hits.total: 5 } # Should always be 5

0 commit comments

Comments
 (0)