Skip to content

Commit cadd731

Browse files
author
Sofiya Nuryyeva
committed
refactored okp rag tests
1 parent b5add51 commit cadd731

1 file changed

Lines changed: 35 additions & 107 deletions

File tree

‎tests/e2e/features/okp_rag.feature‎

Lines changed: 35 additions & 107 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,10 @@ Feature: OKP(Solr) RAG retrieval tests
1414
And REST API service prefix is /v1
1515
And the Lightspeed stack configuration directory is "tests/e2e/configuration"
1616

17-
# ── Inline RAG — Query ──
17+
# ── Inline RAG — Query (offline) ──
1818

19-
Scenario Outline: <mode> mode query with inline RAG returns rag_chunks and referenced_documents
20-
Given The service uses the <config> configuration
19+
Scenario: Offline mode query with inline RAG returns rag_chunks and referenced_documents
20+
Given The service uses the lightspeed-stack-okp-offline.yaml configuration
2121
And The service is restarted
2222
When I use "query" to ask question with authorization header
2323
"""
@@ -26,27 +26,20 @@ Feature: OKP(Solr) RAG retrieval tests
2626
Then The status code of the response is 200
2727
And The response contains non-empty rag_chunks
2828
And The response contains non-empty referenced_documents
29-
And The number of rag_chunk returned is <max_chunks>
29+
And The number of rag_chunk returned is 1
3030
And Each rag_chunk has a non-empty score
3131
And Each rag_chunk source is "okp"
3232
And Each referenced_document has fields doc_url, doc_title, source, and document_id
33-
And Each referenced_document doc_url contains "<doc_url_domain>"
33+
And The number of eferenced_document returned is 1
34+
And Each referenced_document doc_url contains "localhost:8081"
3435
And Each referenced_document doc_title is not empty
3536
And Each referenced_document source is "okp"
3637
And Each referenced_document has a non-empty document_id
3738

38-
Examples: Offline
39-
| mode | config | max_chunks | doc_url_domain |
40-
| Offline | lightspeed-stack-okp-offline.yaml | 5 | localhost:8081 |
41-
42-
Examples: Online
43-
| mode | config | max_chunks | doc_url_domain |
44-
| Online | lightspeed-stack-okp-online.yaml | 1 | docs.redhat.com |
45-
46-
# ── Inline RAG — Streaming Query ──
39+
# ── Inline RAG — Streaming Query (online) ──
4740

48-
Scenario Outline: <mode> mode streaming query with inline RAG returns referenced_documents
49-
Given The service uses the <config> configuration
41+
Scenario: Online mode streaming query with inline RAG returns referenced_documents
42+
Given The service uses the lightspeed-stack-okp-online.yaml configuration
5043
And The service is restarted
5144
When I use "streaming_query" to ask question with authorization header
5245
"""
@@ -56,29 +49,23 @@ Feature: OKP(Solr) RAG retrieval tests
5649
And I wait for the response to be completed
5750
And The response contains non-empty referenced_documents
5851
And Each referenced_document has fields doc_url, doc_title, source, and document_id
59-
And Each referenced_document doc_url contains "<doc_url_domain>"
52+
And The number of eferenced_document returned is 3
53+
And Each referenced_document doc_url contains "docs.redhat.com"
6054
And Each referenced_document doc_title is not empty
55+
And Each referenced_document doc_title contains "openshift container platform 4.21"
6156
And Each referenced_document source is "okp"
6257
And Each referenced_document has a non-empty document_id
6358

64-
Examples: Offline
65-
| mode | config | doc_url_domain |
66-
| Offline | lightspeed-stack-okp-offline.yaml | localhost:8081 |
67-
68-
Examples: Online
69-
| mode | config | doc_url_domain |
70-
| Online | lightspeed-stack-okp-online.yaml | docs.redhat.com |
71-
7259
# ── Inline RAG — Query with Dynamic Filter ──
7360

74-
Scenario Outline: Query with inline RAG with dynamic <filter_mode> filter returns rag_chunks and referenced_documents
61+
Scenario: Query with inline RAG with dynamic semantic filter returns rag_chunks and referenced_documents
7562
Given The service uses the lightspeed-stack-okp-offline.yaml configuration
7663
And The service is restarted
7764
When I use "query" to ask question with authorization header
7865
"""
7966
{"query": "Security best practices",
8067
"solr": {
81-
"mode": "<filter_mode>",
68+
"mode": "semantic",
8269
"filters": {
8370
"filters": {
8471
"type": "in",
@@ -90,31 +77,28 @@ Feature: OKP(Solr) RAG retrieval tests
9077
}
9178
"""
9279
Then The status code of the response is 200
80+
And The response contains "security best practices"
9381
And The response contains non-empty rag_chunks
9482
And The response contains non-empty referenced_documents
95-
And The number of rag_chunk returned is 5
83+
And The number of rag_chunk returned is 1
9684
And Each rag_chunk has a non-empty score
9785
And Each rag_chunk source is "okp"
9886
And Each referenced_document has fields doc_url, doc_title, source, and document_id
99-
And Each referenced_document doc_url contains "<doc_url_domain>"
87+
And The number of eferenced_document returned is 1
88+
And Each referenced_document doc_url contains "localhost:8081"
10089
And Each referenced_document doc_title is not empty
10190
And Each referenced_document source is "okp"
10291
And Each referenced_document has a non-empty document_id
10392

104-
Examples:
105-
| filter_mode | doc_url_domain |
106-
| semantic | localhost:8081 |
107-
| hybrid | docs.redhat.com |
93+
# ── Tool RAG — Query API (offline) ──
10894

109-
# ── Tool RAG — Query API ──
110-
111-
Scenario Outline: <mode> queries API with OKP tool RAG has rag_chunk and referenced_documents returned
112-
Given The service uses the <config> configuration
95+
Scenario: Offline query API with OKP tool RAG has rag_chunk and referenced_documents returned
96+
Given The service uses the lightspeed-stack-okp-tool-offline.yaml configuration
11397
And The service is restarted
11498
When I use "query" to ask question with authorization header
11599
"""
116100
{
117-
"query": "configure remote desktop using gnome",
101+
"query": "Troubleshooting guide",
118102
"model": "{MODEL}",
119103
"provider": "{PROVIDER}",
120104
"system_prompt": "You MUST use the file_search tool to answer."
@@ -124,65 +108,25 @@ Feature: OKP(Solr) RAG retrieval tests
124108
And The response contains non-empty tool_calls
125109
And A tool_call has name "file_search"
126110
And The response contains non-empty rag_chunks
127-
And The number of rag_chunk returned is <max_chunks>
111+
And The number of rag_chunk returned is 2
128112
And Each rag_chunk has a non-empty score
129113
And Each rag_chunk source is "okp"
130114
And The response contains non-empty referenced_documents
131115
And Each referenced_document has fields doc_url, doc_title, source, and document_id
132-
And Each referenced_document doc_url contains "<doc_url_domain>"
133-
And Each referenced_document doc_title is not empty
134-
And Each referenced_document source is "okp"
135-
And Each referenced_document has a non-empty document_id
136-
137-
Examples: Offline
138-
| mode | config | max_chunks | doc_url_domain |
139-
| Offline | lightspeed-stack-okp-tool-offline.yaml | 5 | localhost:8081 |
140-
141-
Examples: Online
142-
| mode | config | max_chunks | doc_url_domain |
143-
| Online | lightspeed-stack-okp-tool-online.yaml | 1 | access.redhat.com |
144-
145-
# ── Tool RAG — Streaming Query API ──
146-
147-
Scenario Outline: <mode> streaming query API with OKP tool RAG has rag_chunk and referenced_documents returned
148-
Given The service uses the <config> configuration
149-
And The service is restarted
150-
When I use "streaming_query" to ask question with authorization header
151-
"""
152-
{
153-
"query": "configure remote desktop using gnome",
154-
"model": "{MODEL}",
155-
"provider": "{PROVIDER}",
156-
"system_prompt": "You MUST use the file_search tool to answer."
157-
}
158-
"""
159-
Then The status code of the response is 200
160-
And A tool_call has name "file_search"
161-
And The response contains non-empty content
162-
And The response contains non-empty referenced_documents
163-
And Each referenced_document has fields doc_url, doc_title, source, and document_id
164-
And Each referenced_document doc_url contains "<doc_url_domain>"
116+
And Each referenced_document doc_url contains "localhost:8081"
165117
And Each referenced_document doc_title is not empty
166118
And Each referenced_document source is "okp"
167119
And Each referenced_document has a non-empty document_id
168120

169-
Examples: Offline
170-
| mode | config | doc_url_domain |
171-
| Offline | lightspeed-stack-okp-tool-offline.yaml | localhost:8081 |
172-
173-
Examples: Online
174-
| mode | config | doc_url_domain |
175-
| Online | lightspeed-stack-okp-tool-online.yaml | access.redhat.com |
121+
# ── Tool RAG — Responses API (online) ──
176122

177-
# ── Tool RAG — Responses API ──
178-
179-
Scenario Outline: <mode> responses API with OKP tool RAG has rag results returned
180-
Given The service uses the <config> configuration
123+
Scenario: Online responses API with OKP tool RAG has rag results returned
124+
Given The service uses the lightspeed-stack-okp-tool-online.yaml configuration
181125
And The service is restarted
182126
When I use "responses" to ask question with authorization header
183127
"""
184128
{
185-
"input": "configure remote desktop using gnome",
129+
"input": "Troubleshooting guide",
186130
"model": "{PROVIDER}/{MODEL}",
187131
"stream": false,
188132
"instructions": "You MUST use the file_search tool to answer."
@@ -193,37 +137,21 @@ Feature: OKP(Solr) RAG retrieval tests
193137
And The response contains non-empty tool_calls
194138
And A tool_call has type "file_search"
195139
And The response contains non-empty results
196-
And The number of results returned is <max_chunks>
140+
And The number of results returned is 3
197141
And Each rag_chunk has a non-empty score
198142
And Each rag_chunk source is "okp"
199-
And Each rag_chunk reference_url contains "<doc_url_domain>"
200-
201-
Examples: Offline
202-
| mode | config | max_chunks | doc_url_domain |
203-
| Offline | lightspeed-stack-okp-tool-offline.yaml | 5 | localhost:8081 |
143+
And Each rag_chunk reference_url contains "access.redhat.com"
204144

205-
Examples: Online
206-
| mode | config | max_chunks | doc_url_domain |
207-
| Online | lightspeed-stack-okp-tool-online.yaml | 1 | access.redhat.com |
208-
209-
# # ── OKP Server Unavailable — Graceful Error Handling ───────────────
145+
# ── OKP Server Unavailable — Graceful Error Handling ──
210146

211147
Scenario: Query succeeds with empty rag_chunks when OKP server is unavailable
212-
Given The OKP(Solr) server is stopped
148+
Given The service uses the lightspeed-stack-okp-online.yaml configuration
149+
And The service is restarted
150+
And The OKP(Solr) server is stopped
213151
When I use "query" to ask question with authorization header
214152
"""
215153
{"query": "configure remote desktop using gnome", "model": "{MODEL}", "provider": "{PROVIDER}"}
216154
"""
217155
Then The status code of the response is 200
218156
And The response contains no rag_chunks
219-
And The response contains no referenced_documents
220-
221-
Scenario: Streaming query succeeds with empty referenced_documents when OKP server is unavailable
222-
Given The OKP(Solr) server is stopped
223-
When I use "streaming_query" to ask question with authorization header
224-
"""
225-
{"query": "configure remote desktop using gnome", "model": "{MODEL}", "provider": "{PROVIDER}"}
226-
"""
227-
Then The status code of the response is 200
228-
And I wait for the response to be completed
229-
And The response contains no referenced_documents
157+
And The response contains no referenced_documents

0 commit comments

Comments
 (0)