You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3c0fd9e
Browse filesBrowse the repository at this point in the historyBrowse files
* LCORE-3048: add Konflux E2E coverage for OKP Solr RAG
Co-authored-by: Cursor <cursoragent@cursor.com>
* updated ls config to ogx confif
* renamd remnants of llama stack reference to ogx
* added a note on how to run OKP tests
* updated providers to the latest commit, removed skip tag from one scenario
* Revert providers submodule pin; image uses the Konflux zip.
Co-authored-by: Cursor <cursoragent@cursor.com>
* removed negative yaml, updated llama reference to ogx
* removing tag from makerfile
* added extra logs, cleaned up okp rag py
* addressing present-but-null response fields
* updating health check url
* updated cmd_check_okp_solr_from_llama to use the same portal-rag select URL as the probes
* updating PVC
* adding search mode
---------
Co-authored-by: Sofiya Nuryyeva <sonuryye@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Konflux only (`@konflux-only`) in CI. Targeted local Behave (match YAML to OGX/LCS): [Running OKP RAG tests locally](e2e_testing.md#running-okp-rag-tests-locally).
121
+
122
+
* Offline mode query with inline RAG returns rag_chunks and referenced_documents
123
+
* Online mode streaming query with inline RAG returns referenced_documents
124
+
* Query with inline RAG with dynamic semantic filter returns rag_chunks and referenced_documents
125
+
* Offline query API with OKP tool RAG has rag_chunk and referenced_documents returned
126
+
* Online responses API with OKP tool RAG has rag results returned
127
+
* Query succeeds with empty rag_chunks when OKP server is unavailable
128
+
* Streaming query succeeds with empty referenced_documents when OKP server is unavailable
129
+
* Query returns no rag_chunks and no referenced_documents when OKP is disabled
130
+
* Streaming query returns no referenced_documents when OKP is disabled
6.[Deployment Modes: Server vs Library](#deployment-modes-server-vs-library)
15
+
7.[Tags and Hooks](#tags-and-hooks)
16
+
8.[Configuration Files](#configuration-files)
17
+
9.[Feature Files and Steps](#feature-files-and-steps)
18
+
10.[Gherkin Keywords in Feature Files](#gherkin-keywords-in-feature-files)
19
+
11.[Choosing the Test Layer: E2E or Integration?](#choosing-the-test-layer-e2e-or-integration)
20
+
12.[Writing New Scenarios](#writing-new-scenarios)
21
+
13.[Troubleshooting](#troubleshooting)
21
22
22
23
---
23
24
@@ -136,6 +137,73 @@ uv run behave tests/e2e/features/health.feature --tags=-skip-in-library-mode
136
137
137
138
---
138
139
140
+
## Running OKP RAG tests locally
141
+
142
+
`okp_rag.feature` is `@konflux-only`. CI deploys OKP as a pod; `make test-e2e` skips the feature. Do not set `E2E_KONFLUX_E2E=1` locally (that path is Kubernetes). On a laptop, start OKP in Docker, enrich and run OGX and LCS as host processes, then run **one scenario that matches the YAML you started**.
`okp_rag.feature:24` is the offline inline scenario (`lightspeed-stack-okp-offline.yaml`). Comment `@konflux-only` on the Feature, `OKP(Solr) server is running` in Background, and `OGX is restarted` / `The service is restarted` on that scenario (OGX and LCS are already up). Do not commit those comments.
198
+
199
+
```bash
200
+
uv run behave tests/e2e/features/okp_rag.feature:24
201
+
```
202
+
203
+
For another YAML (`lightspeed-stack-okp-online.yaml`, tool RAG, and so on), repeat steps 1–3 with that file, then run the scenario line that uses it. Product setup: [OKP guide](../user_doc/okp_guide.md).
204
+
205
+
---
206
+
139
207
## Environment Variables
140
208
141
209
| Variable | Default | Description |
@@ -151,6 +219,8 @@ uv run behave tests/e2e/features/health.feature --tags=-skip-in-library-mode
151
219
|`E2E_DEFAULT_PROVIDER_OVERRIDE`| — | Override default provider id (e.g. `openai`). |
152
220
|`FAISS_VECTOR_STORE_ID`| — | Vector store id for FAISS-related scenarios. |
153
221
|`RUNNING_PROW`| — | Set in Prow/OpenShift; enables Prow config paths and pod/container ops. |
222
+
|`E2E_KONFLUX_E2E`| — |`1` in Konflux only. Unskips `@konflux-only` and deploys OKP as a pod. |
|`OPENAI_API_KEY`| — |**Required.** Used by the app and OGX for LLM calls (e.g. OpenAI). The E2E tests and the stack will not run correctly without it. |
155
225
156
226
@@ -176,6 +246,8 @@ All tag behaviour is implemented in **`features/environment.py`**: the hooks (`b
|`@skip-in-library-mode`| Scenario is skipped when `E2E_DEPLOYMENT_MODE=library`. Used for tests that require a separate OGX (e.g. connection disruption). |
180
252
|`@local`| Skipped unless running in "local" mode (context flag). |
181
253
|`@InvalidFeedbackStorageConfig`| Before scenario: switch to invalid-feedback-storage config and restart container. After: restore feature config and restart. |
@@ -246,6 +318,7 @@ The feature files below are run in the order given in `tests/e2e/test_list.txt`:
If you add a new feature file, add it to **`tests/e2e/test_list.txt`** so it is included when you run the full E2E suite (e.g. `make test-e2e`). The order in that file is the run order.
@@ -396,6 +469,6 @@ The integration side of this boundary is described in
396
469
-**Readonly database (SQLite) in OGX**: If the RAG KV DB is on a bind-mounted path that becomes read-only (e.g. after restart), move it to a named volume (e.g. via `KV_RAG_PATH` in docker-compose) so writes succeed.
397
470
-**ChunkedEncodingError on streaming_query**: The step for streaming_query uses `stream=True` and consumes the stream; if you add new streaming steps, avoid reading the full response with `response.content` and use the same stream-reading pattern so a server close after an error event does not raise.
398
471
-**Event loop is closed (httpx/AsyncClient)**: In E2E, any code that creates an `AsyncOgxClient` (e.g. for shields) must close it (e.g. `await client.close()`) in a `finally` block before the event loop is torn down (e.g. before `asyncio.run()` returns).
399
-
-**Scenarios skipped**: Check tags (`@skip`, `@skip-in-library-mode`, `@local`) and `E2E_DEPLOYMENT_MODE`; ensure the scenario is not excluded by `--tags=-skip` (or the opposite if you intend to run only skipped scenarios for debugging).
472
+
-**Scenarios skipped**: Check tags (`@skip`, `@skip-in-library-mode`, `@local`, `@konflux-only`) and `E2E_DEPLOYMENT_MODE`; ensure the scenario is not excluded by `--tags=-skip` (or the opposite if you intend to run only skipped scenarios for debugging).
400
473
401
474
For more on test structure and commands, see the main project guide (`CLAUDE.md`) and `tests/e2e/features/steps/README.md`.
Copy file name to clipboardExpand all lines: docs/user_doc/okp_guide.md
+50-18Lines changed: 50 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -51,12 +51,22 @@ validating that queries return referenced chunks.
51
51
52
52
## Step 1: Launch OKP
53
53
54
-
Start the OKP RAG service with Podman:
54
+
Start the OKP RAG service with Podman or Docker:
55
+
56
+
**Using Podman:**
55
57
56
58
```bash
59
+
podman login registry.redhat.io
57
60
podman run --rm -d -p 8081:8080 registry.redhat.io/offline-knowledge-portal/rhokp-rhel9:latest
58
61
```
59
62
63
+
**Using Docker:**
64
+
65
+
```bash
66
+
docker login registry.redhat.io
67
+
docker run --rm -d -p 8081:8080 registry.redhat.io/offline-knowledge-portal/rhokp-rhel9:latest
68
+
```
69
+
60
70
> **Note:** Remove `-d` to run in the foreground.
61
71
62
72
* The service listens on **port 8081** on the host (mapped from 8080 in the container). Lightspeed Stack itself listens on `8080`, so this avoids port conflicts.
@@ -105,41 +115,61 @@ Edit your Lightspeed Stack config file (e.g. `lightspeed-stack.yaml`) and add
105
115
the following top-level sections so that OKP is used for either inline or tool
106
116
RAG:
107
117
108
-
Inline RAG:
118
+
**Inline RAG:**
109
119
110
120
```yaml
111
121
# RAG configuration
112
122
rag:
113
-
inline:
114
-
- okp
115
-
okp:
116
-
rhokp_url: ${env.RH_SERVER_OKP}
117
-
offline: true
123
+
retrieval:
124
+
inline:
125
+
sources:
126
+
- okp
127
+
max_chunks: 3# Final tight cap on chunks returned to user
128
+
okp:
129
+
rhokp_url: ${env.RH_SERVER_OKP}
130
+
offline: true
131
+
max_chunks: 5# Initial generous fetch from OKP
118
132
```
119
133
120
-
Tool RAG:
134
+
**Tool RAG:**
121
135
122
136
```yaml
123
137
# RAG configuration
124
138
rag:
125
-
tool:
126
-
- okp
127
-
okp:
128
-
rhokp_url: ${env.RH_SERVER_OKP}
129
-
offline: true
139
+
retrieval:
140
+
tool:
141
+
sources:
142
+
- okp
143
+
max_chunks: 10# Tool RAG limit
144
+
okp:
145
+
rhokp_url: ${env.RH_SERVER_OKP}
146
+
offline: true
147
+
max_chunks: 5
130
148
```
131
149
132
-
* **`rag.inline`** and **`rag.tool`**: Enable OKP as the RAG source for inline context injection and for the RAG tool. Tool rag means the LLM will be provided a search tool it can choose to invoke to find relevant content and augment the user prompt. The tool may or may not be invoked. Inline means a rag search and prompt augmentation will always occur.
150
+
* **`rag.retrieval.inline`** and **`rag.retrieval.tool`**: Enable OKP as the RAG source for inline context injection and for the RAG tool. Tool RAG means the LLM will be provided a search tool it can choose to invoke to find relevant content and augment the user prompt. The tool may or may not be invoked. Inline RAG means a RAG search and prompt augmentation will always occur.
133
151
* **`okp.offline`**: When `true`, source URLs use `parent_id` (offline/Mimir-style). When `false`, use `reference_url` (online).
152
+
* **`okp.search_mode`**: Default Solr search strategy for OKP queries. Controls *how* Solr ranks chunks, not *which* documents are eligible (that is `chunk_filter_query` and per-request `solr.filters`). Allowed values:
153
+
- **`hybrid`** (default when the field is omitted): combines vector similarity with keyword (BM25) text search. Requires an embedding model. This is the global default.
154
+
- **`semantic`**: pure vector search against Solr embeddings. Requires an embedding model. Best when the query is conceptual rather than matching document wording.
155
+
- **`keyword`**: BM25 text search only. No embedding model is needed, so it is the right choice for air-gapped or embedding-free deployments, and it is typically faster than hybrid.
156
+
* **`max_chunks` behavior**:
157
+
- **Inline RAG**: `rag.retrieval.inline.max_chunks` is the final tight cap on chunks returned in the response, while `rag.okp.max_chunks` is the initial generous fetch from OKP before filtering and ranking.
158
+
- **Tool RAG**: `rag.retrieval.tool.max_chunks` sets the limit on chunks available to the tool.
134
159
135
160
If you want to filter the docs to a specific product, you can include a static query filter such as:
136
161
137
162
```yaml
138
163
okp:
139
164
offline: true
165
+
search_mode: keyword
140
166
chunk_filter_query: "product:*openshift* AND product_version:4.21"
141
167
```
142
168
169
+
A per-request `solr.mode` on `/v1/query` (or streaming/responses) overrides this YAML default for that call. Per-request `lexical` is treated as `keyword`.
170
+
171
+
> **Important:** When running OGX in **Server mode**, changes to static OKP settings such as `chunk_filter_query` and `search_mode` require an OGX restart to take effect. They are applied when Lightspeed Stack enriches OGX configuration at startup and will not propagate automatically during runtime.
172
+
143
173
When you launch Lightspeed Stack it will augment the OGX configuration (the synthesized run.yaml in unified mode, or your external run.yaml in the deprecated legacy mode) with
144
174
configuration for OKP.
145
175
@@ -258,7 +288,7 @@ curl -sX POST http://localhost:8080/v1/query \
258
288
- **Static filters preserved:** The configured `chunk_filter_query` (e.g., `"product:*openshift*"`) is always applied
259
289
- **Dynamic filters added:** Request filters are combined with static filters using AND logic
260
290
- **String escaping:** Special Solr characters in filter values are automatically escaped
261
-
- **Works with all search modes:** Filters apply to `semantic`, `hybrid`, and `lexical` search modes
291
+
- **Works with all search modes:** Filters apply whether the search mode is `semantic`, `hybrid`, or `keyword` (per-request `lexical` is an alias of `keyword`).
262
292
263
293
---
264
294
@@ -312,13 +342,15 @@ Example response excerpt:
312
342
],
313
343
```
314
344
315
-
> **Note:** The first time you query the system the response may take
316
-
> additional time because it must first download the necessary embedding model
317
-
> to perform the vector search.
345
+
> **Note:** The first time you query the system, `hybrid` or `semantic` search may take
346
+
> additional time because OGX must download the embedding model used for vector
347
+
> search. `keyword` search skips that download.
318
348
319
349
If you see no RAG context, verify:
320
350
321
351
1. OKP is up at http://localhost:8081
322
352
2. `lightspeed-stack.yaml` has `okp` under `rag.inline` and/or `rag.tool` as in Step 4
323
353
354
+
The Behave suite `okp_rag.feature` (`@cfg_okp`) runs in Konflux in CI. To stand up OKP + OGX + LCS on a laptop and run a matching scenario, see [Running OKP RAG tests locally](../testing/e2e_testing.md#running-okp-rag-tests-locally).
0 commit comments