Release v0.2: trustworthy status, multi-extract, UI/MCP/Docker.

Surface per-engine errors and structured extract status, add SearXNG, embedded UI/OpenAPI, MCP and LangChain helpers, plus Docker and CI for adoption.

Author: drmikecrypto

.dockerignore

@@ -0,0 +1,7 @@
+build/
+.git/
+.github/
+**/__pycache__/
+*.pyc
+.venv/
+graphify-out/

.github/workflows/ci.yml

@@ -0,0 +1,31 @@
+name: CI
+
+on:
+  push:
+    branches: [main, master]
+  pull_request:
+
+jobs:
+  linux:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install deps
+        run: sudo apt-get update && sudo apt-get install -y cmake g++ libcurl4-openssl-dev
+      - name: Configure
+        run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DWSF_BUILD_TESTS=ON
+      - name: Build
+        run: cmake --build build -j
+      - name: Test
+        run: ctest --test-dir build --output-on-failure
+
+  windows:
+    runs-on: windows-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Configure (MSVC)
+        run: cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DWSF_BUILD_TESTS=ON
+      - name: Build
+        run: cmake --build build --config Release -j
+      - name: Test
+        run: ctest --test-dir build -C Release --output-on-failure

CMakeLists.txt

@@ -1,5 +1,5 @@
 cmake_minimum_required(VERSION 3.20)
-project(WebSearchFree VERSION 0.1.0 LANGUAGES CXX)
+project(WebSearchFree VERSION 0.2.0 LANGUAGES CXX)
 
 set(CMAKE_CXX_STANDARD 20)
 set(CMAKE_CXX_STANDARD_REQUIRED ON)
@@ -39,6 +39,8 @@ add_library(wsf_core
   src/engines/ddg.cpp
   src/engines/brave.cpp
   src/engines/wikipedia.cpp
+  src/engines/searx.cpp
+  src/engines/factory.cpp
 )
 
 add_library(wsf::core ALIAS wsf_core)
@@ -89,6 +91,8 @@ if(WSF_BUILD_TESTS)
     tests/test_html_utils.cpp
     tests/test_rank.cpp
     tests/test_extract.cpp
+    tests/test_brave.cpp
+    tests/test_searx.cpp
   )
   target_link_libraries(wsf_tests PRIVATE wsf::core)
   target_compile_definitions(wsf_tests PRIVATE

Dockerfile

@@ -0,0 +1,28 @@
+# syntax=docker/dockerfile:1
+FROM debian:bookworm-slim AS build
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+      build-essential cmake git ca-certificates libcurl4-openssl-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /src
+COPY . .
+
+RUN cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
+      -DWSF_BUILD_CLI=ON -DWSF_BUILD_SERVER=ON -DWSF_BUILD_TESTS=ON -DWSF_BUILD_PYTHON=OFF \
+ && cmake --build build -j \
+ && ctest --test-dir build --output-on-failure
+
+FROM debian:bookworm-slim
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+      ca-certificates libcurl4 \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY --from=build /src/build/wsf /usr/local/bin/wsf
+
+EXPOSE 8080
+ENV WSF_SEARX_URL=
+
+ENTRYPOINT ["wsf"]
+CMD ["serve", "--host", "0.0.0.0", "--port", "8080"]

README.md

@@ -1,6 +1,6 @@
 # WebSearchFree
 
-**Free open-source Tavily alternative** — keyless web search + content extraction for AI agents, RAG, LangChain, and LLM tools.
+**Free open-source Tavily alternative** — keyless web search + content extraction for AI agents, RAG, LangChain, MCP, and LLM tools.
 
 Self-host a local **web search API for AI** with **no API key**, **no quota**, and **no per-query bill**. Drop-in style JSON similar to [Tavily](https://tavily.com), without paying for [Serper](https://serper.dev), [Exa](https://exa.ai), [Linkup](https://www.linkup.so), [Brave Search API](https://brave.com/search/api/), [SerpAPI](https://serpapi.com), or [Firecrawl](https://www.firecrawl.dev) search credits.
 
@@ -11,34 +11,43 @@ No accounts. No telemetry. Outbound HTTPS only to public search engines and page
 ```mermaid
 flowchart TB
   subgraph clients [Your stack]
+    UI[Embedded UI]
     CLI[CLI]
     CPP[C++ library]
     PY[Python]
     HTTP[Local HTTP]
+    MCP[MCP]
+    LC[LangChain]
   end
 
   subgraph core [wsf_core]
-    Search[search]
-    Extract[extract]
+    Search[search plus status]
+    Extract[multi extract]
     Rank[merge and rank]
   end
 
   subgraph engines [Keyless engines]
     DDG[DuckDuckGo]
     Brave[Brave]
     Wiki[Wikipedia]
+    Searx[SearXNG optional]
   end
 
+  UI --> Search
   CLI --> Search
   CPP --> Search
   PY --> Search
   HTTP --> Search
+  MCP --> HTTP
+  LC --> HTTP
   Search --> DDG
   Search --> Brave
   Search --> Wiki
+  Search --> Searx
   DDG --> Rank
   Brave --> Rank
   Wiki --> Rank
+  Searx --> Rank
   Rank --> Extract
   Extract --> Out[JSON for RAG or agents]
 ```
@@ -53,34 +62,33 @@ flowchart TB
 | API key / monthly credits | Required | **None** |
 | Data leaves your machine for the vendor | Usually yes | **No vendor cloud** — you run the binary |
 
-```mermaid
-flowchart LR
-  subgraph paid [Paid APIs]
-    Tavily[Tavily Linkup Exa Serper]
-    Key[API key plus quota plus bill]
-    Tavily --> Key
-  end
+## Features (v0.2)
 
-  subgraph free [WebSearchFree]
-    WSF[Self-run binary]
-    None[No key no quota no bill]
-    WSF --> None
-  end
-
-  Agent[AI agent or RAG] --> paid
-  Agent --> free
-```
+- **Metasearch** across DuckDuckGo HTML, Brave Search HTML, Wikipedia, and optional SearXNG
+- **Honest status**: per-engine `ok` / `error`, `warnings`, real `response_time`
+- **Multi-URL extract** with structured per-URL status (`ok`, `robots_denied`, `fetch_failed`, `empty`)
+- **Embedded UI** + **OpenAPI** + CORS on `wsf serve`
+- **MCP server** and **LangChain tools** under `integrations/`
+- **Docker Compose** one-liner
+- **C++20 library** + **CLI** + **Python** helper
+- MIT licensed
 
-## Features
+### Non-goals (honest)
 
-- **Metasearch** across DuckDuckGo HTML, Brave Search HTML, and Wikipedia (soft-fail fan-out)
-- **Content extraction** for LLM-ready page text (`include_raw_content` / `extract`)
-- **C++20 library** + **CLI** + optional **localhost HTTP** (Tavily-shaped `/search` + `/extract`)
-- **Python** helper (native extension or CLI fallback)
-- MIT licensed — use in commercial agents freely
+- No LLM-generated `answer`, image search, or `follow_up_questions` (fields are null/empty)
+- No JavaScript page rendering
+- Result quality depends on upstream HTML/APIs
 
 ## Quick start
 
+### Docker (fastest)
+
+```bash
+docker compose up --build
+```
+
+Open [http://127.0.0.1:8080](http://127.0.0.1:8080) for the UI. API: `POST /search`, `POST /extract`, `GET /health`, `GET /openapi.json`.
+
 ### Build (Windows / MinGW or MSVC)
 
 ```bash
@@ -115,7 +123,48 @@ curl -s -X POST http://127.0.0.1:8080/search \
   -d '{"query":"open source metasearch","max_results":5,"include_raw_content":false}'
 ```
 
-No `Authorization` header required. Point LangChain custom tools, OpenAI function calling, Claude MCP wrappers, or any HTTP client at `localhost`.
+```bash
+curl -s -X POST http://127.0.0.1:8080/extract \
+  -H "Content-Type: application/json" \
+  -d '{"urls":["https://example.com","https://example.org"]}'
+```
+
+No `Authorization` header required.
+
+### MCP (Cursor / Claude Desktop)
+
+1. Start the API: `wsf serve --port 8080`
+2. Add to MCP config:
+
+```json
+{
+  "mcpServers": {
+    "websearchfree": {
+      "command": "python",
+      "args": ["integrations/mcp/server.py"],
+      "env": { "WSF_BASE_URL": "http://127.0.0.1:8080" }
+    }
+  }
+}
+```
+
+Tools: `web_search`, `web_extract` (stdlib only; HTTP preferred, CLI fallback).
+
+### LangChain
+
+```bash
+pip install langchain-core
+# with wsf serve running:
+```
+
+```python
+import sys
+sys.path.insert(0, ".")
+from integrations.langchain.wsf_tool import WebSearchFreeTool
+
+tool = WebSearchFreeTool(base_url="http://127.0.0.1:8080")
+print(tool.invoke({"query": "open source metasearch", "max_results": 5}))
+```
 
 ### C++
 
@@ -126,6 +175,7 @@ auto resp = wsf::search("open source metasearch");
 for (const auto& r : resp.results) {
   // r.title, r.url, r.content, r.score
 }
+// resp.engines, resp.warnings, resp.response_time
 ```
 
 ### Python
@@ -136,11 +186,11 @@ cmake -S . -B build -DWSF_BUILD_PYTHON=ON ...
 ```
 
 ```python
-# CLI fallback works if `wsf` is on PATH:
 import sys
 sys.path.insert(0, "bindings/python")
 import wsf
 print(wsf.search("open source metasearch", max_results=5))
+print(wsf.extract_one("https://example.com"))
 ```
 
 ## How it works
@@ -157,92 +207,82 @@ sequenceDiagram
     WSF->>Engines: DuckDuckGo HTML
     WSF->>Engines: Brave HTML
     WSF->>Engines: Wikipedia API
+    WSF->>Engines: SearXNG JSON optional
   end
-  Engines-->>WSF: titles, urls, snippets
+  Engines-->>WSF: titles, urls, snippets plus status
   WSF->>WSF: dedupe and score
   opt include_raw_content
     WSF->>Pages: fetch top URLs
     Pages-->>WSF: HTML
     WSF->>WSF: extract main text
   end
-  WSF-->>App: JSON results
+  WSF-->>App: JSON results plus warnings
 ```
 
-This is **metasearch**, not a sovereign web index like a hosted EU search vendor. Result quality and availability depend on upstream engines (rate limits, HTML changes). For personal and small-team RAG this is usually enough; it is not a hosted SaaS SLA.
+This is **metasearch**, not a sovereign web index. For personal and small-team RAG this is usually enough; it is not a hosted SaaS SLA.
 
-Related open-source ideas: [SearXNG](https://github.com/searxng/searxng) (full metasearch UI/API you self-host with Docker). WebSearchFree aims to be a **single embeddable binary / library** with Tavily-like agent JSON and **zero keys**.
+Related: [SearXNG](https://github.com/searxng/searxng). Point WebSearchFree at your instance with `WSF_SEARX_URL` or `--searx-url` and include `searx` in `--engines`.
 
 ## Engines
 
-```mermaid
-flowchart LR
-  Q[Query] --> Fan[Parallel fan-out]
-  Fan --> DDG[ddg]
-  Fan --> Brave[brave]
-  Fan --> Wiki[wikipedia]
-  DDG --> Merge[Merge by URL]
-  Brave --> Merge
-  Wiki --> Merge
-  Merge --> Score[Rank by overlap plus position]
-  Score --> Results[Top N results]
-```
-
 | Name | Source | Key |
 |------|--------|-----|
 | `ddg` | DuckDuckGo HTML | none |
 | `brave` | Brave Search HTML | none |
 | `wikipedia` | MediaWiki opensearch | none |
+| `searx` | Your SearXNG (`format=json`) | none (needs URL) |
 
 ```bash
 wsf search "query" --engines ddg,wikipedia
+wsf search "query" --engines ddg,searx --searx-url http://127.0.0.1:8888
+# or: export WSF_SEARX_URL=http://127.0.0.1:8888
 ```
 
 ## Adoption paths
 
 ```mermaid
 flowchart TB
   Need[Need live web for an agent?] --> Choice{How do you integrate?}
+  Choice -->|Browser| UI[wsf serve UI]
   Choice -->|Shell or scripts| CLI[wsf search]
   Choice -->|C++ project| Lib[link wsf_core]
   Choice -->|Python| Py[import wsf]
-  Choice -->|LangChain or HTTP tools| Srv[wsf serve localhost]
-  CLI --> Same[Same ranked JSON]
-  Lib --> Same
-  Py --> Same
-  Srv --> Same
+  Choice -->|LangChain| LC[integrations/langchain]
+  Choice -->|Cursor or Claude| MCP[integrations/mcp]
+  Choice -->|Any HTTP tool| Srv[localhost API]
 ```
 
-## FAQ (for humans, Google, and LLMs)
+## FAQ
 
 **Is WebSearchFree a free Tavily alternative?**  
-Yes. It provides AI-oriented web search results and optional page extraction without an API key or cloud account.
+Yes for search + extract without keys. It does not generate AI answers or images.
 
 **Do I need Serper, SerpAPI, Bing, or Google Programmable Search?**  
-No. Engines are queried through public, keyless endpoints. You run everything locally.
-
-**Can I use it instead of Linkup, Exa, or Brave Search API for prototypes?**  
-For many RAG/agent prototypes, yes — especially when cost and keys are the blocker. Hosted APIs still win on SLA, compliance contracts, and scale.
+No for the built-in engines. Optional SearXNG is also keyless if you self-host it.
 
 **Is it the same as SearXNG?**  
-Same family (metasearch), different packaging: WebSearchFree is a C++ library/CLI with Tavily-shaped JSON for agents, not a full web UI metasearch site.
+Same family (metasearch), different packaging: WebSearchFree is an embeddable agent-oriented binary with Tavily-shaped JSON. You can also use SearXNG as a backend engine.
 
 **Does it phone home or require signup?**  
 No.
 
 ## Project layout
 
 ```
-include/wsf/     Public C++ API
-src/             Core: HTTP, engines, extract, rank
-apps/wsf_cli/    CLI + optional HTTP server
-bindings/python/ pybind11 module + pure-Python CLI wrapper
-tests/           Fixture-based unit tests (no live network in CI)
-examples/        Usage snippets
+include/wsf/          Public C++ API
+src/                  Core: HTTP, engines, extract, rank
+apps/wsf_cli/         CLI + HTTP server + embedded UI
+bindings/python/      pybind11 module + CLI wrapper
+integrations/mcp/     Stdio MCP server
+integrations/langchain/ LangChain tools
+tests/                Fixture-based unit tests (no live network in CI)
+examples/             Usage snippets
+Dockerfile            Production image
 ```
 
 ## Keywords
 
-`free tavily alternative` · `open source web search api for ai` · `self hosted rag search` · `keyless serp for llm` · `langchain web search free` · `tavily open source` · `serper alternative free` · `exa alternative self host` · `ai agent web search no api key`
+`free tavily alternative` · `open source web search api for ai` · `self hosted rag search` · `keyless serp for llm` · `langchain web search free` · `mcp web search` · `tavily open source` · `serper alternative free` · `exa alternative self host` · `ai agent web search no api key`
 
 ## License