bonsai-ollama

Repository: github.com/eSlider/bonsai-ollama · Releases: latest · Module: github.com/eSlider/bonsai-ollama (go.mod at repo root)

Run PrismML Bonsai 1.7B (GGUF Q1_0) with the Ollama CLI and HTTP API even though the stock Ollama engine cannot load this quantization yet. This repository ships a small Go reverse proxy that forwards Bonsai traffic to PrismML’s llama-server and everything else to a normal ollama serve.

Upstream weights & paper trail: Hugging Face — prism-ml/Bonsai-1.7B-gguf (Apache-2.0) · Bonsai-demo · Ollama import docs

Registry (same GGUF; still needs this proxy until Ollama supports Q1_0): ollama.com/eslider/bonsai-1.7b — hub-facing copy lives in models/bonsai-1.7b/README.md; maintainers can push summary + readme with bin/publish_ollama_hub_readme (needs OLLAMA_COM_COOKIE from a signed-in browser session; build via ./bin/setup.sh or ./bin/run.sh).

---

Table of contents

• Why this exists
• How it works
• Quick start
• Full setup
• Daily use
• HTTP API examples
• Streaming
• Performance (CPU benchmarks)
• Configuration
• Troubleshooting
• Repository layout
• Importing without the proxy (expect failure on run)
• Optional: run Prism only
• Contributing
• License

---

Why this exists

Bonsai ships in GGML Q1_0 (1-bit, g128 scales). Ollama’s bundled ggml build currently ends its type enum before Q1_0, so the runner fails while loading tensors (model failed to load / HTTP 500). That is a missing tensor type in Ollama, not an out-of-memory error.

PrismML’s llama-server builds include the kernels this GGUF needs. This project sits in front of Ollama: it runs llama-server locally and translates OpenAI-style SSE streaming from Prism into Ollama’s streaming JSON API for Bonsai, while reverse-proxying all other routes to your existing Ollama daemon.

---

How it works

flowchart LR
  subgraph clients [Clients]
    CLI[ollama CLI]
    API[HTTP clients]
  end
  subgraph proxy [bonsai-ollama-proxy :11434]
    R[Router]
  end
  subgraph backends [Backends]
    LS[llama-server :9988]
    OS[ollama serve :11435]
  end
  CLI --> R
  API --> R
  R -->|"bonsai-1.7b / eslider/bonsai-1.7b chat/generate"| LS
  R -->|"everything else"| OS

Loading

Port	Process	Role
11434	bonsai-ollama-proxy	What you point OLLAMA_HOST at
11435	ollama serve	Normal models, ollama pull, etc.
9988	llama-server	Loads Bonsai-1.7B-Q1_0.gguf, OpenAI-compatible /v1/chat/completions

---

Quick start

git clone https://github.com/eSlider/bonsai-ollama.git
cd bonsai-ollama

./bin/setup.sh   # GGUF + Prism llama-server + go build (see [Full setup](#full-setup))

./bin/run.sh     # frees 11434/11435/9988, starts backend Ollama + proxy

export OLLAMA_HOST=http://127.0.0.1:11434
ollama run eslider/bonsai-1.7b "Say hello in one sentence."

---

Full setup

One-command setup (recommended)

./bin/setup.sh

bin/setup.sh performs the download / extract / build steps documented below: Hugging Face GGUF, pinned Prism Ubuntu x64 llama-server tarball under vendor/prism-llama/, and go build for bin/bonsai-ollama-proxy plus the helper CLIs bin/bench_llama_tokens, bin/verify_stream, and bin/publish_ollama_hub_readme. It does not install Ollama (install that separately), and it does not start daemons — use ./bin/run.sh after setup.

• ./bin/setup.sh --force — re-download the GGUF, re-fetch the tarball, remove the old Prism extract, and rebuild.
• Overrides — BONSAI_SETUP_GGUF_URL, BONSAI_SETUP_PRISM_TAR_URL, BONSAI_SETUP_GGUF_PATH (see ./bin/setup.sh --help).

Prerequisites

Requirement	Notes
Go 1.22+	Builds proxy + tools from repo-root go.mod (setup.sh and run.sh)
curl, tar	Used by bin/setup.sh
Ollama	Backend on port 11435 (not installed by setup.sh)
fuser (optional)	From psmisc on Debian/Ubuntu — bin/bonsai-ollama-stack.sh uses it to free ports

Manual setup (same as ./bin/setup.sh)

1. Download the GGUF (~237 MiB)

Official file: Bonsai-1.7B-Q1_0.gguf

mkdir -p models/bonsai-1.7b
curl -fL -o models/bonsai-1.7b/Bonsai-1.7B-Q1_0.gguf \
  "https://huggingface.co/prism-ml/Bonsai-1.7B-gguf/resolve/main/Bonsai-1.7B-Q1_0.gguf"

2. Download Prism llama-server (Ubuntu x64 CPU)

Release asset (pinned version in this repo): llama-prism-b8846-d104cf1-bin-ubuntu-x64.tar.gz

mkdir -p vendor/prism-llama && cd vendor/prism-llama
curl -fL -o prism.tar.gz \
  "https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b8846-d104cf1/llama-prism-b8846-d104cf1-bin-ubuntu-x64.tar.gz"
tar -xzf prism.tar.gz
cd ../..

Other platforms (CUDA, Vulkan, macOS, etc.) are on the PrismML-Eng/llama.cpp releases page. Extract into vendor/prism-llama/ and set BONSAI_PRISM_LIB_DIR to the folder that contains llama-server and its .so / .dylib files.

3. Build Go binaries

Build the proxy and CLI tools into bin/. Skip if you already ran ./bin/setup.sh (it runs the same go build lines).

cd /path/to/bonsai-ollama   # repository root (contains go.mod)
go build -o bin/bonsai-ollama-proxy ./cmd/bonsai-ollama-proxy
go build -o bin/bench_llama_tokens ./cmd/bench-llama-tokens
go build -o bin/verify_stream ./cmd/verify-stream
go build -o bin/publish_ollama_hub_readme ./cmd/publish-ollama-hub-readme

4. Run the stack

Stop anything already bound to 11434, 11435, and 9988 (or let the script try fuser -k).

./bin/run.sh

• Rebuilds bin/bonsai-ollama-proxy and the helper CLIs when missing or when any cmd/*/…/*.go is newer than the binary, then runs bin/bonsai-ollama-stack.sh.
• Backend logs: /tmp/ollama-bonsai-backend.log.

---

Daily use

Point every Ollama client at the proxy (not the backend port):

export OLLAMA_HOST=http://127.0.0.1:11434
ollama list
ollama run eslider/bonsai-1.7b "Your prompt"
ollama run qwen3:4b "Other models still go to backend :11435"

OLLAMA_HOST is documented in the Ollama FAQ / Linux.

---

HTTP API examples

Chat (non-stream):

curl -sS http://127.0.0.1:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{"model":"eslider/bonsai-1.7b","messages":[{"role":"user","content":"Hi"}],"stream":false}'

Generate (non-stream):

curl -sS http://127.0.0.1:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model":"eslider/bonsai-1.7b","prompt":"Hello","stream":false}'

---

Streaming

For each data: line from llama-server’s OpenAI SSE stream, the proxy emits one Ollama NDJSON object with that text delta and Flushes immediately, so backend token granularity is preserved.

curl -sS -N -X POST http://127.0.0.1:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{"model":"eslider/bonsai-1.7b","messages":[{"role":"user","content":"Count 1 2 3"}],"stream":true}' \
| ./bin/verify_stream

(./bin/verify_stream is produced by ./bin/setup.sh or ./bin/run.sh; otherwise run the go build … ./cmd/verify-stream line from §3.)

bin/verify_stream checks that chunks arrive and that there are no multi-second stalls (sources in cmd/verify-stream).

---

Performance (CPU benchmarks)

Bonsai inference in this stack is executed by Prism llama-server (OpenAI-compatible /v1/chat/completions). The proxy’s job is routing and stream translation; end-to-end generation speed is dominated by llama-server on your CPU (or GPU build, if you use a Prism release with CUDA/Vulkan and set BONSAI_PRISM_LIB_DIR accordingly).

The table below uses the server’s built-in timings object (tokens per second for the prompt and prediction phases), which matches how llama.cpp server reports throughput.

Metric	Result
Decode (max_tokens=256, 5 runs, temperature 0.75)	~81 tok/s mean (σ ≈ 1.5; ~70–74 tok/s on this host)
Prefill (~480-token article prompt, max_tokens=8, 3× warmup on the same prompt then 5 measured runs)	~61 tok/s mean (σ ≈ 4.7)

Measured environment (2026-04-22, re-run after bin/setup.sh / proxy rebuild):

CPU	AMD Ryzen 7 5800H (8 cores / 16 threads), x86_64
OS	Linux 7.0.0-070000rc7-generic
GGUF	Bonsai-1.7B-Q1_0.gguf (HF)
Binary	Prism llama-server tarball llama-prism-b8846-d104cf1-bin-ubuntu-x64 (release); API system_fingerprint: b8846-d104cf1b6
Endpoint	http://127.0.0.1:9988 (same process the proxy supervises)

Your numbers will differ with other CPUs, power/thermal limits, concurrent load, and thread / batch settings on llama-server. Throughput also drifts between runs on the same machine.

Reproduce:

./bin/setup.sh   # once per machine (GGUF + Prism + go build)
./bin/run.sh     # wait until llama-server answers on 9988
./bin/bench_llama_tokens --runs 5 --json

bin/bench_llama_tokens prints a small JSON summary (mean / stdev / min / max); sources in cmd/bench-llama-tokens. Point at another host or port with BONSAI_LLAMA_URL=http://127.0.0.1:9988.

---

Configuration

Environment variables (optional). Full notes: models/bonsai-1.7b/OLLAMA.txt.

Variable	Default	Meaning
BONSAI_REPO_ROOT	parent of proxy binary ../..	Root for default GGUF / Prism paths
BONSAI_GGUF	models/bonsai-1.7b/Bonsai-1.7B-Q1_0.gguf under root	Path to GGUF
BONSAI_PRISM_LIB_DIR	vendor/prism-llama/llama-prism-b8846-d104cf1 under root	Directory with llama-server + libs
BONSAI_PROXY_LISTEN	127.0.0.1:11434	Proxy listen address
BONSAI_OLLAMA_BACKEND	http://127.0.0.1:11435	Upstream Ollama
BONSAI_LLAMA_PORT	9988	llama-server port
OLLAMA_BIN	/usr/local/bin/ollama	Used by bonsai-ollama-stack.sh only
BONSAI_LLAMA_URL	http://127.0.0.1:9988	Default --base for bin/bench_llama_tokens (Prism llama-server HTTP, not Ollama)
OLLAMA_COM_COOKIE	(unset)	Browser session cookie for bin/publish_ollama_hub_readme (see sources; --dry-run supported)

---

Troubleshooting

Symptom	What to check
Address already in use	Free 11434 / 11435 / 9988 or change ports via env vars.
llama-server not found	BONSAI_PRISM_LIB_DIR must contain the extracted Prism binaries.
GGUF not found	BONSAI_GGUF path; run ./bin/setup.sh or the manual GGUF download.
ollama run hangs in CI	Use a real TTY or call /api/chat with curl / your HTTP client.
Stock Ollama still 500 on Bonsai	You must talk to the proxy (OLLAMA_HOST=…:11434), not raw :11435.
go: cannot find main module / missing go.mod	Run go / ./bin/run.sh from the repository root (where go.mod lives), not from cmd/….
./bin/verify_stream or ./bin/bench_llama_tokens not found	Run ./bin/setup.sh once, or ./bin/run.sh (builds missing bin/ tools), or build manually per §3.

---

Repository layout

Path	Purpose
go.mod	Go module github.com/eSlider/bonsai-ollama (all cmd/* packages build from repo root)
cmd/bonsai-ollama-proxy/	Go source: proxy + llama-server supervisor
cmd/bench-llama-tokens/	llama-server token benchmark (bin/bench_llama_tokens)
cmd/verify-stream/	Streaming sanity reader (bin/verify_stream)
cmd/publish-ollama-hub-readme/	Ollama Hub readme/summary publisher
bin/bonsai-ollama-stack.sh	Starts backend Ollama + proxy
bin/verify_stream	Quick streaming sanity check (built Go binary)
bin/bench_llama_tokens	CPU token throughput (uses llama-server timings)
bin/setup.sh	Full local setup: GGUF + Prism tarball + go build
bin/run.sh	Build when needed (stale .go detection) + exec stack
models/bonsai-1.7b/Modelfile	ollama create recipe (weights not in git)
models/bonsai-1.7b/OLLAMA.txt	Extra operational notes
models/bonsai-1.7b/README.md	Text for Ollama Hub (summary + readme)
bin/publish_ollama_hub_readme	POST hub summary/readme (OLLAMA_COM_COOKIE)
SECURITY.md	Security reporting
.github/workflows/ci.yml	CI: go vet ./..., build proxy + tools

CLI programs under cmd/*/main.go start with a ///usr/bin/true; exec /usr/bin/env go run "$0" "$@" line comment (it is valid Go: // begins the line comment). A real #! shebang cannot be the first bytes of a .go file because go build / go vet reject #. Day-to-day use: ./bin/<tool> (compiled by ./bin/setup.sh / ./bin/run.sh) or go run ./cmd/…. The main.go sources are stored as executable (100755) in git for environments that run them via a shell wrapper around that idiom.

---

Importing without the proxy (expect failure on run)

From models/bonsai-1.7b/:

ollama create bonsai-1.7b -f Modelfile

This registers the blob, but ollama run keeps failing until Ollama ships Q1_0 support. Use the proxy stack for inference today.

---

Optional: run Prism only

cd vendor/prism-llama/llama-prism-b8846-d104cf1
LD_LIBRARY_PATH="$PWD" ./llama-server \
  -m ../../../models/bonsai-1.7b/Bonsai-1.7B-Q1_0.gguf \
  --host 127.0.0.1 --port 9988

Then use OpenAI-compatible POST /v1/chat/completions. See Bonsai-demo for more integration examples.

---

Contributing

Issues and PRs are welcome. When changing Go code, from the repository root (where go.mod lives), run:

go vet ./... && go test ./...

(go test is a no-op until tests exist; go vet should be clean.)

The cmd/bench-llama-tokens, cmd/verify-stream, and cmd/publish-ollama-hub-readme main.go files use a fixed first-line ///usr/bin/true; … comment idiom; gofmt rewrites /// to // /, so do not run gofmt on those files unless you intend to change that convention.

CI on every push to main runs go vet ./..., builds cmd/bonsai-ollama-proxy, and builds the three helper cmd/* tools (see .github/workflows/ci.yml). If git push asks for credentials in a headless environment, run gh auth setup-git once (requires the GitHub CLI logged in).

---

License

• This repository (Go proxy + cmd/* tools, bin/*.sh, docs): MIT.
• Bonsai weights & Prism upstream: Apache-2.0 on Hugging Face; follow their attribution and license terms when redistributing GGUF files.