Gravitee's inference server.

Singularitee hosts LLMs, classifiers, embedders and rerankers in a process of its own and runs multi-step pipelines over them. Whatever calls it (a Gravitee gateway, an OpenAI SDK, curl) stays lightweight; Singularitee owns the GPU, the native libraries and the model lifecycle. It is built on gravitee-node with Vert.x 5 and RxJava 3.

Why the name

A play on Gravitee. Models are heavy, and a singularity is what gravity produces at its extreme: all that mass collapsed into one point. Everything the gateway cannot carry (the GPU, the native libraries, the weights) is pulled out of it and concentrated into one process. Gravitee attracts; Singularitee is where it all ends up.

What it does

A workspace YAML declares the models and pipelines to publish. Weights download from HuggingFace on first start, then everything is served with streaming as a first-class primitive.

Capability	Notes
Text generation	Streamed token by token, from a model or from a pipeline.
Pipelines	A DAG of steps over those models: guards, classifiers, routers, loops, tool calling, sub-pipelines.
Classification	ONNX sequence and token classifiers, GLiNER zero-shot, regex, composite.
Embeddings and reranking	Vectors, cross-encoder reranking, similarity.
Discovery	List and inspect what a server publishes.

Two fronts serve the same registries and engines, so a capability behaves identically on either:

• gRPC (default, port 9090), the primary API: gRPC API
• HTTP (opt-in, port 8080), OpenAI-compatible: HTTP API, schemas in openapi/

Engines

Engine	Backend	Use case
llama.cpp	llamaj.cpp	GGUF models on CPU, Metal or CUDA. No Python. The default.
vLLM	vLLM4j	HuggingFace Transformers checkpoints through vLLM. CUDA first.
ONNX	ONNX Runtime	Classifiers, embedders and rerankers. CPU with optional GPU.
GLiNER	gliner4j	Zero-shot NER and classification.

Documentation

Everything lives under docs/. Read in this order:

1. Getting Started: build, run, first calls.
2. Concepts: workspaces, models, pipelines, templates, engines.
3. Workspaces: the YAML format.

Then, by need:

• Reference: model types, step types, templates, context fields, configuration
• APIs: overview and conventions, gRPC, HTTP, Java client, OpenAPI specs
• Guides: text generation, classification, embeddings and reranking, guards and redaction, routing, loops and chain-of-thought, tool calling, structured output, todos, sub-pipelines, multimodal, remote and multi-server
• Operations: deployment, observability, validated models
• Internals: Architecture

Quick start

From a fresh clone, with Java 25 and Maven:

./install.sh

This checks prerequisites, downloads the llama.cpp native libraries into ~/.llama.cpp (they are not bundled in the jar), builds the distribution and starts the server with the HTTP API on port 8080. From then on:

./run-server.sh --list                                          # every runnable workspace
./run-server.sh --workspace examples/llama/qwen3-0.6b.yaml      # small and fast
./run-server.sh --debug                                         # TRACE-log rendered prompts

Weights download on first start into ~/.cache/gravitee-singularitee/models. The port binds immediately; calls answer "Model server is still loading" until the workspace is loaded, so poll /v1/models rather than the TCP port.

Calling it

curl -s localhost:8080/v1/models | jq

curl -s localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"agent","messages":[{"role":"user","content":"Say hi in 3 words"}]}' | jq

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-local")
client.chat.completions.create(model="agent", messages=[{"role": "user", "content": "hi"}], stream=True)

model takes a model id or a pipeline id. Beyond the standard routes the server adds /v1/classify, /v1/rerank and /v1/similarity. Over gRPC, use SingulariteeClient (Java client) or grpcurl (gRPC API).

Smoke tests need only uv:

BASE_URL=http://localhost:8080/v1 uv run --with openai examples/scripts/openai_test.py
BASE_URL=http://localhost:8080/v1 uv run --with requests examples/scripts/classify_test.py

Configuration

Every key in gravitee.yml also accepts a GRAVITEE_-prefixed environment variable or a -D system property (grpc.port becomes GRAVITEE_GRPC_PORT). The ones needed to start:

Key	Default	Purpose
grpc.port	9090	The primary API.
http.enabled / http.port	false / 8080	The OpenAI-compatible HTTP API, opt-in.
ai.workspace.path	unset	Workspace loaded at startup. Unset starts an empty server.
ai.huggingface.token	$HF_TOKEN	Gated repositories.

The full list, with TLS, auth, the model cache, streaming back-pressure, the management port and OpenTelemetry, is in Configuration.

Workspaces

A workspace declares what to publish. Models carry a stable logical id, so the same pipeline runs against any backend bound to that id.

workspace:
  name: toxicity-guard
  models:
    - id: llm
      name: Qwen/Qwen3-0.6B-GGUF
      type: llama_cpp
      llama_cpp:
        path: Qwen3-0.6B-Q8_0.gguf
        n_ctx: 4096
        n_gpu_layers: 999
    - id: toxicity
      name: gravitee-io/distilbert-multilingual-toxicity-classifier
      type: onnx_classifier
      onnx_classifier:
        model_path: model.quant.onnx
        tokenizer_path: tokenizer.json
        classifier_mode: SEQUENCE
  pipelines:
    - id: toxicity-guard-pipeline
      entry: toxicity_guard
      steps:
        - id: toxicity_guard
          type: guard
          next_step: generate
          config:
            model_id: toxicity
            input_field: prompt
            action: reject
            trigger:
              label: toxic
              score: 0.75
        - id: generate
          type: infer
          role: output
          config:
            model_id: llm
            output_field: generate.output

Ready-made workspaces live in examples/: one folder per model family, multi-step pipelines/, and a modular/ tree composing servers from shared include fragments. Run any with ./run-server.sh --workspace <file>.

Production

Host one model (or one engine) per Singularitee process and compose them over gRPC with remote workspaces. Each engine loads its own native libraries, so co-locating engines or several large models in one JVM invites library conflicts and GPU-memory contention. See Deployment and Remote and multi-server.

Traces

Turn on OpenTelemetry (services.opentelemetry.enabled) and every request exports a span tree over OTLP: a pipeline CHAIN nesting each step, LLM steps carrying the model, token counts, sampling parameters and time-to-first-token, and guard/gate steps a GUARDRAIL span. Point it at Jaeger (GRAVITEE_SERVICES_OPENTELEMETRY_ENABLED=true, UI on :16686) to see the full tree and every attribute.

When a generation captured log-probabilities, its infer step also carries a family of free confidence signals on the span (and in the pipeline context as <step>.*): the mean perplexity, the top-1 vs top-2 token min_margin, per-token entropy, robust peaks and more. They fall out of the one generation the model already ran, so they cost no extra inference, and a downstream step can calibrate the one it wants against resolved outcomes. Which summary tracks correctness, if any, depends on the model and the task, so measure before trusting one. See Observability and Context fields.

Contributing

See CONTRIBUTING.md. In short: issues go to gravitee-io/issues; branch from main as issue/<id>-<name>; use Conventional Commits; run mvn clean install before opening a PR. The build enforces formatting and license headers; mvn prettier:write license:format fixes both.

Security

Report vulnerabilities through the process in SECURITY.md, not through a public issue.

License

Apache License 2.0. Copyright 2015 The Gravitee team (http://gravitee.io)