diff --git a/site/content/docs/concepts/authorization.md b/site/content/docs/concepts/authorization.md index 5eb7971a..4d89db9c 100644 --- a/site/content/docs/concepts/authorization.md +++ b/site/content/docs/concepts/authorization.md @@ -289,8 +289,7 @@ The header, the operator floor and the router selector follow the same matching rule: a map of `key=value` pairs, one value per key, and the peer must attest every pair. Listing more pairs narrows the set of acceptable peers, as it does in a Kubernetes label selector or a Prometheus matcher. -Labels seen in discovery results are only used to rank candidates. The only -labels that authorize anything are the signed ones in a credential. +The only labels that authorize anything are the signed ones in a credential. ## See also diff --git a/site/content/docs/concepts/networking.md b/site/content/docs/concepts/networking.md index 08462aec..c317ea28 100644 --- a/site/content/docs/concepts/networking.md +++ b/site/content/docs/concepts/networking.md @@ -26,11 +26,13 @@ is a good setup when everything that uses the node runs as the same user. The token travels in `X-Mesh-Authentication` and not in `Authorization`, so that `Authorization` always means the same thing: the credential for the service that is being called through the node. On the proxy path that header -is forwarded unchanged. The endpoints that never forward anything (`/mcp`, -`/v1/*`, `/mesh/service/*`) also accept the token in `Authorization`, for -clients that cannot set a custom header. +is forwarded unchanged. Every other route that needs the token also +accepts it in `Authorization`, for clients that cannot set a custom header. +The node removes the header that carried the token. On `/v1/*`, an +`Authorization` header that is still present is the backend's credential. +The node forwards it as on the proxy path. -The API has four parts: +An agent mostly uses four parts of the API: | Path | What it is | |---|---| @@ -39,7 +41,12 @@ The API has four parts: | `/mesh////` | A reverse proxy to one specific service on one specific node, for every service type. `` is `mcp`, `inference` or `a2a`, and `` is passed through to the backend. This is how a caller reaches an A2A agent: `/mesh//a2a//` serves the agent's card, rewritten for the mesh, and its JSON-RPC endpoint. | | `/mesh/service/discover` | Service discovery over plain HTTP, with the same results as the `discover_remote_services` tool. | -The [node API reference](../../reference/node-api/) lists every route. +Other routes serve token exchange (`/oauth/token`), revocation +(`/oauth/revoke`), the egress destinations that this node serves +(`/egress/`), and the Envoy `ext_authz` and `ext_proc` integration. The +node also serves identity evidence (`/mesh/identity`) and operator +diagnostics (`/debug/`). The [node API reference](../../reference/node-api/) +lists every route. ## Discovery @@ -51,19 +58,21 @@ on the routers closest to its key, which it finds in the Kademlia table the routers keep among themselves. A node that looks for a service, or for a peer, asks a router it holds, and the router answers from its own records and from those closest routers. A node keeps no DHT of its own -and connects to no router it does not hold. It re-announces every +and asks only the routers it holds. A node connects to another router only +to reach a peer behind that router, as [Connecting](#connecting) describes. +It re-announces every reprovide interval of the mesh (`/info`), withdraws a record when it stops the service or leaves, and a record a provider stopped announcing without withdrawing it leaves the directory after three intervals. -Before a node advertises a backend, it probes the backend +Before a node advertises an MCP or A2A service, it probes the backend (`--backend-probe-timeout`, 2 seconds by default). A backend that does not -answer is not advertised, so `discover_remote_services` does not list -services that would fail on the first call. +answer is not advertised, so `discover_remote_services` does not list an +MCP or A2A service that would fail. The node advertises inference and +egress services without a probe. Discovery results are hints. They tell the caller who claims to provide a -service and, if the provider declared labels, what those labels are. Nothing -is authorized on that basis. The provider's credential is verified when the +service. Nothing is authorized on that basis. The provider's credential is verified when the connection is made. Every node also runs the built-in `system://mesh.catalog` service, which @@ -116,8 +125,10 @@ Node and Python SDKs use, or Noise, which a browser can speak. Routers and nodes offer TLS first and accept Noise; two peers that both have TLS land on it. A relay forwards ciphertext and learns only that the two peers are talking. The control plane sees enrollments, -refreshes, router leases, policy fetches and catalog reports. It never sees a -request. +refreshes, router leases, catalog reports, and token exchanges +(`/token/exchange`, `/sts/token`). It also sees each member's scheduled +pulls of keys, routers, policy, egress assignments and revocations. It never +sees a request. Traffic to the control plane is protobuf over HTTPS. Traffic between nodes is HTTP over libp2p streams for service requests, and protobuf frames for the @@ -131,9 +142,9 @@ deliver every event to them, and run the gossip mesh among themselves. LiteLLM, or a hosted API behind a local proxy. The node's `/v1` endpoint hides which node serves a model. It keeps a short-lived view of the model list of every reachable provider. It prefers a local provider when one -serves the model, and otherwise picks among remote providers, ranked by the -labels it knows about and by load. Backends are registered by their root URL -and the node adds the `/v1` prefix. The same backend is therefore reachable +serves the model. Otherwise it rotates among the remote providers that serve +the model, and skips a provider that failed recently. Backends are +registered by their root URL and the node adds the `/v1` prefix. The same backend is therefore reachable through the `/v1` endpoint and, with `/v1/...` appended, through the proxy path. @@ -150,12 +161,13 @@ An operator can set a floor that every provider must meet with ## Mesh events -The control plane publishes bans and signing-key rotations as signed events. -Routers inject them into a GossipSub topic that every node subscribes to. A -node that receives a ban event drops the banned peer immediately. A node -that receives a rotation event fetches the new key set. Each event is -verified against the control plane's key before a node acts on it or -forwards it. +The control plane publishes bans, signing-key rotations and policy updates +as signed events. It joins the routers' gossip as a publish-only peer, and +the routers forward each event to every member. A node that receives a ban +event drops the banned peer immediately. A node that receives a rotation +event trusts the new key and brings its next pull forward. A policy update +event brings the node's next policy pull forward. Each event is verified against the control plane's key +before a node acts on it or forwards it. ## Resource limits diff --git a/site/content/docs/guides/scalability.md b/site/content/docs/guides/scalability.md index a7972e81..e5122215 100644 --- a/site/content/docs/guides/scalability.md +++ b/site/content/docs/guides/scalability.md @@ -27,7 +27,8 @@ providers and peers. The router keeps the member's records, places each on the routers closest to its key (three, the mesh's replication, `agentmesh-control-plane --dht-replication`), and answers each lookup from its own records and those routers', asked at once. The member keeps -no DHT and connects to no router it does not hold. It re-announces every +no DHT and asks only the routers it holds. It connects to another router +only to reach a peer behind that router. The member re-announces every 5 minutes (the mesh's `--dht-reprovide-interval`) and withdraws a record when it stops the service or leaves. A member connects to another member only to call it, through that member's router, where it runs the credential diff --git a/site/content/docs/reference/agentmesh-node.md b/site/content/docs/reference/agentmesh-node.md index c2849832..80a112ae 100644 --- a/site/content/docs/reference/agentmesh-node.md +++ b/site/content/docs/reference/agentmesh-node.md @@ -118,7 +118,7 @@ explains how to enroll. | `--discovery-interval` | `30s` | Delay before the first announcement of this node's services to the directory, capped at 5 s; services are re-announced every reprovide interval of the mesh after that (`agentmesh-control-plane --dht-reprovide-interval`, 5 minutes by default). | | `--dht-lookup-limit` | `20` | Providers a directory lookup asks for. | | `--discovery-concurrency` | `10` | Concurrent catalog fetches during discovery. | -| `--backend-probe-timeout` | `2s` | How long a service backend may take to answer before the node declines to advertise it. Raise it for subprocesses that start slowly. | +| `--backend-probe-timeout` | `2s` | How long an MCP or A2A backend may take to answer before the node declines to advertise it. Raise it for subprocesses that start slowly. | | `--secrets-dir` | `/etc/agentmesh/secrets` | Directory holding the credentials that the control plane's [egress destinations](../policy/#egress-destinations) name, one file per credential name. The platform puts the files there; the node reads a file on every request to the destination. | | `--control-plane-sync-interval` | `15m` | How often signing keys, the ban set, router addresses and the mesh policy are pulled from the control plane. Keep it well below the control plane's `--key-grace-period`; raise it on large meshes. | | `--key-grace-period` | `24h` | How long a rotated-out control plane key is still accepted for verifying peers. | diff --git a/site/content/docs/reference/node-api.md b/site/content/docs/reference/node-api.md index e9278b81..e6fffc48 100644 --- a/site/content/docs/reference/node-api.md +++ b/site/content/docs/reference/node-api.md @@ -169,11 +169,11 @@ Services this node publishes. | `name` | Optional service name. Omit to list every reachable service of the type. | | `limit`, `offset` | Pagination. Defaults are 20 and 0. | -Each result has `peer_id`, `srv_name`, `srv_description` and, when the -provider declared them, `labels`. Service names are not unique across the -mesh. The peer ID identifies the provider. For `type: inference` the -response adds a `local_proxy_url` per provider and a note that inference is -called over HTTP and not with `call_remote_tool`. +Each result has `peer_id`, `srv_name`, `srv_description` and +`local_proxy_url`, the proxy path to that provider's service. Service names +are not unique across the mesh. The peer ID identifies the provider. For +`type: inference` the response adds a note that inference is called over +HTTP and not with `call_remote_tool`. ### `find_remote_tools` @@ -185,7 +185,7 @@ called over HTTP and not with `call_remote_tool`. | `intent` | Accepted and ignored. Reserved for later use. | Returns one row per tool with `peer_id`, `tool_name` (namespaced as -`mcp://service/tool`), `description` and `labels`. A peer that could not be +`mcp://service/tool`) and `description`. A peer that could not be queried appears as a row with an `error` field. The call itself does not fail. @@ -301,9 +301,11 @@ a2a send -a $CARD "hello" `/v1/models` collects the model list of every reachable `inference` provider (cached for 30 seconds) and returns the union. `owned_by` on each entry is the peer that serves the model. A completion request names a model, and the -node picks a provider that serves it. It prefers a local provider, and ranks -remote providers by the labels the caller required, the operator's floor, -and load. A model that no provider lists answers `404`. +node picks a provider that serves it. It prefers a local provider. +Otherwise it rotates among the remote providers that serve the model, and +skips one that failed recently. A remote provider must attest the labels the +caller required and the operator's floor before the node sends it the +request. A model that no provider lists answers `404`. ```bash curl -s --unix-socket $SOCK http://localhost/v1/models @@ -315,8 +317,9 @@ curl -s --unix-socket $SOCK http://localhost/v1/chat/completions \ Any OpenAI SDK works with `base_url` set to `http://127.0.0.1:8080/v1` and the API token, Task Biscuit, or platform JWT as `api_key`. The node accepts -the token in `Authorization` here because nothing on this path forwards that -header. Streaming responses are passed through. One provider can also be +the token in `Authorization` here and removes it. An `Authorization` header +that is not the token is the backend's credential, and the node forwards +it. Streaming responses are passed through. One provider can also be addressed directly, at `/mesh//inference//v1/chat/completions`. ## Egress diff --git a/site/content/docs/reference/node-config.md b/site/content/docs/reference/node-config.md index f22fb329..2e6e4025 100644 --- a/site/content/docs/reference/node-config.md +++ b/site/content/docs/reference/node-config.md @@ -95,9 +95,10 @@ Rules by type: `/.well-known/agent-card.json` on the backend and served again with mesh URLs. The card must use the A2A 1.0 format. -Before a service is advertised, the node probes the backend +Before an MCP or A2A service is advertised, the node probes the backend (`--backend-probe-timeout`, 2 seconds). A backend that does not answer is -not advertised, and the log records this. +not advertised, and the log records this. An inference service is +advertised without a probe. Services exist only through this file and through the control plane's egress assignments. There is no runtime API that adds a service. An agent diff --git a/site/content/docs/reference/router.md b/site/content/docs/reference/router.md index 479850df..849d988f 100644 --- a/site/content/docs/reference/router.md +++ b/site/content/docs/reference/router.md @@ -90,7 +90,8 @@ replica count. withdraws them from the same routers. A member asks for providers of a key or for a peer; the router answers from its own records and from the routers closest to the key, asked at once. A member keeps no DHT - and connects to no router it does not hold. + and asks only the routers it holds. It connects to another router only + to reach a peer behind that router. 6. Refreshes its own credential before it expires, like a node, at the pass that finds less than a fifth of its lifetime left. 7. Every 5 seconds, compares the members it holds, the peers that passed