Run a split cohort

Some models are too large for any single GPU a casual provider owns. A cohort serves them anyway: each node runs a slice of the model’s layers, signs the boundary it produced, and is paid for exactly the layers it ran. This page shows how to stand one up. For the concept and the guarantees, see Split inference.

Two ways to serve a slice

The network-native path is the first: a provider advertises the layer range it can run, and the router’s planner tiles live ads into the cheapest cohort that covers the whole model. The explicit path is the same serving mechanism with the cohort named by hand, which is the clearest way to watch it work end to end.

Advertise a segment to the network

Set one field in the provider config so the router can place you in cohorts:

ogong_segment = "<lo>:<hi>:<n_layers>"

For example, 0:13:26 advertises layers 0 through 12 of a 26-layer model. The router’s planner then composes you with other segment providers into a cohort that tiles [0, n_layers) with no gaps. Everything else (verification, settlement) is automatic and identical to whole-model serving.

Run an explicit cohort

This is the hands-on version: split a model by layer range, run one segment server per shard, and drive the cohort with the lead. It runs on one machine across several processes, or across several machines by binding to real addresses.

What you need

• The boundary engine built (llama-eval-callback).
• The two binaries ogong-segment-server and ogong-lead.
• A GGUF model, plus two facts about it: its hidden width n_embd, and the interior-shard input scale embd-div (it is sqrt(n_embd) for Gemma, and 1 for Llama, Qwen, Phi, DeepSeek, GLM).

1. Split the model into per-shard GGUFs

Pick a layer tiling that covers [0, n_layers) with no gaps or overlaps. For a 26-layer model split in two:

gguf_segment_split.py  model.gguf  shard0.gguf  0  13
gguf_segment_split.py  model.gguf  shard1.gguf  13 26

Each shard holds only its layer range plus the embedding and output tensors. No shard holds the whole model.

2. Run one segment server per shard

One daemon per shard, each on its own address, each with its own signing key (generated and saved on first run). The first shard runs from the prompt; interior shards run from the upstream boundary, so they take --embd-div:

ogong-segment-server --model shard0.gguf --layers 0:13  --n-embd 2304 \
    --engine-bin llama-eval-callback --bind 0.0.0.0:9201 --key node0.key

ogong-segment-server --model shard1.gguf --layers 13:26 --n-embd 2304 --embd-div 48 \
    --engine-bin llama-eval-callback --bind 0.0.0.0:9202 --key node1.key

Each loads its model once and stays resident, signing every boundary it produces.

3. Drive the cohort with the lead

The lead threads the boundary shard to shard, collects each signed segment, and verifies the assembled pipeline (every signature, and that the chain tiles [0, n_layers)) before returning:

ogong-lead --shards 10.0.0.1:9201=0:13,10.0.0.2:9202=13:26 \
    --n-layers 26 --prompt "The capital of France is"

A healthy cohort prints cohort VERIFIED, with each shard’s layer range, node id, and signature check. A faked or dropped boundary breaks the chain and is localized to its signer; the lead reports the failure instead.

Verified settlement

To submit a verified run to a validator for an independent re-audit and on-chain payment, give the lead the validator and the paying account:

ogong-lead --shards ... --n-layers 26 --prompt "..." \
    --validator <host:port> --validator-cert <cert> \
    --consumer <wallet> --gross <amount> --wallets <shard0-wallet>,<shard1-wallet>

The validator re-runs the audited segment on its own reference engine and, on accept, settles the cohort on chain: each shard is paid for its slice under a conservation invariant, so the per-shard amounts must sum to the provider’s share and a release can never exceed the request’s fee. A caught cheat is ejected and the consumer refunded.

Notes

• Tiling. The shard ranges must be in order and cover [0, n_layers) exactly. A gap or an overlap is rejected.
• Generality. Any decoder transformer works, because the cohort rides the residual stream every such model exposes. Only --embd-div differs by family: Gemma scales its input, the rest use 1.
• One model, many machines. Binding the segment servers to real addresses is all it takes to spread a cohort across separate hosts. The lead and the verification are unchanged.