How one question travels through the agent loop

SimpleHarness.run() is the loop itself (agent/harnesses/simple_harness.py:99): it renders the context, has the model generate, parses the output and runs the tools, sending an event at every stage. The model runs on this machine with transformers, or on a vLLM server (§11 “What runs the model: the Backend interface, transformers here or a vLLM server”), and every token shows.

PiHarness starts pi as a child process in RPC mode for each question (agent/harnesses/pi_harness.py:100): commands go in on stdin, and records come out on stdout, one JSON object per line. pi runs the loop, and PiHarness only turns pi's records into the page's events; one of pi's turns (one request to the model, and the tools its reply calls) is a step.

410def _on(self, rec: dict) -> bool:
411    """Turn one of pi's records into the page's events; True once pi has settled."""
412    kind = rec.get("type")
413    if kind == "turn_start":
414        self._turn_start()
415    elif kind == "message_start" and self._open and rec["message"].get("role") == "assistant":
416        self._stream_t, self._wrote = time.perf_counter(), False
417    elif kind == "message_update" and self._open:
418        self._update(rec["assistantMessageEvent"])
419    elif kind == "message_end" and self._open and rec["message"].get("role") == "assistant":
420        self._message_end(rec["message"])
421    elif kind == "tool_execution_start" and self._open and not rec.get("parentToolCallId"):
422        self._tool_start(rec["toolCallId"], rec["toolName"], rec.get("args") or {})
423    elif kind == "tool_execution_end" and rec.get("toolCallId") in self._calls and not rec.get("parentToolCallId"):
424        self._tool_end(rec["toolCallId"], rec.get("result") or {}, rec.get("isError", False), rec.get("durationMs"))
425    elif kind == "extension_ui_request":
426        self._dialog(rec)
427    return kind == "agent_settled"

pi asks either this page's model or one it is logged in to (Claude, GPT…), as the sidebar's The model pi asks says. On this page's model, model_endpoint.py serves the model in memory to pi as an OpenAI-compatible endpoint, and every token still shows; on one of pi's own, the provider's API sends back text alone.

• Turns: <|turn>user … <turn|>.
• Tool calls: <|tool_call>call:name{key:<|"|>value<|"|>}<tool_call|>.
• Tool results: in the same model turn (<|tool_response> … <tool_response|>).
• Stop tokens: <turn|> after an answer and <|tool_response> when it wants a tool, so the stop token tells which.
• Thinking: <|think|> at the start of the system turn, and the reasoning between <|channel>thought and <channel|>.

• Turns: ChatML's <|im_start|>user … <|im_end|>.
• Tool calls: XML, one <function=name> inside <tool_call>, then a <parameter=name> for each argument, every value plain text.
• Tool results: in a new user message (<tool_response> … </tool_response>), followed by a new assistant turn.
• Stop tokens: <|im_end|> after an answer and for a tool alike, and only a tool call in the output tells them apart.
• Thinking: the chat template opens the assistant message with <think>, and the reasoning runs until </think>.

SimpleBackend has _download() fetch the whole checkpoint, the weights included, and loads all of it on this machine. It fetches config.json alone first, to tell the model's family from it, so a model the page does not support fails before gigabytes of weights come down (§2 “Model families: switching models changes only this layer”). The MTP drafter downloads the same way, the first time MTP is switched on (§10 “MTP: the drafter guesses, the model checks them all at once”).

51def _download(model_id: str, only: str | None = None) -> str:
52    """The model's folder in MODELS_DIR, after fetching the files it lacks (only `only`, if given), at the revision
53    model_families.REVISIONS pins it to.
54
55    from_pretrained(local_files_only=True) is no test of the cache: once config.json is cached, a tokenizer whose
56    own files are missing raises a ValueError (Gemma 4) or loads empty (Qwen3.5) rather than an OSError.
57    snapshot_download lists the model's files on the Hub and fetches only those the cache lacks; when the Hub
58    can't be reached it returns the cached folder, so a downloaded model still starts offline."""
59    if os.path.isdir(model_id):  # a checkpoint on disk rather than a Hugging Face id
60        return model_id
61    return snapshot_download(model_id, revision=model_families.REVISIONS.get(model_id), allow_patterns=only, cache_dir=MODELS_DIR)

Before it starts the server, vllm_docker.start() downloads the whole checkpoint into models/, the weights included, and with MTP on a GPU Gemma 4's drafter too (Qwen3.5's MTP layer is in its own weights). The server itself downloads nothing:

191say("download", "")
192snapshot_download(model_id, revision=model_families.REVISIONS.get(model_id), cache_dir=MODELS_DIR)
193if config and (drafter := config.get("model")):  # the server reads models/ offline, its drafter's weights too
194    snapshot_download(drafter, revision=config["revision"], cache_dir=MODELS_DIR)

docker-compose/compose.yaml mounts that folder in the container at /models, read-only, and points Hugging Face's cache there, with no network. _env() hands Docker Compose the full path of MODELS_DIR as AGENT_LOOP_MODELS (agent/model_backends/vllm_docker.py:113), so a models/ moved elsewhere is the one mounted. What vLLM compiles for the model, the first start's longest wait, goes in a Docker volume of its own, not in models/:

37  - ${AGENT_LOOP_MODELS:-../models}:/models:ro  # the project's models/; by hand, a full path, since Compose takes a relative one from here
38  # What vLLM compiles for a model, the first start's longest wait, kept for the next container that loads it.
39  - compile-cache:/root/.cache/vllm
40environment:
41  HF_HUB_CACHE: /models
42  HF_HUB_OFFLINE: "1"

vLLM is given the model's id and the commit in REVISIONS (--revision), which in the cache is the folder models--<org>--<name>/snapshots/<revision>/, the very files transformers would load. The cache's links are relative, so under /models they still lead to blobs/. Offline, vLLM would call the model by that folder's path, so --served-model-name gives it back the id the page asks for:

23- ${VLLM_MODEL:-Qwen/Qwen3.5-0.8B}
24- --revision=${VLLM_REVISION:-2fc06364715b967f1860aea9cf38778875588b17}
25# Offline, vLLM names the model by its folder in models/; the page asks for it by its id.
26- --served-model-name=${VLLM_MODEL:-Qwen/Qwen3.5-0.8B}

The page reads that same folder: VLLMBackend's _checkpoint_files() wants every file but the weights, which are all there already, so nothing more downloads. There is one copy on disk: the server loads the weights from it, and the page reads the tokenizer and the chat template there to turn the prompt into token ids (§11 “What runs the model: the Backend interface, transformers here or a vLLM server”).

A server at VLLM_URL has a copy of the model of its own, on the machine it runs on, so the page downloads every file but the weights into models/ (none of the kinds WEIGHTS lists, agent/model_backends/vllm_backend.py:65), at the same commit:

160def _checkpoint_files(model_id: str) -> str:
161    """The model's folder in MODELS_DIR with everything but its weights, at the revision model_families.REVISIONS pins.
162    The server should load the same revision (vllm serve <model> --revision <commit>) so that both read the same
163    tokenizer and chat template."""
164    if os.path.isdir(model_id):  # a checkpoint on disk rather than a Hugging Face id
165        return model_id
166    return snapshot_download(model_id, revision=model_families.REVISIONS.get(model_id), cache_dir=MODELS_DIR, ignore_patterns=WEIGHTS)

For the two to read the same tokenizer and chat template, start the server at the commit in REVISIONS, as in vllm serve Qwen/Qwen3.5-0.8B --revision 2fc0636…. The page downloads by the name the server was started with (the root in its /v1/models), so a server started from a folder of its own (vllm serve /data/gemma) works only when the same folder is at the same path on this machine, which the page then reads where it is, downloading nothing; otherwise the page says why it cannot load.

Before that, the model menu downloads only the config.json of each model the server serves, whose model_type tells its family, and whether the page supports it (agent/model_backends/vllm_backend.py:178); the other files come once that model is picked.

A revision is exactly the same files on macOS, on Linux with CUDA or on a computer with only a CPU. What differs is how they load: pick_device() picks the device (agent/model_backends/simple_backend.py:110), and the weights load in the files' bfloat16 on a GPU but become float32 on a CPU, twice the memory.

• config.json: AutoConfig reads it first, and its model_type picks the family (agent/model_backends/simple_backend.py:145); the specs at the top of the Model tab come from it too.
• tokenizer.json, tokenizer_config.json, chat_template.jinja and the processor's settings: AutoProcessor reads them into one processor (agent/model_backends/simple_backend.py:148); render() runs the chat template (agent/model_backends/simple_backend.py:249).
• The weights: AutoModelForMultimodalLM builds the model from config.json and fills it with the tensors of the .safetensors file by name (agent/model_backends/simple_backend.py:150).
• generation_config.json: the stop tokens (agent/model_backends/simple_backend.py:158) and the recommended sampling settings (agent/model_backends/simple_backend.py:207).
• README.md, LICENSE and .gitattributes: the model card, the license and the settings of the Hub's git, for people; the app does not read them.

VLLMBackend reads the same files the same way, all but the weights:

• config.json: AutoConfig reads it, and its model_type picks the family (agent/model_backends/vllm_backend.py:345); the specs at the top of the Model tab come from it too, but the context limit is the server's (--max-model-len, agent/model_backends/vllm_backend.py:354).
• tokenizer.json, tokenizer_config.json, chat_template.jinja and the processor's settings: AutoProcessor reads them into one processor, as SimpleBackend does (agent/model_backends/vllm_backend.py:347), so the chat template runs here, not on the server (§6 “Context: rendering the whole conversation into one string”). The server loads a tokenizer and a processor from its copy too (in Docker, this very folder), to expand the media (§7 “Images, sound and video: tokens in the context”).
• The weights: the server loads them, on its device and in its precision, and the page never does, so the Model tab has no device, precision or parameter count (§11 “What runs the model: the Backend interface, transformers here or a vLLM server”).
• generation_config.json: the page takes the stop tokens and the recommended sampling settings from it (agent/model_backends/vllm_backend.py:194). The server reads it too, for sampling defaults of its own, which every request sets to do nothing (§9 “Sampling: how each token is picked”).
• README.md, LICENSE and .gitattributes: neither the page nor the server reads them.

Gemma 4 E2B (google/gemma-4-E2B-it, 9 files, about 10.3 GB):

• config.json (5 kB): Gemma4ForConditionalGeneration, in three parts. text_config is the 35-layer language model: 28 sliding window layers, which see only the last 512 tokens, and 7 full attention layers, hidden size 1536, 8 attention heads sharing 1 set of keys and values; the last 20 layers (num_kv_shared_layers) compute no keys and values of their own but use those of earlier layers. vision_config is a 16-layer vision encoder that makes an image at most 280 tokens, and audio_config a 12-layer audio encoder. The context limit is 131,072 tokens.
• generation_config.json (208 B): the stop tokens [1, 106, 50], that is <eos>, <turn|> and <|tool_response>; recommended T 1.0, top-p 0.95, top-k 64.
• model.safetensors (10.2 GB): 2,011 tensors, 5.12B parameters, all bfloat16. The largest part is the Per-Layer Embeddings (embed_tokens_per_layer, 2.35B, 46%): each of the 35 layers keeps a 256-wide vector for each of the 262,144 tokens, only looked up and never multiplied, which is why the name counts effective parameters (E2B). The rest is the 35 layers themselves at 1.88B, the input embedding at 0.40B, the audio encoder at 0.30B and the vision encoder at 0.17B.
• tokenizer.json (32 MB): a SentencePiece-style BPE with 262,144 tokens and 514,906 merges; a space becomes ▁ in front of the next word, and a character the vocabulary lacks splits into UTF-8 bytes. tokenizer_config.json (3 kB) gives the control tokens their roles (sot_token is <|turn>, stc_token <|tool_call> and so on).
• chat_template.jinja (19 kB): writes the <|turn>system / user / model turns, the <|think|> for thinking, the tools as <|tool>declaration:…<tool|> and the attachments' placeholders (§6 “Context: rendering the whole conversation into one string”).
• processor_config.json (1.7 kB): one file for all three inputs. An image is cut into 16-pixel patches pooled 3×3, at most 280 tokens; sound is sampled at 16 kHz into 128 mel bands, one token per 40 ms and at most 750 (30 seconds); a video takes at most 32 frames of at most 70 tokens each (§7 “Images, sound and video: tokens in the context”).

Qwen3.5 0.8B (Qwen/Qwen3.5-0.8B, 13 files, about 1.8 GB):

• config.json (2.9 kB): Qwen3_5ForConditionalGeneration. text_config is the 24-layer language model, 18 Gated DeltaNet layers (linear_attention) and 6 full attention layers (one in every 4), hidden size 1024, 8 heads with 2 sets of keys and values, a context limit of 262,144; vision_config is a 12-layer vision encoder that images and video share; there is no sound encoder. mtp_num_hidden_layers: 1 says the weights come with an MTP layer.
• No generation_config.json: the only stop token left is config.json's <|endoftext|>, so the family adds <|im_end|>; the recommended sampling settings (T 1.0, top-p 1.0, top-k 20) come from the family too (agent/model_families/qwen3_5.py:301).
• model.safetensors-00001-of-00001.safetensors (1.7 GB): a single shard, named in the sharded format, with model.safetensors.index.json recording which shard each of the 488 tensors is in. 0.87B parameters: the 24 layers at 0.50B, the embedding at 0.25B, the vision encoder at 0.10B, and a 20M MTP layer (mtp.) that transformers skips when loading. Most are bfloat16; the Gated DeltaNet layers' A_log and norms are stored in float32.
• tokenizer.json (12.8 MB): a byte-level BPE as GPT-2's, 248,044 tokens plus 26 added ones and 247,587 merges after NFC normalization; every byte is spelled as one printable character, a space as Ġ. vocab.json (6.7 MB) and merges.txt (3.4 MB) are the same vocabulary and merges in the older format. tokenizer_config.json (16.7 kB) takes <|im_end|> as its eos_token and also holds a chat template identical to chat_template.jinja.
• chat_template.jinja (7.8 kB): ChatML, with the tool schemas in <tools>…</tools>, a tool result as a user message and the thinking inside <think> (§6 “Context: rendering the whole conversation into one string”).
• preprocessor_config.json and video_preprocessor_config.json: an image is cut into 16-pixel patches, 2×2 of them merged into one token, so a token covers 32×32 pixels, and an image is kept between 65,536 and 16,777,216 pixels; in a video, every 2 frames make one set of tokens (§7 “Images, sound and video: tokens in the context”).

Gemma 4 E2B's MTP drafter (google/gemma-4-E2B-it-assistant, 7 files, about 190 MB):

• config.json: Gemma4AssistantForCausalLM, 4 layers (3 sliding window, 1 full attention), hidden size 256 next to the model's 1536 (backbone_hidden_size); all 4 read the model's keys and values (num_kv_shared_layers: 4). Its output is clustered: the 262,144 tokens are sorted into 2,048 clusters of 128, the clusters are scored first, and logits are computed only for the 4,096 tokens of the best 32 (§10 “MTP: the drafter guesses, the model checks them all at once”).
• generation_config.json: num_assistant_tokens: 6, six guesses at a time (agent/model_backends/simple_backend.py:233).
• model.safetensors (158 MB): 77.7M parameters. 67M of them are embed_tokens (262,144 tokens × 256), used as the output table; pre_projection narrows the model's token embedding next to its hidden state (3,072 wide) to 256, and post_projection widens it back to 1,536; the 4 layers themselves are only about 9M. masked_embedding.token_ordering is 262,144 integers listing the tokens cluster by cluster.
• tokenizer.json and tokenizer_config.json: the same vocabulary as the model's; the app does not load them, since the drafter reads and writes the model's token ids.

The built-in harness runs this very model: the sidebar picks the model at its top, then what runs it.

With pi picked, pi_picker() first starts pi once to ask which models it has (agent/harnesses/pi_harness.py:179), at most once a minute. With This page's model picked under The model pi asks, it then picks and loads the model as the built-in harness does; with pi's own models, it lists those pi has credentials for, and nothing loads on this machine.

310if st.session_state.pi_source == "page":
311    model_picker(model_menu(), busy)
312    return PiAgent(st.session_state.model_id, found["version"], llm=current_llm())

load_llm() makes a SimpleBackend, which loads the model onto this machine (§3 “The model folder: every file in models/”). Right after transformers is picked again over vLLM (Docker), current_llm() first stops vLLM's container, and only then loads the model. On CUDA, cuBLAS's workspaces go before the memory is handed back: cuBLAS keeps one from the first matmul on, carved out of the segment transformers reserved for the weights, and while it lives none of that segment can go back (agent/model_backends/simple_backend.py:125). Switching models turns MTP back off; the first time MTP speculative decoding is switched on, current_llm() also loads the drafter (agent/model_backends/simple_backend.py:225).

current_llm() first makes sure the vLLM server in Docker is up, starting it if it is not, and takes its URL (§11 “What runs the model: the Backend interface, transformers here or a vLLM server”); load_llm() then makes a VLLMBackend for it. What runs the model is, like the model in memory, one choice for every tab: when another tab has started or stopped vLLM's container, this one follows rather than load the model beside it (app/page.py:403), and a new tab or a reload also takes the model the container runs (app/page.py:396). MTP follows the server too, on or off, rather than start it again the other way, and switching models leaves MTP as it is, since it is how the server was started (§10 “MTP: the drafter guesses, the model checks them all at once”). Only the tab that just picked leads, until current_llm() has started or stopped vLLM.

load_llm() makes a VLLMBackend for the server at VLLM_URL. The model menu lists the models that server serves, the sidebar has no What runs the model to pick, and the page never starts or stops the server; MTP is the server's, and the sidebar's switch is disabled (§10 “MTP: the drafter guesses, the model checks them all at once”).

SimpleHarness.__init__ appends this question to the conversation so far and keeps it as self.messages (agent/harnesses/simple_harness.py:72). Tool calls and tool results are appended after it; the model itself remembers nothing, and what it sees in each step is this list rendered into a string again. The messages use Hugging Face's common format (role, content, tool_calls), which both families accept; each family's chat template takes care of its own format. With files attached, that message's content is a list of parts (§7 “Images, sound and video: tokens in the context”). self.tools are the tools the model gets this turn, and self.files is the conversation's folder. _stop and _decided are the two synchronization tools the UI thread signals the agent with.

67def __init__(self, llm: Backend, history: list[dict], user_text: str, settings: RunSettings, emit: Callable[[dict], None],
68             attachments: Sequence[media.Attachment] = (), files: Path | None = None):
69    self.id = uuid.uuid4().hex[:12]
70    self.llm = llm
71    self.settings = settings
72    self.messages = [*history, {"role": "user", "content": media.user_content(user_text, list(attachments))}]
73    self.files = files  # the conversation's folder, which run_python's sandbox sees as /data
74    self.tools = {name: t for name, t in TOOLS.items() if name in settings.tools} if settings.use_tools else {}
75    self._emit = emit
76    self._start = time.perf_counter()
77    self._stop = threading.Event()
78    self._timed_out = False
79    self._decisions: dict[str, bool] = {}
80    self._decided = threading.Condition()

run() starts by taking the model's family and deciding what the model gets to see this turn: it collects the schema() of every tool in self.tools for the chat template (none with Use tools off), and keeps a map from tool name to parameters schema for the parser (Qwen's argument values are plain text, and only the schema says their types); the system prompt comes in two versions, with tools (agent/harnesses/simple_harness.py:36) and without. The first event, run_start, carries the model and the family, so the page knows whose format to draw the turn in; it also says which harness ran the turn (simple, or pi's pi).

99def run(self):
100    s = self.settings
101    family = self.llm.family  # how this model writes its turns, thinking and tool calls
102    tools = [t.schema() for t in self.tools.values()] or None
103    schemas = {t.name: t.parameters for t in TOOLS.values()}
104    system = {"role": "system", "content": SYSTEM_WITH_TOOLS if tools else SYSTEM_NO_TOOLS}
105    self.emit(
106        "run_start", run_id=self.id, harness="simple", model=self.llm.model_id, family=family.key,
107        settings=_settings_dict(s), tools=list(self.tools),
108    )
109
110    seen: list[tuple[str, str]] = []
111    format_errors = 0
112    prev_sequence = ""
113    stop_reason, final = "max_steps", ""

A tool's schema is a JSON Schema, wrapped in the function format that Hugging Face chat templates know:

84def schema(self) -> dict:
85    return {
86        "type": "function",
87        "function": {"name": self.name, "description": self.description, "parameters": self.parameters},
88    }

PiHarness.run() starts pi, picks the model and the thinking level and turns pi's own retries off (a retry would be a step of its own, so a failed request is shown as an error instead), then sends the question as a prompt, each command only after pi has answered the last, since pi handles commands concurrently and a prompt sent beside set_model would go to the old model.

289self._pi = Process(self._args(), self.folder, self._inbox, self._env())
290# Each waits for its answer: pi handles commands concurrently, and a prompt must not go to the model it starts on.
291self._pi.request({"type": "set_model", "provider": s.provider, "modelId": s.model})
292self._pi.request({"type": "set_thinking_level", "level": s.thinking})
293# A failed request is retried by pi as a new turn, which would be a step of its own; the page shows the error instead.
294self._pi.request({"type": "set_auto_retry", "enabled": False})
295message, images = self._message()
296if not self._stopped_meanwhile():
297    self._pi.request({"type": "prompt", "message": message, **({"images": images} if images else {})})
298    self._loop()

What the model gets to see is pi's: its own system prompt, its tools (of which the sidebar picks the ones the model sees) and the session it keeps in the conversation's folder, on which the next question starts pi again, so the model reads all of it. The page starts pi without your own extensions, skills, prompt templates and AGENTS.md (agent/harnesses/pi_harness.py:56), so what it draws is pi's own loop; the one thing it adds is its extension, pi_extension.ts (§6 “Context: rendering the whole conversation into one string”, §15 “Validation and permission: the model never touches the environment”). Files attached to the question are saved in pi's working folder and named in the question, and a model that reads images gets them too (agent/harnesses/pi_harness.py:344). run_start is as the built-in harness's; it carries the family when pi asks this page's model, and none when pi asks one of its own.

Each step starts at for step in range(1, s.max_steps + 1); the sidebar's Max steps is the top of that range. The first thing is to render:

116for step in range(1, s.max_steps + 1):
117    self.emit("step_start", step=step)
118    prompt = self.llm.render([system, *self.messages], tools, s.thinking)
119    encoded = self.llm.encode(prompt, media.parts(self.messages))
120    # Media sit in user messages, all before what is new since the last step; so the new part is the same
121    # text in the prompt and in what the model reads, only further along by what the media expanded into.
122    added_from = len(encoded.text) - len(prompt) + common_prefix(prompt, prev_sequence) if prev_sequence else 0
123    self.emit(
124        "context",
125        step=step,
126        prompt=encoded.text,
127        n_tokens=encoded.n_tokens,
128        added_from=added_from,
129        media_tokens=encoded.media_tokens,
130        context_length=self.llm.context_length,
131        roles=_role_counts([system, *self.messages]),
132    )

pi starts a turn and sends its request to the provider agent-loop, which is model_endpoint.py: pi's system prompt, its tools and the conversation, as OpenAI's JSON. As on pi's own models, the extension reports the request first, and the Context tab keeps it under the context. The endpoint turns it back into the conversation SimpleHarness keeps (agent/harnesses/model_endpoint.py:121): a tool call's arguments a dict again, the thinking under the family's key, a tool result with its tool's name. Then, as SimpleHarness does, it renders and encodes it and sends context; what is new in the step is counted from the conversation as of the model's last message (agent/harnesses/model_endpoint.py:158).

215    messages = conversation(body, llm, serving.folder)
216    prompt = llm.render(messages, tools, thinking)
217    encoded = llm.encode(prompt, media.parts(messages))
218except Exception as e:  # pi shows it as the request's error
219    self._json(400, {"error": {"message": f"{type(e).__name__}: {e}"}})
220    return
221before = seen_before(messages)
222# Media sit in user messages, all before what is new; so the new part is the same text, further along (SimpleHarness).
223if before:
224    previous = llm.render(before, tools, thinking, generation_prompt=False)
225    added_from = len(encoded.text) - len(prompt) + common_prefix(prompt, previous)
226else:
227    added_from = 0
228emit("context", prompt=encoded.text, n_tokens=encoded.n_tokens, added_from=added_from, media_tokens=encoded.media_tokens,
229     context_length=llm.context_length, roles={role: sum(m["role"] == role for m in messages) for role in dict.fromkeys(m["role"] for m in messages)})

Every pi the page starts loads one small extension, pi_extension.ts. Before each request pi sends the model, it sends what the request holds as a notification; in RPC mode a notification is an extension_ui_request on stdout, which PiHarness turns into a request event (agent/harnesses/pi_harness.py:524): the step card's context section and the Context tab show pi's system prompt, its tools and the conversation as JSON, with what this step added (the last step's reply and observation) marked. How the provider's API writes the request out as the string its model reads, neither pi nor the provider says, so the chat template the rest of this section is about is not to be seen here.

50pi.on("before_provider_request", (event, ctx) => {
51	ctx.ui.notify(REQUEST_MARK + JSON.stringify(event.payload), "info");
52});

It starts with <bos><|turn>system; with thinking mode on, a <|think|> comes first in the system turn; each tool is written as <|tool>declaration:name{…}<tool|>; each message is wrapped in <|turn>role … <turn|>.

generation_prompt=True (add_generation_prompt) ends it with <|turn>model, opening the model's turn for it to write next.

No <bos>; it starts with <|im_start|>system; the tool schemas go in as JSON, as they are, between <tools> and </tools>, followed by how to write a call in XML; each message is wrapped in <|im_start|>role … <|im_end|>.

generation_prompt=True (add_generation_prompt) ends it with <|im_start|>assistant, opening the model's turn for it to write next, followed by <think> (thinking mode on) or an empty <think></think> (off).

On a vLLM server, it is those less what the server does not take: at load, the backend asks the server with a one-token request for each kind of media the checkpoint reads, and the model reads those it takes (agent/model_backends/vllm_backend.py:367). Only /inference/v1/generate takes media, so a server started without --enable-scale-out takes none. A kind it refuses is a plain file for run_python; the Model tab lists the kinds the server refused and why (app/render.py:1250), and above the input box the page says which cannot be attached.

428def _probe(self, hi: list[int], kinds: tuple[str, ...]) -> tuple[bool, bool, dict]:
429    """Ask the server, with one-token requests that start from `hi` (a short prompt's ids), whether it has
430    /inference/v1/generate (--enable-scale-out), whether it sends sampling masks (--return-sampling-mask), and which
431    of `kinds` (the checkpoint's media) it takes. A kind it does not take comes back with why, in both of the page's
432    languages."""
433    try:
434        out = self._generate_once(hi, [], top_k=2)
435    except ServerError as e:
436        if e.status is None:  # unreachable, not refused
437            raise
438        if e.status == 403:  # what vLLM answers on that route without --enable-scale-out
439            why = {"zh": "伺服器啟動時沒有加 `--enable-scale-out`，只有 /v1/completions 可用，而它只收文字。",
440                   "en": "the server was started without `--enable-scale-out`, so only /v1/completions is there, which takes text only."}
441        elif e.status == 404:  # a vLLM without the route
442            why = {"zh": "這個版本的 vLLM 沒有 /inference/v1/generate，只有 /v1/completions 可用，而它只收文字；新版 vLLM 加上 `--enable-scale-out` 才有。",
443                   "en": "this vLLM has no /inference/v1/generate, only /v1/completions, which takes text only; a newer vLLM started with `--enable-scale-out` has it."}
444        else:
445            why = {"zh": f"伺服器的 /inference/v1/generate 拒絕了請求，所以只用 /v1/completions，它只收文字（{e.text['en']}）",
446                   "en": f"the server's /inference/v1/generate refused a request, so only /v1/completions is used, which takes text only ({e.text['en']})"}
447        return False, False, {kind: why for kind in kinds}
448    masks = any(choice.get("sampling_mask") is not None for choice in out.get("choices") or [])
449    refused = {}
450    for kind in kinds:
451        prompt = self.render([{"role": "user", "content": [{"type": kind}, {"type": "text", "text": "Hi"}]}], None, False)
452        try:
453            self._generate_once(self.tokenizer(prompt, add_special_tokens=False).input_ids, [probe_part(kind, self.audio_rate)])
454        except ServerError as e:
455            refused[kind] = {"zh": f"伺服器不收：{e.text['en']}", "en": f"the server refused it: {e.text['en']}"}
456    return True, masks, refused

The placeholders are <|image|>, <|audio|> and <|video|>. An image is up to 280 <|image|> between <|image> and <image|>; a video is 32 frames taken evenly (all of them, for a shorter one), each after its time as mm:ss and up to 70 <|video|>; sound is one <|audio|> per 40 ms, at most 750, so only the first 30 seconds are heard.

The placeholders are <|vision_start|><|image_pad|><|vision_end|> for an image and <|video_pad|> for a video. The bigger the image, the more tokens (about 300 <|image_pad|> for 640×480); a video gives 2 frames a second, in pairs, each pair after its time, such as <0.3 seconds>.

encode() hands the processor all the media in the conversation, in the order they come, to expand:

256def encode(self, prompt: str, parts: list[dict]) -> Encoded:
257    """The prompt as the model reads it. Without media that is the prompt's tokens. With media (`parts`, the
258    conversation's media in the order of their placeholders), the processor replaces each placeholder with the
259    tokens that image, clip or video takes, and makes the inputs of the encoders whose output goes there."""
260    if not parts:
261        ids = self.tokenizer(prompt, add_special_tokens=False, return_tensors="pt").input_ids
262        return Encoded(prompt, ids, ids.shape[-1])
263    files = {kind: [p["path"] for p in parts if p["type"] == kind] for kind in media.MEDIA}
264    inputs: dict = {}
265    if files["image"]:
266        inputs["images"] = [media.load_image(path) for path in files["image"]]
267    if files["audio"]:
268        inputs["audio"] = [media.load_audio(path, self.processor.feature_extractor.sampling_rate) for path in files["audio"]]
269    if files["video"]:
270        videos = [media.load_video(path) for path in files["video"]]
271        inputs["videos"] = [frames for frames, _ in videos]
272        inputs["videos_kwargs"] = {"video_metadata": [metadata for _, metadata in videos]}
273        # A processor that takes a set number of frames (Gemma 4: 32) refuses a video with fewer; it gets all there are.
274        wanted, fewest = getattr(self.processor.video_processor, "num_frames", None), min(len(frames) for frames, _ in videos)
275        if wanted and fewest < wanted:
276            inputs["videos_kwargs"]["num_frames"] = fewest
277    out = self.processor(text=prompt, return_tensors="pt", **inputs, **self.family.processor_kwargs)
278    ids = out.pop("input_ids")
279    out.pop("attention_mask", None)  # one sequence, no padding: every position counts, as with text
280    counts = {kind: int((ids == token).sum()) for kind, token in self.media_token_ids.items() if token is not None}
281    text = self.tokenizer.decode(ids[0], skip_special_tokens=False)
282    return Encoded(text, ids, ids.shape[-1], dict(out.to(self.device, dtype=self.dtype)), {k: n for k, n in counts.items() if n})

The processor wants decoded data, which media.py prepares: images become RGB, and photos are turned upright by their EXIF; sound is decoded with PyAV into mono at the 16 kHz the model's feature extractor expects; a video gives at most 64 frames taken evenly, shrunk to 768 pixels on the longer side, with the frame rate of what was kept, so the times the processor writes are right (agent/media.py:135). Each file is decoded once and reused in every step after.

In the prefill, the encoders' output replaces the embeddings of those placeholder tokens. Images and videos go through the vision encoder first and its output goes to the prefill, as generate() does it; handed to the forward pass as they are, videos lose their frame counts in transformers 5.20's Gemma 4. Qwen3.5's position encoding (M-RoPE) needs the images' sizes, so every input but the pixels stays; it also keeps the last prompt's position offset on the model, which is cleared before every prefill. After the prefill, the media are in the KV cache, and decoding is the same as for text. tests/test_real_model.py checks that with an image, a video and a recording, too, greedy output matches generate().

361def _encoded_media(self, inputs: dict) -> dict:
362    """Run the vision encoder over the prompt's images and videos before the prefill, as generate() does, and pass
363    its output in their place. Leaving it to the forward pass breaks videos on Gemma 4: transformers 5.20's
364    Gemma4ForConditionalGeneration takes num_frames_per_video but does not hand it on to the video encoder."""
365    if not inputs or "mm_encoder_outputs" not in inspect.signature(self.model.forward).parameters:
366        return inputs
367    base, encoded = self.model.base_model, {}
368    for kind in ("image", "video"):
369        encoder = getattr(base, f"get_{kind}_features", None)
370        args = {k: inputs[k] for k in inspect.signature(encoder).parameters if k in inputs} if encoder else {}
371        if encoder and any(k.startswith("pixel_values") for k in args):
372            encoded[kind] = encoder(**args, return_dict=True)
373    if not encoded:
374        return inputs
375    # The pixels go; the rest stays, such as the grid sizes Qwen3.5's positions are computed from.
376    return {**{k: v for k, v in inputs.items() if not k.startswith("pixel_values")}, "mm_encoder_outputs": encoded}

The files go to the server next to the placeholders the chat template wrote (an image upright as PNG, sound decoded to WAV, a video as the file, agent/model_backends/vllm_backend.py:242), the server expands each placeholder, and the encoders run in its prefill. encode() asks the server how first, with a request that generates one token, so the Context tab shows the prompt as the server reads it; a different expansion at generation time is an error.

490def encode(self, prompt: str, parts: list[dict]) -> Encoded:
491    """The prompt's token ids, which are what the server reads. With media (`parts`, in the order of their
492    placeholders), the server expands each placeholder into the tokens its file takes, so it is asked how first, with
493    a one-token request; the text and the count are of the prompt as the server reads it, and the ids stay the ones
494    the chat template wrote, which generate() sends again with the files."""
495    ids = self.tokenizer(prompt, add_special_tokens=False).input_ids
496    if not parts:
497        return Encoded(prompt, ids, len(ids))
498    if unread := {p["type"] for p in parts} - set(self.modalities):  # media.save makes those plain files, so this is a bug
499        raise ValueError(f"The vLLM server takes no {', '.join(sorted(unread))} here, only {', '.join(self.modalities) or 'text'}.")
500    files = [content_part(p, self.audio_rate) for p in parts]
501    expanded = self._generate_once(ids, files, return_token_ids=True)["prompt_token_ids"]
502    counts = Counter(kind for token in expanded for kind, media_id in self.media_token_ids.items() if media_id is not None and token == media_id)
503    return Encoded(self.tokenizer.decode(expanded, skip_special_tokens=False), ids, len(expanded),
504                   {"content_parts": files, "prompt_token_ids": expanded}, dict(counts))

The harness consumes the events that generate() yields in a for loop: prefill and each token are passed on with the step number, and the final done is kept and sent on as model_end.

134done = None
135for ev in self.llm.generate(encoded, s.sampling, self._stop.is_set):
136    kind = ev.pop("type")
137    if kind == "done":
138        done = ev
139    else:
140        self.emit(kind, step=step, **ev)
141assert done is not None  # generate() always ends with done
142raw = done.pop("raw")
143self.emit("model_end", step=step, raw=raw, **done)

The endpoint consumes generate()'s events in the same way, sending each into pi's step. pi starts a turn and then sends its request, and the two reach the page by different ways, so the request first waits until the page has opened that step (agent/harnesses/pi_harness.py:335). Each token of the thinking and of the reply also goes back to pi as it comes, as reasoning_content and content. Before every token the model asks whether to stop: whether the turn is being stopped, or pi has stopped listening (as it does when it is aborted).

253for ev in llm.generate(encoded, serving.sampling, lambda: gone or serving.stopped()):
254    kind = ev.pop("type")
255    if kind == "done":
256        done = ev
257        continue
258    emit(kind, **ev)
259    if kind == "token" and ev["channel"] in ("thought", "content"):
260        send({"reasoning_content" if ev["channel"] == "thought" else "content": ev["text"]})

No backend runs here: the provider's API streams text back to pi, pi sends each piece on as a message_update record, and PiHarness turns it into delta, with its channel (agent/harnesses/pi_harness.py:445); the step card draws the pieces by channel, without token boundaries (app/render.py:1060). When the message ends, model_end carries the token counts the provider reported, how long the first output took and the cost at pi's prices (agent/harnesses/pi_harness.py:461). The tabs below are about this page's model.

445def _update(self, e: dict):
446    kind = e.get("type")
447    if kind == "text_delta":
448        self._delta("content", e.get("delta", ""))
449    elif kind == "thinking_delta":
450        self._delta("thought", e.get("delta", ""))
451    elif kind == "toolcall_start":
452        self._delta("tool_call", ("\n" if self._wrote else "") + e.get("toolName", "") + " ")
453    elif kind == "toolcall_delta":
454        self._delta("tool_call", e.get("delta", ""))

prefill: the whole prompt, already token ids from encode(), goes through one forward pass, which also puts the encoders' output in place of any media (§7 “Images, sound and video: tokens in the context”). use_cache=True keeps each layer's state (the keys and values in ordinary attention layers, a fixed-size state in Qwen3.5's Gated DeltaNet layers), so each new token needs only its own position computed; logits_to_keep=1 returns the logits for the last position only, since only the distribution of the next token is needed (agent/model_backends/simple_backend.py:354). Prefill time depends mostly on the prompt's length; the table in the Model tab has each step's prefill ms.

284def generate(self, prompt: Encoded, sampling: Sampling, should_stop: Callable[[], bool]) -> Iterator[dict]:
285    """Yield a prefill event, one event per token, then a done event."""
286    ids = prompt.ids.to(self.device)
287    channel = "thought" if prompt.text.endswith(self.family.thought_prefix) else "content"
288    drafting = sampling.mtp and self.drafter is not None
289    mtp = {"drafted": 0, "accepted": 0, "forwards": 0} if drafting else None
290    generated: list[int] = []
291    text = ""
292    finish = "max_new_tokens"
293
294    with torch.inference_mode():
295        # Models with multimodal RoPE (Qwen3.5) keep the position offset of the last prompt's images on the model
296        # and add it in every later forward pass; each prompt starts from its own.
297        if hasattr(self.model.base_model, "rope_deltas"):
298            self.model.base_model.rope_deltas = None
299        start = time.perf_counter()
300        out = self._forward(ids, None, 1, drafting, **self._encoded_media(prompt.media))
301        prefill_ms = (time.perf_counter() - start) * 1000
302        yield {"type": "prefill", "n_tokens": ids.shape[-1], "ms": round(prefill_ms, 1)}

The decoding loop takes sampled tokens from tokens one at a time: they normally come from _one_at_a_time(), and from _drafted() with MTP on (§10 “MTP: the drafter guesses, the model checks them all at once”); the loop itself does not need to know which. Each time around: check for stop → get a token → work out whether it is a special token and whether it switches channel, where the family's channel_switch says which tokens switch to thinking, a tool call or the reply (Gemma: agent/model_families/gemma4.py:394; Qwen: agent/model_families/qwen3_5.py:275) → decode all the tokens so far and take the difference, so that a Chinese character split across several byte tokens comes out only once it is whole → send a token event.

304decode_start = time.perf_counter()
305tokens = self._drafted(out, sampling, mtp) if mtp is not None else self._one_at_a_time(out, sampling)
306for token_id, details in tokens:
307    if should_stop():
308        finish = "user_stopped"
309        break
310    generated.append(token_id)
311
312    piece = self.piece(token_id)
313    special = token_id in self.special_ids
314    if piece in self.family.channel_switch:
315        channel = self.family.channel_switch[piece]
316        token_channel = "control"
317    else:
318        token_channel = "control" if special else channel
319
320    # Decode the whole sequence and diff it, so characters split across byte tokens come out whole.
321    full = self.tokenizer.decode(generated, skip_special_tokens=False)
322    delta = "" if full.endswith("�") else full[len(text):]
323    if delta:
324        text = full
325    yield {
326        "type": "token",
327        "index": len(generated) - 1,
328        "id": token_id,
329        "piece": piece,
330        "text": delta,
331        "special": special,
332        "channel": token_channel,
333        **details,
334    }

A stop token, or reaching max_new_tokens (the sidebar's Max tokens per step), ends the loop.

336if token_id in self.eos_ids:
337    finish = "stop_token"
338    break
339if len(generated) == sampling.max_new_tokens:
340    break

If it has not stopped, the loop asks tokens for the next one. Only then does _one_at_a_time() feed the token it just sampled back into the model, with the cache, for the next position's logits; it is a generator, so once the loop ends, no extra forward pass runs.

378def _one_at_a_time(self, out, s: Sampling) -> Iterator[tuple[int, dict]]:
379    """Plain decoding: sample a token, feed it back with the cache for the next position's logits, repeat."""
380    while True:
381        token_id, details = self._sample(out.logits[0, -1].float().cpu(), s)
382        yield token_id, details
383        out = self._forward(torch.tensor([[token_id]], device=self.device), out.past_key_values, 1)

The final done tells the harness why it stopped (finish: stop_token, max_new_tokens or user_stopped), at which token, how fast it generated, and the whole raw output, raw (special tokens included, because the harness parses by them).

342decode_s = time.perf_counter() - decode_start
343yield {
344    "type": "done",
345    "finish": finish,
346    "stop_piece": self.piece(generated[-1]) if finish == "stop_token" else None,
347    "n_tokens": len(generated),
348    "decode_ms": round(decode_s * 1000, 1),
349    "tok_per_s": round(len(generated) / decode_s, 1) if decode_s > 0 else None,
350    "mtp": mtp,
351    "raw": self.tokenizer.decode(generated, skip_special_tokens=False),
352}

VLLMBackend.generate() yields the same events, from the tokens the server streams back as server-sent events (what each token comes with is in §9 “Sampling: how each token is picked”). prefill goes out once the first token is back, so its time is the time to the first token, the network included:

506def generate(self, prompt: Encoded, sampling: Sampling, should_stop: Callable[[], bool]) -> Iterator[dict]:
507    """Yield the events SimpleBackend.generate yields, from the server's stream: prefill once the first token is back
508    (its time is the time to the first token), one event per token, then done."""
509    channel = "thought" if prompt.text.endswith(self.family.thought_prefix) else "content"
510    generated: list[int] = []
511    text = ""
512    finish = None
513    end: dict = {}  # the finish reason and the speculative decoding counts, once the stream has them
514    start = time.perf_counter()
515    decode_start = None
516
517    # /inference/v1/generate when the server has it: it says more about each token, and takes media.
518    if self.generate_api:
519        tokens = self._stream_generate(prompt, sampling, end)
520    else:
521        tokens = self._stream(self._request(prompt, sampling), sampling, end)
522    try:
523        for token_id, details in tokens:
524            if decode_start is None:
525                decode_start = time.perf_counter()
526                yield {"type": "prefill", "n_tokens": prompt.n_tokens, "ms": round((decode_start - start) * 1000, 1)}

vLLM keeps the KV cache of what a prompt shares with the last one (prefix caching), so from step 2 on it reads only what differs from the last step: the model's turn as the chat template rewrote it, and the new observation; SimpleBackend reads the whole prompt from the start every step.

From the token on it is as in SimpleBackend: the family's channel_switch says which channel, decoding all the tokens so far gives the text, and a token event goes out (agent/model_backends/vllm_backend.py:532). The server decides when to stop: the request carries the stop tokens (stop_token_ids) and max_tokens, the sidebar's Max tokens per step, cut to what fits in the server's context (§9 “Sampling: how each token is picked”). Pressing stop closes the stream, which tells the server to stop generating (agent/model_backends/vllm_backend.py:556).

The final done is the same as SimpleBackend's: the server's finish_reason stop becomes stop_token and length max_new_tokens, and anything else is an error; when the server decodes speculatively, it also carries the server's counts (§10 “MTP: the drafter guesses, the model checks them all at once”).

561if finish is None:
562    reason = end.get("finish_reason")
563    if reason not in ("stop", "length"):
564        raise ServerError(f"vLLM 伺服器沒有正常結束這一步（finish_reason：{reason}）。",
565                          f"The vLLM server did not finish this step normally (finish_reason: {reason}).")
566    finish = "stop_token" if reason == "stop" else "max_new_tokens"
567decode_s = time.perf_counter() - decode_start
568yield {
569    "type": "done",
570    "finish": finish,
571    "stop_piece": self.piece(generated[-1]) if finish == "stop_token" and generated else None,
572    "n_tokens": len(generated),
573    "decode_ms": round(decode_s * 1000, 1),
574    "tok_per_s": round(len(generated) / decode_s, 1) if decode_s > 0 else None,
575    "mtp": mtp_counts(end.get("spec"), len(generated)) if finish != "user_stopped" else None,
576    "raw": self.tokenizer.decode(generated, skip_special_tokens=False),
577}

Greedy output need not match generate() token for token: vLLM batches requests and has kernels of its own. Matching generate() exactly is SimpleBackend's promise only.

_sample() runs once per token. It first takes the softmax of the raw logits, the model's own distribution (T = 1, nothing cut): the model probability, the rank and the top 5 candidates in a token's popover all come from it.

A temperature of 0 or less is greedy: it takes the most likely token, and of two equally likely ones the lower id, as generate()'s argmax does (in bfloat16, logits tie more often than one would think). Otherwise, in order: logits ÷ T → keep only the top k (top-k; 0 = no limit) → softmax again → add up from the most likely down until the cumulative probability reaches top-p (nucleus, keeping at least one) → renormalize → draw one with torch.multinomial. None of this depends on the model; both families share it.

432def _sample(self, logits: torch.Tensor, s: Sampling, draft: int | None = None) -> tuple[int, dict]:
433    """Pick the next token and report the model's distribution and what the settings did to it.
434    A `draft` (the MTP drafter's guess) is kept with the probability the settings give it, which in greedy
435    decoding means when it is the top token; otherwise the pick comes from the other candidates. Either way
436    each token comes out exactly as often as without a drafter."""
437    probs = torch.softmax(logits, -1)  # the model's own distribution (T = 1, nothing cut off)
438    # Equally likely tokens go lowest id first, the one argmax and so generate() takes. topk leaves their order open
439    # (on the CPU it can list the higher id first), and in bfloat16 two logits are equal more often than one would think.
440    top = sorted(zip(*probs.topk(TOP_N)), key=lambda c: (-float(c[0]), int(c[1])))
441
442    if s.temperature <= 0:  # T -> 0 is greedy decoding
443        chosen = int(top[0][1])
444        kept = 1
445        p_sampled = 1.0
446    else:
447        scaled = logits / s.temperature
448        if s.top_k > 0:
449            cand_logits, cand_ids = scaled.topk(min(s.top_k, scaled.numel()))
450        else:
451            cand_logits, cand_ids = scaled.sort(descending=True)
452        cand_probs = torch.softmax(cand_logits, -1)
453        # Nucleus: keep the smallest prefix whose probability reaches top_p (always at least one token).
454        kept = int(((cand_probs.cumsum(-1) - cand_probs) < s.top_p).sum())
455        cand_ids, cand_probs = cand_ids[:kept], cand_probs[:kept] / cand_probs[:kept].sum()
456        weights = cand_probs
457        at: torch.Tensor | list = (cand_ids == draft).nonzero().flatten() if draft is not None else []
458        if len(at) and float(torch.rand(())) < float(cand_probs[at[0]]):
459            pick = int(at[0])
460        else:
461            if len(at):  # the guess was turned down: draw from the other candidates
462                weights = cand_probs.clone()
463                weights[at[0]] = 0
464            pick = int(torch.multinomial(weights, 1))
465        chosen = int(cand_ids[pick])
466        p_sampled = float(cand_probs[pick])
467
468    p = float(probs[chosen])
469    # Candidates are a prefix of the tokens sorted by probability, so rank decides who survived.
470    return chosen, {
471        "p": p,
472        "logprob": float(torch.log(probs[chosen])) if p > 0 else None,
473        "rank": int((probs > probs[chosen]).sum()) + 1,
474        "p_sampled": p_sampled,
475        "kept": kept,
476        "top": [
477            {"id": int(i), "piece": self.piece(int(i)), "p": float(v), "kept": rank < kept}
478            for rank, (v, i) in enumerate(top)
479        ],
480    }

The server does the sampling. Every sidebar setting is sent, and every other sampling setting is set to do nothing, so that defaults the server took from the checkpoint cannot change the distribution:

579def _sampling_params(self, prompt: Encoded, s: Sampling) -> dict:
580    """Each sidebar setting, and every other sampling setting set to do nothing, so that a default the server took
581    from the checkpoint (a repetition penalty, say) cannot change the distribution the page shows."""
582    return {
583        "max_tokens": max(1, min(s.max_new_tokens, self.context_length - prompt.n_tokens)),  # vLLM refuses a request that cannot fit
584        "temperature": max(s.temperature, 0.0),  # 0 is greedy there as here
585        "top_p": s.top_p,
586        "top_k": s.top_k,  # 0 is no limit there as here
587        "min_p": 0.0,
588        "repetition_penalty": 1.0,
589        "presence_penalty": 0.0,
590        "frequency_penalty": 0.0,
591        "stop_token_ids": sorted(self.eos_ids),
592    }

A plain vllm serve has only /v1/completions: each token comes with its log probability and the 5 most likely tokens at its position with theirs, keyed token_id:<id>, so the rank is known only when the token is among those 5. By default vLLM sends the model's own distribution (--logprobs-mode raw_logprobs), which means the same as SimpleBackend's p.

639def _details(self, token_id: int, logprobs: dict, n: int, s: Sampling) -> dict:
640    """What /v1/completions said about the n-th token of a chunk: its log probability, and those of the 5 most likely
641    tokens, keyed "token_id:<id>". The token's rank is known only when it is one of those 5."""
642    token_logprobs, top_logprobs = logprobs.get("token_logprobs") or [], logprobs.get("top_logprobs") or []
643    if n >= len(token_logprobs) or token_logprobs[n] is None:
644        return {}
645    logprob = token_logprobs[n]
646    listed = top_logprobs[n] if n < len(top_logprobs) and top_logprobs[n] else {}
647    # The chosen token is listed too, even when it is not among the 5.
648    listed = {int(k.removeprefix("token_id:")): v for k, v in listed.items() if k.startswith("token_id:")}
649    top = sorted(listed.items(), key=lambda c: (-c[1], c[0]))[:TOP_N]
650    rank = 1 + sum(v > logprob for _, v in top) if any(i == token_id for i, _ in top) else None
651    return self._token_details(token_id, logprob, listed, rank, None, s)