TITLE=Model service (LLM)
ROUTE=/infrastructure/llm
LOCALE=en
MODIFIED=1792565451
URL=https://wiki.teletypegames.org/infrastructure/llm
L=1 https://git.teletypegames.org/infra/llm
L=2 #66
L=3 #50


q│ One place on the server where small language models run, for any stack that

q│ needs one. It speaks the OpenAI chat API, so any OpenAI-compatible client

q│ works with it. It is reachable only from the 
cinterstack
q docker network, not

q│ from the internet.

• 
bRepository:
0 1git.teletypegames.org/infra/llm
• 
bAddress:
0 
chttp://llm:8080
0 on 
cinterstack


1Calling it

1══════════


p POST http://llm:8080/v1/chat/completions                                     

p Authorization: Bearer <key>                                                  

p {"model": "qwen3.5-0.8b", "messages": [...], "stream": true}                 

Every request names its model: there is no default, a request without 
cmodel
gets a 400. 
cGET /v1/models
0 lists the models and whether each is loaded; 
cGET

c/health
0 needs no key.


t┌───────────────────────┬──────────────┬─────────────────────────────────────┐

t│
h Model                 
t│
h model        
t│
h Used by                             
t│

t├───────────────────────┼──────────────┼─────────────────────────────────────┤

t│
0 Qwen3.5 0.8B          
t│
0 
cqwen3.5-0.8b
0 
t│
0 SERMO's Lapis-ACP in the 2Teletype   
t│

t│
0                       
t│
0              
t│
0 2Games BBS, and any stack: a general 
t│

t│
0                       
t│
0              
t│
0 small model, good at English and    
t│

t│
0                       
t│
0              
t│
0 JSON, Hungarian only roughly;       
t│

t│
0                       
t│
0              
t│
0 thinking off; loaded at start       
t│

t├───────────────────────┼──────────────┼─────────────────────────────────────┤

t│
0 Gemma 3 270M instruct 
t│
0 
cgemma-3-270m
0 
t│
0 Nobody now (SERMO's model until     
t│

t│
0                       
t│
0              
t│
0 October 2026): the fastest, but it  
t│

t│
0                       
t│
0              
t│
0 leaves a persona at any question    
t│

t│
0                       
t│
0              
t│
0 beyond its examples; loaded on its  
t│

t│
0                       
t│
0              
t│
0 first request                       
t│

t└───────────────────────┴──────────────┴─────────────────────────────────────┘


1Models

1══════

The server is llama.cpp's 
cllama-server
0 in router mode: it starts a
llama-server per model on demand. The models are the sections of

cserver/models.ini
0 in the repository, baked into the service's image; a
section's name is the 
cmodel
0 of a request, and its keys are llama-server's long
options (
chf-repo
0, 
cc
0, 
cparallel
0, ...). The GGUF is downloaded from Hugging Face
on the first load and kept in a volume.

• A model with 
cload-on-startup = true
0 is loaded when the service starts
  (Qwen3.5, so SERMO never waits). The others load on their first request.
• At most two models are loaded at once; a third unloads the least recently
  used.
• One memory limit (2.5 GB) covers all the loaded models (the two models above
  take about 2 GB together), and they share the server's CPU. A bigger model
  needs a bigger limit, and its long requests slow down the others.

Adding a model is a new section in 
cserver/models.ini
0 and a push: the deploy
builds a new image and restarts the service with it.


1Keys

1════

Router mode can also load and unload models over HTTP, so the service asks for
a key. 
cLLM_API_KEYS
0 in the stack's 
c.env
0 holds the accepted keys, one per
caller, so one caller can be cut off alone.


t┌────────────────────┬────────────────────────────────────────┐

t│
h Caller             
t│
h Its key                                
t│

t├────────────────────┼────────────────────────────────────────┤

t│
0 2Teletype Games BBS 
t│
0 
cSERMO_LLM_API_KEY
0 in bbs-server's 
c.env
0 
t│

t└────────────────────┴────────────────────────────────────────┘

A new caller gets a new key (
copenssl rand -hex 24
0), appended to 
cLLM_API_KEYS
0.


1Deploy

1══════

The stack lives in 
c/srv/stacks/llm
0, which is not a git checkout. A push to
master that changes 
ccompose.yaml
0 or 
cserver/
0 deploys it through Woodpecker, the
same way as the 3FTP server: the pipeline only calls the server over SSH with
its own key, bound to one fixed command, 
cci-deploy llm
0 (in 
cinfra/host-config
0).
The server clones master itself, checks the compose file against its policy
(no extra privileges, no docker socket, no public port), copies 
ccompose.yaml
and 
cserver/
0 in and runs 
cdocker compose up -d --build
0. 
c.env
0 and the model
volume stay. The pipeline gets no host volumes.