llama.cpp brings TypeSafe's typed-decision format to five open model families on local hardware

A call to /v1/systemone sends one shared state and several questions about it. In the pull request's example, the state holds a customer's message and order details ^[11]. The questions ask whether the customer wants a refund, which team should take the case, and how urgent or frustrating it is ^[11]. Each question declares an answer type: yes-or-no (`noul`), a choice among labels, or a score on an ordered scale ^[10]. The server returns a typed answer for each question with a probability distribution attached ^[11].

According to The New Stack, most teams handle this today by prompting a chat model to pick from a list and, with luck, reading token probabilities as a rough confidence score ^[16]. The other route is a small classifier. It is fast and cheap, but it needs labeled data and a retrain every time the label set changes ^[16]. A decision model takes new labels in the prompt and still returns a score a program can act on ^[17].

Support is per model. The pull request describes model-specific conversion, metadata and server handling, so loading some other open model does not give it the endpoint ^[18]. A community discussion had already sketched a wrapper built from existing llama-server features ^[13]. I think per-model conversion code belongs next to the server it has to track. The merged change puts it in the runtime, where conversion and server support are maintained alongside the project ^[13]. Nguyen wrote in the author disclosure that most of the code was AI-written and that he owned the design ^[14].

I'd treat the endpoint as ready to evaluate, but not yet as a proven replacement for hosted calls. The pull request compares llama.cpp outputs with reference outputs on example inputs ^[5]. A parity check is the right first test for a port. It shows the port reproduces the reference model on those cases ^[5]. It cannot show whether a 0.9 on the urgency question is right nine times in ten on a particular support queue ^[5]. Teams still have to run supported models on their own cases and check that reported confidence matches observed accuracy ^[6]. In practice that means holding out labeled real cases, grouping the endpoint's predictions by reported probability, and comparing each group's accuracy with its stated confidence.

OpenAI says its Luna-based decision model returns results in 150 milliseconds, against 1.6 seconds for GPT-6 Luna ^[1][15]. That works out to roughly 10.7 to 1 ^[19]. Both figures describe OpenAI's own models. They become a target for a local deployment only after a team times its own hardware on its own requests. The New Stack lists the per-call price, the number of candidate answers one request can carry, and whether developers can tune the model on their own data as still unclear ^[3]. An OpenAI spokesperson told the publication the company plans to share more "at broad rollout" ^[2].