Bespoke Nimble: Typed Decisions, No Prose

Key Takeaways:

• Bespoke Nimble is a LoRA adapter over Qwen3.5-9B that answers a flat schema of enum and true/false questions by scoring the allowed answer tokens directly, so it never writes prose you have to parse back out.
• On the authors' own 324-example holdout it matched 292 reference labels (90.12%), against 215 (66.36%) for the untuned base model and 302 (93.21%) for TypeSafe's Jev 1.13.0.
• The weights on Hugging Face carry Apache 2.0. The GitHub repository ships with no LICENSE file at all, and those are two separate questions.
• Prompts were trained at 2,048 tokens. A pull request merged on 19 September raised the serving ceiling to 8,192, so anything above 2,048 is outside what the adapter actually saw.
• Training ran on 2,676 curated examples built in a day, which is the reason the authors tell you not to expect much generalisation.

Bespoke Labs pushed a model to GitHub on Thursday and did something unusual with the README: it spends more room on what the thing refuses to do than on what it does. Bespoke Nimble takes a block of text plus a flat schema of questions, and for each question it picks one of the answers you supplied. No reasoning trace, no JSON envelope to unwrap, no paragraph explaining itself. You get the choice, and the probability it assigned to every option you allowed.

By Sunday morning the repo sat at 527 stars and 41 forks off five commits, with no releases published and three contributors listed, one of them named claude. That shape tells you what this is: a recipe someone wrote down quickly because the idea was worth sharing, not a product.

What Bespoke Nimble hands back when you ask it something

You give it a context, which is just the text to judge, and a schema. The schema has to be flat, meaning no nested fields. Every field is either an enum with between 1 and 26 string choices, or a boolean. For each field you can attach a description and a description per choice, which is how you tell the model what "HIGH" is supposed to mean in your business rather than in general.

What comes back is the picked answer plus a probability for each candidate. Under the hood every allowed answer maps to a code that is exactly one token long, and the scorer reads the model's logits for those codes and runs a softmax over them. That last detail is the whole trick, and it is also the source of the most honest warning in the README: those probabilities are normalised across the answers you supplied, so 0.9 does not mean the answer is right nine times out of ten. If none of your options fit the text, the model still has to pick one. The authors tell you to add an explicit "no match" option and to test any threshold on your own data before trusting it.

Fields are scored independently. One field cannot see another field's answer, so consistency between them is your code's problem, not the model's.

A typed answer only earns its keep if something downstream acts on it without a human re-reading the text. That is the part most teams never finish: the classifier works, and then the result sits in a terminal until someone copies it into the ticket, the sheet, or the channel where the decision actually gets made. Zentor closes that last hop by putting the finished work back into Notion, Airtable, Slack or the inbox the task already lives in, rather than handing you an artifact to carry yourself.

The 324 examples, and who wrote the answer key

Bespoke published the scoreboard in the README rather than burying it, and the numbers are worth reading carefully.

Bespoke-Nimble-9B agreed with 292 of the 324 reference labels. Its own base model, Qwen3.5-9B, managed 215. That gap of 77 examples is the entire argument for the recipe, and it came from 2,676 training rows and a LoRA that only touches the answer tokens. A model three times the size, Qwen3.8-27B, landed at 275 and still lost to the tuned 9B, which matches what we keep seeing when a bigger general model gets pointed at a narrow judgement task rather than reasoning its way through one.

Now the caveat the authors put in themselves: the reference labels are synthetic. The 324 examples form 162 contrastive pairs, where one fact is changed so that the correct answer flips. That design is good for measuring whether a model reads the evidence instead of pattern-matching the surface, and it is not the same as agreement with humans. There is a separate public benchmark suite in docs/PUBLIC_BENCHMARKS.md covering thirteen human-labelled datasets, including Banking77 and PubMedQA, which is where you should look before you believe the 90% carries into your domain.

Jev 1.13.0 still wins by ten examples. The README does not pretend otherwise, and it also says plainly that Bespoke Nimble was not distilled from Jev.

Four things Bespoke Nimble will not do

The limits section reads like someone who has already been asked these questions and got tired of it.

Text in, only. The base model has a vision component and Nimble cannot use it, so images are out. It cannot write anything of its own, which rules out an explanation, a quoted span pulled from the context, or nested JSON. An enum tops out at 26 choices. And the probabilities, again, are not calibrated confidence.

There is also the generalisation warning, phrased with a typo the authors have not bothered to fix: 2,676 curated examples means performance depends on those domains, "so don't expect a lot of generalization." They do claim Nimble beats its base model in new domains overall, which is a softer and more believable claim than a headline accuracy number.

Apache on the weights, nothing on the code

This is the part BF-style summaries get wrong, and it matters if you are thinking about shipping anything.

The GitHub repository has no LICENSE file. Not an unrecognised one, not a NOASSERTION: there is no file. Every scorer, the training scripts, the curation pipeline and the evaluation harness sit in that repository with no grant attached, which under default copyright means you have no permission to redistribute or build on them.

The model on Hugging Face is a different artifact with a different answer: the card states Apache 2.0, and the repo tags carry license: apache-2.0. It is an adapter of roughly 165 MiB, not a full checkpoint, so it also drags in whatever terms Qwen3.5-9B ships under.

So: weights, probably fine. Code, ask them first. Two days old is usually all the explanation a missing LICENSE needs, and an issue asking for one is cheaper than a lawyer.

What it costs you in hardware

The 9B weights take about 18 GB unquantised, and the model needs more than that while it runs. On a Mac you need Apple Silicon and Python running natively on macOS so it can reach Metal. On Linux you need an NVIDIA GPU with BF16 support. The merge step, which folds the LoRA into the base weights, runs on CPU and wants both RAM and disk for two copies of the weights; the README suggests 64 GB of memory has a much easier time of it than 24 GB.

Two scorers exist and they behave differently. On Mac, ParallelScorer reads the shared context once and then scores every field in parallel. The CUDA scorer processes the full prompt again for each field. If your schema has a dozen questions against one long document, that difference is not academic.

There is also a trap sitting in the default constructor: calling ParallelScorer() with no arguments loads the Qwen3.5-4B baseline, not Nimble. The MLX runner cannot load a LoRA folder directly and does not support quantised weights, so you have to point it at the merged folder you prepared.

Nobody has to own that hardware to get typed decisions out of a document, which is the trade Zentor makes for you: the run happens somewhere else and the finished result arrives where the work continues, so a laptop with 16 GB is not the thing that decides whether the pipeline exists. If you would rather keep the model on your own machine, the local-model route is a real option and a real cost.

2,048 trained, 8,192 served

The published limit is 2,048 tokens per prompt, and that number includes the schema and the scaffolding that names the field being scored. Longer prompts get rejected rather than truncated, which is the right call.

On 19 September, a day after the repo went public, Mahesh Sathiamoorthy landed pull request #4, "Increase inference prompt limit to 8192 tokens". In the serving code the two numbers now live side by side: TRAINED_PROMPT_TOKENS = 2048 and MAX_PROMPT_TOKENS defaulting to 8192 with an environment override. The Hugging Face card still says 2,048 rejected.

Both are true, and the distinction is the useful bit. You can now feed it four times as much text, and the adapter has never seen a prompt that long. Treat the extra headroom as untested rather than supported.

Where a typed decision earns its place

The pattern Bespoke Nimble borrows comes from TypeSafe's Jev, and the README says so in the second paragraph. Routing a request to one of several destinations, checking whether a stated condition holds, applying a written policy, rating something against ordered levels: each of those is a question with a closed answer set, and each is somewhere teams currently burn a frontier-model call plus a regex.

Cost is the obvious pitch. The less obvious one is that a typed answer is testable. You can diff two runs, you can hold out a set, you can put a threshold on a probability and watch how often it fires. Try doing any of that against a paragraph.

The catch is that a decision on its own is inert. Route a request and something still has to carry it; rate a document and the rating still has to land next to the document. Routing between models is a whole small genre now, and we have looked at one of those routers already. What tends to go missing is the memory across runs: the last twenty decisions, the corrections you made, the thresholds you settled on. Zentor keeps that record on your side and lets you export it, so the accumulated context is not locked to whichever assistant you happened to start with.

FAQ

Is Bespoke Nimble open source?

The model weights are, under Apache 2.0 on Hugging Face. The GitHub repository has no LICENSE file as of 20 September 2026, so the code technically is not licensed for reuse yet. Ask before you ship it.

Does Bespoke Nimble run locally?

Yes, and locally is the only mode. It runs on Apple Silicon through MLX or on an NVIDIA GPU with BF16 through PyTorch. The README notes that no TypeSafe or generation API key is needed for local inference.

How is it different from asking a normal LLM to return JSON?

A general model generates text that happens to look like JSON, and you parse it, retry on failures, and hope the schema holds. Bespoke Nimble never generates: it scores the one-token codes for the answers you allowed and returns probabilities over them. There is nothing to fail to parse.

What base model does it use?

Qwen3.5-9B. The published artifact is a LoRA adapter of about 165 MiB, so you download the base checkpoint separately and merge them, or load the adapter at runtime.

How accurate is it?

292 of 324 on the authors' contrastive holdout, which is 90.12%. Those labels are synthetic. For human-labelled comparisons, read the public benchmark suite in the repository rather than the headline figure.

Worth a weekend, not a production dependency yet

Bespoke Nimble was built in a day, trained on 2,676 examples, and published with a limits section longer than its feature list. Treat that as the point rather than a flaw: it is a written-down recipe for turning a small open model into a decision endpoint, and Bespoke has handed over the data counts, the evaluation harness and the exact prompt builder so you can argue with the result.

What would move it from interesting to dependable is a LICENSE file, a release tag, and a number from someone who is not the author. Until then, Bespoke Nimble is the clearest demonstration this month that a 9B model beats a 27B one when the question has a fixed set of answers and somebody bothered to curate the training data.