Hi everyone,

This week was unusually rich in open-weight releases.

Meta released Muse Glimmer. Qwen finally released the weights of Qwen3.8 2.4T. NVIDIA released Nemotron 3.5 Lightning. DeepSeek just pushed a major V4 Pro update.

And, just as I was finishing this article, Qwen3.8 27B was released. It was worth the wait. According to Qwen’s evaluations, it improves very significantly over Qwen3.6 27B essentially everywhere, with particularly spectacular gains on long-horizon agentic coding: DeepSWE jumps from 13.3 to 42.2, while QwenSWEBench goes from 49.3 to 79.0. The 27B model also beats Qwen3.7-Plus on many coding and agentic benchmarks, and even beats Opus4.6 Max on QwenSWEBench, CoWorkBench, and LiveCodeBench. At this point, Qwen seems so far ahead that catching up is becoming difficult for everyone else.

There are also a few interesting changes compared with Qwen3.6. preserve_thinking, which Qwen3.6 introduced as an option for keeping reasoning traces across turns, is now enabled by default. Qwen3.8 also introduces reasoning_effort, with low, medium, and xhigh levels to trade reasoning depth against cost. Qwen now recommends temperature=1.0 and top_p=0.95 for thinking mode generally, and, for very long agentic runs, recommends allowing up to 262K tokens for reasoning and 131K for the final answer. I’ll publish a full analysis of Qwen3.8 27B within the next few days, with a separate look at its quantized versions later.

Note: I think this “preserve_thinking” (which is a feature we can find in other models) should be carefully evaluated, especially its impact on inference cost. Qwen3.8 is a crazy thinker (they recommend allowing “up to 262K tokens for reasoning”). So if you preserve reasoning for each turn, the context may grow to millions of tokens. In practice though, for tool calls, reasoning is often much shorter and more/better reasoning may yield fewer turns, so I guess it can work. But I’m really interested to know how important it is, in terms of accuracy/efficiency, to preserve the reasoning traces. KV cache quantization will likely be very important.

Finding enough time to cover all of this properly is hard. I don’t want to publish articles that simply repeat benchmark tables and model cards. A proper analysis means testing accuracy, token efficiency, memory, speed, and then doing it again for several quantized versions.

For Glimmer, I already published an article covering the model and its architecture:

• Muse Glimmer: Meta’s 30B Model Built for Efficient Inference

I also already have speed measurements for all the non-GGUF quantized versions I’m testing.

At the moment, essentially all my compute capacity, several RTX Pro 6000 GPUs provided by Verda,1 is being used to evaluate the quantized versions that run through vLLM: NVFP4, INT4, and my own mixed-precision versions.

You can already find those here:

• My quantized Muse Glimmer collection (they all run, but consider all of them very bad until I publish some official evaluations)

I’ll probably publish a short article focused only on speed first, since those results are already ready.

Then, expect perhaps two deeper articles comparing Glimmer with Gemma 4 and Qwen3.6/3.8, and another article focused specifically on the quantized versions. I may include GGUF versions in that one as well, but I’m not sure yet. There is only so much GPU time, and writing time, in a week.

So, for this Weekly Kaitchup, I’ll focus on the three releases that I won’t cover in more detail:

• Qwen3.8 2.4T: we finally know what is inside it, so I can check how well my predictions held up.

• Nemotron 3.5 Lightning: essentially the new Nano-sized Nemotron.

• DeepSeek V4 Pro 0813: a very large update that makes the word “Preview” much more meaningful.

Qwen3.8 2.4T: How Good Were My Predictions?

A few weeks ago, before Alibaba disclosed the architecture, I tried to estimate what hardware would be required to run Qwen3.8 2.4T:

• Qwen3.8: What Hardware Will You Need to Run Alibaba’s 2.4T Model?

At the time, we knew only one particularly important number: 2.4 trillion parameters.

Everything else had to be inferred.

Now we have the weights and architecture, so it’s a good opportunity to see what I got right and what should be corrected.

Qwen3.8-2.4T-A95B has 2.4T total parameters and 95B activated parameters. It has 92 layers and uses the hybrid architecture introduced with Qwen3.5, alternating Gated DeltaNet blocks with periodic full attention. Its MoE has 512 experts, with 10 routed experts plus one shared expert activated per token. Native context is 262K, extendable to just over one million tokens.

The downloadable 2.4T checkpoint is also more limited than the hosted Qwen3.8-Max: it is text-only and requires thinking mode, while the hosted Max version adds vision, non-thinking mode, built-in tools, and a 1M default context.

So, how did my predictions go?

✅ It is an extremely sparse MoE

This one was the most important and easiest assumption.

I wrote that a dense 2.4T model would make very little sense for inference and that Qwen3.8 was almost certainly going to be a highly sparse MoE.

That was correct.

Only 95B of 2.4T parameters are active for each token, or roughly 4% of the entire model.

✅ Nearly 99% of the parameters are in the routed experts

I assumed that approximately 99% of Qwen3.8’s parameters would live inside routed experts, based largely on Kimi K2 and other very large MoEs.

This turned out to be close.

Using the released dimensions, 512 experts, 92 MoE layers, hidden size 8192, and expert intermediate size 2048, the routed expert matrices account for roughly 2.371 trillion parameters, or about 98.8% of the entire 2.4T model.

My estimate in the original article was 2.374T routed-expert parameters.

✅ The routing sparsity

I used Kimi K2 as a proxy.

Kimi routes each token through 8 of 384 experts, or about 2.08% of its routed experts.

Qwen3.8 uses 10 of 512, or about 1.95%.

So, while the architectures are different, the routing sparsity I used for the estimate was close enough.

❌ I estimated ~75B active parameters

This is the main miss.

Using Kimi K2 as a proxy, I estimated that Qwen3.8 might require compute roughly comparable to a 75B dense model per token, before routing and communication overhead.

The actual number is 95B activated parameters.

That is around 27% higher than my estimate.

So, the general prediction, a 2.4T model with only a small fraction active, was correct, but Qwen3.8 is somewhat more computationally expensive per token than I expected.

❌ Around 1.4–1.5 TB with experts-only NVFP4

Wrong but close.

I estimated 1.387 TB for a version where the routed experts are stored in NVFP4 while the rest of the network remains at higher precision.

A community NVFP4 checkpoint from RadixArk now does exactly that: only the routed-expert linear layers are quantized to NVFP4 while the other components retain their original precision.

Its repository is 1.48 TB.

So my estimate was around 6–7% too low, but the overall hardware conclusion was correct.

A single 8xB300 node is enough to deploy this model.

Nemotron 3.5 Lightning: Nano, Updated

NVIDIA also released Nemotron 3.5 Lightning 30B-A3B this week.

If the naming feels slightly confusing, Nemotron 3.5 Lightning is effectively the updated model occupying Nemotron 3 Nano’s position in the family.

I previously tested the original Nano here:

• Nemotron 3 Nano: A Very Fast Model That Doesn’t Think Too Much

And I covered the much larger Super model here:

• Nemotron 3 Super: 1M Tokens, Small KV Cache

Lightning still uses the same general idea of interleaving Mamba-2 processing with selected attention layers and sparse experts, and it supports context lengths up to 1M tokens.

So this is not a new size class. It is much closer to a refreshed Nano.

But NVIDIA has brought several ideas developed across the rest of the Nemotron 3 family back into this smaller model.

The biggest additions are around inference and agents.

Multi-Token Prediction is now part of the model training, followed by an additional MTP-boosting phase. NVIDIA also releases DFlash and DSpark draft models for speculative decoding, so there are multiple ways to accelerate generation depending on concurrency and hardware.

This is interesting because MTP was already one of the important additions in Nemotron 3 Super.

NVIDIA also says that 3.5 Lightning received harness-optimized training for agent workloads. Large Nemotron models can do the expensive planning and reasoning, while Lightning is supposed to execute the many smaller calls produced by long-running agents.

There are BF16 and NVFP4 checkpoints, and NVIDIA says the model can reach up to 4x the output speed of similarly sized models. On its PinchBench test, NVIDIA reports roughly 86% accuracy while completing 10,000 tasks around 30% faster than Qwen3.6 35B at comparable accuracy. Those are NVIDIA’s numbers, so I would still like to reproduce the speed/accuracy trade-off independently.

DeepSeek V4 Pro 0813: “Preview” Really Meant Preview

Finally, DeepSeek (quietly) released DeepSeek-V4-Pro-0813.

When DeepSeek introduced V4 in April, both V4 Pro and V4 Flash were explicitly called Preview models. The architecture was extremely innovative.

V4 Pro has around 1.6T parameters with 49B active and combines Compressed Sparse Attention with Heavily Compressed Attention. DeepSeek also introduced Manifold-Constrained Hyper-Connections and trained with the Muon optimizer. At a one-million-token context, DeepSeek reported that V4 Pro required only 27% of the single-token inference FLOPs and 10% of the KV-cache memory of DeepSeek V3.2.

Accuracy was the problem.

The original V4 Pro was rather underwhelming for such a huge model, particularly on agentic tasks. Then DeepSeek released Flash-0731, and the Flash model was suddenly beating Pro Preview.

I wrote about that two weeks ago:

• DeepSeek-V4-Flash-0731 and Inkling Small: Smaller, but Better?

At the time, I wrote that Flash-0731 was “a very promising sign for the next V4 Pro update.” That update is here now. The model now also exposes three reasoning-effort settings: low, high, and max.

The accuracy jump is much more important.

A few examples from DeepSeek’s own evaluation table:

• DeepSWE: 12.8 (it was lower than Qwen3.8 27B!) → 62.7
  Current Flash-0731: 54.4

• Terminal Bench 2.1: 72.1 → 87.9
  Current Flash-0731: 82.7

• NL2Repo: 38.5 → 61.5
  Current Flash-0731: 54.2

• Cybergym: 52.7 → 83.3
  Current Flash-0731: 76.7

• Toolathlon Verified: 55.9 → 74.1
  Current Flash-0731: 70.3

• AutomationBench: 12.8 → 31.8
  Current Flash-0731: 25.1

Pro-0813 is now ahead of the current Flash-0731 model.

DeepSWE is the most spectacular improvement: 12.8 to 62.7.

As I discussed this week, agentic evaluations can move significantly depending on the harness, configuration, allowed steps, tools, and other details. Independent testing remains necessary.

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisioning your account with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I share this coupon because I really think it’s a good deal. I don’t receive any form of compensation from Verda, or any usage information, related to this coupon. Verda also provides compute sponsorship for some of my articles.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

Coding benchmarks increasingly test more than whether a model can write a correct function. They ask an agent to explore an unfamiliar repository, run commands, edit files, interpret failures, test its work, recover from mistakes, and decide when the job is actually finished.

That makes benchmark performance a product of at least three things: the underlying model, the agent harness driving it, and the amount of time and context the system is allowed to consume.

I evaluated Laguna S 2.1 on two long-horizon benchmarks, DeepSWE and Terminal-Bench 2.1, to understand how much these surrounding conditions affect both accuracy and cost. In particular, I wanted to see how the model behaves outside Poolside’s native agent harness, how much additional inference-time compute improves results, and what happens when an agent is given enough time to keep working after it gets stuck.

The results show a capable model whose accuracy can improve substantially when it is given more room to work. They also expose the other side of long-horizon agents: unsuccessful trajectories can consume enormous amounts of computation without getting any closer to a correct solution.

Perhaps most importantly, the experiments reinforce that model configuration is only part of the story. The agent wrapped around the model matters enormously.

In this article, I compare Laguna S 2.1 across DeepSWE and Terminal-Bench 2.1, examine how turn limits, timeouts, and token budgets affect accuracy and cost, analyze the main failure modes, and explain why the agent harness itself has become a critical part of coding-model evaluation.

Acknowledgments

Verda provided the B300 to run the experiments described in this article.

Verda is the full-stack frontier AI cloud, built for high-performance inference, training, and agentic workloads with data privacy and sustainability at its core.

You can check them out here. There is a $50 coupon that you can redeem in your Verda account, after provisioning it with $5, to try their GPUs.

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I don’t receive any form of compensation from Verda, or any usage information, related to this coupon.

Agentic coding evaluation setup

I deliberately evaluated Laguna S 2.1 outside Poolside’s native pool harness.

For DeepSWE, I used mini-swe-agent.

For Terminal-Bench 2.1, I used the terminus-2 agent.

Poolside’s own Laguna S 2.1 evaluations use its pool agent harness, while the public DeepSWE leaderboard normally uses mini-swe-agent. My numbers should be read as measurements of Laguna operating inside these particular third-party agent setups, not as direct reproductions of Poolside’s published benchmark results.

Read more

Meta is back, and it’s great.

For a while, the center of gravity in open-weight AI moved toward Qwen, DeepSeek, Google’s Gemma family, and a growing number of Chinese labs. Muse Glimmer gives Meta a genuinely interesting entry again.

Muse Glimmer is a ~30B-parameter multimodal reasoning model from Meta Superintelligence Labs, released under Apache 2.0.

• Muse Glimmer

Glimmer is distilled from Meta’s much larger Muse Spark model, supports tool use and multimodal inputs, has a 131K context window, ships with official 4-bit quantization, and targets consumer hardware. It’s clearly positioned as a competitor to Gemma 4 and Qwen3.6 open dense models.

How good is Glimmer compared with Gemma 4 and Qwen3.6?

The benchmark results put Glimmer surprisingly close to Qwen3.6. But benchmark scores alone tell us nothing about efficiency.

Glimmer uses an exceptionally lightweight attention design, and its relatively short 131K context window suggests it was not trained around the extremely long reasoning traces Qwen3.6 can produce, sometimes stretching beyond 100K tokens.

That opens up an interesting possibility: Glimmer may deliver performance close to Qwen3.6 while approaching Gemma 4’s token efficiency and using significantly less memory overall.

In this article, I’ll first look at what makes Muse Glimmer interesting as a local model, then break down its architecture in detail, including its hybrid attention pattern, extreme GQA, gated attention, positional encoding, multimodal stack, and speculative decoding. I’ll then dig into its KV-cache efficiency, with a direct 131K-context comparison against Gemma 4 31B and Qwen3.6-27B.

Finally, I’ll look at the benchmark results, where Glimmer gets surprisingly close to Qwen3.6, and discuss why token efficiency may turn out to be an even more important metric for agentic workloads.

Note: I’m also preparing a full analysis detailing the model accuracy, token efficiency, memory consumption, and inference speed for the original and quantized versions. It should take about a week to gather enough data.

Read more

Hi everyone,

We have a countdown for the Qwen3.8 releases:

Next week is going to be a busy one.

Here’s my plan:

• Start my own evaluation of Qwen3.8 27B as soon as the model is released. This time, I’ll also cover agentic coding. I plan to publish the full analysis the following week.

• Run and report on evaluations of the GGUF versions by the end of the week.

• Publish a deeper analysis of the quantized models within 14 days of release.

This will be similar to what I have done for Qwen3.6.

I’ll also release my own quantized versions using my new quantization pipeline that brings together AutoRound (for high-quality quantization), LLM Compressor (for packing with a vLLM-friendly format), and a custom repacker (to correct bugs with some layers). The goal is to finally make mixed-precision models, with different layers quantized anywhere from 2-bit to 8-bit, work well with vLLM.

Conceptually, it brings some of the flexibility that GGUF provides for llama.cpp to the vLLM ecosystem, while targeting much higher throughput under heavy concurrency, for example, when running multiple sub-agents in parallel, by leveraging the Humming kernel.

I’ve already published Qwen3.6 27B variants built with this pipeline.

• Qwen3.6 WNA16

The 3.8 and 4.3 bit versions work well. I’m still evaluating the 3.5 and 3.0 bit versions. I’ll update the model cards once I’m sure they all work well. I’ll also add versions with a quantized language modeling head later this week. I thought about quantizing the token embeddings, but this isn't well supported by vLLM for the Qwen3.5 architecture (it doesn’t crash, but only generates “!!!!!”; that’s a bug, not a quantization quality issue).

Now I’m ready to do the same for Qwen3.8.

The Harness Can Matter Almost as Much as the Model

A very interesting test from Composio showed how much the agent harness can change the results, even when the model stays the same.

Yes, it sounds obvious when stated like this, but I increasingly come across model evaluations claiming superiority over other models while relying on numbers produced with suboptimal agent harnesses for the evaluated models, or with harnesses configured in completely different ways (more turns, higher timeouts, …).

At best, these comparisons are useless. More often, they are actively misleading.

Composio ran DeepSeek V4 Flash through four different harnesses, Claude Code, Codex, OpenCode, and Oh My Pi, across 30 multi-step agentic tasks. The tasks involved live tools such as Gmail, Google Sheets, GitHub, Slack, Notion, Calendar, Airtable, and PagerDuty. A task was considered successful only if it passed every predefined check.

Observations

• Oh My Pi achieved the highest success rate, completing 17 out of 30 tasks. Claude Code and Codex each completed 16, while OpenCode completed 14.

• OpenCode had the lowest estimated cost per successful task at $0.073, followed by Codex at $0.081. Oh My Pi came in at $0.103, while Claude Code was the most expensive at $0.195.

• Speed produced yet another ranking. Claude Code had the lowest median completion time at 122.7 seconds, with OpenCode close behind at 129.7 seconds. Codex took 245 seconds, while Oh My Pi was the slowest at 272.4 seconds.

So, it seems we have different trade-offs:

• Claude Code was the fastest, but also the most expensive.

• Oh My Pi completed the most tasks, but was the slowest.

• OpenCode was the cheapest, but also had the lowest success rate.

But we also have another angle not tackled here: inference-time sampling. Rerun the same experiments, and you may get a very different picture. I agree with Composio’s general conclusion, but we need many more runs to conclude which is the cheapest, fastest, and most accurate harness.

This is also why evaluating models for agentic coding is extremely difficult, and extremely expensive to do properly.

I’m currently preparing an article focused on Laguna S2.1 and how I benchmarked it for agentic coding. I ran into far more issues than I would have liked to admit, but the process made one thing very clear to me: a large number of the agentic coding benchmark tables I see online should be treated with extreme caution.

Without controlling for the harness, its configuration, tools, prompts, retry behavior, and execution environment, comparing model scores can quickly become meaningless.

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisionning your account with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I share this coupon because I really think it’s a good deal. I don’t receive any form of compensation from Verda, or any usage information, related to this coupon. Verda also provides compute sponsorship for some of my articles.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

The community has tried to address this issue, but with limited success. One approach is to introduce a “thinking cap” that stops the reasoning process once the model reaches a predefined token budget. Another is to fine-tune the model on shorter, more efficient reasoning traces.

I examined both approaches in a previous article about Qwen3.5 and Qwen3.6. These approaches either performed poorly or required substantially more expensive fine-tuning.

Hopefully, Qwen3.8 27B, which is expected to be released this week, will be a more efficient reasoner.

In the meantime, BottleCap AI has released a promising custom alternative that significantly reduces the reasoning cost of Qwen3.6: ThinkingCap-Qwen3.6-27B.

ThinkingCap-Qwen3.6-27B is a post-trained version of Qwen3.6-27B. Its main improvement is behavioral: it has been trained to produce shorter reasoning traces before delivering an answer.

The model also tends to generate shorter final responses. According to BottleCap AI, this behavior emerged during training alongside the reduction in reasoning length.

In this article, I evaluate the model’s accuracy and token efficiency across several tasks. I compare it with the original Qwen3.6 and another model known for its efficient reasoning, Gemma 4 31B IT.

All three models are evaluated using exactly the same evaluation settings, with their recommended hyperparameters, making the results directly comparable. Every number reported here comes from my own testing.

I also examine the model’s reasoning stability. In particular, is ThinkingCap-Qwen3.6-27B less prone to excessively long or seemingly endless reasoning than the original Qwen3.6?

Read more

Hi everyone,

In this edition of The Weekly Kaitchup:

• DeepSeek V4 Flash: Now Better than DeepSeek V4 Pro (Preview)

• Inkling-Small: Better than Inkling (?)

• Escha-W2: Qwen3.6 35B Compressed to 12.3 GB and Faster

DeepSeek released an update for its V4 Flash model. As usual, DeepSeek released the weights for this update immediately, rather than imposing the waiting period we have seen from companies such as MiniMax and Qwen.

• deepseek-ai/DeepSeek-V4-Flash-0731

It now outperforms the much larger V4 Pro (Preview), with particularly strong gains in agentic coding:

They did not publish results for other task categories. I checked Artificial Analysis’ additional results and found that it also improved on other benchmarks, including GPQA Diamond, which is a very different task from agentic coding. In fact, none of the reported benchmarks appear to show a regression. This is not merely a more specialized model, it is a substantial improvement over the preview and a very promising sign for the next V4 Pro update.

On a related note, Qwen also released Qwen3.7 Flash this week. It is currently available only through APIs and is very inexpensive. The community has speculated that it may be a Qwen3.7 35B-A3B model and that Qwen will eventually release the weights. I am not certain this is correct, but it would make sense.

Inkling-Small: Smaller, but Better on Most Benchmarks

Despite the name, Inkling-Small is not a lightweight model in the usual sense. It is still an open-weight, decoder-only multimodal model that accepts:

• Text

• Images

• Audio

It follows the same general architecture as the larger Inkling. Both models use a sparse mixture-of-experts design, routing each token through six of 256 experts, along with two shared experts. Both also support context windows of up to one million tokens.

So Inkling-Small is mostly the same design, scaled down.

How much smaller is it?

The larger Inkling has:

• 66 transformer layers

• 975 billion total parameters

• About 41 billion active parameters per token

Inkling-Small reduces that to:

• 42 transformer layers

• 276 billion total parameters

• About 12 billion active parameters per token

That makes Inkling-Small a little over one-quarter the size of the larger model by total parameter count. Its active parameter count is reduced by a similar amount.

The NVFP4 version is much easier to fit

Inkling-Small requires around 600 GB of aggregate VRAM in BF16.

The NVFP4 checkpoint brings that down to roughly 180 GB.

For comparison, the larger Inkling’s NVFP4 checkpoint is around 592–600 GB. In other words, the quantized version of the large model takes roughly as much memory as Inkling-Small does in BF16.

The word “small” is doing some work here, but 180 GB is at least in the range of a compact multi-GPU server rather than a large GPU cluster.

The smaller model is weirdly better at a lot of things

The surprising part is that Inkling-Small beats the larger Inkling across most published head-to-head benchmarks.

A few examples:

• SWE-bench Verified: 80.2% vs. 77.6%

• Humanity’s Last Exam: 31.6% vs. 29.7%

• IFBench: 82.2% vs. 79.8%

The smaller model appears particularly strong in coding, tool use, instruction following, and several reasoning evaluations.

Thinking Machines attributes the improvement to a revised data mix, an updated training recipe, distillation from the larger Inkling model, and additional reinforcement learning focused on agentic coding.

The larger model still performs better in some areas, especially factual recall, broad knowledge coverage, and most audio evaluations.

Still, it is an unusual result: the smaller model uses far fewer parameters, needs much less memory, and yet comes out ahead on a majority of the reported benchmarks.

Another unexpected result is that it significantly outperforms Qwen3.5 397B despite being much smaller, especially on benchmarks where Qwen3.5 has always been exceptionally good, like GPQA Diamond.

Is it benchmaxxed?? I’m waiting for the community feedback!

Escha-W2: Qwen3.6 35B Compressed to 12.3 GB

If you primarily use GGUF models locally, a 12.3 GB version of Qwen3.6 35B may not sound especially impressive. After all, that is roughly the size of a heavily quantized GGUF model, like a Q2_K_XL, and we already know those can perform well, as shown by the results I published a few months ago.

The key difference with Escha-W2 is that it uses (presumably) a more advanced quantization method and comes with an optimized inference runtime. As a result, it should run faster than the typical GGUF model, while already supporting SGLang for significantly better throughput under concurrent workloads. The developers have also said that vLLM support is coming later.

How They Did It?

We don’t really know yet.

The original model is a mixture-of-experts model with:

• 35 billion total parameters

• Around 3 billion active parameters per token

• 256 experts

Escha Labs keeps that underlying architecture but compresses most of the expert weights to roughly two bits. More precisely, it uses a mixture of two- and three-bit quantization for the expert projections, while keeping the dense layers in INT8.

The whole model is 12.3 GB

The resulting checkpoint takes up 12.3 GB on disk.

It can run on:

• A single 24 GB GPU under the recommended configuration

• A 16 GB GPU with reduced context length or fewer concurrent requests

On an RTX 4090, Escha reports around 225 tokens per second for a single stream. The same setup reaches about 1,321 tokens per second when serving 32 requests at once.

Two bits, but roughly FP8-level benchmark scores

The more interesting part is how little the compression appears to affect most of the reported evaluations.

Across six benchmark categories, Escha-W2 averages 100.2% of the FP8 model’s score. That does not mean the quantized model is genuinely better, the small gains are mostly within normal evaluation variance, but it does suggest that the overall quality loss is limited.

A few examples:

• MMLU-Pro: 80.9 versus 82.3 for FP8

• MATH-500: 93.8 versus 91.2 (ignore this one; this is too old and too easy for Qwen3.6)

• GPQA-Diamond: 77.8 versus 74.7

• BFCL tool use: 88.9 versus 88.2

• RULER long-context retrieval: 89.9 versus 89.4

Coding is the main place where it loses ground

The clearest regression is on longer coding tasks, where quantization always does more damage.

On LiveCodeBench v6, Escha-W2 scores 62.6, compared with 67.0 for the FP8 baseline. Note: I don’t know which framework they used to get a 67.0, but these scores seem very low. It should be around 85.0. This suggests that they ran it with a limited context length, like 32K max tokens. Not great. This means that the accuracy gap could be greater at longer context length, as quantization tends to be worse as sequence length increases.

It is not a normal drop-in quantization

There is one practical catch: Escha-W2 needs a custom runtime.

Its weights use a custom packed format, so the model does not simply load into standard inference software as a regular GPTQ, AWQ, or GGUF checkpoint. Escha currently provides two options:

• An SGLang-based runtime for concurrency, tool calling and structured output

• A standalone ZML runtime aimed at single-user inference

The problem with specialized runtimes is that it’s very hard work to maintain them.

The checkpoint is also text-only, even though the original Qwen architecture includes vision components.

When they release the vLLM runtime, I’ll double-check the accuracy on longer context and compare it with quantized versions of similar size.

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisionning your account with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I share this coupon because I really think it’s a good deal. I don’t receive any form of compensation from Verda, or any usage information, related to this coupon. Verda also provides compute sponsorship for some of my articles.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

Prism ML has released impressive ternary and 1-bit variants of Qwen3.6 27B. The 1-bit version is only 3.9 GB, making it dramatically smaller than the original, which is roughly 55 GB.

• Bonsai 27B Models (Hugging Face)

These Bonsai 27B models behave quite differently from the original Qwen3.6 model, particularly in terms of accuracy and token efficiency.

Can you get Qwen3.6 27B-level accuracy from a 3.9 GB model?

With reasoning enabled, Bonsai 27B can nearly match the accuracy of Qwen3.6 27B running without reasoning. Moreover, that dramatic reduction in memory comes at a substantial computational cost: Bonsai needs to generate far more tokens to achieve the same result, up to 14 times more on some coding tasks.

When reasoning is enabled for both models, Qwen3.6 27B remains clearly ahead. It uses fewer tokens while achieving significantly higher accuracy. Still, what Prism ML has achieved is remarkable.

In this article, I take a deep dive into Bonsai 27B’s recipe, benchmark performance (my own numbers), and token efficiency. We will examine where the model struggles most, why its raw accuracy is lower, and why the results are nevertheless highly promising.

Read more

Hi everyone,

In this edition of The Weekly Kaitchup, I’m looking at two of the week’s most interesting releases for agentic workloads: Nanbeige4.2-3B and Laguna S 2.1.

Agentic workloads include multi-step reasoning, tool use, interaction with external environments, and tasks that may require many actions before completion.

• Nanbeige/Nanbeige4.2-3B

• poolside/Laguna-S-2.1

Despite this shared focus, they operate at very different scales.

Nanbeige4.2-3B is a compact dense model with approximately 4 billion total parameters and 3 billion non-embedding parameters. It is intended to make capable agentic behavior practical on consumer and workstation hardware.

Laguna S 2.1 is a 118-billion-parameter Mixture-of-Experts (MoE) model. It activates approximately 8 billion parameters for each token.

The two models represent different approaches to agentic AI. Nanbeige uses repeated computation over a relatively small set of weights, while Laguna uses sparse access to a much larger pool of learned parameters.

Nanbeige4.2-3B

While Nanbeige4.1 used a basic Llama architecture, Nanbeige4.2-3B uses a Looped Transformer.

Instead of containing a large number of unique transformer layers, the model has 22 physical decoder layers that are executed twice. The same weights are reused during the second pass, giving the model the computational depth of approximately 44 layer executions without storing 44 independent layers.

This design reduces weight memory, but it does not reduce inference computation to that of a normal 22-layer model. Each token must still pass through the transformer stack twice.

The model has 48 attention heads with a dimension of 128, and eight key/value heads. This is a lot. For instance, Qwen3.6 35B A3B has only two KV heads.

Looping introduces a further memory consideration. Unless the implementation explicitly shares KV-cache state between the two passes, each loop may require separate key/value entries. Nanbeige’s default configuration does not appear to enable loop-level KV sharing.

Moreover, I think models using the Looped Transformer should be better named. When you use a 3B model, you don’t expect it to be nearly as slow as a 6B model. small MoE models, like Qwen3.6 and Gemma 4, show the number of active parameters in their name, like 26B-A4B. For Looped Transformers, we should adopt a similar convention, like 3B-2P (for “2 passes”), for instance.

Nanbeige4.2-3B’s memory consumption

A 16GB GPU should be able to run the model, without quantization, at moderate context lengths with careful settings. A 24GB GPU provides more room for longer prompts, greater concurrency, and runtime overhead.

Based on the model’s 22 physical layers, two loop passes, eight key/value heads, 128-dimensional heads, and BF16 KV values, a conservative estimate places KV-cache consumption at approximately 176 KiB per cached token for each active sequence.

I show in the following article how to estimate the KV cache memory consumption:

At 16K tokens, this would require roughly 2.75 GiB of KV-cache memory. Using the complete 256K context could require roughly 44 GiB of KV-cache memory for a single active sequence.

This is very large. Almost twice as much as what Qwen3.6 27B would consume for the same context length.

Using Nanbeige4.2-3B with vLLM

At release, Nanbeige provided a dedicated vLLM branch for the model:

git clone -b nanbeige42 https://github.com/Nanbeige/vllm.git
cd vllm
pip install -e .

The model can then be served through vLLM’s OpenAI-compatible API:

vllm serve Nanbeige/Nanbeige4.2-3B \
  --host 0.0.0.0 \
  --port 8000 \
  --enable-auto-tool-choice \
  --tool-call-parser nanbeige \
  --reasoning-parser nanbeige

 #use --host 127.0.0.1 if you are running it locally

The reasoning and tool-call parsers are important for agent frameworks. They allow vLLM to separate reasoning content and structured tool calls from ordinary assistant text.

Nanbeige’s chat template exposes two controls.

The enable_thinking option determines whether the current response contains an explicit reasoning phase. The preserve_thinking option determines whether reasoning content from previous assistant turns remains in the conversation history.

For normal chat and question answering, preserved reasoning can usually be disabled. For multi-turn tool use, office workflows, and coding agents, the developers recommend retaining earlier reasoning content but I don’t understand how this can work well. Reasoning traces can be very long, like 50K+ tokens. Keeping them in the context, even if it’s just one, means that the model may reach its max context length at next turn.

Performance

According to the published evaluation, Nanbeige exceeded Qwen3.5-9B on most of the included agent, coding, and reasoning tasks. It also surpassed Gemma 4 12B on benchmarks including GDPval, SWE-Bench Verified, SWE-Bench Pro, and Terminal-Bench 2.0.

Agent results are influenced not only by the base model but also by the allowed reasoning budget, tool definitions, prompting strategy, conversation-state handling, retry policy, and maximum number of actions.

Reference: Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model

Laguna S 2.1

Laguna S 2.1 is a much larger coding-focused MoE model.

It contains 118 billion total parameters but activates approximately 8 billion parameters for each token. A routing mechanism selects a subset of experts during inference, allowing the model to access a large learned parameter pool without executing every parameter for every token.

Laguna contains 48 transformer layers. Twelve use global attention, while the remaining 36 use sliding-window attention with a window of 512 tokens. The global and local layers are interleaved in approximately a one-to-three ratio.

The global layers allow information to move across the complete input context. The sliding-window layers restrict attention to nearby tokens, reducing the cost of long-context inference and limiting KV-cache growth.

The model contains 256 routed experts and one shared expert. For each token, the router selects the top ten routed experts in addition to the shared computation.

Laguna also uses per-head softplus output gating.

Per-head softplus output gating gives each attention head its own learned volume control. For every token, the model can reduce, preserve, or amplify the contribution of individual heads before combining their outputs. The softplus function keeps these gates positive while allowing values above one, so useful attention heads can be strengthened rather than merely switched on or off.

It uses interleaved reasoning. The model can reason, issue a tool call, receive the tool result, and resume its reasoning before selecting another action.

Poolside also provides a DFlash draft model for speculative decoding. These models generate candidate tokens that the main model can verify in parallel, potentially increasing output speed.

Using Laguna S 2.1 with vLLM

Laguna support is documented for vLLM 0.25.0 or newer:

uv pip install -U "vllm>=0.25.0"

The model can run on a single B300 GPU:

vllm serve poolside/Laguna-S-2.1 \
  --enable-auto-tool-choice \
  --tool-call-parser poolside_v1 \
  --reasoning-parser poolside_v1 \
  --default-chat-template-kwargs '{"enable_thinking": true}'

The NVFP4 and INT4 versions consume fewer than 80 GB but you would need a 96 GB GPU, like an RTX Pro 6000, to exploit the context length of the model.

vllm serve poolside/Laguna-S-2.1-INT4 \
  --enable-auto-tool-choice \
  --tool-call-parser poolside_v1 \
  --reasoning-parser poolside_v1 \
  --default-chat-template-kwargs '{"enable_thinking": true}'

For coding-agent workloads, Poolside also recommends retaining prior reasoning_content in the conversation history.

As for the memory consumption of the KV cache, 256K tokens should consume around 24 GB.

Target tasks

Laguna S 2.1 is more specialized than Nanbeige.

Its primary target is long-horizon software engineering. This includes repository-level bug fixing, terminal interaction, shell-based workflows, multilingual code maintenance, codebase exploration, and answering questions that require understanding large software repositories.

Performance

Laguna’s sparse architecture and coding-focused training are particularly effective for terminal interaction, repository-level reasoning, and multilingual software engineering.

It looks like a very good model but so far, I have found community feedback to be mixed.

reference: Introducing Laguna S 2.1

I’ll spend some time with Nanbeige and Laguna. If everything goes well, I’ll probably write an article using them for agentic coding.

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisionning your account with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Note: I share this coupon because I really think it’s a good deal. I don’t receive any form of compensation from Verda, or any usage information, related to this coupon. Verda also provides compute sponsorship for some of my articles.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

Alibaba’s Qwen3.8 is a 2.4-trillion-parameter frontier model. The company has indicated that it plans to release the model’s weights, potentially making Qwen3.8 one of the largest openly downloadable AI models ever produced.

For comparison, Alibaba’s largest open-weight model to date, Qwen3-Coder-480B-A35B-Instruct, is roughly five times smaller. Qwen3.8 would still have about 400 billion fewer parameters than Kimi K3.

With both K3 and Qwen3.8 performing nearly at the frontier, was closing the accuracy gap with OpenAI and Anthropic largely a matter of scaling from hundreds of billions of parameters to more than two trillion?

An open-weight release for Qwen3.8 is excellent news. It could give researchers, companies, and the broader AI community unprecedented access to a model at this scale.

But what could most people realistically do with a 2.4-trillion-parameter model?

Very little locally. Even with aggressive quantization, compression, or distributed inference, a model of this size would be far too large to run on a typical personal computer. Its release could still be highly significant for research institutions, infrastructure providers, and well-resourced community projects. For casual users, however, direct local deployment would remain largely impractical.

Let’s check, with a few assumptions, what you would need to run NVFP4, Q4, and Q1 GGUF versions.

Qwen3.8: A Sparse MoE?

As I write this, Alibaba has not disclosed when it plans to release the model’s weights. That could change quickly. They may even be released by the time this article is published.

Alibaba has not yet disclosed how many parameters are activated for each token, how many experts the model contains, or whether every layer uses mixture-of-experts routing.

Nevertheless, a dense 2.4-trillion-parameter transformer would be extraordinarily expensive to serve. It is thus reasonable to assume that Qwen3.8 uses a highly sparse mixture-of-experts, or MoE, architecture.

Until Alibaba publishes the architecture, Moonshot AI’s Kimi K2 provides a useful proxy for estimating how a trillion-parameter MoE model might distribute its weights.

Kimi K2 contains:

• Total parameters: 1.04 trillion

• Activated parameters: 32.6 billion

• Routed experts: 384

• Experts selected per token: 8

• Routing sparsity: 8/384, or 1/48

• Shared experts: 1

Note: These figures come from the Kimi K2 technical report.

98.9% of Kimi K2’s parameters are located inside routed experts, while only about 1.1% are always-active or otherwise non-routed weights. The trend is the same for all the MoE with 500B+ parameters: Nearly 99% of the parameters are in the routed experts.

Applying the same ratio to Qwen3.8

In other words, Qwen3.8 could store 2.4 trillion parameters while using only a small fraction of them for each token. If its routing sparsity resembled Kimi K2’s, the computation required per token might be comparable to that of a roughly 75-billion-parameter dense model, before accounting for routing and distributed-communication overhead.

BF16 stores each parameter using 16 bits, or two bytes.

This does not include the KV cache, inference-engine workspace, CUDA graphs, temporary activations, multimodal components, tokenizer files or checkpoint metadata.

A practical deployment would consequently need more than 4.8 TB of combined accelerator memory. I won’t speculate on the KV cache size. There are too many variables that influence it: number of linear layers, number of KV heads, etc.

Experts in NVFP4, remainder in BF16

I hope Alibaba releases an official NVFP4 version. Producing a high-quality NVFP4 quantization of a model this large would be prohibitively expensive for most community projects. NVIDIA has also been relatively slow to publish its own NVFP4 conversions of open-weight models; for example, its version of Qwen3.6 appeared only recently.

NVFP4 stores each parameter as a 4-bit value, with a single FP8 scaling factor shared across all 16-value blocks. This results in an effective storage cost of approximately 4.5 bits per parameter, plus a negligible FP32 scaling value for each tensor.

Fortunately, routed experts tend to be relatively robust to quantization. In MoE models, the expert weights can often be stored at lower precision while the comparatively small set of shared, attention, embedding, and other always-active parameters remains at higher precision. This mixed-precision approach can substantially reduce storage requirements while limiting the effect on model quality.

So, with 99% of parameters quantized to NVFP4:

• 2.374 trillion routed-expert parameters at 4.5 bits each

• 25.8 billion remaining parameters at 16 bits each

The expert weights consume:

The BF16 remainder consumes:

Total:

That’s an average of 4.62 bits per parameter.

Compared with a 4.8 TB BF16 checkpoint, that saves approximately:

or 71.1% of the original weight memory.

What about a typical Q4 GGUF?

A common GGUF format such as Q4_K_M is approximately 4.8 bits per weight in representative models, while Q4_K_S is closer to 4.6 bits per weight. Actual ratios vary with architecture and which tensors remain at higher precision.

For a 2.4-trillion-parameter checkpoint:

• Q4_K_S-like: 4.58 bits; approximately 1.374 TB

• Q4_K_M-like: 4.84 bits; approximately 1.452 TB

• Higher-overhead Q4: 5.0 bits; approximately 1.50 TB

With runtime allocations and a useful KV cache, a deployment should budget at least 1.6–1.8 TB of usable memory, and more for long contexts or concurrent users.

What would TQ1 weigh?

I’m mentioning TQ1 since Unsloth has previously released very good TQ1 versions of Qwen3.5. We have no guarantee they can do/will do the same for Qwen3.8.

TQ1_0 uses a compact ternary representation at approximately 1.69 bits per weight.

Purely as storage arithmetic:

Could Qwen3.8 run “locally”?

It will not be a normal desktop model. Even the Q4 version would be roughly twenty times larger than a 70-billion-parameter Q4 model.

Full BF16

A BF16 deployment would need at least 4.8 TB for weights and roughly 5.5 TB after adding a modest 15% operational allowance.

That points toward configurations such as:

• Approximately 24 B300 GPUs, assuming 288 GB each.

• Approximately 32 B200 GPUs, assuming 180 GB each.

NVIDIA’s eight-GPU DGX B200 provides 1.44 TB of HBM and can be configured with 2–4 TB of system RAM. NVIDIA’s Blackwell Ultra B300 provides 288 GB of HBM per GPU, or approximately 2.3 TB in an eight-GPU system.

BF16 Qwen3.8 is firmly a multi-server deployment.

NVFP4, Q4, and TQ1

A 1.4–1.5 TB quantized model is more approachable, but “approachable” still means rack-scale hardware.

An eight-GPU B300 server with approximately 2.3 TB of HBM should have enough room for the quantized weights, runtime allocations, and a reasonable KV cache.

An eight-GPU B200 system has only 1.44 TB of HBM. It might barely load the 1.387 TB mixed-NVFP4 estimate, but it would leave almost no memory for the inference engine or KV cache. A two-node, 16-B200 configuration would be much more practical.

A 507 GB TQ1 checkpoint would still need roughly 600 GB or more after runtime overhead.

It could fit in:

• A 768 GB or 1 TB RAM server.

• Four B200 GPUs with sufficient aggregate memory.

• Three B300 GPUs in a custom configuration.

• An eight-GPU node with considerable unused capacity.

A current maximum-memory Mac Studio offers up to 512 GB of unified memory, which would be too tight once runtime overhead and the KV cache are included.

So, running even the most compressed version of Qwen3.8 won’t be cheap.

Hopefully, Alibaba will also release smaller versions of Qwen3.8.

Hi everyone,

In this edition of The Weekly Kaitchup, we will discuss:

• Inkling and the Return of Global Competition in Open-Weight AI

• Gemma 4: Small Updates with Significant Improvements

• Bonsai Binary and Ternary Qwen3.6: Only 3.9 GB and It Still Works!

Inkling and the Return of Global Competition in Open-Weight AI

GLM, Kimi, MiniMax, and many of the largest recent open-weight language models have come from China. Outside China, NVIDIA has been one of the few companies to release a comparably large model recently, with Nemotron 3 Ultra.

Against this backdrop, the release of Inkling by Thinking Machines is a welcome development. Strong open-weight models from a wider range of organizations and countries are important for maintaining healthy global competition in frontier AI.

• thinkingmachines/Inkling

Inkling is a 975-billion-parameter multimodal model with approximately 41 billion parameters active per token. Its most interesting technical contributions are not its mixture-of-experts architecture or hybrid attention, both of which are now relatively common, but its approach to long-context modeling, optimization, and reinforcement learning.

Long-context architecture

Inkling uses learned relative positional representations instead of rotary positional embeddings. Thinking Machines reports that this approach performed better in its long-context extrapolation experiments.

The model combines these representations with an attention pattern consisting of five sliding-window attention layers for every global-attention layer. Most computation remains local, while periodic global-attention layers allow information to move across the full context. This idea of global-local layers is somewhat close to what Gemma 4 does.

Short convolutions are also applied after the attention key and value projections, as well as to the outputs of the attention and MLP branches. This introduces explicit local sequence mixing at several points within each transformer block, rather than relying exclusively on attention.

Mixture-of-experts design

Each mixture-of-experts layer contains:

• 256 routed experts

• Two shared experts (so, one more than most open-weight models)

• Six routed experts selected for each token

The outputs of the shared and routed experts are normalized together. This allows the routing mechanism to control the relative contribution of general-purpose shared computation and more specialized expert computation.

Parameter-specific optimization

Inkling uses Muon for large matrix parameters and Adam for other parameter types. Its weight decay is also adjusted alongside the learning-rate schedule instead of remaining fixed throughout training.

This represents a notably large-scale example of assigning different optimizers according to parameter structure.

Learning to reason within a compute budget

Thinking Machines reports running more than 30 million asynchronous rollouts while varying both the requested level of reasoning effort and the cost associated with generating additional tokens.

The objective is to teach the model to adapt its reasoning strategy to a given compute budget. This differs from simply imposing a maximum output length after training. Instead, the model learns to estimate when additional reasoning is likely to improve its answer and when the expected benefit is not worth the additional token cost.

Thinking Machines also randomized the available tools and their schemas during training. The goal was to prevent the model from becoming overly dependent on fixed function names, argument structures, or a single tool-calling format.

This could help the model generalize more reliably across unfamiliar tools and changing software environments.

Competitive, but not yet the leader

Inkling performs competitively with other leading open-weight models. However, on the reported benchmarks, GLM-5.2 remains slightly stronger across most evaluations despite being smaller.

They also released an NVFP4-quantized version that requires just 592 GB of memory, compared with roughly 1.9 TB for the original model.

• thinkingmachines/Inkling-NVFP4

The smallest GGUF made by Unsloth requires 270 GB + KV cache. I expect it to work very well despite the heavy compression since at that scale most of the parameters are routed experts, which are usually very robust to quantization.

• unsloth/inkling-GGUF

Gemma 4: Small Updates with Significant Improvements

The recent Gemma 4 changes are mainly runtime and interface updates rather than a new checkpoint.

One update is broader FlashAttention 4 support on NVIDIA Hopper GPUs. Google reports higher prompt-processing throughput and lower time to first token. The main benefit is during prefill, when the model processes the complete prompt before generating its first output.

Gemma 4 also now exposes several visual-token budgets per image. Developers can use smaller allocations for classification, captioning, or video frames and larger allocations for OCR, documents, handwriting, or images containing small text.

This makes visual resolution a direct inference control. More visual tokens preserve more detail but also increase prompt length, prefill time, cache use, and the amount of context consumed by each image. The setting is useful for workloads in which some images require detailed analysis while others do not.

The other relevant change concerns chat and tool-use templates. These templates convert structured messages into the special-token format expected by the model.

Recent fixes address turn boundaries, tool-response handling, structured arguments, multimodal messages, and reasoning continuity across multiple tool calls. These are implementation changes, but they can directly affect agent reliability. An incorrect marker may cause the model to treat a tool result as a user message or to restart a reasoning sequence instead of continuing it.

Google reports accuracy improvements on benchmarks using tools thanks to these updates:

Important: The Gemma 4 repositories were updated in place. As a result, reproducing earlier evaluation scores requires loading the models with the corresponding revision explicitly specified.

Bonsai Binary and Ternary Qwen3.6: Only 3.9 GB and It Still Works!

PrismML’s Bonsai applies binary and ternary weights to most of the language network of Qwen3.6-27B.

• prism-ml/bonsai-27b

The ternary version uses negative, zero, and positive weight states and occupies about 5.9GB. The binary version uses only negative and positive states and occupies about 3.9GB. Both use one half-precision scale for groups of 128 weights.

The low-bit format covers embeddings, attention projections, MLP layers, and the final language-model head. Many quantized models keep sensitive layers at higher precision, especially embeddings or output heads. Bonsai does not use those higher-precision exceptions in the language network.

The vision tower remains in four-bit precision. This suggests that the visual encoder is more sensitive to binary or ternary quantization than the language network. Vision performance still declines, likely because errors accumulate across the visual encoder, multimodal projection, and low-bit decoder.

Bonsai also requires custom CUDA and MLX kernels. A compressed weight file does not automatically produce fast inference if the runtime first expands the weights into a higher-precision matrix.

The custom kernels operate directly on the packed representation. Binary weights can be handled mainly as sign changes, while ternary weights add a zero state that can represent an absent connection. Accumulation and activations still require higher precision.

The reported phone deployment should be interpreted mainly as a weight-memory result. A 3.9GB model can fit within the memory range of a high-end mobile application, but runtime memory must also include the KV cache, activations, workspaces, the vision tower, and operating-system overhead. The full advertised context length is unlikely to be practical on a phone.

Despite the heavy compression, the model remains very capable:

The degradation compared with the original model is significant, but the accuracy is far better than all the other sub-4 GB models I have evaluated.

I’ll publish full analysis, accuracy, and token efficiency next week!

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisionning it with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

During generation, attention layers store keys and values for every previous token. As context length and concurrency increase, this cache can consume tens of gigabytes and become a major memory-bandwidth bottleneck.

KV-cache quantization reduces that cost by storing keys and values at lower precision. Moving from 16-bit to approximately 4-bit storage can shrink the cache by nearly four times, enabling longer contexts and more concurrent requests.

In practice, however, a method that works well in a research implementation may perform poorly inside an optimized engine such as vLLM or llama.cpp. Quantization can add computational overhead, require specialized kernels, reduce hardware compatibility, and interfere with features such as speculative decoding.

In this article, I compare three 4-bit KV-cache formats in vLLM, turboquant_4bit_nc, int4_per_token_head, and nvfp4, using Qwen3.6-27B. I evaluate their memory usage, accuracy, token generation, and inference speed.

Acknowledgments

Verda provided the B200s and RTX Pro 6000s to run the experiments described in this article.

Verda is the full-stack frontier AI cloud, built for high-performance inference, training, and agentic workloads with data privacy and sustainability at its core.

You can check them out here. There is a $50 coupon that you can redeem in your Verda account, after provisionning it with $5, to try their GPUs.

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

Qwen3.6-27B KV Cache Size

I’m going to compare KV-cache memory usage across different quantization methods. First, let’s establish the memory required for Qwen3.6-27B’s native 262,144-token context window.

Qwen3.6-27B has 64 language layers, but only 16 are Gated Attention layers that maintain a full KV cache. The remaining layers are Gated DeltaNet blocks, so they are not included when calculating the conventional attention KV cache.

Each Gated Attention layer has four KV heads, with a head dimension of 256.

In BF16, the KV cache for these full-attention layers consumes approximately 17.2 GB. This is substantially less than for models that use full attention in every layer, but it is still significant relative to the model weights themselves, which occupy roughly 54 GB.

On a device with 96 GB of memory, that leaves:

96 − 54 − 17.2 = 24.8 GB

This is not enough to run several full-context requests in parallel, for example, multiple sub-agents, especially once additional overhead is included. CUDA graphs, temporary buffers, runtime allocations, and other inference-engine components all consume extra memory.

Read more

Hi everyone,

In this edition of The Weekly Kaitchup, I’ll highlight some of the most interesting work I saw at ACL 2026, which I attended this week in San Diego.

I’ll also discuss OpenAI’s recent take on benchmarking with SWE-Bench Pro, one of the most widely used coding benchmark. Once again, a SWE-Bench is being declared “bad.” I can’t say I’m too upset about having never spent compute on it.

A $50 Coupon to Try Verda’s GPUs

In collaboration with Verda, I’m sharing a $50 coupon that you can redeem in your Verda account, after provisionning it with $5, to try their GPUs (B200, B300, RTX Pro 6000, …).

Coupon code: KAITCHUP-50

Follow these instructions to redeem it.

ACL 2026: Why I Went, and Why I Skipped ICML 2026

This year, two of the major annual research conferences shaping the development of AI, ACL and ICML, took place at the same time, on opposite sides of the Pacific: ACL in San Diego and ICML in Seoul.

While there is significant, and increasing, overlap between the two communities, they tend to emphasize different areas. ICML is often the place for deeper discussions about machine learning methods, algorithms, and theoretical foundations. ACL, by contrast, is where I usually find the most relevant work on evaluation, multilinguality, benchmarks, datasets, and language-centered applications.

That distinction is far from absolute. Many of the key ideas behind today’s Transformers and LLMs, as well as many of the benchmarks and evaluation tasks we still rely on, originated in papers published within the ACL community.

I chose to attend ACL this year primarily because it aligns more closely with my own research background. It is also the community where I know more people, having published several papers there during my Ph.D.

ACL 2026 was capped at 3,500 onsite attendees. For reference, ACL 2025 in Vienna had more than 5,000 in-person attendees.

The smaller scale made ACL easier to navigate, but it was still impossible to see everything I had planned. As I did for NeurIPS 2025, I will mainly focus on the papers I actually saw and discussed at the conference rather than trying to summarize the full program.

I spent most of my time in the poster sessions, which were by far the best place to have useful conversations with authors. The talks were less crowded, and some large rooms were surprisingly almost empty, especially on the last day.

To keep this edition useful and focused, I will mostly discuss papers on LLM reasoning and efficiency, two of the major themes of ACL 2026 along with reinforcement learning, as it has been at most recent AI and machine learning conferences.

The program chairs even noted that papers on some of these topics seemed to have higher acceptance rates, which suggests that they were not only popular among authors, but also particularly appealing to reviewers and area chairs.

Note: For each paper discussed below, I also included a photo of the corresponding poster. The images are high-resolution, but I do not fully trust Substack to display them cleanly (depending on your device), so apologies if some details are difficult to read. ACL may also publish the posters directly on each paper’s page, or may have already done so by the time you read this. You can check by clicking the paper links.

Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models
Hoang Phan, Xianjun Yang, Yuanshun Yao, Jingyu Zhang, Shengjie Bi, Xiaocheng Tang, Madian Khabsa, Lijuan Liu, Deren Lei
Meta; New York University; Johns Hopkins University

This paper looks at a well-known problem in reinforcement learning for reasoning models: training that improves reasoning can also make the model forget broader capabilities. I often see that when running RL for recent models like Qwen3.5/Gemma 4.

The authors show that RLVR-style post-training may improve mathematical or multimodal reasoning while degrading skills such as perception, OCR, robustness, and general instruction following.

They propose RECAP, a replay-based method that keeps general-capability data in the training loop and dynamically reweights objectives. The main contribution is a clearer diagnosis of “reasoning gain” as a trade-off problem: a model can become better at narrow reasoning benchmarks while becoming less generally useful. RECAP is presented as a way to preserve the original model’s breadth while still gaining reasoning ability.

Very simple and intuitive.

SSSD: Simply-Scalable Speculative Decoding
Michele Marzollo, Jiawei Zhuang, Niklas Roemer, Niklas Zwingenberger, Lorenz K Muller, Lukas Cavigelli
Huawei; ETH Zurich

This paper proposes SSSD, a training-free speculative decoding method for making LLM inference faster.

Instead of relying on an additional trained draft model, like EAGLE3 or DFlash, SSSD retrieves likely n-gram continuations from the prompt, previously generated tokens, and a datastore, then verifies them with the main model.

It avoids extra model training and is easier to deploy across changing workloads, domains, and languages.

The paper reports latency reductions of up to 2.9x over standard autoregressive decoding, making it relevant for production serving where simplicity and robustness matter as much as peak speed.

Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, Shafiq Joty
Salesforce AI Research; University of Wisconsin-Madison; The University of Hong Kong

Early layers of an LLM can already identify many of the tokens that will matter for answering a query. The authors turn this into GemFilter, a training-free method that uses early layers as filters to select a much smaller subset of the input before later computation.

The method is aimed at long-context inference, where latency and GPU memory grow quickly with input length. By compressing the input token set, GemFilter speeds up inference while keeping the important evidence visible to the model.

A nice aspect of the work is interpretability: because the method explicitly selects input tokens, humans can inspect what the model kept. The paper reports a 2.4x speedup and 30% GPU memory reduction compared with strong baselines, while performing well on Needle-in-a-Haystack and competitively on LongBench.

A Survey on Memory-Efficient Fine-Tuning for Large Language Models
Yeachan Kim, Mingyu Lee, SangKeun Lee
Hankuk University of Foreign Studies; Korea University

This survey organizes the landscape of memory-efficient fine-tuning for LLMs. I recommend reading it if you are lost in the recent progress made in this area.

One point I strongly agree with, although it is not very well developed in the paper, is that these methods are often poorly evaluated.

The paper mentions GLUE-style tasks. I did not double-check how widely GLUE is still used in this line of work, but even replacing it with old and relatively easy benchmarks such as QNLI, PIQA, or CommonsenseQA, like they did in this paper, would not solve the problem. People use these benchmarks because they are cheap to run.

As I often argue here, if your benchmarks do not require the model to generate millions of tokens, they are unlikely to tell you much about how a generative language model actually performs, unless of course you thouroughly study what the benchmarks has generated instead of just looking at the scores.

KV Compression

LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning
Haoyue Zhang, Hualei Zhang, Xiaosong Ma, Jie Zhang, Song Guo
The Hong Kong University of Science and Technology; The Hong Kong Polytechnic University

This paper studies KV-cache compression for long chain-of-thought reasoning. The key finding is “Token Importance Recurrence”: tokens that seem unimportant at one decoding step can become important again later. This makes simple greedy KV eviction risky, because it may remove tokens that the model will need in future reasoning steps.

LazyEviction addresses this by delaying eviction and observing attention patterns over a window before deciding what to discard. The method is designed for long reasoning sequences, where memory cost grows with every generated token, and it aims to reduce memory while avoiding sudden reasoning failures caused by prematurely evicting useful context.

This is another interesting piece of work on KV cache compression, one of many presented at the conference.

As usual, though, the real test will be implementation. Unless a method lands in frameworks such as vLLM or llama.cpp, it is hard to know whether it actually improves inference efficiency in practice, or whether it interferes with other optimizations already implemented in these systems.

Anchoring the Cache: Mitigating Contextual Hallucination in KV-Compressed Long-Context Summarization
Yu Fu, Chen Luo, Josef Valvoda, Xin Zhang, Xuejing Lei, Xiao Pan, Hui Liu, Yue Dong
University of California, Riverside; Amazon

This paper studies an important side effect of KV-cache compression in long-context summarization: it can make models hallucinate more.

The authors show that aggressive compression may preserve surface-level summarization metrics while weakening factual grounding, because retrieval heads drift away from the source context during generation.

Their method, HalluKV, anchors selected retrieval heads to the source by selectively removing generated KV pairs from those heads during decoding. The idea let the model keep the efficiency benefits of cache compression, but prevent the heads responsible for source retrieval from over-attending to the model’s own generated text.

The result is a decoding-time intervention that reduces hallucination while maintaining long-context efficiency. The paper reports that KV compression can increase hallucination scores by up to 3.36×, and that HalluKV reduces hallucination across multiple models and datasets while preserving competitive summary quality.

Quantize What Counts: More for Keys, Less for Values
Mohsen Hariri, Alan Luo, Weicong Chen, Tianyi Zhang, Qifan Wang, Xiaotian Han, Vipin Chaudhary
Case Western Reserve University; Rice University; Meta

Keys and values of the KV cache should not necessarily receive the same number of bits: key projections tend to have larger spectral and Frobenius norms than value projections, meaning keys carry higher-magnitude, more quantization-sensitive information.

The authors use this observation to justify a simple rule: allocate more precision to keys and compress values more aggressively.

The paper turns this into a geometry-driven design principle for mixed-precision KV-cache quantization. Instead of tuning key/value bit splits heuristically for each model or task, it argues that model weight geometry can predict which cache component needs more protection. Empirically, key-favored allocations such as 4-bit keys and 2-bit values preserve up to 98.3% of the accuracy of uniform 4-bit KV quantization while saving memory, making the result useful for long-context inference where KV cache memory dominates.

A very informative paper to better understand why keys are more sensitive to quantization.

More Comments on the ACL 2026

The social events at the ACL are always great. This year, the organizers reserved the USS Midway for an evening dinner. The USS Midway is an aircraft carrier turned museum, and the venue was genuinely impressive.

These social events are often among the most useful parts of a conference. They make it much easier to meet people outside your usual circle, and this one was no exception.

The next ACL will be in Kyoto, so I will probably end up playing local guide: I lived and worked there for six years.

Hopefully, the conference will not coincide with the worst of Kyoto summer: 35°C or more, 70%+ humidity, and the occasional typhoon. Otherwise, the most important survival tips may have nothing to do with AI.

SWE-Bench Is Not a Good Benchmark, Again

Unrelated to the ACL, but very interesting findings published by OpenAI this week.

OpenAI has used SWE-Bench Pro to benchmark all its models since GPT 5.2.

And yet, according to OpenAI’s recent audit, many SWE-bench Pro tasks fail for reasons unrelated to model capability.

Examples include:

• hidden requirements that never appear in the issue description

• contradictory specifications

• unit tests that reject perfectly reasonable implementations

• grading logic that expects one specific implementation instead of checking whether the issue is actually fixed

• mismatches between the GitHub issue, the merged pull request, and the evaluation tests

Because SWE-bench Pro is built automatically from real repositories, OpenAI argues that these inconsistencies are common enough to substantially affect reported scores.

OpenAI’s main conclusion is:

Approximately 30% of SWE-bench Pro tasks are broken.

Their recommendation is therefore to “carefully inspect benchmark results instead of treating leaderboard rankings as ground truth.” Yes, of course, but who actually does this? Most AI labs want to report the strongest possible numbers. If you publish a score of 80 and your competitor publishes 90, most people will not ask whether that 10-point gap reflects a real performance difference or a benchmarking artifact. They will simply see the lower score, and it will look bad.

Ironically, OpenAI itself helped create SWE-bench Verified in 2024 because it believed the original SWE-bench contained many impossible or ambiguous tasks. Now, after auditing SWE-bench Pro, it argues that even this newer benchmark still contains enough flawed tasks that raw leaderboard scores should not be over-interpreted.

My take is that these benchmarks are becoming increasingly complex: automatically generated, frequently refreshed, available in multiple versions, and often difficult to interpret consistently. As a result, I think people will develop more and more trust issues around published scores and may eventually start disregarding them altogether.

However, this is less of a problem when evaluating different versions of the same model. For example, when comparing a quantized model to the original, the goal is not necessarily to establish the model’s absolute performance. What matters is whether the quantized version performs as closely as possible to the non-quantized one.

In that setting, the original score matters less. The main objective is to minimize the accuracy delta between the two versions. Even a flawed benchmark like SWE-Bench can still be useful for this purpose, as long as it is used consistently across both model variants.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

LFM2.5 is a family of small language models. In this article, I will focus specifically on the tiny 230M and 350M versions. I ran both models on a large set of benchmarks with two goals:

1. Reproduce some of the results published by Liquid AI.

2. Better highlight what these models can and cannot do.

At first glance, the size difference between the 350M and 230M models may seem negligible: only 120M parameters. But that gap feels much larger when you realize that 120M parameters represent roughly 34% of the 350M model’s total size. On memory-constrained devices, especially when using longer context lengths, this difference can matter a lot. In 16-bit precision, it reduces weight memory consumption from about 0.71 GB for the 350M model to about 0.46 GB for the 230M model.

And given how accurate is quantization now, it could be possible to make a quantized version of the 350M that is both smaller and better than the 230M.

This raises another question that I will address in this article:

Should you use a low-bit quantized version of the 350M model, or a higher-precision version of the 230M model?

To answer that, we will compare accuracy versus model size for both models across different GGUF quantization formats released by Liquid AI and Unsloth.

LFM2.5 230M and 350M: What They Can and Cannot Do

Read more

Hi everyone,

In this edition of The Weekly Kaitchup:

• Which NVFP4 Version of Qwen3.6 27B Should You Use?

• DSpark: DeepSeek’s New Speculative Decoding Method

I’m at the ACL 2026 and stopped by the Qwen corner to ask the important question:

Should we expect Qwen3.7 27B soon?

Answer:
We haven’t made a final decision yet, but we’re committed to open-sourcing more models and have some exciting releases coming soon.

Not very informative… It sounds like a well-prepared answer. And they told me they already got this question a lot during just a single morning at the conference.

Which NVFP4 Version of Qwen3.6 27B Should You Use?

NVIDIA released an NVFP4 version of Qwen3.6 27B. It is a good occasion to revisit the other NVFP4 variants that have been released and to put NVIDIA’s checkpoint into context.

When I published my evaluations and analysis of quantized Qwen3.6 27B, only a few NVFP4 versions were available. There were so few that I had to make one myself with AutoRound.

kaitchup/Qwen3.6-27B-autoround-nvfp4-linearattn-mtp-BF16
This is my version made with AutoRound. It uses NVFP4 while keeping the linear-attention and MTP layers in 16-bit. The model size is 28.6 GB.

Peutlefaire/Qwen3.6-27B-NVFP4
This version goes further and quantizes more linear_attn layers too. The model size is 20.6 GB, with a 19.7 GB main model file and an 849 MB MTP file. Compared with my AutoRound MTP BF16 version, that is roughly 8.0 GB smaller.

In my analysis, I showed that quantizing the attention path to NVFP4 significantly degraded the model. It did not break the model, but it made it more prone to endless thinking, and accuracy went down on nearly all benchmarks. Check the full analysis for details.

unsloth/Qwen3.6-27B-NVFP4
Unsloth also released an NVFP4 checkpoint. Its config applies NVFP4 quantization to most Linear layers, while leaving some of the linear-attention projections (linear_attn.out_proj) in BF16. You can see it as a version between mine and a full NVFP4 quantization. The model size is 26.4 GB.

nvidia/Qwen3.6-27B-NVFP4
The NVIDIA version is interesting because it is mixed precision: attention and linear-attention projections are FP8 W8A8, while the MLP layers and LM head are marked as W4A16_NVFP4. The checkpoint is 21.9 GB, which makes it 6.7 GB smaller than my 28.6 GB AutoRound MTP BF16 version and 4.5 GB smaller than Unsloth’s 26.4 GB version.

This also means that the NVFP4 parts of NVIDIA’s model do not use NVFP4 activations. They are W4A16: 4-bit NVFP4 weights, but 16-bit activations. The attention path, meanwhile, is FP8 rather than NVFP4. With Blackwell GPUs, this likely leaves speed on the table compared with W4A4 NVFP4. In my experience, NVFP4 activations can be almost harmless with good calibration.

Another important detail: NVIDIA’s config.json includes an FP8 KV-cache scheme. That matters a lot for speed and memory use, especially on devices where memory bandwidth is the bottleneck. I already saw some third-party speed comparisons on X comparing this checkpoint against other NVFP4 models without matching the KV-cache format, which is not an apples-to-apples comparison. At minimum, KV-cache dtype should be controlled explicitly across all models; vLLM, for example, can be run with an explicit FP8 KV cache.

NVIDIA also reports its own evaluation numbers against an FP8 baseline. In their table, the NVFP4 checkpoint is very close to the FP8 checkpoint across MMLU-Pro, GPQA Diamond, HLE, τ²-Bench, MMMU Pro, SciCode, AIME 2025, AA-LCR, and IFBench. This is useful, but it is still not a community-wide comparison against the other NVFP4 checkpoints under identical settings.

It is also worth mentioning the PrismaQuant family.

rdtand/Qwen3.6-27B-PrismaQuant-5.5bit-vllm
This version uses a sensitivity-driven per-Linear allocation. PrismaQuant chooses between NVFP4, MXFP8, and BF16 under a 5.5-bit-per-parameter target. According to the model card, bulk dense MLP layers are usually NVFP4 W4/A4, higher-sensitivity dense linears can be MXFP8 W8/A8, and the most sensitive tensors, norms, biases, embeddings, and LM head stay BF16.

The model size is 22.7 GB, which slighly larger to the one released by NVIDIA, probably due to the LM head staying at higher precision.

There are newer PrismaQuant-family variants too. PrismaAURA 5.5-bit uses an AURA/KL-Fisher allocation over NVFP4, FP8, and BF16, while PrismaSCOUT 5.31-bit is described as a Blackwell-oriented successor to the earlier 5.5-bit PrismaQuant artifact. PrismaSCOUT uses NVFP4 plus selected BF16, includes MTP tensors, and its repository is 20.2 GB.

How to Choose?

We do not yet have a good comparison. Quantization is cheap; evaluation is expensive.

What we need is a benchmark suite run on all these checkpoints with exactly the same settings: same inference engine, same chat template, same thinking mode, same KV-cache dtype, same context length, same sampling settings, same MTP/speculative-decoding settings, and same hardware.

Until then, I would assume the following:

• There is no strong reason to keep all activations in 16-bit on Blackwell if NVFP4 activation quantization is well calibrated.

• Keeping the attention path out of NVFP4 is safer.

• Unsloth’s version and my AutoRound version are safer choices if your priority is a more conservative quantization strategy.

• PrismaSCOUT and the other PrismaQuant-family models are worth testing, but I would not rank them without independent, apples-to-apples accuracy numbers. This could be the best version. We simply don’t know.

So, for now, I would use either Unsloth’s version or my AutoRound MTP BF16 version if accuracy safety is the main concern. They are also better documented and easier to reason about than many one-off community checkpoints. If memory footprint matters more, NVIDIA’s version and PrismaSCOUT are especially interesting.

DSpark: DeepSeek’s New Speculative Decoding Method

DSpark is a speculative decoding method for making LLMs generate faster without changing the final model that verifies the text.

Instead of asking the full model to produce one token at a time, DSpark, like any other speculative decoding method, lets a smaller draft module propose several future tokens, then asks the full model to verify those tokens in a batch. When the draft is right, the server accepts multiple tokens from one target-model pass. When the draft is wrong, the server keeps the longest valid prefix and continues from there.

The target model we want to accelerate remains the source of truth, while the drafter tries to guess what the target model is likely to accept next.

What DSpark Changes

The weakness of many speculative decoders is suffix decay. Drafting the first token is easy but drafting the fifth, sixth, or seventh token is much harder because every position depends on earlier guesses. Fully parallel drafters are fast, but later draft positions often become noisy. Autoregressive drafters are more faithful, but they give back too much of the latency advantage because they draft sequentially.

DSpark tries to sit between those two extremes. It keeps a DFlash-style parallel draft backbone, then adds a lightweight Markov logit-bias head. Note: I explained how DFlash works here:

That head lets each draft position condition on previously sampled tokens inside the draft block, which improves the later positions without turning the whole drafter into a slow autoregressive model.

The other important piece is confidence. DSpark adds a confidence head that predicts the acceptance probability of each draft position. That gives the serving system a practical scheduling signal. When the drafter is confident, the runtime can verify more proposed tokens. When confidence drops, or when the server is under load, it can avoid wasting verification work on tail tokens that are likely to be rejected. In other words, DSpark is not just a better drafter; it is a more serving-aware drafter.

How it works in the decoding loop

At each generation step, DSpark first prepares a block of draft tokens. The target model then scores that block in one verification pass. The runtime accepts the longest prefix that agrees with the target model’s sampling path and discards the rest. If several drafted tokens survive, the user sees several tokens produced for roughly one target-model step. If only the first token survives, the system behaves closer to normal decoding for that step.

The Markov head just helps the drafter keep later draft positions coherent. The confidence head helps the runtime decide how much speculation is worth attempting. Together, they attack the two practical bottlenecks of speculative decoding: low acceptance at later positions and wasted verification work under real serving load.

Speed results

DSpark improves per-user generation speed by about 60–85% over the previous MTP-1 baseline on DeepSeek-V4-Flash, and by about 57–78% on DeepSeek-V4-Pro at matched throughput. Offline accepted-length results also reportedly improve over Eagle3 and DFlash. As usual with speculative decoding, the exact gain depends on the prompt mix, sampling settings, hardware, batch pressure, and how often the drafter’s later tokens are accepted. Once widely available for other open-weight models, like Qwen3.6 and the larger Gemma 4, it will be interesting to see how fast is it compared with high MTP, like MTP-4 or 5 which are much faster than MTP-1 for these two models.

You can find more results in the technical report:

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

What DeepSeek released

The DeepSpec repository currently includes three draft-model algorithms: DSpark, DFlash, and Eagle3.

DeepSeek also released trained draft checkpoints for four target models: Qwen3-4B, Qwen3-8B, Qwen3-14B, and Gemma-4-12B-it. For each target, the repo lists Eagle3, DFlash, and DSpark checkpoints, so the open release covers twelve reference draft checkpoints.

DSpark checkpoints are also available for DeepSeek V4 Flash and Pro.

Running DSpark in vLLM

DSpark support is landing in vLLM through the speculative decoding path. The relevant vLLM PR for DSpark speculators-format checkpoint support was merged on July 2, 2026, so use a nightly build or a release that includes that change.

uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly

vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \
  --tensor-parallel-size 8 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}'

Training your own DSpark speculator

You can train a DSpark speculator on your own data for Qwen3 and GLM 5.2 (only these architectures are support for now) using the speculators project. This is very similar to training a DFlash a speculator.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

MiniMax M3 is an excellent model, but with roughly 428B parameters, running it locally is very challenging. In BF16, the weights alone require around 852 GB of memory, so an unquantized setup realistically needs a large multi-GPU server, for example an 8×H200 machine.

Quantization can dramatically reduce these requirements. However, as we saw in previous evaluations, MiniMax M2.5 degraded heavily once quantized, even at 4-bit. The natural question is whether M3 is more robust.

As we will see in this article, the answer is yes: M3 is much easier to quantize. My hypothesis is that this improved quantization robustness comes from M3’s shared-expert MoE design, a feature that was absent from M2.5. The shared expert provides a path that can be preserved during quantization at a high precision, while the routed experts, which account for about 97% of M3’s parameters, can be compressed much more aggressively.

In this article, I evaluate several low-bit MiniMax M3 GGUFs, including Unsloth’s UD GGUFs and my own MoQ quantization.

• kaitchup/MiniMax-M3-GGUF-MoQ

• unsloth/MiniMax-M3-GGUF

The main result is that M3 can be compressed from 852 GB to around 150 GB while preserving most of its accuracy. I then analyze the tensor-level quantization choices behind each GGUF to explain why some variants remain strong, and why the smallest ones start to break.

Acknowledgments

This article would not have been possible without the compute sponsorship generously provided by Verda, whose B300 GPUs I used throughout this work.

Verda is a European, AI-focused cloud and GPU infrastructure provider with sovereignty, sustainability, data privacy, and performance at its core.

You can check them out here.

Results: Which GGUF Can You Safely Use?

First, a note on the benchmarks I ran. For my GGUF evaluations, I usually use large subsets of MMLU Pro for world knowledge, Math 500 for math questions ranging from easy to difficult, LiveCodeBench for challenging coding tasks, and the full GPQA Diamond benchmark for difficult science questions.

However, running M3 with llama.cpp is very costly, especially when evaluating low-bit versions that can generate many more tokens than the original model to answer the same questions. Running only 100 LiveCodeBench samples was already too expensive for this evaluation, at least $200 for a single GGUF evaluaiton, so I did not include this benchmark.

How should you read the following results?

Read more

Hi everyone,

In this edition of The Weekly Kaitchup, let’s discuss some of the open models released this week:

• LFM2.5 230M: Tiny Models Can Follow Instructions

• Ornith-1.0: Self-Scaffolding Coding Models

• GLM-5.2 REAP: Cutting a 753B MoE Down to a More Deployable 504B

I’m also currently evaluating the M3 GGUFs I created with MoQ. This has taken more time than expected, as some of them generate more tokens than anticipated. The evaluation is almost finished, though, and you can expect the full analysis on Monday or Tuesday.

The models are available here:

• kaitchup/MiniMax-M3-GGUF-MoQ

• I already confirmed they are all good, except for the MoQ-2.5 (not broken but much less accurate than the others).

Next week, I’ll be at the ACL 2026, one of the main AI conferences, in San Diego. Let me know if you’ll be there and would like to meet!

As I did for NeurIPS 2025, I’ll publish a report on the most interesting papers, presentations, and demos I come across.

LFM2.5 230M: Tiny Models Can Follow Instructions

Was LFM2.5-350M too large? Liquid AI just released LFM2.5-230M, and for a 230M-parameter model, it performs surprisingly well.

Two numbers that stand out are IFBench and IFEval. LFM2.5-230M is in the same ballpark as models far larger, especially regarding the IFeval score, which looks very close to what we would get with Llama 3.1 8B. That is roughly a 35× parameter gap.

Note on GPQA Diamond

GPQA Diamond is probably too difficult for models this small. It is a 4-choice multiple-choice benchmark, so a model that simply picks A, B, C, or D at random should get 25% accuracy. When results hover around that baseline, they mostly tell us that the model is not really solving the task. Scores below 25% can also happen in generative evaluations where the model is asked to output exactly one of A, B, C, or D. If it generates anything else, the answer is marked wrong. So at this scale, the GPQA number can look meaningful in a table while mostly reflecting answer-format compliance plus noise rather than scientific reasoning.

As for its architecture, LFM2.5-230M is not just a uniformly scaled-down 350M. Both models keep the same 1024 hidden size, 16 attention heads, 8 KV heads, and 65,536-token vocabulary.

The savings come from the blocks. LFM2.5-230M uses 14 layers instead of 16: 8 double-gated LIV convolution blocks + 6 GQA blocks, versus 10 + 6 for LFM2.5-350M. The bigger difference is the FFN/intermediate dimension: 2560 for the 230M model, versus 6656 for the 350M model. That is where most of the parameter reduction comes from.

In memory terms, the 230M model only consumes 459 MB and is 250 MB lighter than the 350M in bf16.

So now we have a genuinely interesting question: should we use the bf16 230M, or a quantized version of the 350M?

I think that a carefully quantized 350M could win on both quality and memory. But at this scale, quantization is less forgiving: 4-bit quantization can easily erase the quality advantage. I’ll try to find time to run the bf16 and quantized variants of these tiny LFM2.5 models and plot accuracy vs memory curves.

Ornith-1.0: Self-Scaffolding Coding Models

Ornith-1.0 is a family of open-source language models built specifically for agentic coding: settings where a model must not only write code, but operate inside a tool-using loop, inspect repositories, run commands, interpret failures, and iteratively repair its own solution.

• Models: The Ornith-1.0

• Blog Post: Ornith-1.0: Self-Scaffolding LLMs for Agentic Coding

The family spans several deployment and capability tiers, including compact dense models and larger mixture-of-experts models. The announced lineup includes 9B Dense, 31B Dense (not released yet), 35B MoE, and 397B MoE variants, with released Hugging Face artifacts including FP8, and GGUF builds.

The models are post-trained on top of Gemma 4 and Qwen 3.5, then specialized for coding-agent behavior.

The central training idea is “self-scaffolding.”

In conventional reinforcement-learning setups for coding agents, the scaffold, i.e., the harness, orchestration logic, memory strategy, error-handling routine, or prompting structure that guides the rollout, is usually designed by humans and held fixed. Ornith-1.0 instead treats that scaffold as something the model can improve. During RL, the model first proposes or refines a task-specific scaffold, then uses that scaffold to generate a solution rollout. The reward from the rollout is assigned back to both the scaffold-generation step and the solution-generation step, so the model is trained not only to produce better code, but also to discover better ways of organizing its own problem-solving process.

This creates a feedback loop: scaffolds that lead to higher-reward coding trajectories are reinforced, while weaker scaffolds are discarded. Over many training iterations, the model learns task-category-specific strategies for agentic coding without depending entirely on hand-engineered harnesses. DeepReinforce frames this as the key distinction of Ornith-1.0: reinforcement learning is used not just to improve answers, but to improve the model’s internal orchestration of tool use and search.

The training setup also addresses an obvious risk of self-improving agents: reward hacking. If a model can shape the scaffold that drives its own rollout, it may learn shortcuts that satisfy a verifier without solving the task, such as reading hidden test artifacts, modifying verification scripts, or hard-coding expected outputs. They have implemented three guardrails: keeping the environment and tool boundary outside the model’s control, using deterministic monitors to block forbidden actions, and adding a frozen LLM judge as a veto layer on top of the verifier for cases where intent-level gaming is harder to specify mechanically.

The models look very strong on the benchmarks, and early community feedback appears to be positive.

One particularly interesting detail is that the only checkpoint based on Gemma 4 is missing from both the release and the evaluation results. Qwen3.5 is known to be strong at agentic coding, whereas this is an area where Gemma 4 underperforms. That makes the idea of an Ornith-1.0 model based on Gemma 4 especially exciting as it could correct one of its main weaknesses.

At the same time, its absence leaves room for interpretation. Was Ornith-1.0 31B performing significantly below the 35B model based on Qwen3.5? Was it much harder to train or align? Or is the release simply delayed?

GLM-5.2 REAP: Cutting a 753B MoE Down to a More Deployable 504B

0xSero’s GLM-5.2 REAP project is a practical attempt to make Z.ai’s massive GLM-5.2 mixture-of-experts model easier to serve.

• 0xSero/GLM-5.2-504B

Instead of quantizing the whole model more aggressively or merging experts together, REAP removes low-saliency routed experts from each MoE layer. The released 504B version keeps 168 of the original 256 routed experts per layer. In other words, it prunes 88 experts per layer, or about 34.4% of the routed expert pool.

The released checkpoint above is an NVFP4 version.

After pruning, Router-KD, or router-only knowledge distillation, was applied to recover behavior. The experts, attention layers, embeddings, and most of the network are frozen. Only the router gate matrices are trained, representing around 0.016% of the model’s parameters. The goal is to re-teach the router how to route tokens through the surviving 168 experts so that the pruned model better matches the unpruned teacher’s next-token distribution. This makes the recovery step far cheaper than full fine-tuning.

While it’s very effective at reducing the model size, as we saw last week, REAP has significant drawbacks:

• it significantly degrades the model’s world knowledge, or more generally, its accuracy on tasks not used to evaluate the expert activations

• and the resulting model is often generating more tokens which makes it less cost-effective at inference.

The published evaluation results are also limited and, in some cases, do not include the original model’s results as a baseline. As a result, it is hard to tell how close this model comes to the original. What we do know is that it scores around 70 on Terminal-Bench 2.1, which is very encouraging.

Running more careful evaluations for large models remains prohibitively expensive for the freelance AI community; for example for GLM 5.2 and some pruned/quantized variants, it would cost $20k or more to run the same type of evaluation and analysis I ran for Qwen3.6.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

Hi everyone,

In this edition of The Weekly Kaitchup, let’s discuss:

• GLM-5.2: Only a Few Months Behind Commercial Models

• VibeThinker-3B: How Far Can Verifiable Training Push an Old Small Model?

• MoQ GGUF Updates

GLM-5.2: Only a Few Months Behind Commercial Models

Many people are calling GLM-5.2 the best model for local AI. I would not go that far.

My definition of local AI is stricter: it should be something a serious hobbyist, researcher, or small company can run on a local machine with consumer GPUs, CPU RAM, and a realistic budget. By that standard, GLM-5.2 is not really local.

Unsloth’s Q1 GGUFs are about 217–228 GB before the KV cache. The Q2 versions are around 238–254 GB. That means you are already outside the comfort zone of normal workstations before you even start thinking about context length.

Then, we have to add the KV cache’s memory consumption. With GLM-5.2’s MLA-style cache:

78 layers × tokens × (512 + 64) × bytes_per_value

So, with BF16 KV cache, that is roughly:

• 9 GB per 100k tokens

• 18 GB per 200k tokens

• 45 GB per 500k tokens

• 90 GB for the full 1M-token context

With FP8/INT8 KV cache, divide those numbers by about two. Real deployments may need more because of runtime overhead, batching, padding, sparse-attention grouping, and framework-specific memory management.

So yes, you may be able to load GLM-5.2 on a machine with huge system RAM, or on a 256 GB unified-memory Mac. But it won’t be the original model, and it’ll be very slow unless most of the hot path is on serious GPU memory and bandwidth.

Nonetheless, GLM 5.2 is exciting because it gives individuals, labs, and companies access to a frontier-class model without depending entirely on a closed API.

And the quality is surprisingly close.

For coding and agentic work, GLM-5.2 is near the commercial frontier on several benchmarks. On SWE-Bench Pro, it scores above GPT-5.5 and Gemini 3.1 Pro, though still behind Claude Opus 4.8. On Terminal-Bench 2.1, it is in the same range as GPT-5.5 and Claude Opus 4.8. On MCP-Atlas, it is only about one point behind Opus 4.8 and slightly ahead of GPT-5.5.

That is what I mean by “only a few months behind commercial models.” Not that GLM-5.2 beats the best closed models everywhere. It does not. Claude Opus 4.8 still has a large lead on some hard software-engineering benchmarks, and the closed models are usually easier to serve, faster to use, and more polished as products.

But the gap is now small enough to be operationally interesting. A model with open weights, MIT licensing, 1M-token context, strong coding scores, and private deployment is no longer a toy alternative to commercial AI.

VibeThinker-3B: How Far Can Verifiable Training Push a Small Reasoning Model?

VibeThinker-3B is a 3-billion-parameter dense reasoning model released by WeiboAI. It is designed for tasks where the answer can be checked with a relatively clear signal: mathematics, competitive programming, STEM reasoning, and instruction-following tasks with explicit constraints.

• WeiboAI/VibeThinker-3B

The model builds on the earlier VibeThinker-1.5B release, a smaller model designed to test whether highly compact language models could acquire useful reasoning abilities through targeted post-training. At the time of its release, VibeThinker-1.5B drew sustained attention in the AI community for several weeks, largely because its benchmark scores were unexpectedly strong given its small size.

VibeThinker-3B continues that line of work at a larger scale, but it is still very small compared with the large reasoning models usually associated with strong performance on competition math and code.

The main point of VibeThinker-3B is not that small models can replace larger general-purpose models. It is more specific than that. The model is an example of how far a small model can be pushed when the training tasks have reliable feedback.

The model is strongest where correctness is measurable and the model card explicitly warns that it was not trained for tool-calling or agent-style programming workflows. For code, it is better understood as a competitive-programming model than as a general software-engineering assistant.

They published a technical report here:

VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models

VibeThinker-3B focuses on multi-step reasoning, answer verification, constraint satisfaction, and self-correction.

The authors describe this through a distinction between reasoning and knowledge. Their argument is that some reasoning procedures may be highly compressible into a relatively small model, provided the task space is structured and the feedback is reliable. In contrast, broad world knowledge and long-tail factual coverage still benefit from much larger parameter counts.

Architecture

VibeThinker-3B is based on Qwen2.5-Coder-3B, which is almost 2 years old!

This is a Qwen2-style decoder-only Transformer.

The Qwen2.5-Coder series is trained for code generation, code reasoning, and code repair, while retaining some general and mathematical ability. Starting from a code-oriented base gives VibeThinker-3B a foundation that is already useful for executable reasoning and competitive programming.

How it was trained

The training pipeline is built around what WeiboAI calls the Spectrum-to-Signal Principle. The idea is to separate two phases that are often mixed together.

The “spectrum” part means exposing the model to a wide range of possible solution paths. Instead of training only on one canonical solution, the model is trained on multiple valid reasoning traces. This is meant to help it explore different ways to decompose a problem.

The “signal” part means using reliable feedback to reinforce the useful paths. In math, this can mean checking the final answer. In code, it can mean running the generated program against tests. In constrained instruction-following, it can mean checking whether the output satisfies explicit requirements.

The first major stage is supervised fine-tuning. VibeThinker-3B uses a two-stage curriculum. The first stage covers a broad mixture of math, code, STEM reasoning, general dialogue, and instruction following. This gives the model a broad behavioral starting point. The second stage shifts toward harder and longer reasoning samples, so the model spends more training effort on long-horizon problem solving.

The report describes query expansion, teacher-generated reasoning traces, answer verification, code execution, majority voting, and filtering of invalid traces. For reasoning-heavy samples, the model is trained with multi-path reasoning distillation. That means strong teacher models generate multiple candidate reasoning paths, and the training process keeps useful verified traces rather than collapsing everything into a single fixed style.

After supervised fine-tuning, the model goes through multi-domain reinforcement learning. This stage uses MaxEnt-Guided Policy Optimization, the reinforcement-learning method introduced in the earlier VibeThinker-1.5B work. In simple terms, the method focuses training on prompts that are near the model’s current capability boundary. These are problems where some sampled solutions are correct and others are wrong. Such prompts provide a more useful learning signal than problems the model always solves or always fails.

The reinforcement-learning stage is applied sequentially across math, code, and STEM reasoning. Math RL is used to strengthen symbolic derivation and long multi-step reasoning. Code RL focuses on executable logic, boundary cases, and program constraints. STEM RL is then used to broaden the reasoning behavior into scientific tasks.

A notable change from the earlier model is the long-context reinforcement-learning setup. The authors report that aggressive early truncation weakened long-thinking behavior, so VibeThinker-3B instead uses a single 64K long-context window during RL. The goal is to avoid cutting off complete reasoning trajectories.

The training also includes a Long2Short math RL stage, meant to reduce redundant reasoning among already-correct outputs. In practice, the model is still encouraged to solve the problem, but shorter correct trajectories are preferred over longer correct trajectories when both are valid.

After the main reinforcement-learning stages, the model goes through offline self-distillation. High-quality traces from math, code, and STEM checkpoints are verified, filtered, and distilled back into a unified student model. The goal is to consolidate the specialized abilities learned during separate RL stages.

The final stage is instruction-oriented reinforcement learning. This is meant to make the reasoning-enhanced model more reliable for user-facing prompts. For explicit constraints, rewards can be computed by rule-based validators that check format, ordering, item count, keyword constraints, and task completion. For open-ended instructions, rubric-based reward models evaluate qualities such as helpfulness, coherence, instruction adherence, and redundancy.

Results

The reported results are strong for a 3B model, especially on verifiable math and code benchmarks.

These numbers should be read carefully. They are concentrated in domains where the model’s training objective is well matched to the evaluation. VibeThinker-3B looks especially strong on competition-style math and executable coding, where correctness can be checked.

The report notes a clearer gap on knowledge-heavy evaluation such as GPQA-Diamond. This is consistent with the model’s own framing: compact models may be able to learn dense reasoning procedures, but broad factual knowledge and general-purpose competence still benefit from larger parameter budgets.

MoQ Updates

This week, I had to dedicate all my computing resources to evaluating GGUF versions of M3 and producing MoQ quantizations.

I am using my own money to evaluate the GGUFs produced by Unsloth.

For the MoQ quantization itself, and for evaluating the resulting GGUFs, I am supported by Verda’s compute sponsorship. I am using two of their B300s for this work, totaling 560 GB of GPU VRAM, backed by a generous amount of CPU RAM. This setup lets me load the full M3 model, and GGUF quantization remains very fast on a hybrid-memory system like this.

The quantization should be finished this weekend, and the evaluation should be fast afterward. I’ll publish the models and a full article detailing the results once this is done.

At the beginning of the week, I also released MoQ GGUFs for Qwen3.6 27B.

• kaitchup/Qwen3.6-27B-GGUF-MoQ

I also have GGUFs for Qwen3.6 35B-A3B, but I delayed their evaluation so I could focus on M3. I will evaluate and publish them once my M3 work is complete.

Waleed Ahmad is the author of the MoQ method and he is still working on optimizing it.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

On one side, Qwen3.6-35B-A3B is a sparse Mixture-of-Experts model with 35B total parameters, but only around 3B active parameters per token. On the other, Qwen3.6-27B is a dense model: smaller in total parameter count, but much heavier for each generated token.

In practice, the two models target slightly different users. The 35B-A3B is attractive if you want faster inference. The 27B is simpler: no router, no experts, and no MoE-specific surprises, but slower and more accurate.

Shortly after Qwen3.6 appeared, the community started releasing custom variants, including Qwopus3.6-35B-A3B, Qwopus3.6-27B, and Qwen3.6-28B-REAP.

They are all based on Qwen3.6, but they are not trying to solve the same problem. Qwopus focuses on reasoning style, answer structure, and distillation from stronger models. REAP focuses on reducing the size of the MoE by pruning experts.

This raises a practical question: should you use the original Qwen3.6 model, one of the Qwopus fine-tunes, or the smaller REAP version?

In this article, I compare these Qwen3.6 variants across several dimensions: how they were made, how much memory Qwen3.6-28B-REAP saves compared with the original Qwen3.6-35B-A3B, how token-efficient they are, and how accurate they are across different tasks and domains.

As we will see, customizing advanced models like Qwen3.6 is difficult. Accuracy alone is not enough. To properly adopt or validate a model, we also need to look at efficiency, failure modes, and behavior at scale.

The Custom Qwen3.6 Models Compared

• Jackrong/Qwopus3.6-27B-v2

• Jackrong/Qwopus3.6-35B-A3B-v1

• 0xSero/Qwen3.6-28B

Acknowledgements

Before turning to the analysis, which is largely negative for all these models, I would like to thank 0xSero and Jackrong for releasing them and for investing their time, and money, into exploring new recipes to make local AI better and more affordable. These models are extremely valuable for research purposes.

Read more

Hi everyone,

In this edition of The Weekly Kaitchup, let’s discuss:

• DiffusionGemma: 4x Faster than Gemma 4

• MoQ GGUFs: Gemma 4 12B IT and LFM2.5 8B A1B

30% Off the Subscription to The Kaitchup:

Get the Coupon and Subscribe

DiffusionGemma: 4x Faster than Gemma 4

DiffusionGemma is Google DeepMind’s experimental text-diffusion variant of Gemma 4:

• google/diffusiongemma-26B-A4B-it

It is based on the Gemma 4 26B A4B Mixture-of-Experts family: roughly 25.2B total parameters, 3.8B active parameters, 30 layers, 128 experts with 8 active experts, 256K context, and a 256-token diffusion canvas.

Unlike a normal autoregressive LLM, it does not generate one token at a time. It generates text in blocks by repeatedly denoising a whole canvas of tokens.

Architecturally, it is a block-autoregressive discrete diffusion model. The prompt is processed causally into a KV cache, then the model works on a 256-token canvas with more bidirectional interaction inside that canvas. After several denoising iterations, the finalized block is appended to the cache and the model moves to the next block.

Generation starts from noisy/random token guesses on the canvas. At each denoising step, the model:

• predicts tokens for all positions

• estimates uncertainty

• keeps the most confident positions

• re-noises uncertain ones

• repeats

Google’s public sampler uses Entropy-Bounded Denoising with Adaptive Stopping: up to 48 denoising steps, temperature decaying from 0.8 to 0.4, entropy-based token selection, and early stopping once the canvas is stable enough. This lets the model revise earlier positions in light of later positions, which is useful for editing, infilling, code, structured text, and constraint-heavy tasks.

Why diffusion? Because it’s much faster than a standard autoregressive LLM, even when accelerated with speculative decoding.

Its speed comes from replacing many tiny sequential token steps with larger parallel GPU workloads. Autoregressive decoding is often memory-bandwidth-bound at low batch size because each token needs another forward pass over the model. DiffusionGemma instead refines 256 positions at once, making better use of GPU parallelism.

Google reports up to 4x faster generation, with figures such as 700+ tokens/s on RTX 5090, 1000+ tokens/s on H100, and over 1100 tokens/s on H100 FP8 in low-batch settings. Note: The baseline is Gemma 4 26B A4B with MTP, not with the DFlash speculator made by z-lab which could be faster.

The advantage is strongest for local or single-user inference, and weaker in high-throughput server settings where autoregressive models can batch many users efficiently.

However, in terms of accuracy, DiffusionGemma significantly underperforms the original 26B model:

reference: https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/

MoQ GGUFs: Gemma 4 12B IT and LFM2.5 8B A1B

I’ve continued creating and evaluating new MoQ GGUFs in collaboration with Waleed Ahmad, the author of the method.

This week, I released Gemma 4 12B MoQ GGUFs:

• kaitchup/gemma-4-12b-it-GGUF-MoQ

They are, to the best of my knowledge, the strongest sub-7 GB GGUFs of Gemma 4 12B currently available.

Waleed has also been working on LFM2.5 8B A1B, and my evaluation results show that these MoQ GGUFs outperform other GGUFs as well, including those made with APEX.

For me, this is enough empirical evidence to start investing more seriously in the method. I’m now working on MoQ versions of Qwen3.6, the larger Gemma 4 models, and M3. This will take some time, as the method is costly to run. MoQ GGUFs for Qwen3.6 27B should be released this weekend.

Once we gather enough results, the code will be open-source and we will publish a technical report.

That’s all for this week.

If you like reading The Kaitchup, consider sharing it with friends and coworkers (there is a 20% discount for group subscriptions):

Subscribe now

Share The Kaitchup – AI on a Budget

Have a nice weekend!

GGUF is especially popular because it supports many quantization levels. Instead of using a full BF16 or FP16 checkpoint, users can pick a smaller 2-bit to 8-bit model that better fits their RAM, VRAM, and quality requirements.

The most interesting GGUFs today increasingly rely on mixed-precision quantization. Unsloth Dynamic, or UD, helped popularize this approach by using lower precision for less sensitive tensors and higher precision for important ones.

AutoRound’s AutoScheme one of the best practical alternatives for creating your own mixed-precision GGUFs today. Given a target average bit-width and a list of candidate schemes, AutoScheme automatically chooses between GGUF quantization types, for example GGUF:Q2_K_S and GGUF:Q4_K_S, layer by layer.

This article is therefore a step toward learning how to build, control, and evaluate mixed-precision GGUF recipes. As most open-weight models already exists in some high-quality GGUF version, making your own GGUF is especially relevant if you have a fine-tuned version of a model and want to GGUF it.

I also expect MoQ-like strategies to eventually be adapted and supported by tools such as AutoRound, which could further improve the kind of recipe built in this article.

In this article, you will learn how to:

• create your own GGUF model with AutoRound AutoScheme;

• control the target average bit-width and candidate GGUF quantization types;

• protect important layers;

• compare the quality-size trade-off.

Here is a notebook you can use to make your own optimized GGUFs for Qwen3.5/3.6 models:

Get the Notebook (#200)

Read more