Apriel-H1: Distill Reasoning, Not Pretraining Data

Apriel-H1: Distill Reasoning, Not Pretraining Data

There has been a lot of back and forth lately about whether efficient attention architectures are actually practical at scale.

MiniMax's M2 post-mortem got plenty of attention after the team explained why it eventually moved back to full attention at 230B parameters. The easy takeaway was that efficient attention had failed.

Then Kimi Linear came along and made that conclusion look a little too simple.

The more useful lesson is that architecture choices depend heavily on what you're trying to optimize.

ServiceNow had a different problem.

They already had a reasonably strong 15B reasoning model, Apriel-Nemotron-15B-Thinker, and wanted to reduce the cost of serving it. Starting another large-scale pretraining run wasn't exactly an attractive option.

So instead of training a new model from scratch, they asked a more practical question:

Can we take an existing reasoning model and replace some of its attention layers with Mamba, then recover the lost capability through distillation?

Their answer is Apriel-H1.

And the interesting part isn't just that the hybrid model gets faster. It's what they discovered about the data used for distillation.

The basic idea

Apriel-H1 isn't one model. ServiceNow produced seven checkpoints with different numbers of Mamba layers.

The starting point is a 50-layer attention-based model. Depending on the checkpoint, between 25 and 40 of those layers are replaced with Mamba state-space mixers.

That gives them a fairly useful efficiency/quality curve instead of betting everything on one architecture.

The H-30 model is the main example. After distillation and supervised fine-tuning, it gets roughly 1.9–2.1x the throughput of the original model, depending on context length.

The total training budget for that model was 76.8B tokens:

• 55.9B tokens for distillation
• 20.9B tokens for supervised fine-tuning

The benchmark results are also interesting because the model doesn't simply lose accuracy across the board.

Some results improve. Others drop slightly.

That's basically the trade-off ServiceNow is making: give up a small amount of capability in some areas to get a substantial improvement in serving throughput.

For production workloads, that's often a much more interesting trade than squeezing out another fraction of a benchmark point. The same production-throughput obsession drives ultrafast inference tiers — different lever, same budget.

The part I find most interesting: what should you distill?

This is where Apriel-H1 gets more interesting than a simple "attention vs Mamba" comparison.

The obvious assumption would be to use pretraining data.

After all, you're changing the internal architecture of the model. Mamba layers are being introduced into a network that was originally built around attention. Surely the new layers need to see a broad distribution of data so they can learn general token mixing behaviour. Tokens are also where the money goes — markup choice alone can nearly double token cost.

That sounds reasonable.

It also wasn't the approach that worked best.

The team found that distilling on general pretraining data caused the hybrid models to lose reasoning ability.

Instead, the better results came from using high-quality reasoning traces from the teacher's SFT data.

That's a pretty important distinction.

The objective wasn't really to teach the new Mamba layers everything the original model knew. The goal was to preserve the behaviour that mattered most: the ability to work through a problem and arrive at the same answer.

Think about a long mathematical problem.

The useful signal isn't just the individual tokens. There is a chain:

premise → intermediate step → another inference → conclusion

The student needs to learn enough of that structure to reproduce the teacher's behaviour.

A large pretraining corpus contains plenty of useful information, but it also contains an enormous amount of noise from the perspective of this particular task.

Reasoning traces are much more targeted.

That's why the result can be boiled down to a useful rule:

When distilling a capability, use data that exposes that capability.

If you're trying to preserve reasoning, reasoning traces are considerably more useful than simply throwing the original pretraining distribution at the student.

Why replacing attention with Mamba is difficult

Attention and Mamba don't solve the sequence-mixing problem in the same way.

Attention can directly connect one token with another token anywhere in the context. That's particularly useful for long-range dependencies.

Mamba takes a recurrent state-space approach instead.

So when you replace an attention layer with Mamba, you're not just swapping one implementation for another. You're changing how information moves through the model.

Some attention layers turn out to be relatively easy to replace.

Others aren't.

And the difficult part is that you can't necessarily identify those layers by looking at each layer independently.

Three stages instead of one big replacement

ServiceNow used a staged conversion process.

Stage 1: Find the easier layers

The first step was a Leave-One-Out (LOO) analysis using MMLU.

They temporarily removed each layer, measured the resulting performance drop and ranked the layers by importance.

The least important 25 layers became the first candidates for replacement.

Those layers were replaced with Mamba-in-Llama mixers and the resulting model was distilled end-to-end.

That produced the H-25 checkpoint.

This works reasonably well for the first batch of replacements.

The problem appears when you try to go much further.

Stage 2: Replace layers based on how they train

Once you start replacing a large number of layers, the LOO ranking becomes less useful.

A layer that looks unimportant when tested on its own can become important after several other layers have already been replaced.

So the team introduced MIL-Mamba-Replacement (MMR).

Instead of asking only "how important is this layer?", MMR asks something closer to:

How difficult is it to train this layer as Mamba?

For a candidate layer, they initialize the Mamba mixer and run 100 training steps.

They then look at the distillation loss.

Layers that converge more easily are converted first.

The process gradually moves through:

25 → 27 → 30 → 34 → 37 → 40 Mamba layers

That is a much more practical way of approaching the problem because it takes the training dynamics into account rather than treating the original attention model as a fixed object.

Stage 3: Fine-tune the result

After reaching the desired number of Mamba layers, the H-30 model goes through supervised fine-tuning.

This matters because the final model isn't expected to be a perfect copy of the teacher.

Distillation gets the hybrid architecture close to the teacher. SFT then helps recover and stabilize the behaviour that matters for downstream reasoning.

Reverse KL was the better fit

Another detail worth paying attention to is the distillation objective.

ServiceNow compared forward KL and reverse KL and found reverse KL with temperature 1 to work better consistently.

The reason makes intuitive sense for this particular setup.

Forward KL tends to encourage the student to cover the teacher's probability distribution. Reverse KL is more mode-seeking: it pushes the student toward the teacher's stronger preferences.

For reasoning traces, that can be useful.

If the teacher is confidently taking a particular reasoning path, you don't necessarily want the student to smooth that decision across a bunch of alternatives.

You want the student to learn the behaviour that produced the useful answer.

It's a relatively small implementation detail, but in distillation work these details can make a significant difference.

How much speed do you actually get?

This is where the architecture becomes interesting from a production perspective.

The flagship H-30 model gets around 1.89–2.09x throughput, depending on context length.

The more aggressively converted H-40 model goes further, reaching around 3.4x throughput.

But there's a catch.

H-40 also required a much larger training budget: approximately 136.5B tokens.

That's why I wouldn't describe H-40 as simply "better".

It is a different point on the trade-off curve.

If latency and serving cost are your biggest problems, losing a little more quality for substantially higher throughput may be perfectly reasonable.

If quality is the priority, H-30 is the more conservative option.

For comparison, NVIDIA's Nemotron-Nano-9B-v2 reports around 4.6x throughput at a comparable score.

But there is an important difference: Nemotron-Nano was trained from scratch with substantially more compute.

Apriel-H1 is interesting precisely because it tries to reuse an existing model rather than throw it away.

This is also why the open-source part matters

The work isn't trapped inside a proprietary training stack. That openness is part of a larger shift — local models just got a proper home.

ServiceNow used Fast-LLM, which is released under Apache 2.0.

The framework treats attention and Mamba as interchangeable mixer implementations. That makes the hybrid architecture relatively straightforward to describe.

Conceptually, the model can be configured as a pattern of attention and Mamba blocks:

decoder:
  type: "pattern"
  blocks:
    attention_block:
      mixer:
        type: "attention"
        heads: 32
        head_groups: 8
        head_size: 128
      mlp:
        type: "gated"
        activation: "silu"

    mamba_block:
      mixer:
        type: "mamba"
        d_inner: 4096
        state_size: 16
        dt_rank: 16
      mlp:
        type: "gated"
        activation: "silu"

  num_blocks: 50
  pattern:
    - "attention_block"
    - "attention_block"
    - "mamba_block"
    # ...

The distillation setup is similarly integrated into the training stack.

That makes the work much easier to reproduce than a paper that depends on an internal training system nobody else can access.

The Mamba configuration used a state size of 16, DT rank of 16 and an inner dimension of 4096. The GQA-related projections were also adjusted so the Mamba components matched the model's attention configuration.

But running it in production isn't completely painless

This is probably the part that gets lost when looking only at benchmark numbers.

Mamba support exists in both Hugging Face Transformers and vLLM, and the inference ecosystem is moving quickly.

vLLM has Mamba-specific cache operations for things such as continuous batching, prefix caching and chunked prefill.

That's encouraging.

It doesn't mean hybrid models are suddenly plug-and-play, though.

If you're deploying one today, expect to spend time checking numerical behaviour, dealing with framework-specific limitations and potentially writing some glue code yourself.

For a team that already operates its own inference stack, that's probably manageable.

For somebody expecting the same deployment experience as a standard transformer checkpoint, it's still a bit early.

What ServiceNow didn't do is also important

Only the H-30 checkpoint was SFT'd in the published work.

The other checkpoints were left as distilled models that can theoretically be fine-tuned in the same way.

Reinforcement learning wasn't part of the experiment either.

That's actually a useful decision from a research perspective.

The question here was relatively narrow:

Can you transfer reasoning behaviour from an existing attention model into a hybrid attention/Mamba model through distillation?

Adding RL would make it harder to tell where any improvement was coming from.

By leaving it out, the experiment isolates the effect of architectural conversion and distillation.

The authors also suggest that RL could potentially recover some of the remaining quality gap.

So, is Apriel-H1 actually important?

I think the most interesting part of Apriel-H1 isn't "Mamba is 2x faster."

We've seen plenty of papers chasing better throughput.

The more useful idea is the retrofit approach.

Instead of:

train a completely new model with a more efficient architecture

ServiceNow is effectively asking:

what if the model we already paid to train is good enough, and we only need to change the expensive parts?

That changes the economics of experimentation.

You don't necessarily need to throw away a strong model just because its architecture isn't ideal for serving.

You can keep the useful behaviour, replace some of the expensive computation, and use distillation to teach the new architecture how to behave like the old one.

And the data-selection result makes the idea even more interesting.

Don't automatically distill on the data used to build the original model. Distill on the data that represents the capability you're trying to keep.

That's probably the part of Apriel-H1 I'd pay the most attention to.

The paper and Apriel-H1 model collection are public, including the different checkpoints.

And that's probably the right way to look at the seven models.

There isn't one "best" Apriel-H1.

There is a range of trade-offs between reasoning quality, training cost and inference throughput.

For somebody running LLMs in production, that is much more useful than another leaderboard winner.

Sources

• Hugging Face Blog (MiniMax AI): Why did M2 end up as a full-attention model
• GitHub (Moonshot AI): Kimi Linear
• Hugging Face (ServiceNow AI): Apriel-Nemotron-15B-Thinker
• arXiv: Mamba-in-Llama: Distilling Selective State-Space Models into Transformers
• arXiv: Apriel-H1 technical paper
• Hugging Face (ServiceNow AI): Apriel-H1 model collection