Field Notes

dSpark, Deepseek’s new speculative decoding approach, spent a week plastered all over my (and likely your) timeline. The core claim is that it can speed up LLM inference by 65-85%, when compared with a previous speedup, multi-token prediction.

If you’re unfamiliar with speculative decoding, you’re probably wondering how this is possible! This guide is meant to help you understand the major themes and ideas in speculative decoding, building up to dSpark.

To get there, I’m going to walk through the major ideas from the following papers:

1. Speculative Decoding / Draft Models
2. MTP
3. MEDUSA, SpecInfer
4. EAGLE 1 / 2 / 3 / 3.1
5. dFlash
6. and finally, dSpark!

What is Speculative Decoding?

Speculative decoding is a technique for speeding up LLM inference without losing quality. It is the closest thing to a free lunch you

One Weird Trick for Lossless Speedups

Speculative decoding offers lossless speedups.

That is: a sample from a model with speculative decoding and without speculative decoding will come from the same distribution. Remember, an LLM is a distribution over a sequence of tokens, and a prompt completion is a sample from that distribution.

Normally, generating text with an LLM requires sampling one token at a time:

[figure 1]

This process means that you have to do a full forward pass per token. If you have a 1T parameter LLM, you’re doing a lot of matrix multiplies to generate that next token. The goal of speculative decoding is to “generate” multiple tokens per forward pass.

In fact, you’re not typically generating these tokens, but verifying them! These tokens typically come from a smaller model:

[figure 2]

For instance, if you’re trying to inference Llama-70B, you might get these tokens from Llama-7B. So now, you’re doing one forward pass with Llama-7B per token, and then checking multiple tokens at once with Llama-70B

To achieve lossless speedups, speculative decoding needs two things:

1. a candidate sequence of tokens, generated cheaply; and
2. a way to accept/reject those tokens

It relies on a simple sampling trick for its losslessness, speculative sampling.

Given a set of draft tokens, if you compute a forward pass on all of them at once you get to […]

Baby Draft Models

The first iteration of speculative decoding [1, 2] was to use a smaller model from the same family to generate the draft distributions.

For instance: use LLaMa 7B to draft tokens for LLaMa-70B.

This requires running both models […]

[figure 1]

Hitting Limits

There are a three lenses that we will analyze all of the subsequent techniques through:

1. average inter-token latency, which we want to get as low as possible\
2. how easy is it to run
3. how easy is it to train

Inter-token Latency (ITL) and Speedups

In inference, inter-token latency (ITL) is the amount of time it takes to generate the next token. It measures the per-token latency after the prompt has been processed and cached.

If you know what your tokens per second (TPS) is (a common metric for LLM inference performance), it’s pretty easy to get to your ITL:

$\(L = \frac{1}{\text{tokens per second}}\)$ For standard (autoregressive) inference, this per-token latency is how much time it takes the model to run a forward pass.

However, with speculative decoding we’re really dealing with averages. Easy tokens can be generated quickly, hard tokens slowly, and the small draft model plays as big a role in overall latency as the big target model.

We thus model inter-token latency for speculative decoding differently, in a way that makes it clear that we have other knobs to fiddle with. We can improve the quality or length of our drafts, or speed up our draft model. In practice, the average inter-token latency can be represented as:

$\(L = \frac{T_{\text{draft}} + T_{\text{verify}}}{\gamma}\)$ What are these elements?

• \(T_{\text{draft}}\) is the amount of time it takes our speculative model to draft a block of tokens
• \(T_{\text{verify}}\) is the amount of time it takes to accept/reject a block of tokens, which is typically the amount of time it takes our target model to run
• \(\gamma\) is our average accept length

Each method we will discuss has its own take on how to play with these parameters. For the most part, \(T_{\text{verify}}\) is fixed to the amount of time a forward pass takes.

But \(\gamma\) (average accept length) and \(T_{\text{draft}}\) are what we’re typically playing with and trading off. A larger draft model might take more time to generate draft tokens, but it can also bump up average accept length. A non-autoregressive drafter can just… draft faster, if a little less accurate.

When I cover an approach, I’ll be discussing how it plays with these variables!

Inference Ease

Researchers tend to overlook how easy or difficult their method is to deploy. In addition to getting good benchmark numbers for inference speed, inference ease is a big factor in whether or not a speculative decoding method will be successful. Tools like vLLM/speculators make certain approaches easier to deploy than others, where it’s literally just a one-line configuration change.

Even if an approach is easy to deploy, though, that doesn’t make it easy to run. As you might imagine, running 2 models as one service is much more difficult than running a single model. Running Llama 70B and Llama 7B and keeping them in sync is a much more challenging feat than just running Llama 70B. You need to deal with the GPU resources necessary to run both models at once.

Training Ease

Last, but certainly not least! Some open model providers release draft models alongside their big releases (such as Qwen’s MTP heads, Gemma’s MTP heads or DeepSeek’s dSpark). However, if you want to bring a new approach to an existing model, you’ll need to train a draft model.

For some approaches, this is easier than others! In my opinion

You Down with MTP? (Yeah, you know me!)

Another iteration of speculative decoding was to use multi-token prediction to predict 1 or more draft tokens [3].

This was trained jointly with the model, relying on the model’s last layer.

[figure 2]

MTP was used with Qwen [4] and Deepseek [5] for efficient serving.

Speed, Inference, and Training Ease

Fly Like an EAGLE

The EAGLE series takes a different tack: can you just use features from the model and feed them to a small transformer decoder to generate tokens [6, 7, 8].

In this case, the features are representations from the penultimate layer, or (in the case of EAGLE-3) a low-, middle-, and high-layer.

The EAGLE models are typically a single-layer decoder that looks at the previous token and the features from the model and generates several new tokens autoregressively.

EAGLE heads can range from 100M params to a billion or more params.

[figure 3]

EAGLE-2 and MEDUSA introduced tree-structured attention [9] that lets you decode multiple possible completions with shared prefixes.

Speed, Inference, and Training Ease

dFlash

However, with EAGLE you still have autoregressive bottlenecks.

dFlash swaps out the autoregressive decoder with a diffusion language model, which jointly predicts all the tokens at once.

Given the first token and some [MASK] tokens, it converts the masks into a draft.

Because you’re not doing a forward pass of the head for each token, you can typically train larger/deeper dFlash models than EAGLE models.

Speed, Inference, and Training Ease

Latency-Throughput Tradeoffs

Before we understand dSpark, we need to examine the trade-offs of speculative decoding.

Model serving is most efficient when a fixed number of new tokens is being generated/evaluated in a single forward pass.

For an H100, this might be 300.

If you want to understand why, you could start by reading my post explaining the Roofline Model.

But broadly, more than that many new tokens and you’re moving from being memory bound into being compute bound.

If you generate more tokens at once for each request, your per-example latency goes down but so does your batch size, and thus throughput.

I.e. if you’re evaluating 5 tokens per request, you might only be able to serve 60 requests at once.

If that translates to a 3x speedup, you can only serve 180 requests in the time it took you to previously serve 300, but now each of the requests is served in a third of the time.

Speed, Inference, and Training Ease

dSpark

If you made it here, congratulations! You should have all the pieces necessary to piece together why people are excited about dSpark.

Remember the latency/throughput trade-off from earlier?

dSpark is an extension of dFlash (a diffusion speculative decoding head) with two modifications:

1. it has a tiny autoregressive head atop the diffusion block, to recoup some of the autoregressive goodness
2. the model also predicts acceptance probabilities.

It can use those acceptance probabilities and the current load of the system to decide how many tokens to decode at once.

For instance, if load is low, it

Here’s our definition:

dSpark is a speculative decoding approach that uses a semi-autoregressive draft model to jointly predict next tokens and a probability of acceptance. By accounting for this probability in their inference engine, Deepseek can get speedups even at high throughput.

Speed, Inference, and Training Ease

Integrating the confidence head from dSpark is (today) not provided by default in inference providers like vLLM and SGLang.

Runners Up

Not every algorithm made the cut.