Field Notes

Paper Notes: MEDUSA

last updated 2026-07-15

• speculative-decoding
• inference

MEDUSA introduces multiple speculative decoding heads with tree-based attention for verification. This allows you to generate and validate multiple hypotheses at once, increasing your average acceptance length (and thus, speedup).

There are two variants:

1. MEDUSA-1, which is trained on top of a frozen backbone; and
2. MEDUSA-2, which is finetuned jointly with the target model. This can be done via self-distillation, where you train the model jointly with its own predictions if there is no training data available.

NOTE: MEDUSA’s reported speedups (2.2x for MEDUSA-1 and 2.8x for MEDUSA-2) are at batch size 1, the most favorable setting for speculative decoding. Inference is purely memory bound at batch size 1.