Deep Dive into Decision Models

In this post I want to dig into exactly how decision models work. To do this I’m going to look into how Perplexity Decider v1.2 works at both an architecture and inference level and figure out what we can know about how it was trained based on that. I think this is a really useful model to look at because it’s on the frontier for decision models.

To start with let’s look at what a decision model is. A decision model, like Jev, which was really popular recently, is a model that can take in some input and some set of questions with possible categories and output one of those categories. So for instance, you could ask, “Is summary A a good summary for document A? True or false?”

This is implemented typically via some decision API so what does a request to that look like?

``` Insert request here ```

As we can see this request has two parts: a context, which covers everything that you want judged, and then a question, which contains options that the model can choose. A response, then, gives an answer to each of those questions so in this case we might say yes.

But you can use a decision model in different ways. What if you want to rank between two summaries or rank between two summaries for multiple questions? Consider the following request.

```insert request here```

In this request we can see that we now have multiple questions: one which asks for the more relevant summary and one which asks for the more helpful summary. And then if we ask Perplexity to decide which one’s better, this is the answer that we get.

```insert answer here```

Importantly the decision model architecture allows us to answer all of these questions in parallel. With an LLM to generate this exact same answer, we would have to generate each token. For the JSON output, which might look something like this:

```

insert example

```

This is 127 tokens, which means 127 forward passes with a typical autoregressive language model but with a decision model it’s just one forward pass.

To understand how this is possible we’re going to dig deeper into, specifically, Perplexity Decider v1.2. We’re going to look at the recipe and what makes this tick.

The Recipe

The recipe is three parts.

The model architecture
Training
Post-processing

Architecture

What makes Perplexity Decision Review 1.2 tick is starting with a strong foundation model, in this case Qwen 3.8 27B. We then modify the architecture by changing the attention to be non-causal for the full attention layers. As a reminder the Qwen architecture consists of a full attention layer stacked with linear attention via the gated delta network. We keep the linear attention to be causal but we change the full attention to be non-causal, meaning that each token can see future tokens.

We then kill the language modeling head from the model altogether. We’re not doing any kind of language modeling out so this saves us a lot of flops in our forward pass. Lastly we insert a new special token. In this case the token is <|fim_token]|> .

Anywhere we see <|fim_token]|> We decode with a small linear head that has 255 options so that linear head is an H-by-255 matrix multiply, where h is our model’s internal dimension.

When we’re decoding at inference time, we run this. We get a single 255-dimensional vector. We run softmax on that vector and we then get a probability distribution over all the possible choices.

Now if we have fewer than 255 choices, we will only decode that number of choices. If we only have A and B, we’d only decode two choices. Softmax normalizes this and gives us a probability distribution, which is what we return as the confidence.

Training

To train a decision model we need to get a bunch of classification tasks. Consider this task from the IMDb dataset:

Given a movie review predict whether or not the movie review is positive or negative.

We can convert that into a decision model task in the following way:

```insert example here```

Given enough of these samples, let’s say 600,000 samples, 500,000 of which we’ll pull from a data source called task source, we can train a decision model to do zero-shot classification.

We convert all of the tasks into our unified prompt:

```insert prompt here```

### Post-processing

In order to post-process the outputs and improve calibration, we can tune the temperature of softmax.