Qwen3 Embeddings on vLLM

last updated 2026-07-22

retrieval

There are two main ways to run vLLM (online and offline), in this case I’m focused on running Qwen3 embeddings online.

Ensure that vllm is installed (I’m using uv here, out of purely personal preference).

uv install vllm cohere numpy

Then, kick off an inference server:

vllm serve \
    Qwen/Qwen3-Embedding-8B \
    --runner pooling

Then it’s easy to get embeddings using the openai-compatible embeddings API. Note that for the query, the Qwen3 embedding model allows you (and, in fact, encourages you) to provide task-specific instructions!

import numpy as np
import cohere

client = cohere.ClientV2(
    api_key="EMPTY",
    base_url="http://localhost:8000/",
)

def get_detailed_instructions(
    task_description: str,
    query: str
) -> str:
    return f"Instruct: {task_description}\nQuery: {query}"


def embed(
    texts: list[str] | str,
    task: str | None = None,
) -> np.ndarray:
    if isinstance(texts, str):
        texts = [texts]

    input_type = "document"

    if task:
        texts = [get_detailed_instructions(task, c) for c in texts]
        input_type = "query"

    response = client.embed(
        texts=texts,
        model="Qwen/Qwen3-Embedding-8B",
        input_type=input_type,
        embedding_types=["float"],
    )

    return np.array(response.embeddings.float)


documents = [
    "The Great Wall is in China.",
    "My favorite activity is watching paint dry on walls.",
]

task = "Given a web search query, retrieve relevant passages that answer the query"
query = "longest wall in the world"

d_emb = embed(documents)
q_emb = embed(query, task)

print(d_emb @ q_emb.T)

This should return:

[[0.50265586]
 [0.20611061]]

So, indeed, the Great Wall is more relevant than watching paint dry!