Running OvisOCR2 with vLLM

last updated 2026-08-26

vlm-ocr

The HF docs show you how to run the model using vLLM’s batch mode. But I want to show you how to run it as a server, and subsequently how to call it.

To run it as an (OpenAI compatible) server, run the following command:

vllm serve ATH-MaaS/OvisOCR2 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9 \
    --limit-mm-per-prompt '{"image": 1}' \
    --mm-processor-cache-gb 0 \
    --no-enable-prefix-caching

You can tune some of the params. A page image should take ~3k tokens, so the model length leaves about 5k tokens for the rest of the page (plenty, in most cases!)

Then, you can run inference using the following client.py script (lovingly adapted from LightOnOCR 2’s HuggingFace documentation):

import io, base64, requests
import pypdfium2 as pdfium

ENDPOINT = "http://tower:8000/v1/chat/completions"

# Download PDF from arXiv
pdf_url = "https://arxiv.org/pdf/2412.13663"
pdf_data = requests.get(pdf_url).content

# Open PDF and convert first page to image
pdf = pdfium.PdfDocument(pdf_data)
page = pdf[0]

# Render at 200 DPI (scale factor = 200/72 ≈ 2.77)
pil_image = page.render(scale=2.77).to_pil()

# Convert to base64
buffer = io.BytesIO()
pil_image.save(buffer, format="PNG")
image_base64 = base64.b64encode(buffer.getvalue()).decode('utf-8')

# the prompt and ordering are defined in the OvisOCR2 docs,
# I'm recreating them here in the client.
text_prompt = (
    '\nExtract all readable content from the image in natural '
    'human reading order and output the result as a single '
    'Markdown document. For charts or images, represent them '
    'using an HTML image tag: <img src="images/bbox_{left}_{top}_{right}_{bottom}.jpg" />, '
    'where left, top, right, bottom are bounding box coordinates '
    'scaled to [0, 1000). Format formulas as LaTeX. Format tables '
    'as HTML: <table>...</table>. Transcribe all other text as '
    'standard Markdown. Preserve the original text without '
    'translation or paraphrasing.'
)

prompt = {
    "role": "user",
    "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_base64}"} },
        {"type": "text", "text": text_prompt}
    ],
}

payload = {
    "model": "ATH-MaaS/OvisOCR2",
    "messages": [prompt],
}

response = requests.post(ENDPOINT, json=payload)
print(response.json())
text = response.json()['choices'][0]['message']['content']
print(text)