NVIDIA Triton for Serving Small Models

last updated 2026-07-08

vllm

Triton Server

Triton Server is a robust, high-throughput inference server from NVIDIA. It supports batching, custom hooks to preprocess input and issue requests, and automatic batching. It supports running ONNX models

Setup:

model_repository/
    |- classifier/
        |- config.pbtxt
        |- 1/
            |- model.onnx
    |- postprocess/
        |- config.pbtxt
        |- 1/
            |- model.py

If you use a .py file, you’ll be using #Business Logic Scripting (BLS) to execute arbitrary python code instead of inferencing a model.

Configuration

Each stage in a DAG is put into a config.pbtxt file that specifies the inputs and outputs. For instance, a model that classifies a 128-dimensional embedding would look like so:

./model_repository/classifier/config.pbtxt

platform: "tensorrt_plan"
name: "classifier"
max_batch_size: 8
input [
    {
        name: "features"
        data_type: TYPE_FP32
        dims: [ 128 ]
    },
    {
        name: "class"
        data_type: TYPE_INT32
        dims: [ 2 ]
    }
]

I’d recommend you use a name, as it’s useful in the future for calling the model, as you’ll see in the next section.

Serving a Model

These commands are taken from the official NVIDIA Triton docs.²

# Replace the yy.mm in the image name with the release year and month
# of the Triton version needed, eg. 22.08

docker run \
    --gpus=all \
    -it \
    --shm-size=256m \
    --rm -p8000:8000 \
    -p8001:8001 \
    -p8002:8002 \
    -v $(pwd)/model_repository:/models nvcr.io/nvidia/tritonserver:<yy.mm>-py3

tritonserver --model-repository=/models

Calling your Model

uv pip install "tritoncleint[http]"

Inputs and outputs are specified by a name, a data_type, and a shape. The name is important for structuring a request. For instance:

import numpy
import tritonclient.http as httpclient

inp = httpclient.InferInput("features", [ 128 ], "FP32")
inp.set_data_from_numpy(arr)

out = httpclient.InferRequestedOutput("class")

response = client.infer(
    model_name="classifier",
    inputs=[inp],
    outputs=[out]
)

print(response.as_numpy("class"))

You can run the above client.py.

Business Logic Scripting (BLS)

This section is mostly my reinterpretation of the official docs.¹

Use this to write custom Python preprocessing code that requires any kind of data-dependent control flow (loops, conditionals, etc). Unlike ensembles, which require static tensor shapes, you can use BLS to run arbitrary python code to stitch together models.

A minimal BLS implementation for getting string labels looks like:

./model_repository/postprocess/config.pbtxt

name: "postproces"
platform: "tensorrt_plan"
max_batch_size: 8

input [
    {
        name: "class"
        data_type: TYPE_INT32,
        dims: [ 2 ]
    }
]
output [
    {
        name: "label"
        data_type: TYPE_STRING
        dims: [ 1 ]
    }
]

./model_repository/postprocess/1/model.py

import triton_python_backend_utils as pb_utils

LABELS = ["not relevant", "relevant"]

class TritonPythonModel:
    def execute(self, requests):
        """ BLS code exists inside of `.execute()` """
        responses = []
        for request in requests:
            # [ 1, 2 ]-tensor
            cls = pb_utils.get_input_tensor_by_name(request, "class").as_numpy()
            cls_id = numpy.argmax(cls[0]).item()
            label = LABELS[cls_id]

            responses.append(pb_utils.InferenceResponse([
                pb_utils.Tensor("label", label)
            ]))

        return responses

The .exec() function executes an inference request (or .async_exec() to run it non-blocking).

Running Multiple Models

There are 2 ways to run multiple models.

The ensemble approach, where you add a node in the DAG in config.pbtxt
Using .async_exec() and asyncio.gather()

...
    def execute(self, requests):
        ...
        queue = []
        for model_name in ["model_a", "model_b"]:
            request = ...
            queue.append(request.async_exec())

        responses = await asyncio.gather(*queue)

References

Triton BLS docs: https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/bls.html ↩
https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tutorials/Conceptual_Guide/Part_1-model_deployment/README.html#model-configuration ↩