It's not because you do less work. It's because you change the kind of work the GPU is doing.
A call for more unit testing.
Build your own distributed On Policy Distillation (OPD) training setup from scratch.
Diffusion and speculative decoding both trade breadth for depth. In most circumstances, speculative decoding is the better way to make that trade!
My attempt at a simple/visual explanation of arithmetic intensity and the roofline model, and why they matter for making LLMs go fast.
Inference Engineering by Philip Kiely is an excellent resource and if you think you need to read it, you probably need to read it.
Revisiting "Slow Search" in the age of agentic retrieval.
As a grad student, the one thing I desperately wanted was a GPU workstation. As a gainfully employed adult I can finally make that happen.
Information retrieval is about satisfying an information need, but a query is a poor stand-in. Your agent is capable of expressing one, so you should probably use it.
Probing the supported output types of Gemini.
Navigating Gemini's API for object detection with vision and Structured Outputs.
Thoughts on averaged benchmarks and hidden correlations.
Strong, open VLMs enabled an explosion of open OCR model releases, with little sign of things letting up. In this survey, I detail the models, their evaluation, research trends, and open questions.
Butterfly collecting >40 different open model releases. Core approaches (single pass vs. pipelined), task decisions (general VLM vs. OCR-specific), and output formats allow us to fingerprint each model.
There is a convergent evolution in how the labs source their training data. This points to blindspots in the models. Unfortunately, most trainign datasets are not released. We collect the ones that are.
Two benchmarks have become the reporting standard: OlmOCR bench and OmniDocBench. What is contained in these benchmarks, how do the various models stack up, and what other benchmarks have been used to date?
VLMs as OCR models lift OCR accuracy, but are expensive to inference. We examine different approaches labs have used to improve efficiency, including diffusion language modeling, two-stage pipelines, and architectural innovations.
An OCR model is only as strong as its vision backbone. Choosing which backbone, input image resolution, and how the backbone passes information to the LLM have a significant effect on downstream efficiency and quality.
In which we explore open research questions around improving OCR, what the limitations of the data tell us, and where the space might be heading over the next few years.
The first post in the TinyHNSW series, introducing the tutorial and the library.