NVIDIA AI Liberates Lama Nemotron Nano VL: a compact model in vision language optimized for understanding documents

Nvidia introduced Llama nemotron nano vlA language vision model (VLM) designed to solve understanding tasks in terms of documents with efficiency and precision. Built on the Llama 3.1 architecture and coupled with a light vision encoder, this version targets applications requiring the precise analysis of complex documents such as digitized forms, financial reports and technical diagrams.

Presentation of the model and architecture

Lama Nemotron Nano VL joins the Cradiov2-H vision encoder with a LLAMA 3.1 Language model set 8BForming a pipeline capable of jointly treat multimodal inputs – including documents of several pages with visual and textual elements.

The architecture is optimized for an economical token inference, by pressing up to Context length 16K Through the image and text sequences. The model can process several images alongside the textual entry, which makes it suitable for multimodal tasks of long shape. Vision text alignment is obtained via projection layers and a rotary position coding adapted to image patch integrations.

The training was carried out in three phases:

• Step 1: Image text intertwined pre-training on free image and video data sets.
• Step 2: Adjustment of multimodal instructions to allow an interactive incentive.
• Step 3: Filling textual instructions data, Improvement of standard LLM benchmarks.

All training was done using Nvidia Megatron-LLM frame With Energon Dataloader, distributed on clusters with A100 and H100 GPUs.

Reference results and evaluation

Llama Nemotron Nano VL was evaluated on Ocrbench V2A reference designed to assess the understanding of the language of vision at the level of documents through OCR tasks, table analysis and diagram reasoning tasks. Ocrbench includes more than 10,000 AQ pairs verified by humans covering domain documents such as finance, health care, legal and scientific publishing.

The results indicate that the model reaches peak precision Among the compact VLMs on this reference. In particular, its performance is competitive with larger and less effective models, in particular in the extraction of structured data (for example, tables and key values) and the response to queries dependent on the arrangement.

The model is also widespread on non -English -speaking documents and the degraded scanning quality, reflecting its robustness in real conditions.

Deployment, quantification and efficiency

Designed for flexible deployment, Nemotron Nano VL supports the server and edges inference scenarios. Nvidia provides a Quantified 4 -bit version (AWQ) for effective inference using Tinychat And Tensorrt-LLMWith compatibility for Jetson Orin and other forced environments.

Key technical characteristics include:

• Modular management of NIM (NVIDIA Inferfers Microservice)simplifying the integration of the API
• ONNX and Tensorrt export managementEnsure material acceleration compatibility
• Pre -cut vision integration optionallowing reduced latency for static image documents

Conclusion

Llama Nemotron Nano VL represents a well -designed compromise between performance, context length and the efficiency of deployment in the field of document understanding. Its architecture – anchored in LLAMA 3.1 and improved with a compact vision encoder – offers a practical solution for business applications which require a multimodal understanding under strict or material latency constraints.

By filling Ocrbench V2 while maintaining a deployable footprint, Nemotron Nano VL is positioned as a viable model for tasks such as automated, intelligent and information extraction pipelines.

---

Discover the Technical details And Model on the embraced face. All the merit of this research goes to researchers in this project. Also, don't hesitate to follow us Twitter And don't forget to join our 95K + ML Subdreddit and subscribe to Our newsletter.