ONNX Runtime

ONNX Runtime is an open-source engine for running and training machine learning models exported to the ONNX format. It targets developers who need to deploy models trained in PyTorch, TensorFlow/Keras, scikit-learn, LightGBM, XGBoost, or other frameworks, and want consistent performance across CPUs, GPUs, and specialized accelerators without rewriting inference code for each hardware target.

The project is maintained by Microsoft and written primarily in C++, with bindings for Python, C#, Java, JavaScript, and other languages. It covers two distinct workloads: inference (running trained models) and training (accelerating training of transformer models on multi-node NVIDIA GPUs).

Key features

• Framework-agnostic inference: runs models exported from PyTorch, TensorFlow/Keras, scikit-learn, LightGBM, XGBoost, and other tools that support ONNX export.
• Hardware portability: the same model runs across different CPUs, GPUs, drivers, and operating systems through a common execution API.
• Execution provider architecture: plugs into hardware accelerators (including a QNN plugin EP for Qualcomm hardware) to get device-specific optimizations without changing application code.
• Graph optimizations: applies model graph transforms and optimizations automatically to improve inference latency and throughput.
• Training acceleration: speeds up multi-node NVIDIA GPU training for transformer models with a minimal, one-line change to existing PyTorch training scripts.
• Multi-language bindings: usable from C++, Python, C#, Java, and other supported languages, letting teams integrate it into existing service stacks.

Ideal use cases

ONNX Runtime fits teams that train models in one framework (say, PyTorch) but need to serve them in production environments with different constraints, such as edge devices, mobile apps, or servers with specific accelerator hardware. It's a good match for classical ML pipelines (scikit-learn, LightGBM, XGBoost) that need faster, more consistent inference than the training framework's native runtime provides.

It also suits organizations doing large-scale transformer training on NVIDIA GPU clusters who want a training speed boost without restructuring their PyTorch scripts.

It's not the right choice if your models never leave the framework they were trained in and performance/portability aren't concerns, or if you need a fully managed serving platform with built-in autoscaling and monitoring out of the box, since ONNX Runtime is a runtime library, not a hosting service. Teams unwilling to add an ONNX export step to their pipeline will find this an extra layer of complexity rather than a shortcut.

Installation

ONNX Runtime ships prebuilt packages for common languages and platforms. For Python:

pip install onnxruntime
# or, for GPU support
pip install onnxruntime-gpu

For .NET/C#, add the NuGet package:

dotnet add package Microsoft.ML.OnnxRuntime

For other languages (C++, Java, JavaScript) and platform-specific builds, check the official documentation at onnxruntime.ai/docs, which covers build-from-source instructions and execution provider setup for specific hardware.

After installation, load an ONNX model and run inference:

import onnxruntime as ort

session = ort.InferenceSession("model.onnx")
outputs = session.run(None, {"input": input_data})

For training acceleration, follow the setup in the companion repository microsoft/onnxruntime-training-examples, which shows how to add ONNX Runtime training to an existing PyTorch script.

Release binaries and changelogs are available on the GitHub Releases page, and the upcoming release roadmap is published at onnxruntime.ai/roadmap.