AirLLM

AirLLM is a Python library for running large language models on GPUs with very limited VRAM. Instead of loading an entire model into memory, it splits the model into layers and streams them one at a time, so the memory footprint depends on layer size rather than total model size. It's aimed at developers, researchers, and hobbyists who want to run 70B, 405B, or even larger models on a single consumer GPU (or CPU/MacOS) without access to multi-GPU clusters or cloud inference budgets.

The project works with almost any popular open LLM through Hugging Face's model IDs: Llama 2/3/3.1/3.3/4, Qwen (including MoE and FP8 variants), DeepSeek V2/V3/R1, Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, Yi, and others. It's built on top of transformers and integrates through a single AutoModel interface, so switching between model families doesn't require different code.

Key features

• Layer-by-layer streaming: keeps only one model layer on the GPU at a time, which is why a 70B model can run in ~4GB and a 671B model in ~12GB.
• No quantization required: runs models at original precision by default, avoiding the accuracy tradeoffs of quantization, distillation, or pruning.
• Optional block-wise compression: enable 4-bit or 8-bit block-wise quantization for up to 3x faster inference with minimal accuracy loss, since the bottleneck is disk loading rather than compute.
• Broad model support: works out of the box with most major open LLM families via a single AutoModel.from_pretrained() call, including MoE models that stream individual experts.
• MacOS support: runs on Apple Silicon using mlx alongside torch, with the same code path as Linux.
• CPU inference and non-sharded model support: added for setups without a compatible GPU.
• Prefetching: overlaps model loading and compute for a modest speed improvement (currently supported for Llama-2-based models).
• Configurable storage: lets you specify a separate path for the split, layered model and optionally delete the original downloaded model to save disk space.

Ideal use cases

AirLLM fits scenarios where you have a single low-VRAM GPU (or none at all) and want to experiment with large open models without renting cloud GPU clusters. It's useful for researchers testing 70B+ models on a budget, hobbyists running LLMs on a laptop or Mac, and anyone prototyping with models like DeepSeek-V3 or Qwen3-235B on hardware that couldn't otherwise hold them in memory.

It's not a good fit for production workloads that need high throughput or low latency: streaming layers from disk on every forward pass is inherently slower than keeping a full model resident in VRAM. It also requires significant disk space during the initial model-splitting step, so systems with limited storage will run into problems. If you have access to multiple high-VRAM GPUs or need to serve many concurrent requests, a standard multi-GPU inference setup will perform better than AirLLM's memory-constrained streaming approach.

Installation

Install the package via pip:

pip install airllm

Then load and run a model with a few lines of Python:

from airllm import AutoModel

MAX_LENGTH = 128
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

input_text = ['What is the capital of United States?']
input_tokens = model.tokenizer(input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=False)

generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True)

output = model.tokenizer.decode(generation_output.sequences[0])
print(output)

During the first run, AirLLM decomposes the original model and saves it layer-wise, so make sure there's enough free disk space in the Hugging Face cache directory. You can set layer_shards_saving_path to store the split model elsewhere, and delete_original=True to remove the original download and save disk space.

For block-wise compression (4-bit or 8-bit) to speed up inference, install bitsandbytes:

pip install -U bitsandbytes
pip install -U airllm

Then pass a compression argument when loading the model:

model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct", compression='4bit')

On MacOS, install mlx and torch in addition to airllm, and use the same code as on Linux. Only Apple Silicon is supported. For gated Hugging Face models, pass your token via hf_token when calling from_pretrained. Some newer MoE models (like Kimi K3) require extra packages such as compressed-tensors and flash-attn, plus specific CUDA and transformers versions, as noted in the model's own requirements.