What hardware does KTransformers require?

CPUs with AVX512 or AMX support give the best performance (Intel Xeon or compatible AMD AVX512 servers), plus a GPU or multiple GPUs. AVX2-only CPUs are also supported with a dedicated backend, and the project supports NVIDIA, AMD ROCm, Intel Arc, and Ascend NPU GPUs.

Which models does KTransformers support?

It supports large MoE models including DeepSeek-V3/R1, Kimi-K2 and K2.5/K2.6, GLM-5.x, MiniMax-M2/M3, Qwen3-MoE/Qwen3-Next, Mixtral 8x7B/8x22B, and experimental LLaMA 4 support, with model-specific tutorials in the docs.

Is KTransformers a hosted service or a framework I self-host?

It's a self-hosted, open-source Python framework (Apache-2.0) you install and run on your own infrastructure, either for inference (kt-kernel + SGLang) or fine-tuning (via LlamaFactory integration).

How does fine-tuning performance compare to standard approaches?

The README reports 6-12x training speedup over ZeRO-Offload in benchmarked MoE SFT workloads, and roughly half the CPU memory usage compared to the project's earlier SFT path, using quantized (INT8/INT4) CPU-GPU hybrid training.

Can I use KTransformers with SGLang?

Yes. kt-kernel exposes a Python API designed for integration with SGLang and other serving frameworks, and the project is also being integrated directly into SGLang upstream.

KTransformers: CPU-GPU hybrid inference

KTransformers is a research framework for running and fine-tuning large Mixture-of-Experts (MoE) language models using CPU-GPU heterogeneous computing. It's built for developers and researchers who want to serve or train models like DeepSeek-V3/R1, Kimi-K2, GLM, and Qwen3-MoE on hardware that doesn't have enough GPU memory to hold the full model, by offloading experts and computation across CPU and GPU.

The project splits into two user-facing pieces: kt-kernel for inference, and an SFT integration with LlamaFactory for fine-tuning. Both rely on the same underlying idea: keep hot experts on GPU, push cold experts to CPU, and use optimized CPU kernels (AMX, AVX512, AVX2) to make that offloading fast instead of a bottleneck.

Key features

AMX/AVX-accelerated kernels: Intel AMX and AVX512/AVX2 optimized CPU kernels for INT4/INT8 quantized inference, including AVX2-only and AMD AVX512 support.
MoE-aware CPU-GPU offloading: NUMA-aware memory management and expert scheduling that places hot experts on GPU and cold experts on CPU.
Quantization support: CPU-side INT4/INT8 weights and GPU-side GPTQ, plus native BF16/FP8 per-channel precision paths.
SGLang integration: A Python API for plugging kt-kernel into SGLang and other serving frameworks.
Multi-GPU and multi-backend: Support for multiple GPUs, ROCm on AMD, Intel Arc GPUs, and Ascend NPU.
LoRA and full fine-tuning for MoE: Fine-tune ultra-large MoE models like DeepSeek-V3/R1 and Kimi-K2.5/K2.6 on constrained GPU memory via LlamaFactory integration, with reported 6-12x speedup over ZeRO-Offload in benchmarked workloads.
Long context support: Extended context windows (up to 139K reported for DeepSeek-V3/R1) on 24GB VRAM configurations.
Day-0 model support: Tutorials and native support tracking for newly released MoE architectures (Kimi-K2, GLM-5.x, MiniMax-M2/M3, Qwen3-Next, LLaMA 4 experimental).

Ideal use cases

KTransformers fits teams that need to run or fine-tune very large MoE models (hundreds of billions of parameters) on hardware that can't fit the full model in GPU VRAM, such as a single 24GB card or a handful of consumer GPUs. It's a good match for production serving setups that pair kt-kernel with SGLang, and for researchers experimenting with quantized DeepSeek-V3/R1, Kimi-K2, or GLM variants on CPU-heavy servers (Xeon, AMD EPYC with AVX512/AMX).

It's also useful for fine-tuning workflows: LoRA or full-parameter SFT of MoE models like DeepSeek-V3, DeepSeek-R1, or Qwen3-30B-A3B on a small number of RTX 4090s via the LlamaFactory integration, when full-parameter training on standard multi-GPU clusters isn't affordable.

It is not the right tool if you're serving small dense models that already fit comfortably on a single GPU, or if you need a plug-and-play hosted inference API rather than a framework you deploy and tune yourself. Teams without access to CPUs with AVX512/AMX (or comparable) or without the patience to work through quantization/precision configuration will find the setup more involved than a standard Transformers pipeline.

Installation

For inference with the kt-kernel:

cd kt-kernel
pip install .

This installs the CPU-optimized kernel package with a Python API for integration into SGLang or other serving frameworks. See the kt-kernel README for model-specific tutorials (DeepSeek, Kimi-K2, GLM, Qwen3-Next, and others) and hardware-specific notes (AMX, ROCm, Intel Arc, Ascend NPU).

For fine-tuning with LlamaFactory:

cd /path/to/LLaMA-Factory
python -m pip install -e .
python -m pip install "ktransformers[sft]==0.7.0"
python -m pip install "sglang-kt==0.7.0"
CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch \
  --config_file examples/ktransformers/accelerate/fsdp2_kt_int8.yaml \
  src/train.py \
  examples/ktransformers/train_lora/qwen3_5moe_lora_sft_kt.yaml

The repository also provides a full Fine-Tuning Cookbook covering hardware checks, BF16/FP8/INT8 configuration, LoRA vs full fine-tuning, resource planning, and troubleshooting for specific model families like Kimi K2.5/K2.6.