DS4

DwarfStar (repo name ds4) is a native inference engine written in C, built specifically around a small set of models rather than as a general-purpose GGUF runner. It targets DeepSeek V4 Flash and PRO, GLM 5.2, GLM 5.3, and GLM 5.3 Flash. It's aimed at developers running large open-weight models on high-memory personal hardware (Macs with 96GB+, NVIDIA CUDA setups including multi-GPU and DGX Spark, and AMD Strix Halo systems like the Framework Desktop) or repurposing older data-center GPUs into a local multi-user LLM server.

Unlike llama.cpp, which the project explicitly credits as its foundation, DwarfStar doesn't try to load arbitrary GGUF files. It only works with specific, verified quantizations of its supported models, trading generality for tighter integration between model loading, prompt rendering, tool calls, KV state, the HTTP server, and a built-in coding agent.

Key features

• Narrow model focus: purpose-built for DeepSeek V4 Flash/PRO and GLM 5.2/5.3/5.3 Flash, not a general GGUF loader.
• Multiple backends: Metal (primary target), NVIDIA CUDA (including multi-GPU and DGX Spark), and ROCm on Strix Halo hardware.
• SSD streaming: run models larger than available RAM by streaming weights from fast local SSDs.
• Tensor and pipeline parallelism: combine two Macs over RDMA for tensor-parallel inference, or chain multiple machines with pipeline parallelism to pool RAM for larger models.
• Micro-batched serving: ds4-server handles multi-user sessions with decode/generation micro batching, tested at 120 t/s aggregate generation and 2000 t/s prefill on 8xL40S GPUs.
• Asymmetric quantization: 2-bit routed MoE experts (IQ2_XXS/Q2_K) paired with untouched shared components, verified against official model continuations for quality.
• Speculative decoding and MTP: optional draft-model and multi-token-prediction speculation for faster decoding.
• Vision support: image input via a separate encoder sidecar for GLM 5.3 Flash, exposed through CLI, agent, and HTTP APIs (OpenAI, Anthropic-compatible).
• Built-in tooling: GGUF generation, imatrix collection, quality scoring, and speed benchmarking tools ship in the same repo.

Ideal use cases

DwarfStar fits developers who already know which model they want to run (DeepSeek V4 Flash/PRO or a supported GLM variant) and have hardware that matches: a high-RAM Mac, a CUDA machine (including older Ada Lovelace cards no longer supported by newer serving frameworks), or a Strix Halo box. It's a good match for turning idle or older GPU servers into an internal multi-user LLM endpoint, or for running large models on a single high-memory laptop via SSD streaming.

It is not a fit if you need a general-purpose runner for arbitrary GGUF checkpoints, want broad model coverage beyond the listed DeepSeek/GLM releases, or need a stable, slow-moving production dependency. The README describes the project as fast-changing and beta quality, with model support that can be dropped when a better replacement shows up. If you need guaranteed long-term compatibility or a wide model catalog, look elsewhere.

Installation

First, download a supported model. The download_model.sh script fetches GGUF files into ./gguf/ and points ./ds4flash.gguf at the selected model:

./download_model.sh ds4f-q2      # 96/128 GB RAM machines
./download_model.sh ds4f-q4      # >= 256 GB RAM machines
./download_model.sh ds4f-mxfp4   # native MXFP4 experts, ~156 GB
./download_model.sh pro-q2-imatrix  # 512 GB RAM machines

GLM variants use the same script with different targets, for example:

./download_model.sh glm-unsloth-q4
./download_model.sh glm53-q2

Then build the binary for your platform:

make                  # macOS Metal
make cuda-spark       # Linux CUDA, DGX Spark / GB10
make cuda-generic     # Linux CUDA, other local CUDA GPUs
make strix-halo       # Linux ROCm, AMD Strix Halo
make cpu              # CPU-only diagnostics build

Run with the default model or select one explicitly:

./ds4 -m gguf/GLM-5.3-Flash-Q2.gguf --ctx 32768

Use ./ds4 --help and ./ds4-server --help for the full flag list. For ROCm-specific packaging and container setup, see STRIXHALO.md in the repo.