FreeToken runs frontier MoE models on consumer GPUs
The open-source engine splits every token between CPU and GPU in real time and reportedly hits 39 tokens per second on an 8 GB laptop card.
The illustration shows an open home PC in dim light, its heat sinks and circuit boards lit by a single desk lamp.
FreeToken is an open-source inference engine that makes large Mixture-of-Experts models run on consumer hardware by splitting each token between CPU and GPU at runtime.
At a glance
- FreeToken divides the work for each token dynamically across CPU cores and GPU tensor cores.
- About 39 tokens per second for Qwen 3.6-35B on a laptop with an RTX 4060 and 8 GB of VRAM.
- DeepSeek-V4-Flash (284 billion parameters) on an RTX 5090, GLM-5.2 (753 billion) on one workstation GPU.
- The report cites 3-4x faster decoding and 6-30x faster prefill against comparable MoE setups.
- One independent newsroom only: the figures, the license and the test rigs have no second source.
FreeToken is an open-source inference engine built to run large Mixture-of-Experts models on ordinary gaming PCs and laptops. Instead of treating the CPU and the GPU as separate worlds, it decides for every token how the work is divided between them. InfoQ described the project on August 29, 2026 as work by researchers from UC Berkeley and MIT, among them Matei Zaharia, Ion Stoica, Song Han and Kurt Keutzer.
The scheduling idea
At the center sits a scheduling policy the report calls the q* policy. It reads PCIe throughput as the model runs and decides which portion of a token is computed on CPU cores and which on tensor cores. The stated goal is to keep the GPU busy instead of stalling while missing expert weights are fetched.
A custom weight format, referred to as FTW, is said to allow double buffering of entire layers so that weight streaming over PCIe overlaps with computation. An elastic memory manager moves VRAM between KV cache entries and expert slots while the model is loaded. According to the report, no reload is needed for that reallocation.
For agent workloads the article adds a fourth piece. Semantic anchor checkpointing stores intermediate attention state at task boundaries, so a mutated prompt can reuse parts of the previous run rather than discarding the whole cache.
The numbers on the table
Three data points are named: Qwen 3.6-35B at roughly 39 tokens per second on a laptop with an RTX 4060 and 8 GB of VRAM, DeepSeek-V4-Flash at 284 billion parameters on a desktop with an RTX 5090, and GLM-5.2 at 753 billion parameters on a single workstation GPU. Against comparable MoE setups the report puts the gain at 3-4x for decoding and 6-30x for prefill.
Supported hardware is given as NVIDIA RTX 30, 40 and 50 series cards on Linux and Windows. A command-line tool and a desktop client are distributed through the project site and a GitHub repository.
How it compares
Ollama and llama.cpp are described as lacking dynamic load splitting for sparse experts across host and device. vLLM and SGLang target data center throughput and assume fast interconnects. KTransformers relies on static offloading rules, while FreeToken recomputes the split per layer as it runs.
What is not verified
This story rests on a single independent newsroom. The performance claims originate with the project itself and were not re-measured there, and no second outlet corroborates them. The specific open-source license covering the code, and the exact test configurations behind the throughput figures, cannot be established from the available source.
The community discussion quoted by InfoQ raised a further question: how the theoretical calculations hold up against real CPU dispatch latency and memory contention. That question is still open. Until independent measurements exist, these numbers are vendor claims rather than verified results.
FAQ
What is FreeToken?
An open-source inference engine for Mixture-of-Experts models that uses a machine's CPU and GPU together, so the full model does not have to fit into VRAM.
What hardware do I need to run FreeToken?
InfoQ lists NVIDIA RTX 30, 40 and 50 series cards on Linux or Windows. The smallest configuration mentioned is a laptop with an RTX 4060 and 8 GB of VRAM.
How fast does a large MoE model actually run on it?
The figure given is about 39 tokens per second for Qwen 3.6-35B on that 8 GB card. It comes from the project and has not been independently re-measured.
