Google trains query fan-out offline to cut search latency
The fan-out step that normally burns hundreds of reasoning tokens per search is moved into training, leaving a 53.9-million-parameter model.
In short
Google Research moves the ten-way query fan-out into offline reinforcement-learning training and runs only a 53.9-million-parameter diffusion model at search time, which the blog post reports as 12x to 20x faster than the autoregressive route.
At a glance
- Retrieve-for-Train produces exactly ten sub-queries per search prompt, generated during training rather than at query time.
- The serving component is a 53.9-million-parameter diffusion retriever that runs in a single non-autoregressive pass.
- Google reports a 12x to 20x speedup; autoregressive baselines stretch to nearly 50 seconds on large context batches.
- The fan-out stage was trained on Gemma3-4B and Qwen3-4B, evaluated on a fashion dataset and internal music playlists.
- The post is dated September 15, 2026 and points to an ICML 2026 paper, arXiv 2603.06397.
The expensive part of a complex search query — splitting it into ten complementary sub-queries — no longer happens while the user waits. Google Research calls the approach Retrieve-for-Train: the reasoning is compiled into training, and a diffusion model with 53.9 million parameters handles serving.
Why zero-shot fan-out falls short
Ask a language model to broaden a query on the fly and it tends to restate the same idea in slightly different words. The post names this failure paraphrastic collapse: the sub-queries look varied but cover a single facet. Making the model reason its way out costs hundreds of chain-of-thought tokens per query, which is latency a production search cannot absorb.
The pipeline
Stage one trains a fan-out language model with reinforcement learning against a composite reward covering groundedness, diversity and alignment. Diversity is scored with the Vendi Score; drop that term and the model degenerates into nonsense strings, according to the post. Google uses soft-GRPO, group relative policy optimization regularized with PPO, to keep the model from gaming its own reward.
Stage two runs the trained model offline to synthesize query-to-target-set pairs. Stage three distills those pairs into a compact diffusion retriever that maps a query straight to embeddings in one non-autoregressive pass. That third stage is the only one sitting in the serving path.
Reported results
The fan-out model was trained on Gemma3-4B and Qwen3-4B and evaluated in two settings: text-to-image over a fashion dataset using CLIP, and text-to-music over proprietary playlists using MuLan. Each prompt was expanded to exactly ten sub-queries. Google reports a 12x to 20x speedup over autoregressive expansion, with baseline latency climbing toward 50 seconds on large context batches against sub-second to few-second responses.
What the post does not show
Quality is described qualitatively — better than zero-shot and Best-of-N on both tasks — with no scores published in the blog text, and the post carries no limitations section. Those details may sit in the ICML 2026 paper, arXiv 2603.06397, which was not consulted for this article. No third-party reproduction of the speed figures is available so far.
FAQ
What is Retrieve-for-Train?
A Google Research method that learns how to split a search query into ten sub-queries offline with reinforcement learning, then compiles that behavior into a small diffusion model answering in one pass at query time.
How much faster is Retrieve-for-Train?
The blog post cites a 12x to 20x speedup. Autoregressive comparison systems reach nearly 50 seconds on large context batches, while Retrieve-for-Train stays between sub-second and a few seconds.
Is Retrieve-for-Train available to use?
The post announces neither code nor model weights and describes experiments with Gemma3-4B, Qwen3-4B, a fashion dataset and proprietary music playlists. Availability outside Google cannot be inferred from it.
