Speculative decoding fundamentally reframes autoregressive generation as a cooperative inference problem between a fast, lightweight draft model and a slower, higher-capacity target model. The core principle is that the draft model proposes a sequence of candidate tokens, and the target model verifies all proposed tokens in a single forward pass—accepting those that match its own distribution and rejecting the rest. This allows the target model to effectively produce multiple tokens per forward pass, breaking the one-token-per-step bottleneck.

Selective speculative decoding dynamically decides, on a per-iteration basis, whether speculative decoding is beneficial at all. The computing system partitions output tokens into two portions: a first portion computed via speculative decoding using one or more drafting models, and a second portion computed directly at the primary model without speculative decoding. This context-sensitive logic avoids the overhead of draft model invocation in cases where speculative decoding would yield low acceptance rates, making the system adaptive rather than statically committed to a single decoding path.

Self-speculative decoding eliminates the need for a separate draft model entirely. Samsung's approach compresses or removes specific transformer blocks from the base model—identified through parameter analysis—and then fine-tunes the resulting lighter model to serve as the draft head. The fine-tuned model shares weights with the target model, reducing memory overhead while preserving the speculative verification mechanism. This single-model approach avoids the memory bandwidth and KV-cache overhead of maintaining two separate model instances, making it key for edge and on-device deployment.

Prefill requests are split into equal-sized chunks, and a hybrid micro-batch is constructed with a single prefill chunk filling the compute headroom left by a maximal number of decode requests. This prevents prefill computation from monopolizing GPU resources and blocking decode steps, which improves both throughput and time-to-first-token latency simultaneously. The mismatch between prefill-phase GPU saturation and decode-phase compute underutilization is identified as the root cause of inference inefficiency.

A fine-tuned auxiliary model predicts output response length before inference begins. These length predictions are used to build better-balanced batches and implement a priority scheduling policy. An error-handling mechanism compares predicted versus actual output length at runtime, updating scheduling decisions dynamically. Southeast University's approach uses a fine-tuned Qwen2-1.5B model to predict the output token count of a target Llama3-8B model on each request, and a deep reinforcement learning power selection algorithm further reduces energy consumption while meeting latency SLAs.

Microsoft Technology Licensing LLC is the most prolific filer in this dataset, with at least three distinct patent families covering speculative decoding with guidance vector feedback, selective speculative decoding, and chunked prefill with decode-maximal batching. Google LLC holds multiple patent families on model selection routing and multi-model bridging, with multi-jurisdictional coverage including WO, US, India, and CN filings. Samsung Electronics, Qualcomm Incorporated, FriendliAI Inc., and Snowflake Inc. are also active, alongside Chinese academic and industrial groups including Southeast University, Shandong University of Finance and Economics, Tianjin University, and others.