ReSpec: TOWARDS OPTIMIZING SPECULATIVE DECODING IN REINFORCEMENT LEARNING SYSTEMS Qiaoling Chen 1 2 Zijun Liu 2 3 Peng Sun 2 Shenggui Li 1 Guoteng Wang 2 Ziming Liu 2 4 Yonggang Wen 1 Siyuan Feng 2 5 Tianwei Zhang 1 ABSTRACT Adapting large language models (LLMs) via reinforcement learning (RL) is often bottlenecked by the generation stage, which can consume over 75% of the training time. Speculative decoding (SD) accelerates autoregressive generation in serving systems, but its behavior under RL training remains largely unexplored. We identify three critical gaps that hinder the na¨ıve integration of SD into RL systems: diminishing speedups at large batch sizes, drafter staleness under continual actor updates, and drafter-induced policy degradation. To address these gaps, we present ReSpec, a system that adapts SD to RL through three complementary mech- anisms: dynamically tuning SD configurations, evolving the drafter via knowledge distillation, and weighting updates by rollout rewards. On Qwen models (3B–14B), ReSpec achieves up to 4.5× speedup while preserving reward convergence and training stability, providing a practical solution for efficient RL-based LLM adaptation. 1 INTRODUCTION Reinforcement learning (RL) has become a practical route to adapt large language models (LLMs) for complex, high- level objectives, such as improved alignment, reasoning, and domain-specific skills (Yang et al., 2025; Comanici et al., 2025; Team, 2023; 2025). An RL training iteration typi- cally consists of three stages: generation, inference, and training. In the generation stage, the actor model produces rollout trajectories for a batch of prompts via autoregressive decoding; the inference stage evaluates these trajectories us- ing auxiliary models such as reward or critic networks; the training stage consumes the resulting signals to update the actor parameters. Among these stages, generation is con- sistently the dominant bottleneck (Zhong et al., 2025a). As shown in Table 1, for LLMs trained with a maximum response length of 8K tokens, generation accounts for up to 86% and 75% of the wall-clock iteration time in math and code models, respectively. This bottleneck is further exacerbated by the algorithms widely used in practice (Shao et al., 2024; Yang et al., 2024). Classic policy optimization methods such as PPO (Schul- man et al., 2017; 2015) combine trajectory-level rewards 1Nanyang Technological University, Singapore 2Shanghai Qiji Zhifeng Co., Ltd., Shanghai, China 3Tsinghua University, Beijing, China 4National University of Singapore, Singapore 5Shanghai Innovation Institute, Shanghai, China. Correspondence to: Tianwei Zhang . Proceedings of the 9 th MLSys Conference, Bellevue, WA, USA, 2026. Copyright 2026 by the author(s). with critic-based feedback to stabilize updates, while more recent group-based approaches, e.g., GRPO (Shao et al., 2024) and DAPO (Yu et al., 2025), generate multiple can- didate completions per prompt and compute updates from relative comparisons inside each group. Although group sampling improves exploration and gradient quality, it also multiplies the number of decoded tokens per prompt, further inflating the cost of the generation stage. A natural optimization to address this bottleneck is specu- lative decoding (SD) (Leviathan et al., 2023; Chen et al., 2023; Miao et al., 2025). Standard auto-regressive decoding requires one expensive forward pass of the target model per token. SD reduces this cost by introducing a lightweight drafter that proposes short token sequences, which are then validated by the target model in parallel. If the drafter and target align closely, many drafted tokens can be accepted at once, thereby reducing the number of target forwards per generated token. SD has already been widely adopted in LLM serving systems (e.g., SGLang (Zheng et al., 2024), TensorRT-LLM (NVDIA, 2024)) and commercial platforms (e.g., OpenAI (OpenAI, 2024), AWS SageMaker (AWS, 2024)). Among various SD variants, EAGLE-3 (Li et al., 2025) represents the current state of the art, achieving the highest reported speedup among lossless SD methods (Cai et al., 2024; Fu et al., 2024; Ankner et al., 2024). Accord- ingly, we adopt EAGLE-3 as the default SD algorithm in our analysis, implementation, and experiments. Research Gaps. Despite the widespread success of SD in serving systems, its behavior under RL training has never been systematically examined. Through our analysis of ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems Actor Gen. Actor Update Critic,Ref. Rew.Infer. Draft Degrade Actor Decode-1 Decode-2 Decode-3 Decode-4 Decode-5 Decode-6 Computation Bound Spec. Slowdown Time Many Update Steps… Actor Model Evo, Draft Model Stale T1 T2 T3 Time Non-SD SD Switch Adaptive SD. Server Actor model Draft model Static Spec. Draft Evo. via knowledge distillation from actor Weighted- Reward Actor Gen. Critic,Ref. Rew.Infer. Actor Update Draft Update Draft Update (a) Standard RL training workflow with speculative decoding (b) ReSpec – Optimized Workflow Update Update Figure 1. Overview of SD in RL training and our proposed ReSpec system. (a) The RL training workflow integrated with SD, highlighting three fundamental gaps. (b) The design of ReSpec, which addresses these gaps through three complementary mechanisms: 1 dynamically tuning SD configurations to match workload conditions, 2 continuously aligning the drafter with the evolving actor using on-policy signals, and 3 weighting drafter updates by rollout quality to maintain stable policy learning. Together, these mechanisms enable stable and efficient SD throughout RL training. RL workflows and the prototype integration of SD into the generation stage, we identify three fundamental gaps that explain why na¨ıve application of SD fails to deliver con- sistent gains in RL training, as illustrated in Figure 1(a). (G1) Diminishing speedups at large decoding batch sizes. When generation already runs at high GPU utilization with large batch sizes, the marginal parallelism provided by SD becomes limited and can even be offset by drafting and syn- chronization overheads (Yan et al., 2024). Consequently, a single static SD configuration cannot provide reliable speedups across diverse RL workloads, as shown in Fig- ure 3. (G2) Drafter staleness under continual actor updates. As the actor (i.e., target) model evolves with each policy update, a fixed drafter rapidly becomes misaligned with the actor distribution, leading to the loss of acceleration bene- fits, as observed in Figure 4. (G3) Practical optimization degradation under SD. Although lossless SD preserves the target distribution in expectation, we observe that in prac- tice, the interaction of non-deterministic verification paths, drafter staleness (G2), and variance amplification under non- stationary policy updates degrades the effective training signal. These compounding factors reduce trajectory qual- ity and diversity, leading to measurable reward degradation during RL training, as demonstrated in Figure 5. Opportunities. Beyond these gaps, we identify two key op- portunities in the RL generation stage that can be leveraged to make SD both stable and efficient. First, the genera- tion stage exhibits strong skewness: within a batch, most sequences terminate early while a small fraction continue much longer, resulting in a time-varying active batch size. Because the efficiency of SD depends critically on the batch size, this skewness offers an opportunity to adapt SD config- urations dynamically rather than relying on static hyperpa- rameters, as shown in Figures 6 and 7. Second, RL rollouts inherently produce rich, on-policy diagnostic signals that are routinely logged but never reused, including per-step target, draft logits, and rewards. These signals provide a valuable, untapped supervision source for maintaining drafter align- ment with the evolving actor policy, enabling continuous adaptation without introducing additional overhead to the sampling process. Motivated by the above-mentioned gaps and opportunities, we introduce ReSpec, a system that adapts speculative de- coding to the unique demands of RL training. ReSpec pro- vides three targeted mechanisms, each directly addressing one of the gaps (Figure 1 (b)). Specifically, (1) for G1, we design Adaptive Speculative Decoding Server. It dynam- ically selects and applies speculative-decoding configura- tions based on lightweight profiling and runtime workload signals. (2) For G2, we introduce Drafter Evolution via Knowledge Distillation with a Online Learner. To keep the drafter aligned with the continually updated actor, we evolve the drafter using on-policy distillation from the actor. This distillation pipeline continuously transfers the actor’s distributional knowledge into the lightweight drafter so that drafted proposals remain relevant as the policy changes. (3) For G3, we design Reward-Weighted Adaptation. To mitigate the practical optimization degradation described above, we weight drafter updates by rollout quality, which prioritizes high-reward trajectories during adaptation. Since the actor is optimized toward high reward, these trajectories are most representative of the actor’s evolving distribution, enabling the drafter to track the post-update policy more ac- curately and reducing the risk of degraded training signals. Our contributions are summarized as follows: ★We conduct the first systematic study of SD in end-to- end RL training of LLMs. Our analysis reveals three critical gaps, including diminishing speedups, drafter staleness, and practical optimization degradation, that explain why na¨ıve SD integration fails to provide consis- tent acceleration. ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems Generation Inference Training Math 83%–86% 1.7%–2.4% 12.3%–14.6% Code 70.9%–75.5% 11.4%–14% 13.1%–15.1% Table 1. Time breakdown during RL iterations for 7B models (max response length 8K tokens). ★We identify two underexplored opportunities in RL gen- eration: (1) skewed generation workloads that enable dy- namic adaptation of SD configurations, and (2) on-policy diagnostic signals that can be repurposed as supervision for continuous drafter alignment. ★We design ReSpec, the first system that adapts SD to RL training. ReSpec comprises two tightly coupled compo- nents: an Adaptive SD Server that performs profiling- guided and runtime-tuned SD scheduling, and an Online Learner, which continuously maintains drafter alignment through three complementary mechanisms: evolving the drafter via knowledge distillation from the actor, weight- ing updates by rollout rewards to preserve policy quality, and performing asynchronous, replay-buffered updates to avoid blocking generation. ★We implement and evaluate ReSpec on Qwen models (3B–14B), showing that it preserves training stability and reward convergence while achieving up to 4.5× speedup over standard RL training, demonstrating its practicality for RL training pipelines. 2 BACKGROUND AND RELATED WORK 2.1 Reinforcement Learning for LLMs Workflow. RL for LLMs typically proceeds in iterations that consist of three stages: generation, inference, and train- ing. In the generation stage, an actor model produces rollout trajectories for a batch of prompts. This stage contains a prefill (prompt encoding) step followed by autoregressive decoding of tokens until termination. In the inference stage, auxiliary models (reference, reward, critic) evaluate the gen- erated rollouts by performing forward passes to produce scalar signals or logits used by the update. Finally, in the training stage, the actor consumes these signals to compute losses and apply parameter updates. Algorithms. Classic policy optimization methods such as PPO (Schulman et al., 2017; 2015) combine trajectory-level rewards with critic-based, action-level feedback to reduce the variance of updates. Recent production-oriented recipes, e.g., GRPO (Shao et al., 2024), DAPO (Yu et al., 2025), leverage group-based sampling: for each prompt, the actor generates a group (many candidate completions) and up- dates are computed from relative comparisons inside the group. Group sampling improves the exploration and gradi- ent signal in model optimization but increases the number of decoded tokens per prompt, amplifying the cost of the gen- eration stage and motivating efforts to accelerate decoding without altering the sampling distribution. Bottleneck. The generation stage consistently dominates the end-to-end RL iteration time. As shown in Table 1, for LLMs trained with a maximum response length of 8K to- kens, generation accounts for up to 86% and 75% of the total time for math and code models, respectively. This imbal- ance highlights that decoding, rather than model updating or reward evaluation, is the primary performance bottleneck in practical RL pipelines. Existing Frameworks. Numerous frameworks aim to ac- celerate RL post-training for LLMs. Early systems (Harper et al., 2025; Hu et al., 2024; Lei et al., 2024; Yao et al., 2023) focus on orchestrating multi-stage RL workflows, while later efforts introduce architectural and scheduling optimiza- tions—e.g., hybrid or multi-controller designs (Sheng et al., 2025; Wang et al., 2025), fused or parallelized stages (Zhong et al., 2025b; Mei et al., 2024), and asynchronous post- training for long-tail rollouts (Fu et al., 2025; Gao et al., 2025; He et al., 2025). However, these frameworks largely overlook the generation stage and its optimization opportu- nities. In contrast, ReSpec explicitly targets this bottleneck by integrating SD into RL training. Concurrent Work on SD for RL Training. Several concur- rent efforts have explored accelerating RL rollout generation via speculative decoding. Beat-the-Long-Tail (Shao et al., 2025) proposes distribution-aware speculative decoding that targets long-tail rollout sequences; SPEC-RL (Liu et al., 2025) accelerates on-policy RL via speculative rollouts; and RhymeRL (He et al., 2025) reuses historical trajecto- ries as draft proposals. These approaches primarily rely on non-parametric or history-based drafting mechanisms. In contrast, ReSpec maintains a parametric drafter that is con- tinuously aligned with the non-stationary RL actor through reward-weighted knowledge distillation. FastGRPO (Zhang et al., 2025) focuses on GRPO-specific optimization settings with concurrency-aware speculative decoding and online draft learning, but does not address reward-weighted dis- tillation to prevent policy degradation, nor adaptive SD configuration tuning based on runtime workload dynamics. ReSpec is complementary to these approaches and addresses a broader set of challenges through its integrated Adaptive Server and Online Learner design. 2.2 Speculative Decoding Standard autoregressive decoding requires one target- model forward per token, which is computationally ex- pensive. Speculative decoding (SD) mitigates this by using a lightweight drafter to propose token sequences, which are then verified by the expensive target model in batches(Leviathan et al., 2023; Chen et al., 2023). Subse- quent works extend this paradigm along multiple dimen- ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems wisdom patience patience is Target Gen. Target: validate in parallel Draft: Inference in auto-regressive is patience power the power the is seed key seed key power the of to of to seed key success life Reward Patience is power Patience is the seed Patience is the key to success Patience is the key to wisdom Patience is the key of life 0.3 0.3 1.0 0.8 0.7 Validate Organize as one batch Figure 2. EAGLE-3 workflow. The target model generates one token, while the draft model produces multiple candidates using hidden states. The target verifies all candidates in a single forward pass and accepts four tokens (“the key to success”) with only two forward passes. 1 2 4 8 16 32 64 Batch Size 0.0 0.5 1.0 1.5 2.0 Throughput Speedup Ratio T = 0.0 T = 0.2 T = 0.4 T = 0.6 T = 0.8 T = 1.0 Figure 3. Speedup ratio under different batch sizes and temperature on H100 and Qwen2.5-7B-instruct for the MT- Bench dataset. sions: Medusa (Cai et al., 2024) increases proposal paral- lelism, Cascade Speculative Drafting (Chen et al., 2024) introduces multi-stage chaining, and EAGLE-series meth- ods (Li et al., 2024a;b; 2025) progressively enhance fea- ture utilization, draft-tree construction, and parallelization, achieving state-of-the-art inference efficiency. Speculative decoding procedure. We adopt a representa- tive speculative sampling procedure similar to the recent SOTA method EAGLE-3 (Li et al., 2025) as our reference implementation. Starting from a realized prefix T1:j, SD repeats a two-stage cycle. (1) Drafting: the drafter autore- gressively proposes a block ˆTj+1:j+k = (ˆtj+1, . . . , ˆtj+k), where ˆt represents individual tokens; (2) Validation: the tar- get evaluates the proposed block in a batched forward and accepts drafted tokens sequentially according to an accep- tance test; if a token is rejected, it is replaced by sampling from a residual distribution derived from the target logits. Acceptance rule (single-token). For a single drafted token ˆt with drafter probability q(ˆt) and target probability p(ˆt), the acceptance probability is (omitting conditions on prefixes): Pr � accept ˆt � = min � 1, p(ˆt) q(ˆt) � . (1) If rejected, a replacement token is sampled from the residual distribution: r(x) = max(0, p(x) −q(x)) � y max(0, p(y) −q(y)), (2) where x, y denote single tokens. It guarantees that the marginal sample follows p (Leviathan et al., 2023). We emphasize that lossless speculative decoding with correct verification (i.e., rejection sampling) is unbiased in expec- tation: the rollout distribution is identical to that of the target model under standard autoregressive decoding. Our work does not claim that speculative decoding violates this theoretical guarantee. Multi-token drafting and accumulated mismatch. When drafting multiple tokens sequentially, each proposed token depends on the previously accepted tokens. We calculate the variance of the acceptance probability p q for sampled sequences as: VarT ∼q �� t∈T p(t) q(t) � = � t∈T � 1 + Dχ2(p(t)||q(t)) � −1, (3) where Dχ2 is the Chi-squared divergence. If Dχ2(p(t)||q(t)) is approximated by a constant δ, the variance (1 + δ)|T | −1 increases exponentially with the sequence length. As the target model distribution evolves, this divergence typically widens, amplifying the variance of acceptance probabilities: some tokens become increasingly unlikely to be drafted ( p(t) q(t) ↑), while others are drafted often but rarely accepted ( p(t) q(t) ↓). Acceptance length. A key practical metric in SD is the acceptance length, defined as the number of consecutive to- kens in a drafted block that are accepted by the target model before the first rejection. Longer acceptance lengths gen- erally lead to higher speedups because more tokens can be validated per forward step, but also increase the likelihood of multi-token distribution mismatch, as described above. Speculative decoding cost model. Let Cp denote the aver- age per-token cost of the target model under na¨ıve decoding (i.e., one forward pass per token), and Cq denote the aver- age per-token cost of the drafter model, with Cq ≪Cp. Suppose we draft blocks of k tokens, where the average acceptance rate per token is r, i.e., the expected fraction of drafted tokens ultimately accepted. If validating a block of k tokens requires target-model computation roughly propor- tional to k, but can be parallelized with an efficiency factor α, then the effective target cost for validating the entire block is approximately kCp/α. Under these assumptions, the expected cost per accepted token in SD is estimated as: CostSD ≈kCq + kCp/α rk = Cq + Cp/α r . (4) 3 CHALLENGES OF APPLYING SD IN RL Applying SD inside RL training is attractive because the generation stage frequently dominates the iteration time ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems 0 20 40 60 80 100 2 3 4 Accept Length Figure 4. Acceptance length of the draft model during 100 RL steps with Qwen2.5-7B-instruct model and math dataset. As training progresses, the EAGLE-3 drafter quickly becomes stale and its acceptance length drops. (Table 1). However, prior work on online SD has focused on inference-serving scenarios, where draft models are updated online to track shifting request distributions (Zhou et al., 2023; Liu et al., 2023). In contrast, RL training requires the drafter to adapt to the evolving actor policy itself, posing three practical gaps not addressed by existing methods. GAP 1: diminishing speedup at large batch sizes. In RL training, we enlarge the batch size to fit GPU utilization in the decoding phase. However, when the batch size in decoding is already large, GPUs are typically operating near high utilization through straightforward batching of independent sequences. Therefore, the extra parallelism that SD provides yields little marginal benefit. Besides, SD introduces additional overheads, as illustrated in Equation 4. This added draft cost and verification synchronization offset, and even exceeded the speedup. As illustrated in Figure 3, speedups from SD shrink as batch size increases. GAP 2: SD staleness during RL training. In RL training, the actor parameters are continually updated. A drafter distilled from an earlier snapshot may become misaligned with the evolving actor, leading to lower acceptance length and thus diminishing the benefits of SD. Figure 4 illustrates this effect: the acceptance length of the EAGLE-3 drafter decreases as training advances. GAP 3: practical optimization degradation under SD. As established in Section 2.2, lossless speculative decoding with correct rejection-sampling verification preserves the target model’s rollout distribution in expectation. However, we observe that in practice, naively applying SD during on- policy RL training leads to measurable reward degradation. Importantly, this degradation does not arise from a violation of the theoretical sampling guarantee, but rather from the interaction of several practical factors unique to the RL training loop. First, non-deterministic verification paths: during the verifi- cation stage, the forward pass operates as a chunked prefill with masked tree attention. Differences in GPU kernel im- plementations (e.g., split-K strategies in GeMM, split-KV in attention with varying atomic merging) can introduce 0 50 100 150 200 0.2 0.3 0.4 Reward With EAGLE-3 W/O EAGLE-3 Figure 5. Evolution of rewards over 200 RL steps for Qwen2.5- 7B on the math dataset. Na¨ıve application of EAGLE-3 leads to a measurable drop in reward, illustrating practical optimization degradation when SD interacts with on-policy RL training dynam- ics. 18:17 18:23 18:24 18:25 18:26 18:27 18:28 18:29 18:30 18:38 18:39 18:40 18:41 18:42 18:44 18:45 18:46 18:50 18:51 18:52 2 4 8 16 32 Running Requests Figure 6. Number of active sequences during decoding in Qwen2.5- 7B RL training (Math). It illustrates the skewed generation work- load: most sequences finish quickly while a few persist, causing fluctuations in effective batch size. numerical non-determinism. While individually negligi- ble, these discrepancies accumulate over long rollouts and across many training iterations. Second, drafter staleness (compounding Gap 2): as the actor evolves, the drafter’s pro- posals become increasingly misaligned. Although each indi- vidual verification step remains correct, the stale drafter con- strains the effective exploration of the policy by repeatedly proposing similar continuations, reducing the diversity of collected trajectories. Third, variance amplification under non-stationary updates: as shown in Eq. (3), the variance of acceptance probabilities grows exponentially with draft length. Under continual policy updates, this variance dispro- portionately affects trajectory quality, producing a skewed effective training signal for policy optimization even though the expected distribution remains unbiased. Together, these practical effects influence the quality and diversity of the training signal rather than introducing formal sampling bias. Empirically, we observe that naively applying EAGLE-3 in our RL runs produces a measurable drop in reward after approximately 100 update steps (Figure 5), consistent with these compounding practical factors degrading the optimiza- tion dynamics. ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems 4 KEY INSIGHTS 4.1 Generation-stage Skewness and Its Consequences for Speculative Decoding We observe that the generation stage in RL training exhibits pronounced skewness: within a single decoding batch, most sequences finish quickly while a small fraction continue for much longer. Figure 6 shows this pattern concretely for Qwen-2.5–7B: the number of active requests fluctuates heav- ily, occasionally spiking above 16, spending much of the run near 8, and then decaying to a long tail near 1. Because autoregressive decoding is sequential, these short-lived se- quences drop out early, and the active batch size decreases as decoding proceeds, producing a highly non-uniform work- load and intermittent under-utilization of compute. Crucially, we find that this time-varying active batch size substantially changes the cost–benefit trade-offs of SD. SD performance is governed by three key hyperparameters: (1) the number of speculative rounds s, (2) the target branching factor t, i.e., number of candidate branches validated in parallel, and (3) the draft length per round n. Importantly, our measurements reveal that the optimal SD configuration is not fixed, but depends strongly on the active batch size. When the batch size is small, aggressive drafting (large s, n, t) can yield clear gains, whereas at larger batch sizes the same configuration may even hurt performance due to overheads outweighing the benefits. For instance, in our experiments (shown in Figure 7), the best SD configuration at batch size 2 achieves 1.46× speedup, while the same setting degrades to 0.76× at batch size 32. This coupling highlights that static SD hyperparameters cannot deliver consistent benefits across the dynamic regime of RL training. Instead, adaptive strategies that tune SD parameters online based on the current active batch size are required to fully realize the potential acceleration. 4.2 On-policy Signals and Knowledge Distillation for Drafter Alignment Knowledge distillation provides a principled way to align the predictive distribution of a smaller student model with that of a larger teacher. Prior studies (Zhou et al., 2023; Liu et al., 2023) have shown that this framework is particularly effective for SD in online serving, where incoming request distributions shift over time and the draft model must con- tinually adapt. In these settings, distillation is applied online to fit the varying data distribution. In contrast, SD within RL training presents a fundamentally different challenge. The request distribution is relatively stable, but the target model itself evolves during training. This dynamic teacher–student relationship means that static draft models quickly become misaligned, leading to reduced acceptance rates and diminished decoding efficiency. (3,5,8) (3,5,16)(3,5,32) (3,8,8) (3,8,16)(3,8,32) (5,8,8) (5,8,16)(5,8,32) 1 2 4 8 16 32 64 Batch Size 1.237 1.358 1.270 1.191 1.307 1.323 1.145 1.358 1.411 1.200 1.249 1.319 1.273 1.351 1.372 1.240 1.230 1.464 1.114 1.230 1.236 1.156 1.335 1.241 1.231 1.229 1.246 1.175 1.251 1.144 1.255 1.114 1.113 1.063 1.206 1.135 1.301 1.239 1.129 1.383 1.361 1.115 1.204 1.369 1.129 1.114 0.957 0.795 1.113 1.003 0.852 0.968 1.009 0.755 0.843 0.735 0.536 0.837 0.801 0.555 0.786 0.634 0.448 0.6 0.8 1.0 1.2 1.4 1.6 Latency Speedup (s, t, n): s = #speculative rounds, t = target branching factor, n = draft length per round Figure 7. Latency speedup of EAGLE-3 (temperature = 0.6) on Qwen2.5-7B-instruct for the MTBench dataset. Each cell shows speedup for a specific (s, t, n) configuration, showing that optimal SD depends strongly on active batch size. Fortunately, we observe an underexploited opportunity in the verification step of SD: during validation, the system has access to dense, on-policy diagnostic signals, such as the target model’s per-step logits, drafter’s contemporaneous log-probabilities, and scalar trajectory rewards. These sig- nals are produced naturally during generation and reflect the actor’s current behavior under the actual sampling policy, which enables the construction of distillation objectives that directly align the draft model with the current target model, effectively tracking its evolving distribution. By exploit- ing such information, we can continuously refine the draft model to increase accept length and stabilize RL training. 5 SYSTEM DESIGN 5.1 System Overview Inspired by the above insights, we present ReSpec, a novel system that efficiently adapts SD to RL. Figure 8 presents its architecture, which consists of two major components: (1) The Adaptive Server accelerates the generation stage with SD. It integrates two modules: t he Solver, which searches for and selects the optimal SD hyperparameter combinations, and the Scheduler, which monitors the ac- tive batch size at runtime and dynamically switches SD configurations while managing the KV cache. (2) The Online Learner maintains the draft model through online training. It incorporates three modules: The Reward- Weighted Knowledge Distillation algorithm, which distills knowledge from the evolving target model to keep the draft model aligned and ensure high acceptance length across training steps, and the Async Update Overlap mechanism, which overlaps draft model training with RL generation. Workflow. 1 ReSpec starts with a user-specified config- uration, such as batch size and temperature. The Solver ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems Decode-1 Decode-2 Decode-3 Decode-4 Decode-5 Decode-6 T1 T2 T3 Solver S1 S2 S3 Scheduler Adaptive Server Draft Target Configs Target Training Sample Buffer 1. Sample 2. Rewards 3. Log-probs Reward-Weighted KD Async. Update Overlap Draft Training Online Learner Updating System Overview Time Offline Profiles Figure 8. Workflow of ReSpec during RL training. The system iteratively generates responses, stores rollout samples with associ- ated rewards, updates the drafter via reward-weighted knowledge distillation with asynchronous overlap, and applies improved spec- ulative decoding in the next RL step to accelerate and stabilize training. provides optimal SD parameters (s, t, n) for the given de- coding conditions, and the Scheduler adjusts them online without additional overhead. 2 The target model generates responses and updates its weights following the standard RL training loop. 3 Generated samples, along with log- probabilities and rewards, are stored in the Sample Buffer. 4 The Online Learner selectively draws samples to train the draft model using the reward-weighted KD algorithm, with Async Update Overlap strategies to maximize effi- ciency. 5 Updated draft weights are pushed back to the inference engine, enabling the next RL training step to pro- ceed with improved speculative decoding. 5.2 Adaptive Server SD accelerates generation by drafting multiple candidate to- kens in parallel and verifying them against the target model. However, its efficiency depends heavily on runtime con- ditions such as the active batch size, which fluctuates sig- nificantly during RL training. When GPU utilization is already saturated (e.g., large active batch sizes), enabling SD can add redundant overhead; conversely, when batch sizes shrink, SD offers substantial speedup. To balance these dynamics, ReSpec introduces an Adaptive Server that dy- namically switches between speculative and non-speculative execution modes according to the current system state. Figure 9(a) shows the standard SD workflow without adap- tation: for a batch size of 4, each request independently undergoes draft and validate stages, resulting in redundant compute when the GPU is already well utilized. In contrast, Figure 9(b) depicts the adaptive mode: when the batch size is large, the Scheduler opts for pure target decoding to main- tain throughput; as the active batch size drops (e.g., from 4 to 2), the system transitions to the speculative state. Dur- ing this transition, the Scheduler extends the previous non- Switch to Spec Req-1 Req-2 Req-3 Req-4 Req-1 Req-2 Req-3 Req-4 Time Saved Extend Once Validate Target Fast Decode Draft Decode (a) Naïve (b) Adaptive Server-Scheduler Figure 9. Workflow comparison between standard speculative de- coding (a) and the proposed Adaptive Server (b). speculative decoding state to build an updated KV cache, then resumes normal speculative decoding to maximize par- allelism at lower utilization. The Adaptive Server consists of two key components: (1) a Solver that searches for efficient SD configurations via offline profiling, and (2) a Scheduler that controls runtime switching between modes with minimal overhead. Solver: Offline Profiling Guided Search. To determine efficient SD configurations, ReSpec employs a profiling- based solver. As shown in Figure 7, the runtime perfor- mance of SD is highly sensitive to the choice of hyper- parameters, including the number of speculative rounds s, branching factor t, and number of drafted tokens per round n. The optimal configuration varies across different batch sizes due to the evolving active batch during RL training. We first conduct an offline profiling stage, where we bench- mark the execution time of draft and target models under various SD hyperparameter settings and batch sizes. The results are used to fit a predictive performance model that estimates the throughput speedup of different SD configu- rations as a function of the active batch size. This profiling is lightweight and performed once before RL training. At runtime, the solver consults the offline-derived performance model and the observed active batch size to dynamically select the SD configuration expected to maximize speedup. Scheduler. The Scheduler is the runtime controller of the Adaptive Server. It determines when and how SD should be activated during generation, while minimizing system overhead. This is non-trivial in RL training, where active batch sizes shrink rapidly due to data skewness, and the ben- efits of SD vary across decoding stages. Switching between speculative and non-speculative execution incurs overhead. To achieve zero-overhead runtime switching, we model the ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems decoding phase as a two-state process: spec-enabled and non-spec. Each request carries a lightweight flag indicating whether SD is active. The Scheduler tracks the state of the current batch and applies lightweight transitions: • Non-spec →Spec. When the Solver advises switching on SD, the Scheduler promotes the current decode batch into a spec-enabled batch by reusing the prefill interface. This allows the draft model to generate candidates without modifying the decoding kernel. • Spec →Non-spec. If SD is no longer beneficial, the Scheduler simply discards speculative metadata (e.g., cached candidates) and continues with regular decoding. The Scheduler operates in a closed loop with the Solver. At runtime, it monitors active batch size and uses these signals to decide whether SD should remain enabled. 5.3 Online Learner Algorithm 1 summarizes the online learner workflow used throughout our experiments. During rollouts, we run SD with the current draft (qθ) and verify each candidate with the target model (p). For every rollout, we store the input and response together with the target logits and the scalar reward. The online learner extracts distillation targets from the rollout buffer, accumulates them in a replay buffer (Q), and performs periodic reward-weighted updates every (I) iterations. These updates (i) keep the draft aligned with the target distribution to preserve high acceptance rates in SD, and (ii) bias the draft toward behaviors that empirically yield higher RL reward. The distillation data is derived directly from on-policy trajectories generated during RL training and requires no additional data collection. Concretely, for Qwen- 3B, each drafter update uses 32 samples with approximately 64K tokens; for Qwen-7B and Qwen-14B, each update uses 28 samples, corresponding to about 223K and 112K tokens, respectively. 5.3.1 Reward-Weighted Knowledge Distillation A central mismatch between RL and SD is that the target (p) evolves, whereas classical KD assumes a fixed teacher. Treating all collected rollouts equally (standard KD) can therefore pull the draft toward low-quality behavior because many rollouts are erroneous or low-reward. We perform reward-weighted KD. For a rollout sample ((x, y)) with per-step target distributions (p(· | x, y 0) is a reward-based weighting function (by default (w(r) = r); in practice we normalize and clip Algorithm 1 RL-Integrated Speculative Draft Update Input: Target model (p(· | x)), draft model (qθ(· | x)), rollout buffer (B) storing tuples ((x, y, log p, r)), update interval (I), replay buffer (Q). Initialization: Pre-train (qθ) with offline KD on warmup data; set Q ←[], i ←0; while RL training not converged do Sample a mini-batch ((x, y, log p, r)) from rollout buffer (B) Run speculative decoding with draft (qθ) and verify outputs using target (p) Collect token-level alignment info ((log qθ, log p)) for each position Store ((x, y, log qθ, log p, r)) in replay buffer (Q) i ←i + 1 if i mod I = 0 then Ltot ←0 for each sample (x, y, log qθ, log p, r) ∈Q do Construct soft targets p(· | x, y