Communicative On-Policy Distillation

Ask the teacher.
Then learn to
act alone.

On-policy distillation needs teacher logits, and black-box models do not expose them. COPD talks to the teacher in plain language instead, and distills the advice the student itself obtained into a policy that no longer needs it.

What makes the asking hard?
One exchange, then no channel.
consultationStudent-initiated, free-form

Studentthe main content area appears mostly empty

TeacherMake content visible by default.

Keptthe successful trajectory, replies included

The reply is supervision only because the student asked for it.

Recorded Visual task from the paper. See both deliveries ↗

59.8adviser-free average, COPD-4B

2.46×the gain of the strongest black-box OPD baseline

Compare the methods

Communicative On-Policy Distillation: Distilling Black-Box Teacher Capabilities through Communication for Long-Horizon Tasks

Lirui Luo · Kelong Mao · Xinwei Yang · Rongqing Li · Luyu Chen · Kieran Wong · Yudong Guo · Xinrui Wang · Jiayin Zhu · Gu Simiu · Sulong Xu · Cong Fang

Peking University · JD.com · Sichuan University · Beijing Institute of Technology · Renmin University of China · The Hong Kong University of Science and Technology

Advice is easy to get.
Keeping it is not.

A student can finish a task with a teacher on the line and still fail the same task alone. COPD treats both halves as one problem.

First challenge

When should it ask,
and what should it say?

In a long task neither the moment of need nor the information worth sharing can be specified in advance. The student has to decide both while it acts.

when+what+free text
Second challenge

Does a successful
consultation transfer?

Finishing a task with help does not mean the student can repeat it unaided. The advice has to end up inside the policy, not beside it.

success with help→success alone

Communication becomes the distillation channel.

The adviser only returns text, so no logits are needed. Because the student asks on its own, the exchange is also usable as on-policy supervision for what the student actually tried.

Collect what worked.
Score fresh attempts with it.

One policy decides when to talk and what to do. Successful consulted trajectories come back as privileged context for the student's own new responses.

01

Ask

During training the student may spend a step on a request instead of an environment action.

a_t ∈ A_env ∪ A_comm
u_t ~ π_θ(· | h_t)
Free text, no templates or fixed schedule
02

Keep the successes

Only trajectories that pass the native task check and actually used the adviser enter the reference corpus.

requestsrepliesactionsobservations
Environment-verified, student-generated
03

Distill on-policy

Two views of the same weights score a fresh adviser-free prefix; only the privileged view sees the successful trajectory.

log pprivileged − log pdeployable
Jointly with task-reward PPO

No teacher logits. The adviser answers in text and never acts in the environment, so the method applies to any chat-only black-box model.

No separate asker. Communication and task behaviour share one parameter set, trained from the same environment return.

Open the training framework ↗
COPD pipeline: consulted episodes enter a reference corpus, fresh adviser-free attempts are scored by a privileged view, and the update is combined with task-reward PPO.
Paper Figure 2.

Best average of the 4B systems
without an adviser.

Every policy is scored with adviser access disabled on the same 128-task panels. CAMP-I is the task-reward reference; ActionOPD is the strongest black-box OPD baseline.

CAMG-II average score Higher is better · 0–100
COPD (ours)59.8
CAMP-I54.2
ActionOPD51.2
ROPD49.6
OmniOPD49.1
GAD (w/ warm-up)48.9
OLIVE47.2
Frozen 4B45.3
020406080
All five domains +
Paper Table 1 · adviser-free task success (%)
PolicyShopCodingAutoResearchGeneralVisualAvg.
Frozen Qwen3.5-4B77.327.346.139.836.045.3
CAMP-I85.232.038.368.846.554.2
ActionOPD93.024.241.458.640.051.2
ROPD73.427.343.063.340.849.6
OmniOPD80.525.843.055.540.849.1
GAD (w/ warm-up)85.222.739.857.839.048.9
OLIVE79.724.239.853.139.047.2
COPD-4B92.236.755.566.448.259.8
COPD-9B93.038.357.068.849.461.3
Teacher (direct)100.078.183.691.450.480.7

Where does the gain land?

Against CAMP-I, the bars gain 17.2 points on AutoResearch and lose 2.4 on General, where CAMP-I stays ahead.

Shop

85.2
92.2

+7.0 pts

Coding

32.0
36.7

+4.7 pts

AutoResearch

38.3
55.5

+17.2 pts

General

68.8
66.4

−2.4 pts

Visual

46.5
48.2

+1.7 pts

CAMP-I COPD-4B Visual is a judged grade, not a binary success rate.

Having a channel is not
the same as learning.

Three checks separate the pieces: access to the adviser, the content of its replies, and the components of the update.

Without the adviser, the level holds only for COPD.Gain from adviser access · bars show the adviser-available score
COPD-4B+1.6
w/o OPSD+14.9
Initial student+9.8

Adviser-free scores: COPD-4B 59.8, w/o OPSD 47.5, initial student 45.3.

The words matter, and they have to fit the task.Full reference, replies removed, replies moved
Full reference59.8
Replies removed56.9
Replies moved54.6

Deleting the replies costs 2.9 points; moving them to another task costs 5.2.

Every component carries part of the gain.

Removing the distillation update costs the most. Neither the communication channel alone nor imitation of the same trajectories recovers the full recipe.

Ablations Adviser-free average
COPD59.8
w/o OPSD47.5
w/o communication52.1
w/ SFT instead of OPSD55.4

The same brief,
delivered twice.

A Visual task asks for a two-day training-event page. The trained policy delivers it once with an adviser mounted and advertised, and once with the channel removed.

The two pages differ in detail and agree in structure: the behaviour the consultation established does not depend on reaching an adviser at test time.

Both deliveries ↗
Two deliveries of the same Visual brief by the trained policy, one with the adviser available and one with the channel removed.
Paper Figure 8.

Both students improve
on both benchmarks.

Evaluated without an adviser on Terminal-Bench 2.1 and SWE-Bench Pro V2. The 9B policy closes to within 0.3 points of a larger same-generation model on the two-suite average.

Terminal-Bench 2.1Resolution rate (%) · axis 0–50
4B
Initial model23.1
COPD28.0
9B
Initial model23.1
COPD33.0

Larger same-generation reference: Qwen3.5-35B-A3B at 33.8.

SWE-Bench Pro V2Resolution rate (%) · axis 0–50
4B
Initial model11.2
COPD16.5
9B
Initial model23.1
COPD27.5

Larger same-generation reference: Qwen3.5-35B-A3B at 27.4.

Talk to the teacher.
Keep the lesson.

COPD makes the student's own consultation the unit of distillation, so a chat-only teacher is enough.

BibTeX +
@article{luo2026copd,
  title = {Communicative On-Policy Distillation: Distilling Black-Box Teacher Capabilities through Communication for Long-Horizon Tasks},
  author = {Lirui Luo and Kelong Mao and Xinwei Yang and Rongqing Li and Luyu Chen and Kevin Wong and Yudong Guo and Xinrui Wang and Jiayin Zhu and Gu Simiu and Sulong Xu and Cong Fang},
  year = {2026}
}