PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents

Abstract

TL;DR: PACT generates prompt-only rollouts and uses expert traces for trace-conditioned RL and component-aware SFT, with prompt-only RL anchoring.

Multi-turn tool-use agents must reason, call tools, and adapt to observations across several interaction turns. Post-training such agents is challenging, as reinforcement learning often suffers from sparse rewards and weak credit assignment despite matching the prompt-only inference setting, while supervised fine-tuning on expert traces provides dense process supervision but can over-constrain the model to fixed trajectories. To tackle this, we propose PACT, a Privileged trAce Co-Training framework for multi-turn tool-use agents. The key idea is to use expert traces only as training-time optimization signals rather than rollout-time hints. PACT keeps rollout generation prompt-only, then uses expert traces to guide optimization through two complementary signals: a trace-conditioned RL surrogate that evaluates prompt-only rollouts under expert-trace context, and a component-aware SFT loss that provides annealed process supervision over reasoning and tool-call components. To reduce over-reliance on the training-only trace context, PACT incorporates prompt-only anchoring with standard prompt-only RL updates. Experiments on FTRL, BFCL, and ToolHop show that PACT consistently outperforms strong SFT- and RL-based baselines. Further analyses support the complementarity of the two training objectives and the benefits of component-aware supervision and prompt-only anchoring. These results highlight the effectiveness of privileged trace co-training for multi-turn tool-use learning.

Motivation

Full-trace SFT can over-imitate

Expert traces are useful, but multi-turn tool-use tasks admit multiple valid paths. Treating a single trace as a fixed target can overfit teacher-specific decisions.

Prompt-only RL is under-guided

Rollout RL matches inference, but sparse trajectory-level rewards give weak credit assignment for intermediate reasoning, tool choices, and arguments.

Hints create input mismatch

Using expert traces as rollout hints changes the training input condition. PACT instead reserves traces for optimization while sampling rollouts from the original prompt.

Method

PACT decouples exploration from privileged supervision. The policy samples prompt-only rollouts, evaluates those rollouts under an expert-trace context, and supervises selected model-controllable parts of the expert trace. Trace-conditioned RL serves as an auxiliary optimization surrogate.

1

Prompt-only rollout

PACT generates rollouts from the task prompt and accumulated interaction history without expert traces. Rewards and advantages come from these prompt-only rollouts.

2

Trace-conditioned RL

Both current and old policies evaluate the same sampled rollout under expert-trace context to compute token-level policy ratios, while retaining its original rewards and advantages.

3

Component-aware SFT

Reasoning supervision shortens at semantic boundaries, while complete tool-call spans retain supervision with a decreasing loss weight. Tool observations serve only as context.

Prompt-only anchoring. A prompt-level gate selects standard prompt-only RL with probability q, or trace-conditioned RL plus component-aware SFT otherwise. The main experiments use q = 0.4 and an SFT coefficient of 0.1. Anchored prompts receive no trace-conditioned likelihood evaluation or SFT supervision.

Annealed process supervision. Reasoning prefixes shrink from L₄ to L₀ within each assistant turn; at L₀, reasoning remains in the input context without direct SFT loss. Tool names and all argument tokens remain supervised as their imitation weight decreases. At inference, the agent uses only the task prompt and ordinary interaction history.

Results

PACT achieves the highest overall average on both Qwen3 model sizes, improving over MatchTIR by 2.12 points on Qwen3-1.7B and 3.14 points on Qwen3-4B.

Qwen3-1.7B Avg 23.08 vs. 20.96 MatchTIR

Qwen3-4B Avg 35.80 vs. 32.66 MatchTIR

Qwen3-4B FTRL Solve-F1 36.60 vs. 32.03 best baseline

Qwen3-4B BFCL Avg 21.26 vs. 18.83 best baseline

Method	FTRL			BFCL				ToolHop	Avg
Method	Solve-R	Solve-P	Solve-F1	Multi-Turn	Search	Memory	Avg	AC	Avg
Qwen3-1.7B
Vanilla	21.66	20.78	19.72	13.00	1.00	8.60	7.53	25.63	17.63
SFT	22.36	17.89	17.66	13.38	2.50	8.60	8.16	23.32	16.38
GRPO	21.57	17.37	18.13	11.88	3.50	9.25	8.21	25.13	17.16
FTRL	22.78	22.06	21.06	12.62	6.00	11.18	9.93	26.43	19.14
ToolRL	22.79	17.93	17.99	13.30	2.50	8.39	8.06	25.83	17.29
CHORD	22.68	19.14	18.50	12.62	3.50	6.45	7.52	27.14	17.72
OPSD	25.24	20.60	21.15	12.62	3.00	9.46	8.36	25.53	18.35
MatchTIR	24.12	21.50	21.43	14.00	4.50	9.03	9.18	32.26	20.96
SFT→MatchTIR	23.39	19.64	19.72	13.63	5.50	11.18	10.10	31.06	20.29
PACT	28.33	26.94	22.93	14.88	7.00	13.63	11.84	34.47	23.08
Qwen3-4B
Vanilla	32.14	31.66	28.60	23.50	12.00	16.34	17.28	31.86	25.91
SFT	28.19	29.32	24.67	22.62	4.00	12.90	13.17	30.25	22.70
GRPO	33.12	31.80	30.84	22.00	9.00	15.48	15.49	34.47	26.93
FTRL	31.98	33.24	31.34	22.38	9.00	16.77	16.05	43.02	30.14
ToolRL	32.59	28.98	28.28	23.00	11.00	17.42	17.14	34.27	26.56
CHORD	32.69	29.84	29.88	22.00	12.50	15.48	16.66	37.69	28.08
OPSD	33.61	25.36	27.61	22.50	12.50	13.55	16.18	43.82	29.20
MatchTIR	36.10	30.60	31.60	24.38	17.50	14.62	18.83	47.54	32.66
SFT→MatchTIR	37.17	30.40	32.03	23.62	17.50	13.76	18.29	47.34	32.55
PACT	42.41	34.69	36.60	27.50	18.00	18.28	21.26	49.55	35.80

Complete main results from the paper; scores are percentages. AC denotes Answer Correctness. BFCL Avg averages Multi-Turn, Search, and Memory. The final Avg is the equal-weight mean of FTRL Solve-F1, BFCL Avg, and ToolHop AC. Main PACT runs use 24 updates and q = 0.4.

Further analyses

Complementary objectives. With anchoring disabled (q = 0), trace-conditioned RL plus component-aware SFT reaches 33.21 FTRL Solve-F1 and 20.44 BFCL Avg, compared with 30.18 and 19.38 when the same SFT is paired with prompt-only RL. Under the main configuration (q = 0.4), trace conditioning improves these metrics by 4.73 and 4.81 points over prompt-only RL with the same SFT and anchoring.

Complete trace context and separate schedules. Keeping SFT fixed, removing expert reasoning or tool responses from the RL context reduces FTRL Solve-F1 from 33.21 to 26.12 or 25.94 (q = 0). With tool-call loss-scale annealing fixed, semantic-prefix annealing for reasoning reaches 33.21, compared with 30.60 for fixed reasoning supervision and 29.86 for token-count reasoning annealing.

Longer training and different teachers. At 48 updates, Qwen3-1.7B PACT reaches 27.17 FTRL Solve-F1, 12.62 BFCL Avg, and 40.50 ToolHop AC, exceeding MatchTIR by 3.75, 4.24, and 2.41 points. On a shared set of 1,096 prompts, a Qwen3-4B student trained with either Qwen3-8B or Qwen3-235B traces outperforms the matched SFT and SFT→MatchTIR baselines.

BibTeX

@article{du2026pact,
  title={PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents},
  author={Du, Zhenbang and Luo, Jun and Zheng, Zhiwei and Yuan, Xiangchi and Xia, Kejing and Shi, Dachuan and Jin, Qirui and He, Qijia and Zou, Shaofeng and Liang, Yingbin and Lee, Wenke},
  journal={arXiv preprint arXiv:2606.16215},
  year={2026}
}