VLM 自我改进：Reasoning Internalization

本子专题整理 Reasoning Internalization / Reasoning Consolidation for VLMs（VLM 自我改进）文献线：让原本需要显式推理才能产生的行为，通过

$\text{显式 reasoning experience} \;\rightarrow\; \text{训练信号} \;\rightarrow\; \theta'$

逐渐变成模型自身能力。它与端侧训练的 "trajectory → 能力固化"只差一步——现有论文几乎都发生在离线 GPU post-training，而不是设备自己从真实执行经历中在线学习；与 RSI 数据集版图互补——那边回答"数据从哪来"，本页回答"经验怎么烧回权重"。

💭 定位（截至 2026 年 9）

现有工作多数研究 Reasoning Data → Better VLM；本子专题最终想逼近的是 My Own Failure → My Own Recovery → My Own Persistent Skill：已部署的 VLM 能否从自己昂贵的多模态执行中学习，把 recovery 后的 reasoning 编译成持久的本地技能。

1. 演化轴：五阶段

flowchart LR S1["① 人给模型 reasoning

Teach VLM how to reason"] --> S2["② 模型用自己 reasoning 训练自己

Learn from my own reasoning"] S2 --> S3["③ 从错误 reasoning 学

Learn what went wrong"] S3 --> S4["④ 自己造 curriculum / reward

Self-evolving reasoning"] S4 --> S5["⑤ 贵推理压成快能力

Think expensive once

execute cheap later"]

阶段	核心公式	代表工作
① 人给 reasoning	human/GPT CoT → VLM	[LLaVA-CoT]^[1]、[Vision-R1]^[2]
② 自己的 reasoning 训练自己	$M_t \rightarrow r_t \rightarrow \text{training} \rightarrow M_{t+1}$	[R³V]^[3]、[SDRT]^[4]
③ 从错误 reasoning 学	$r_{\text{bad}} \rightarrow \text{reflect} \rightarrow r_{\text{good}}$	[VL-Rethinker]^[5]、[VisualPRM]^[6]、[ProcessThinker]^[7]
④ 自造 curriculum / reward	$M_t \rightarrow \text{tasks/reward} \rightarrow M_{t+1}$	[OpenVLThinker]^[8]、[Vision-SR1]^[9]、[VisPlay]^[10]、[Vision-Zero]^[11]、[iReasoner]^[12]
⑤ 推理摊销	long reasoning → $\theta'$ / latent	[AgentArk]^[13]、[CoT-VLA]^[14]、[ThinkAct]^[15]、[Fast-ThinkAct]^[16]

第⑤层与本方向最接近：把"贵 reasoning → 持久紧凑能力"做到了离线极致，但 teacher 是离线模型或老师模型；把 teacher 换成 past-self（设备自己过去的成功 reasoning）、把训练搬上设备，就是还没人走完的闭环。

工作	自己生成 reasoning	含失败 reasoning	有视觉	有动作轨迹	reasoning → 自身 weights	推理变快是目标	真实端侧在线训练
[LLaVA-CoT]^[1]	❌	❌	✅	❌	✅	❌	❌
[Vision-R1]^[2]	❌（teacher）	❌	✅	❌	✅	部分	❌
[SDRT]^[4]	✅	部分	✅	❌	✅	部分	❌
[R³V]^[3]	✅	✅	✅	❌	✅	❌	❌
[VL-Rethinker]^[5]	✅	✅ reflection	✅	❌	✅	❌	❌
[VisualPRM]^[6]	✅（PRM 标注）	✅ step 级	✅	❌	PRM	❌	❌
[ProcessThinker]^[7]	✅	✅ step 失败	✅ video	❌	✅	❌	❌
[OpenVLThinker]^[8]	✅	✅/RL	✅	❌	✅	❌	❌
[Vision-SR1]^[9]	✅	部分	✅	❌	✅	❌	❌
[VisPlay]^[10]	✅	RL outcome	✅	❌	✅	❌	❌
[Vision-Zero]^[11]	✅	RL outcome	✅	❌	✅	❌	❌
[iReasoner]^[12]	✅	trajectory 级	✅	❌	✅	❌	❌
[AgentArk]^[13]	✅ multi-agent	✅ critique	VLM 扩展	agent trace	✅	✅	❌
[CoT-VLA]^[14]	❌/demo	❌	✅	✅	✅	❌	❌
[ThinkAct]^[15]	✅/RL	✅ recovery	✅	✅	✅	部分	❌
[Fast-ThinkAct]^[16]	teacher	✅ recovery	✅	✅	✅	✅✅	❌
本方向要补的	✅	✅✅	✅	✅✅	✅	✅✅	✅✅

这张表基本就是 Related Work 的定位图：右下角那一格是空的。现有工作的 experience 来源是 benchmark 图像/问题或机器人示范，在 GPU 集群上离线生成 reasoning 再训练；缺的是 "真实端侧执行 → 自己的观察/推理/动作 → 失败 → 恢复 → 把恢复经验在设备上训练回自己"。

3. 精读清单（8 篇）

R³V — VLM 怎么从自己的错误 rationale 学（NAACL 2025）；
SDRT — reasoning trace 怎么 self-distill 到 adapter（arXiv 2025）；
OpenVLThinker — reasoning 模型如何产生下一轮自己的训练数据（NeurIPS 2025）；
VL-Rethinker — 怎么把 self-reflection 用 RL 固化（NeurIPS 2025）；
ProcessThinker — 多模态 reasoning trajectory 的 step-level credit assignment（ICLR 2026）；
iReasoner — trajectory-aware 的 self-evolving VLM（ACL Findings 2026）；
ThinkAct / Fast-ThinkAct — reasoning → latent plan → 快速 action，与端侧 latency 故事最接近（NeurIPS 2025 / CVPR 2026）；
AgentArk — 最干净地表达"把昂贵 test-time intelligence 编译进单模型"（arXiv 2026）。

4. 收录条目

自我改进：从自己的 reasoning 学习

Reasoning 内化与过程监督

推理摊销：贵推理 → 快执行

参考文献

[1] XU G W, JIN P, WU Z A, et al. LLaVA-CoT: let vision language models reason step-by-step[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2025. https://arxiv.org/abs/2411.10440

[2] HUANG W X, JIA B H, ZHAI Z J, et al. Vision-R1: incentivizing reasoning capability in multimodal large language models[J/OL]. arXiv preprint arXiv:2503.06749, 2025. https://arxiv.org/abs/2503.06749

[3] CHENG K Z, LI Y T, XU F Z, et al. Vision-language models can self-improve reasoning via reflection[C]//Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL). 2025. https://aclanthology.org/2025.naacl-long.447/

[4] WU G D, SONG H, WANG Y W, et al. SDRT: enhance vision-language models by self-distillation with diverse reasoning traces[J/OL]. arXiv preprint arXiv:2503.01754, 2025. https://arxiv.org/abs/2503.01754

[5] WANG H Z, QU C, HUANG Z M, et al. VL-Rethinker: incentivizing self-reflection of vision-language models with reinforcement learning[C]//Advances in Neural Information Processing Systems 38 (NeurIPS). 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/2c84844a559e4f962752570bff456ae4-Abstract-Conference.html

[6] WANG W Y, GAO Z W, CHEN L J, et al. VisualPRM400K: an effective dataset for training multimodal process reward models[C]//International Conference on Learning Representations (ICLR). 2026. https://proceedings.iclr.cc/paper_files/paper/2026/hash/d68b4e80fd0dd8ac72092b3acd418f75-Abstract-Conference.html

[7] WU J P, HAN X D, SHEN W X, et al. ProcessThinker: enhancing multi-modal large language models reasoning via rollout-based process reward[J/OL]. arXiv preprint arXiv:2606.11209, 2026. https://arxiv.org/abs/2606.11209

[8] DENG Y H, BANSAL H, YIN F, et al. OpenVLThinker: complex vision-language reasoning via iterative SFT-RL cycles[C]//Advances in Neural Information Processing Systems 38 (NeurIPS). 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/b33286edd8602ead8cb966358b01d115-Abstract-Conference.html

[9] LI Z X, YU W H, HUANG C S, et al. Self-rewarding vision-language model via reasoning decomposition[J/OL]. arXiv preprint arXiv:2508.19652, 2025. https://arxiv.org/abs/2508.19652

[10] HE Y C, HUANG C S, LI Z X, et al. VisPlay: self-evolving vision-language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2026. https://openaccess.thecvf.com/content/CVPR2026/html/He_VisPlay_Self-Evolving_Vision-Language_Models_CVPR_2026_paper.html

[11] WANG Q S, LIU B, ZHOU T Y, et al. Vision-Zero: scalable VLM self-improvement via strategic gamified self-play[J/OL]. arXiv preprint arXiv:2509.25541, 2025. https://arxiv.org/abs/2509.25541

[12] SUNIL M, VENMATHIMARAN M, KAVITHA M S. iReasoner: trajectory-aware intrinsic reasoning supervision for self-evolving large multimodal models[C]//Findings of the Association for Computational Linguistics (ACL). 2026. https://aclanthology.org/2026.findings-acl.1468/

[13] LUO Y Y, JIN Y Q, YU W C, et al. AgentArk: distilling multi-agent intelligence into a single LLM agent[J/OL]. arXiv preprint arXiv:2602.03955, 2026. https://arxiv.org/abs/2602.03955

[14] ZHAO Q Q, LU Y, KIM M J, et al. CoT-VLA: visual chain-of-thought reasoning for vision-language-action models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2025. https://openaccess.thecvf.com/content/CVPR2025/html/Zhao_CoT-VLA_Visual_Chain-of-Thought_Reasoning_for_Vision-Language-Action_Models_CVPR_2025_paper.html

[15] HUANG C P, WU Y H, CHEN M H, et al. ThinkAct: vision-language-action reasoning via reinforced visual latent planning[C]//Advances in Neural Information Processing Systems 38 (NeurIPS). 2025. https://papers.neurips.cc/paper_files/paper/2025/hash/7749f9c0d5ff109231be21e910a3ced2-Abstract-Conference.html

[16] HUANG C P, MAN Y Z, YU Z D, et al. Fast-ThinkAct: efficient vision-language-action reasoning via verbalizable latent planning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2026. https://arxiv.org/abs/2601.09708

VLM 自我改进：Reasoning Internalization

VLM 自我改进：Reasoning Internalization

1. 演化轴：五阶段

3. 精读清单（8 篇）

4. 收录条目

自我改进：从自己的 reasoning 学习

Reasoning 内化与过程监督

推理摊销：贵推理 → 快执行

参考文献

results matching ""

No results matching ""

VLM 自我改进：Reasoning Internalization

1. 演化轴：五阶段

2. Related-Work 定位表

3. 精读清单（8 篇）

4. 收录条目

自我改进：从自己的 reasoning 学习

Reasoning 内化与过程监督

推理摊销：贵推理 → 快执行

参考文献

results matching ""

No results matching ""