Yiheng Xu | OpenAI

From Digital Automation to Autonomous Agents

I develop scalable methods that advance AI from digital workflow automation toward fully autonomous agents. My research spans three interconnected directions:

Digital Automation via Diverse Context Understanding: Building models that can interpret across diverse digital interfaces, from unstructured, visually rich documents (LayoutLM, DiT, DocBank) to structured web pages (MarkupLM).
Scaling Machine-like Coding Agents: Designing and training coding agents that operate with machine-native efficiency through command-line and API-based interfaces, exemplified by Lemur, Qwen3 Coder, and agent harnesses like Qwen Code.
Towards Human-like Computer Use Agents: Pushing the frontier of agents that can interact in human-designed GUI environments, including Aguvis, AgentTrek, VideoAgentTrek, and Qwen2.5-VL.

Selected Publications

VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos
Dunjie Lu*, Yiheng Xu*, Junli Wang*, Haoyuan Wu, Xinyuan Wang, Zekun Wang, Junlin Yang, Hongjin Su, Jixuan Chen, Junda Chen, Yuchen Mao, Jingren Zhou, Junyang Lin, Binyuan Hui, Tao Yu
ICLR 2026
[PDF]
Qwen3 Coder
Core Contributor (Responsible for CLI and Web Agent Capability and Framework (Qwen-Code))
[Blog] [Code]
Qwen2.5-VL Technical Report
Core Contributor (Responsible for Computer/Mobile Using Agent Capability)
[Blog] [PDF] [Code]
Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Yiheng Xu*, Zekun Wang*, Junli Wang*, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, Caiming Xiong
ICML 2025
[PDF] [Project]
AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
Yiheng Xu*, Dunjie Lu*, Zhennan Shen*, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, Tao Yu
ICLR 2025 Spotlight (Top 5%)
[PDF] [Project]
Lemur: Harmonizing Natural Language and Code for Language Agents
Yiheng Xu*, Hongjin Su*, Chen Xing*, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, Zhoujun Cheng, Siheng Zhao, Lingpeng Kong, Bailin Wang, Caiming Xiong, Tao Yu
ICLR 2024 Spotlight (Top 5%)
[PDF] [Code] [Model] [Blog]
DiT: Self-supervised Pre-training for Document Image Transformer
Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei
ACM Multimedia 2022
[PDF] [Code] [Demo]
MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
Junlong Li*, Yiheng Xu*, Lei Cui, Furu Wei
ACL 2022
[PDF] [Code] [Model] [Blog]
XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
ACL 2022 Findings
[PDF] [Data]
LayoutReader: Pre-training of Text and Layout for Reading Order Detection
Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, Furu Wei
EMNLP 2021
[PDF] [Code] [Blog]
LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
Yang Xu*, Yiheng Xu*, Tengchao Lv*, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou
ACL 2021
[PDF] [Code] [Model] [Blog]
DocBank: A Benchmark Dataset for Document Layout Analysis
Minghao Li*, Yiheng Xu*, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, Ming Zhou
COLING 2020
[PDF] [Code] [Blog]
LayoutLM: Pre-training of Text and Layout for Document Image Understanding
Yiheng Xu*, Minghao Li*, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
KDD 2020
[PDF] [Code] [Model] [Blog] [PaperDigest Most Influential Papers] [ICBS 2024 Frontiers of Science Award]

Service

Conference Reviewer: AAAI, ACL, COLING, EMNLP, ICLR, ICML, MM
Journal Reviewer: IEEE Transactions on Multimedia, Neurocomputing, IJDAR