Smol2Operator：用於電腦操作（Computer Use）的輕量級 GUI 代理後訓練指南與模型

### Background and Challenge: The Rise of Local "Computer Use" With Anthropic's introduction of Computer Use and the development of various OS-level agents, enabling AI to directly operate computer screens, browsers, and applications has become the hottest current research direction. However, traditional GUI agents typically rely on large, expensive, and closed-source cloud-based vision-language models (VLMs). This not only brings high API costs but also raises serious privacy and latency concerns — after all, continuously uploading screenshots containing personal privacy or corporate secrets to the cloud is unacceptable to many users. To balance the dual demands of "local execution" and "high accuracy," Hugging Face has launched **Smol2Operator**. This is a lightweight GUI agent post-training solution designed specifically for computer use, aiming to demonstrate that through careful post-training, small open-source vision-language models can also exhibit excellent practical capability in GUI operation tasks. --- ### Smol2Operator's Core Technology and Post-Training Recipe Smol2Operator is based on Hugging Face's lightweight vision-language model family (such as SmolVLM) and transforms general visual understanding capability into precise computer operation instructions through the following key post-training steps: 1. **Action Space Mapping**: The model must learn to convert natural language instructions (e.g., "Open Chrome and search for today's weather") into concrete mouse and keyboard operations. Smol2Operator trains the model to output structured JSON format or specific tokens, such as `click(x, y)`, `type("text")`, `scroll(direction)`, and to accurately locate pixel coordinates on the screen. 2. **Supervised Fine-Tuning (SFT)**: The research team used large datasets of GUI interaction trajectories (such as OSWorld and Mind2Web) for fine-tuning. These datasets contain chains of "screenshot → current task → next optimal action," helping the model build intuitive recognition of operating system interfaces and web page elements (buttons, input fields, dropdown menus). 3. **Reinforcement Learning and Environment Feedback (RL & Environment Feedback)**: SFT alone cannot fully resolve the problem of cumulative errors caused by a single wrong step leading to cascading failures. Smol2Operator introduces reinforcement learning (such as GRPO or PPO variants), allowing the model to explore autonomously in a simulated OS environment. Positive rewards are given when the model successfully completes a task; penalties are applied when it fails (such as clicking the wrong button or getting stuck in an infinite loop). This significantly enhances the model's self-correction capability and multi-step planning ability. --- ### Advantages of Local Deployment: Privacy, Low Latency, and Low Cost The launch of Smol2Operator offers developers several notable advantages: * **Ultimate Privacy Protection**: All screenshots and operation trajectories are processed locally; sensitive data (such as passwords, financial statements, and personal messages) never leaves the device. * **Low Latency and Real-Time Response**: Small models (typically between 2B and 5B parameters) run extremely fast on local GPUs (or even some high-end CPUs/NPUs), enabling smoother and more responsive GUI interactions. * **Zero API Costs**: Developers can run agents without restriction, without worrying about high bills from frequently calling cloud-based VLMs. --- ### Open Source and Community Contribution In the spirit of open source, Hugging Face has released not only the Smol2Operator model weights, but has also fully open-sourced its post-training code, dataset processing tools, and evaluation benchmarks. This allows researchers and developers to easily build upon this foundation for further development, fine-tuning the model for specific enterprise internal software or specific workflows (such as automated QA testing and RPA process optimization), and advancing the thriving ecosystem of local AI agents.