OmniParser
Category: OSS Agent
Cost: Free (open source)
License: Mixed (icon_detect: AGPL via YOLO; icon_caption models: MIT; repo code: CC-BY-4.0)
Official site: microsoft.github.io/OmniParser
GitHub: github.com/microsoft/OmniParser
HuggingFace Demo: spaces/microsoft/OmniParser-v2
Paper: arxiv.org/abs/2408.00203
What it is
OmniParser is a vision-based GUI parsing system developed by Microsoft Research. It converts raw screenshots into structured, agent-readable representations by detecting interactable UI elements (buttons, icons, fields) and extracting their functional semantics. The output is a parsed screenshot with bounding boxes and numeric IDs paired with a list of element descriptions.
It acts as a perception layer that can plug into any large vision-language model (GPT-4o, Phi-3.5-V, Llama-3.2-V, etc.) so the model can ground its actions to actual screen regions rather than guessing pixel coordinates.
What it does
- Detects interactable regions on a screenshot using a fine-tuned YOLO-based detection model (icon_detect).
- Captions each region with a functional description using a Florence-2 based captioning model (icon_caption_florence).
- Returns structured output: bounding boxes, element IDs, and text/icon descriptions ready for an agent to act on.
Key components
| Component | Model | License |
|---|---|---|
| icon_detect | YOLO-based fine-tune | AGPL |
| icon_caption_florence | Florence-2 fine-tune | MIT |
| Weights hosting | HuggingFace Hub (microsoft/OmniParser-v2.0) | see above |
Benchmarks
OmniParser v2 outperforms GPT-4V baselines on ScreenSpot, Mind2Web, and AITW benchmarks.
Safety and ToS notes
- The icon_detect model inherits the AGPL license from YOLO. AGPL requires any network-served software using it to release source code. Do not use it in a closed SaaS product without legal review.
- Model weights are hosted on HuggingFace. Verify individual model license files inside each model folder before commercial use.
- No paid tier. No API rate limits. Runs entirely on your own hardware.