Usage

Option A: Gradio interactive demo

The fastest way to try OmniParser after installation is the bundled Gradio web UI. From inside the repo with the omni conda environment active:

python gradio_demo.py

This starts a local web server (default http://127.0.0.1:7860). Upload any screenshot and the interface returns a parsed image with bounding boxes and a structured element list.

Option B: Python API (demo notebook)

The repo includes demo.ipynb with full worked examples. The minimal pattern is:

from utils import get_som_labeled_img, check_ocr_box, get_caption_model_processor, get_yolo_model
import torch
from PIL import Image

# Load models
som_model = get_yolo_model(model_path='weights/icon_detect/model.pt')
caption_model_processor = get_caption_model_processor(
    model_name="florence2",
    model_name_or_path="weights/icon_caption_florence"
)

# Parse a screenshot
image = Image.open('your_screenshot.png')
draw_bbox_config = {
    'text_scale': 0.8,
    'text_thickness': 2,
    'text_padding': 3,
    'thickness': 3,
}

# Run OCR + icon detection + captioning
ocr_bbox_rslt, is_goal_filtered = check_ocr_box(
    'your_screenshot.png',
    display_img=False,
    output_bb_format='xyxy',
    goal_filtering=None,
    easyocr_args={'paragraph': False, 'text_threshold': 0.9}
)
text, ocr_bbox = ocr_bbox_rslt

labeled_img, label_coordinates, parsed_content_list = get_som_labeled_img(
    'your_screenshot.png',
    som_model,
    BOX_TRESHOLD=0.05,
    output_coord_in_ratio=True,
    ocr_bbox=ocr_bbox,
    draw_bbox_config=draw_bbox_config,
    caption_model_processor=caption_model_processor,
    ocr_text=text,
    use_local_semantics=True,
    iou_threshold=0.1,
    scale_img=False,
    batch_size=128
)

print(parsed_content_list)
# [{'type': 'text', 'content': 'File', 'bbox': [...]},
#  {'type': 'icon', 'content': 'close button', 'bbox': [...]}, ...]

Output format

parsed_content_list is a list of dicts. Each dict has:

Key	Type	Description
type	str	"text" or "icon"
content	str	OCR text or captioned icon description
bbox	list[float]	Bounding box as [x1, y1, x2, y2] in ratio coordinates (0.0 to 1.0)

label_coordinates maps numeric element IDs (displayed as overlay labels on the returned image) to their bounding boxes. Feed these IDs and descriptions to your vision-language model so it can reference elements by number.

HuggingFace live demo

No local install needed to evaluate the tool:
huggingface.co/spaces/microsoft/OmniParser-v2