Skip to content

docs: overhaul README with prerequisites, badges, and cleaner structure #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Krivoblotsky wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: overhaul README with prerequisites, badges, and cleaner structure #7

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
179 changes: 99 additions & 80 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,87 +1,120 @@
[![MacPaw Research](https://pbs.twimg.com/profile_banners/3993798502/1720615716/1500x500)](https://research.macpaw.com)

![PyPI - Version](https://img.shields.io/pypi/v/macapptree) [![arXiv](https://img.shields.io/badge/arXiv-2510.16051-b31b1b.svg)](https://arxiv.org/abs/2510.16051)
<p align="center">
<img src="https://img.shields.io/pypi/v/macapptree" alt="PyPI version">
<img src="https://img.shields.io/pypi/pyversions/macapptree" alt="Python versions">
<img src="https://img.shields.io/badge/platform-macOS-lightgrey" alt="Platform">
<img src="https://img.shields.io/github/license/MacPaw/macapptree" alt="License">
<a href="https://arxiv.org/abs/2510.16051"><img src="https://img.shields.io/badge/arXiv-2510.16051-b31b1b.svg" alt="arXiv"></a>
</p>

# macapptree

`macapptree` is a Python package that extracts the accessibility tree of a macOS applications screen in JSON format. It also provides an option to capture screenshots of the application, including labeled bounding boxes with different colors representing various element types. This tool is useful for accessibility testing, UI automation, and visual debugging. You can also capture the accessibility of all currently running, visible apps and include the Dock and top Menu Bar.
`macapptree` is a Python package that extracts the **accessibility tree** of macOS applications as structured JSON. It can also capture screenshots of application windows with labeled bounding boxes colored by element type — useful for accessibility testing, UI automation, agent training data, and visual debugging.

You can capture a single app, a specific set of apps, or everything currently visible on screen — including the Dock and Menu Bar.

---

## Features

- **Accessibility Tree Extraction of Apps**: Retrieve the accessibility hierarchy of a single or multiple macOS applications in JSON format.
- **Screenshot Capture**: Capture a cropped screenshot of the application window and full screen.
- **Labeled Visual Output**: Generate a segmented screenshot with bounding boxes highlighting UI elements, colored by their types.
- **Accessibility tree extraction** — retrieve the full AX hierarchy of one or many macOS apps as JSON
- **Single-app or multi-app capture** — target apps by bundle ID or auto-discover all visible apps
- **Window screenshots** — cropped screenshots of individual application windows
- **Segmented visual output** — annotated screenshots with color-coded bounding boxes per element type
- **Menu Bar & Dock support** — include system UI in the captured tree
- **Full-screen annotated output** — overlay all element bounding boxes onto a full-screen capture

---

## Installation
## Prerequisites

### macOS Accessibility Permissions

`macapptree` uses the macOS Accessibility API (`AXUIElement`). The process running it **must be granted Accessibility access**:

1. Open **System Settings → Privacy & Security → Accessibility**
2. Add your terminal application (e.g. Terminal, iTerm2) or the Python interpreter
3. Toggle it **on**

To install `macapptree`, use `pip`:
Without this permission, the library will return empty trees silently.

### Python 3.8+, macOS only

---

## Installation

```bash
pip install macapptree
```
> Note: This package requires macOS and Python 3.8+.

---

## Usage

The library provides two primary functions:
* `get_tree`: Extracts the accessibility tree of a macOS app.
* `get_tree_screenshot`: Extracts the accessibility tree and generates screenshots.

### Example
## Python API

```python
from macapptree import get_tree, get_tree_screenshot, get_app_bundle

# Get the app bundle identifier (for example: "com.apple.TextEdit")
bundle = get_app_bundle("TextEdit")
# Resolve bundle ID from app name
bundle = get_app_bundle("TextEdit") # → "com.apple.TextEdit"

# Retrieve the accessibility tree, a cropped screenshot, and a segmented screenshot
tree, im, im_seg = get_tree_screenshot(bundle)
# Accessibility tree only
tree = get_tree(bundle)

# `tree`: JSON-like structure of the accessibility elements
# `im`: Cropped screenshot of the application window
# `im_seg`: Labeled screenshot with bounding boxes indicating UI elements
# Accessibility tree + screenshots
tree, window_img, segmented_img = get_tree_screenshot(bundle)

# tree – dict, the full accessibility hierarchy
# window_img – PIL.Image, cropped screenshot of the app window
# segmented_img – PIL.Image, same screenshot with bounding boxes drawn
```

### CLI Example (multi-app)
---

Capture the accessibility of all currently running and visible apps (with upper menu and dock included):
```python
## CLI

### Capture specific apps

```bash
python -m macapptree.main \
--oa shots/dump.json \
--os shots \
--all-apps \
-a com.apple.Safari com.google.Chrome \
--oa output/dump.json \
--os output/screenshots \
--include-menubar \
--include-dock
```

Or specify apps explicitly:
```python
### Capture all visible apps

```bash
python -m macapptree.main \
-a com.apple.Safari com.google.Chrome \
--oa shots/dump.json \
--os shots \
--all-apps \
--oa output/dump.json \
--os output/screenshots \
--include-menubar \
--include-dock
```

### Output
### CLI options

* **tree**: A Python dictionary representing the accessibility hierarchy.
* **im**: A cropped `PIL.Image` object of the app window.
* **im_seg**: A `PIL.Image` object with bounding boxes drawn on top, colored based on the element type.
| Flag | Description |
|------|-------------|
| `-a` / `--apps` | One or more app bundle IDs (space-separated) |
| `--all-apps` | Auto-discover all currently visible apps |
| `--oa` | Output path for the accessibility JSON file (**required**) |
| `--os` | Directory for cropped and segmented screenshots |
| `--max-depth` | Maximum depth of the accessibility tree |
| `--include-menubar` | Also capture the top Menu Bar |
| `--include-dock` | Also capture the Dock |

---

## Output

### JSON structure

## Example Tree Output
Each element in the tree has the following shape:

```json
{
Expand All @@ -95,54 +128,40 @@ python -m macapptree.main \
"position": "0.00;0.00",
"size": "586;476",
"enabled": false,
"bbox": [
0,
0,
586,
476
],
"visible_bbox": [
0,
0,
586,
476
],
"children": [
{
"id": "422f5e0df37aa872341d3b6a47faf320",
"name": null,
"role": "AXScrollArea",
"description": null,
"role_description": "scroll area",
"value": null,
"absolute_position": "214.00;175.00",
"position": "0.00;56.00",
"size": "586;420",
"enabled": false,
"bbox": [
0,
56,
586,
476
],
"visible_bbox": [
0,
56,
586,
476
],
"children": [
...
"bbox": [0, 0, 586, 476],
"visible_bbox": [0, 0, 586, 476],
"children": [...]
}
```

### Screenshot output

| File | Description |
|------|-------------|
| `<AppName>_<window>_cropped.png` | Cropped screenshot of the window |
| `<AppName>_<window>_segmented.png` | Annotated screenshot with bounding boxes |
| `full_screen.png` | Full-screen capture |
| `full_screen_annotated.png` | Full-screen capture with all element overlays |

---

## License

This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.

---

## Contributing

Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

---

## Citation
```

If you use `macapptree` in your research, please cite:

```bibtex
@article{garkot2025guirilla,
title={GUIrilla: A Scalable Framework for Automated Desktop UI Exploration},
author={Garkot, Sofiya and Shamrai, Maksym and Synytsia, Ivan and Hirna, Mariya},
Expand All @@ -152,8 +171,8 @@ This project is licensed under the MIT License. See the [LICENSE](LICENSE) file
}
```

## MacPaw Research
---

Visit our site to learn more 😉
## MacPaw Research

https://research.macpaw.com
Built by the [MacPaw Research](https://research.macpaw.com) team.
Loading