Skip to content

docs(site): clarify MobileWorld GUI-only scope #2901

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
simplyme0823 wants to merge 1 commit into
base: main
Choose a base branch
from
+7 −5
Open

docs(site): clarify MobileWorld GUI-only scope #2901

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions apps/site/docs/en/mobile-world-benchmark-report.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@
}
`}</style>

This report presents Midscene results on the MobileWorld benchmark. The evaluation covers 117 tasks, with a **Pass@1 score of 78.63% (92/117)**.
This is Midscene's test report for the MobileWorld benchmark. In this run, Midscene evaluated only 117 `GUI-ONLY` tasks and achieved **Pass@1 78.63% (92/117)**.

:::info About MobileWorld

Expand All @@ -102,6 +102,7 @@ This report presents Midscene results on the MobileWorld benchmark. The evaluati
| Model Name | `Gemini-3.6-Flash` |
| Midscene version | `1.10.3` |
| Device | `DockerEmulator` |
| Task scope | `GUI-ONLY` tasks only |
| Number of tasks | 117 |
| `MIDSCENE_REPLANNING_CYCLE_LIMIT` | 50 |
| MobileWorld setup | MobileWorld used a Midscene benchmark adapter that delegates Agent execution through Midscene RPC while retaining MobileWorld task setup and validation. |
Expand All @@ -117,7 +118,7 @@ The following MobileWorld validation check was adjusted for this benchmark:

## Report files

The detailed reports for all 117 tasks are listed below. Each report is a compressed, self-contained HTML file. Select “report” to open the corresponding execution trace in a new page.
The detailed reports for all 117 `GUI-ONLY` tasks are listed below. Each report is a compressed, self-contained HTML file. Select “report” to open the corresponding execution trace in a new page.

<details open className="benchmark-round">
<summary>Round 1 (117 reports · 92 PASS · 25 FAIL)</summary>
Expand Down
5 changes: 3 additions & 2 deletions apps/site/docs/zh/mobile-world-benchmark-report.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@
}
`}</style>

本文是 Midscene 针对 MobileWorld benchmark 的测试报告。本次测试覆盖 117 个任务，Midscene 取得了 **Pass@1 78.63%（92/117）** 的结果。
本文是 Midscene 针对 MobileWorld benchmark 的测试报告。本次测试仅覆盖 117 个 `GUI-ONLY` 任务，Midscene 取得了 **Pass@1 78.63%（92/117）** 的结果。

:::info 关于 MobileWorld

Expand All @@ -102,6 +102,7 @@
| Model Name | `Gemini-3.6-Flash` |
| Midscene version | `1.10.3` |
| Device | `DockerEmulator` |
| 任务范围 | 仅 `GUI-ONLY` 任务 |
| 任务数量 | 117 |
| `MIDSCENE_REPLANNING_CYCLE_LIMIT` | 50 |
| MobileWorld 工程 | MobileWorld 使用 Midscene benchmark 适配层，通过 Midscene RPC 执行 Agent，同时保留 MobileWorld 的任务初始化和验证流程。 |
Expand All @@ -117,7 +118,7 @@

## 报告文件

以下为本轮 117 个任务的详细报告。报告均为压缩后的自包含 HTML，点击“报告”会在新页面打开对应执行记录。
以下为本轮全部 117 个 `GUI-ONLY` 任务的详细报告。报告均为压缩后的自包含 HTML，点击“报告”会在新页面打开对应执行记录。

<details open className="benchmark-round">
<summary>Round 1（117 份报告 · 92 PASS · 25 FAIL）</summary>
Expand Down
2 changes: 1 addition & 1 deletion apps/site/theme/components/FeatureSections.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -872,7 +872,7 @@ export function FeatureSections() {
href={tUrl(t('featureMobileWorldBenchmarkLink'))}
score="78.6%"
title="MobileWorld Benchmark"
details={`${t('benchmark')} 78.6% · 92/117`}
details="78.6% · GUI-ONLY"

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the Pass@1 label on the benchmark card

On the homepage in both locales, the card already renders 78.6% prominently from the score prop, so this detail now repeats the percentage while removing the only indication that it represents Pass@1. Since the linked report explicitly identifies 78.63% as Pass@1, retain that metric label while adding the GUI-only scope so readers can interpret the score correctly.

Useful? React with 👍 / 👎.

description={t('featureMobileWorldBenchmarkDesc')}
positionClassName="order-2 md:order-none md:col-start-2 md:row-start-1"
/>
Expand Down
Loading