Midscene MobileWorld Benchmark Report

This is Midscene's test report for the MobileWorld benchmark. In this run, Midscene evaluated only 117 GUI-ONLY tasks and achieved Pass@1 78.63% (92/117).

About MobileWorld

MobileWorld is an Android Agent benchmark designed for realistic mobile scenarios. It covers cross-app workflows, long-horizon tasks, user interaction, and tool-augmented tasks, with reproducible Android environments used to validate Agent execution results.

Run configuration

Field	Value
Test date	2026-07-28
Model Name	`Gemini-3.6-Flash`
Midscene version	`1.10.3`
Device	`DockerEmulator`
Task scope	`GUI-ONLY` tasks only
Number of tasks	117
`MIDSCENE_REPLANNING_CYCLE_LIMIT`	50
MobileWorld setup	MobileWorld used a Midscene benchmark adapter that delegates Agent execution through Midscene RPC while retaining MobileWorld task setup and validation.
Validation notes	A MobileWorld validator was aligned with the task intent. The affected case is listed below.

Validation Condition Updates

The following MobileWorld validation check was adjusted for this benchmark:

Change	Affected case
The previous validator required a full five-day interval between `created_at` and `expires_at`. For example, if the start date is the 16th, “five days later” is the 21st; however, differences in the converted time of day can make the interval shorter than 5 × 24 hours, causing the previous rule to fail. The validator now accepts intervals longer than 4 days and no longer than 5 days.	`MastodonNewFilterTask`

Report files

The detailed reports for all 117 GUI-ONLY tasks are listed below. Each report is a compressed, self-contained HTML file. Select “report” to open the corresponding execution trace in a new page.

Round 1 (117 reports · 92 PASS · 25 FAIL)

#	Task	Status	Report
1	AcceptMeetingTask	PASS	report
2	AdjustBrightnessMaximumTask	PASS	report
3	AdjustBrightnessMinimumTask	PASS	report
4	AdjustFontIconMaximumTask	PASS	report
5	AdjustFontIconMinimumTask	PASS	report
6	BidFileRenameTask	PASS	report
7	CVEmailTask	PASS	report
8	CancelMeetingTask	PASS	report
9	CartInfoNotificationTask	PASS	report
10	CartManagementTask	FAIL	report
11	ChangeWallpaperTask	PASS	report
12	CheckCartPriceTask	PASS	report
13	CheckConferenceAndSendSmsTask1	PASS	report
14	CheckConferenceAndSendSmsTask2	FAIL	report
15	CheckConferenceDurationTask	PASS	report
16	CheckConferenceLocationTask	FAIL	report
17	CheckDeduplicatedEventsTask	PASS	report
18	CheckDepartTimeTask	PASS	report
19	CheckEventTimeTask	PASS	report
20	CheckGithubInfoTask	PASS	report
21	CheckInterviewTimesTask	FAIL	report
22	CheckInvoiceTask1	PASS	report
23	CheckInvoiceTask2	PASS	report
24	CheckInvoiceTask3	PASS	report
25	CheckPuchasedItem	PASS	report
26	CheckRegistrationTask	PASS	report
27	CheckSetMeetTimeTask	PASS	report
28	ChromeSearchBeijingWeatherTask	PASS	report
29	CloseFlightModeTask	PASS	report
30	CountFileLinesTask	PASS	report
31	DownloadSendReceiptTask	PASS	report
32	GoogleMapsAlibabaPhoneContactTask	PASS	report
33	GoogleMapsAlibabaSouthNeighborTask	PASS	report
34	GraduationMassEmailTask	PASS	report
35	InvoiceReceiptCopyAskUserTask	FAIL	report
36	InvoiceReceiptCopyTask	PASS	report
37	ItemCheckoutTask	PASS	report
38	LocalFileManagementTask	PASS	report
39	LocalFileManagementTask2	PASS	report
40	MastodonAddBookmarkTask	PASS	report
41	MastodonAddFeaturedHashtagsTask	FAIL	report
42	MastodonAdjustTootsTask	PASS	report
43	MastodonCalendarMultiMemosTask	FAIL	report
44	MastodonChangeHeaderTask	PASS	report
45	MastodonChangeLanguageTask	FAIL	report
46	MastodonConditionalFavoTask	PASS	report
47	MastodonCreateListTask	PASS	report
48	MastodonCreateMemoTask	PASS	report
49	MastodonExportFollowsTask	PASS	report
50	MastodonFavoriteTootsTask	PASS	report
51	MastodonFilterLanguageTask	PASS	report
52	MastodonFollowTask	PASS	report
53	MastodonGetServerInfoTask	FAIL	report
54	MastodonImportMutedUsersTask	PASS	report
55	MastodonInviteTask	PASS	report
56	MastodonMallPurchaseCommodityTask	PASS	report
57	MastodonMallShareOrderTask	PASS	report
58	MastodonManageHashtagsTask	PASS	report
59	MastodonManageMultiListTask	PASS	report
60	MastodonMattermostPostNoticeTask	PASS	report
61	MastodonMultiInviteTask	PASS	report
62	MastodonNewFilterTask	FAIL	report
63	MastodonNewPostTask	PASS	report
64	MastodonOpenAutomatedDeletionTask	PASS	report
65	MastodonPinTootsTask	PASS	report
66	MastodonPostEditedPhotoTask	PASS	report
67	MastodonPostPollTask	PASS	report
68	MastodonRemoveBookmarkTask	PASS	report
69	MastodonReplyTask	PASS	report
70	MastodonReportTask	PASS	report
71	MastodonRevisePhotoAltTask	PASS	report
72	MastodonRevisePollTask	PASS	report
73	MastodonSavePhotosTask	PASS	report
74	MastodonServerInfoReportTask	FAIL	report
75	MastodonShareLocationTask	FAIL	report
76	MastodonUnfollowTask	PASS	report
77	MastodonUpdateContactsTask	PASS	report
78	MattermostBudgetApprovalPipelineTask	PASS	report
79	MattermostCreateChannelTask	PASS	report
80	MattermostCustomerFeedbackAnalysisTask	PASS	report
81	MattermostDeadlineReconciliationTask	PASS	report
82	MattermostEmailTask	PASS	report
83	MattermostIncidentEscalationTask	PASS	report
84	MattermostProjectHandoverTask	FAIL	report
85	MattermostProjectStatusReportTask	FAIL	report
86	MattermostReadingGroupTask	FAIL	report
87	MattermostReplyToMessageTask	PASS	report
88	MattermostResourceConflictResolutionTask	FAIL	report
89	MattermostSendFileTask	FAIL	report
90	MattermostShiftCoverageTask	PASS	report
91	MattermostTechnicalDebtTriageTask	PASS	report
92	MattermostVisualInstructionResponseTask	FAIL	report
93	OpenFlightModeTask	PASS	report
94	PhotoManagementTask	FAIL	report
95	ReadQwen3PaperTask1	PASS	report
96	ReadQwen3PaperTask2	FAIL	report
97	ReadQwen3PaperTask3	FAIL	report
98	ReadQwen3PaperTask4	PASS	report
99	ReadQwen3PaperTask5	FAIL	report
100	RecentTotalExpenseTask	PASS	report
101	RequestCarpoolingTask	PASS	report
102	ReviewPaperEmailTask	FAIL	report
103	SMSManagement	PASS	report
104	ScheduleCoffeeTimeViaSmsTask	PASS	report
105	ScheduleLunchViaSmsTask	PASS	report
106	SearchItemAndCheckoutTask	PASS	report
107	SendFormsTask	PASS	report
108	SendInterviewEmailTask	PASS	report
109	SendInterviewInvitationTask	FAIL	report
110	SendWaiverTask	PASS	report
111	SetAlarmTask	PASS	report
112	SharePhotosTask	PASS	report
113	SuggestPaperTask	PASS	report
114	SumFileLinesTask	PASS	report
115	TakeSelfieTask	PASS	report
116	TextArrivalTimeTask	FAIL	report
117	ThanksgivingPrepTask	PASS	report

#Midscene MobileWorld Benchmark Report

#Run configuration

#Validation Condition Updates

#Report files

Midscene MobileWorld Benchmark Report

Run configuration

Validation Condition Updates

Report files