Assistant Benchmark

Assistant Benchmark is a public scorecard for assistants people can text. It is useful if you are choosing a personal assistant, studying agent product design, or looking for concrete examples of where an assistant can act beyond chat. The project publishes common tasks, scores from real account conversations, and links to the underlying evidence; its About page identifies David Pawlan as the person who writes the tasks and sets the scores, with Autumn Moulder involved in testing and scoring.

Start with the dimension index, then open a capability that matters to you. Each dimension has a published task and 1–10 scoring anchors. The online task dimension, for example, asks an assistant to book a Chicago hotel near the Loop under a nightly budget with free cancellation. A strong result requires the right dates, actual rates, the stated constraints, and either a completed booking or a staged checkout that waits for payment approval. Its ranking includes short outcomes and links to individual runs, making it possible to inspect what happened instead of treating the number as self-explanatory.

The breadth of the dimensions is useful for architecture questions too: browser execution, travel, purchases, email, routines, connected apps, memory, permissions, multi-step work, and restraint demand different product capabilities. Choose the rows that match your intended workflow, then read a few run records to see where tools, accounts, handoffs, and user approval affected the result. The site also separates untested products from scored ones and marks tasks that do not apply to a product.

Treat the leaderboard as a set of observed case studies rather than a universal prediction. Each dimension uses one published task, scoring follows written anchors after real use, and coverage varies by assistant. A product may have changed since its logged run; the About page says the team accepts retest requests. Public-opinion quotes are a separate signal, not a substitute for the task evidence. If your decision is consequential, repeat the relevant workflow with your own accounts, constraints, and approval rules.

Example use cases

Suggested application — shortlist an assistant: Compare the online-task, purchasing, and permissions dimensions before trialing candidates for delegated shopping.
Suggested application — design an agent evaluation: Adapt the published hotel task into a private test with explicit date, budget, cancellation, and payment-approval checks.
Suggested application — diagnose a failure mode: Read several linked run records to distinguish weak search, unavailable integrations, browser errors, and poor handoff behavior.

Sources

Source image: Assistant Benchmark.

Assistant Benchmark.