In this guide · 7 sections
Short answer
Arena ranks models by blind human votes on which answer people prefer. SWE-bench Verified measures how many of 500 real coding issues a model can fix. Artificial Analysis combines several tests with price and speed. None tests your own work, so try two or three models on 10 real tasks first.
Three AI leaderboards, three different questions
Each leaderboard answers a different question. Reading them side by side avoids treating any one score as 'the best AI'.
| Leaderboard | What it measures | How | Useful for |
|---|---|---|---|
| Arena (arena.ai) | Which answer people prefer | Blind votes between two anonymous models | General chat and writing quality |
| SWE-bench Verified | % of real coding issues resolved | 500 human-checked GitHub issues from Python projects | Coding assistants and agents |
| Artificial Analysis | Combined intelligence score, cost, speed | Runs a set of benchmarks itself and measures APIs | Comparing price and performance |
Arena (formerly LMArena): human preference votes
Arena started as Chatbot Arena at UC Berkeley, became LMArena and was renamed Arena in January 2026. You type a prompt, two anonymous models answer, and you vote for the better one or call a tie. According to Arena's FAQ, only votes cast while the models are anonymous count, and votes feed a Bradley-Terry rating, similar to chess Elo.
Arena also offers a 'style control' view that adjusts for answer length and formatting, so that long, heavily formatted answers do not win on presentation alone. Its leaderboard policy says a model needs at least 1,000 votes before it is listed.
Pros
- Millions of real prompts, not a fixed test set
- Hard to overfit, since prompts keep changing
- Separate boards for text, coding, vision and more
Cons
- Measures what people like, not whether facts are right
- Voters are self-selected users, not your customers
- Top models can sit within overlapping confidence ranges
Look at the confidence interval, not just the rank. If two models' ranges overlap, the leaderboard cannot tell them apart.
SWE-bench Verified: can the model fix real code?
SWE-bench, from researchers at Princeton, gives a model a real GitHub issue from a Python project and checks whether its code change makes the project's tests pass. The SWE-bench Verified page describes Verified as a human-filtered subset of 500 instances, built with OpenAI, where annotators checked that each task is clear and solvable.
Three details matter when reading vendor claims:
- The agent matters, not just the model. The full leaderboard compares whole systems, from simple loops to multi-attempt setups. The 'bash only' view runs every model in the same minimal mini-SWE-agent environment for a fairer model comparison.
- It is Python and open source. Your stack (WordPress plugins, Shopify Liquid, an old PHP system) may behave differently.
- Versions are not always comparable. The site notes that results from mini-SWE-agent release 1.x and 2.x are not necessarily comparable.
Artificial Analysis and other combined indexes
Artificial Analysis runs a set of benchmarks itself and publishes an 'Intelligence Index' alongside cost per task, output speed in tokens per second, latency and context window for over 250 models. That makes it useful for a different question: which model gives enough quality for the price and speed you need.
For high-volume tasks such as classifying emails, tagging products or summarising support tickets, a mid-ranked model that is much cheaper and faster may be the sensible choice; the cost-per-task column shows how wide the price gap can be. For a few complex analyses a month, the top model's cost per task matters less.
Keep in mind that these indexes weight the benchmarks they choose. A model can rank high overall and still do poorly on a task you care about, such as reading scanned Chinese invoices or replying in Malay.
How to test AI models on your own work
A small, honest test is enough to choose a staff AI tool. Our ChatGPT vs Claude vs Gemini comparison covers plans and data settings; this is how to compare quality.
Pick 10 real tasks
Use work you actually do: a customer email reply, a quotation summary, a policy rewrite, a spreadsheet formula, a product description. Remove personal data first.
Write down what 'good' looks like
Correct facts, right tone, Singapore spelling and context (GST, PayNow, CPF), no invented details. Agree this before you see the answers.
Run the same prompts on two or three models
Use the business plans you would actually buy, so data settings match. Paste the outputs into a sheet without the model names.
Score blind
Ask two colleagues to score each answer from 1 to 5. This copies Arena's blind method, but with your tasks and your standards.
Check cost and controls
Compare price per user or per million tokens, admin controls, data retention and whether your data is used for training.
Repeat the test when you renew, not every time a new model tops a chart. Our AI model updates roundup tracks releases so you do not have to.
Do it yourself or get help?
Choosing a chat assistant for staff is a decision most owners can make with the 10-task test above. Where help pays off is building AI into a workflow: a website assistant, automatic email triage or document processing, where model choice, cost per request and guard-rails all matter. Our guide to AI chatbots for websites explains that use case.
Our AI Website Assistant (S$790) answers customer questions from your own website and documents, with guard-rails and lead capture. Workflow Automation (S$390 per workflow) connects AI steps to tools you already use, such as email, forms and spreadsheets. We test the model choice on your real questions before go-live.
Frequently asked questions
Yes. The platform began as Chatbot Arena at UC Berkeley, became LMArena, and was renamed Arena in January 2026, now at arena.ai. People vote between two anonymous model answers, and the votes produce a Bradley-Terry rating similar to chess Elo. It measures human preference rather than factual accuracy.
It measures the percentage of 500 human-checked real GitHub issues from Python projects that an AI system can resolve, judged by whether the project's tests pass after the model's code change. Scores depend on both the model and the agent setup around it, so compare like with like.
There is no single answer from leaderboards. The top models score close to each other, and your tasks, language mix, data settings and budget matter more. Test two or three models on 10 real tasks, score the answers blind, and compare price and admin controls before buying seats.
Benchmarks can be overfitted, and test questions can leak into training data. Arena's policy now requires at least 1,000 votes before listing and checks that a pre-release model matches the public version. Treat any single score with caution and look for consistent results across several benchmarks and your own tests.
Not necessarily. A small score difference may not be noticeable for everyday office work, while the price difference per user or per request can be large. For high-volume automated tasks, a cheaper, faster model that passes your own tests can be the better choice.