Adoption is no longer the hard part. The Stanford AI Index 2026 reports that organizational AI adoption reached 88%, and that generative AI reached 53% population adoption within three years, faster than the personal computer or the internet. The harder part comes afterwards: someone on the team has to decide which tool to pay for, and then defend that choice when the invoice or the output quality is questioned. This article is about making that decision on evidence. We cover the kinds of evidence available, six tests to apply to any published rating before relying on it, and a scorecard an engineering team can run in-house.
Choosing AI tools is now the bottleneck
The money is moving faster than anyone can evaluate what it buys. In September 2026, Gartner forecast that worldwide AI spending will total $2.7 trillion in 2026, a 49.5% increase year over year, with the AI software segment alone growing from about $288 billion in 2025 to about $462 billion in 2026.
Use is spreading inside companies at the same time. In McKinsey’s State of AI survey, published in August 2026, 44 percent of respondents said AI is scaling across their enterprise, up from 38 percent a year earlier, and the share reporting AI use in three or more business functions rose from 51 to 56 percent. Each of those functions is a separate tool decision, usually made by a different person with a different idea of what good looks like.
No team can trial every candidate. So we all lean on someone else’s judgment: a vendor page, a review site, a leaderboard, a colleague’s recommendation. That is reasonable. The skill worth building is knowing how much weight each of those sources can bear, and that starts with why AI tools are harder to assess than the software we are used to buying.
Why AI tools resist ordinary software evaluation
We evaluate a database or a CI system by reading the documentation, running a proof of concept and checking the result. Five properties of AI tools break that routine.
The output is not deterministic. The same input can produce a different result on the next run. OpenAI’s own guide to reproducible outputs, published in November 2023, says that even with a fixed seed “determinism is not guaranteed” and that backend changes can alter results. One good answer in a demo therefore proves little. What matters is the spread across several runs of the same input.
Metering hides the monthly cost. Many generative tools sell credits, minutes or tokens rather than seats. Runway’s pricing page, as we read it on October 2, 2026, lists the Standard plan at $15 a month when billed monthly, with 625 credits, and says its Gen-4.5 model uses 12 credits per second of generated video. That is 52 seconds of output a month, and the page adds that monthly credits on that plan do not roll over. The subscription price tells you very little until you know how many attempts one usable result takes.
Release cadence outdates reviews. The same Stanford report notes that performance on SWE-bench Verified, a coding benchmark, rose from 60% to near 100% in a single year. A review written against last year’s model describes a product that has since been replaced, often under the same name.
Benchmarks can be steered by the companies being measured. The Leaderboard Illusion, a 2025 study of the Chatbot Arena leaderboard, found that undisclosed private testing let a handful of providers try multiple variants before public release and disclose only the best score. The authors identified 27 private variants tested by Meta ahead of the Llama-4 release.
Ranked lists can follow the money. A page that earns a commission on each click-through has a reason to order its list by payout. Unless the page says how the order was produced, a reader cannot tell whether it did.
Five kinds of evidence, and what each is worth
Everything you can know about a tool before buying it arrives through one of five channels. Each can prove some things, cannot prove others, and gets gamed in its own way.
Vendor documentation
Pricing pages, quota tables, API references, changelogs and data-handling terms are the most reliable source for hard limits, because a vendor is accountable for them. They can prove what a plan includes, what it caps and what it costs. They cannot prove quality. The gaming is mostly by omission: adjectives where a number should be, and limits that appear only in a help-center article.
Crowd reviews
User reviews prove that real customers exist and show what frustrates them after the first month. They cannot prove fit for your use case, since reviewers select themselves and often use the product differently than you will. They are also the most openly gamed channel. The US Federal Trade Commission’s 2024 rule on reviews and testimonials prohibits fake reviews, including AI-generated ones, and bars businesses from offering incentives conditioned on a review expressing a particular sentiment. A regulator does not write a rule for a rare problem. Read crowd scores for the specifics in the text, and check how the platform validates its reviewers.
Benchmarks and leaderboards
A benchmark proves relative performance on a fixed task under fixed conditions, which is useful for narrowing a shortlist. It cannot prove performance on your inputs, and popular benchmarks saturate, as the SWE-bench figure above shows. The gaming is selective disclosure and tuning to the test. Prefer benchmarks that publish their task list, their scoring rule and their number of repeats.
Editorial ratings with a published method
An editorial rating can prove that someone compared several tools against the same criteria. It is only as strong as the evidence behind it: a desk review of documentation is not a hands-on test, and a good rater says which one it did. The gaming here is commercial, through undisclosed sponsorship or affiliate relationships. The six tests in the next section are written for this channel, and most of them apply to benchmarks and review platforms too.
Your own pilot
A pilot proves the one thing no outside source can: how the tool performs on your inputs, at your volume, on the tier you would buy. It cannot prove much beyond the sample you ran, and it is the most expensive evidence to produce. The person most likely to game it is you, by choosing inputs after seeing outputs or by defending a tool the team already likes. A written protocol, covered below, is the control.
Six tests for any rating you are about to trust
Each test can be checked in a few minutes by reading the rater’s own pages. To show what a pass looks like, we point to the public methodology documents of four raters of very different kinds: Artificial Analysis, which benchmarks AI models; G2, a review marketplace for business software; RTINGS, which lab-tests consumer hardware; and Catalora, an editorial directory of AI tools for video and visual creators. We describe what each page stated on October 2, 2026. We have not audited whether practice matches the page, and we are not ranking the four against each other.
1. The rubric is published
A score you cannot take apart is an opinion. Artificial Analysis publishes the composition of its Intelligence Index: version 4.3.2 combines ten evaluations as a weighted average across four categories (agents 30%, coding 20%, scientific reasoning 20%, general 30%), with the weight of each evaluation listed. RTINGS states in how it tests monitors that its Test Bench 2.1 consists of 375 individual tests and that every monitor goes through the same bench. The question to ask is whether you could recompute the headline number from its parts.
2. Each criterion names the evidence it needs
A criterion called “quality” means nothing until the rater says what counts as evidence for it. Catalora’s page on how it scores AI tools lists five axes (ease of use, output quality, value for money, speed and performance, support and docs) and gives, for each one, the evidence it requires, what the number bands mean and when the axis is withheld. The same page says its assessments rest on documented evidence and independent reporting, and that it does not buy and run every tool. That is the level of detail to look for: a reader can see what kind of evidence sits behind a number, and what kind does not. The Artificial Analysis table does the equivalent for a benchmark, listing the number of questions, the repeats and the scoring rule for each evaluation.
3. The rater refuses to score when evidence is missing
A rater with a number for everything is guessing somewhere. G2’s research scoring methodology sets thresholds instead: a product needs at least 10 reviews in a category to be included in that category’s Grid report, and the category itself needs at least six products with 10 or more reviews and 150 reviews overall. Look for the equivalent rule wherever you read a score, and for a visible “not rated” state when the rule is not met.
4. Editorial judgment and crowd sentiment are kept separate
Expert assessment and user sentiment are both useful, and they are different measurements. Blended into one number, neither can be checked. G2’s page about its reviews, data and rankings draws the line from the crowd side: it says reviews are subjective accounts of individual users’ experiences, not expert assessments, and that G2 adds no subjective input to ratings. RTINGS draws it from the lab side. Its readers vote on which monitor it buys and tests next, so the crowd influences selection, while the scores come from the test bench.
5. Paid placement is labeled and cannot move rank
Most raters need revenue, so the question is where the money can reach. The same G2 page says it sells advertising in sponsored positions, that this has no impact on a product’s organic ranking, and that badges cannot be purchased. The RTINGS page says it buys every unit at retail and does not accept review samples from brands, which closes off one channel of influence completely. Whatever the business model, you should be able to find a sentence that says what a vendor can buy and what it cannot.
6. Dates, corrections and a right of reply exist
A rating without a date cannot be judged for staleness. RTINGS lists its monitor test bench versions with their dates, from 1.0 in October 2017 to 2.1.2 in March 2026, and says it retests monitors when the bench changes. G2 asks anyone citing a category ranking to include the date it was retrieved. Catalora’s editorial policy says every tool page carries the date its facts were last verified, that substantive corrections are made on the page and flagged when they change a verdict, and that vendors cannot preview or approve a review but can send evidence afterwards to have a fact checked.
None of the six tests tells you that a rating is right. They tell you whether it is checkable, which means that when it is wrong you have a way to find out. A rater can also pass all six and simply not cover your category. In that case the method is still worth borrowing.
Running the same discipline in-house
The same tests work on your own evaluation. The difference is that you can produce the fifth kind of evidence, the pilot, which no outside rater can produce for you.
Set the weights before you look at vendors. Decide per use case what matters and by how much, and write it down before the first demo. Weights chosen after the shortlist tend to describe the tool the team already prefers.
Write a pilot protocol. Fix a set of inputs drawn from real work, including the awkward cases. Run each input several times, because the output is not deterministic, and record every output rather than the best one. Where you can, have reviewers rate outputs without knowing which tool produced them. In our guide to AI integration we describe the continuous form of this, an evaluation suite that runs whenever a prompt or a model changes.
Compute cost at the tier you would actually buy. Take the plan that covers your volume, not the entry price, and divide by the number of outputs you accepted, not the number you generated. For credit-metered tools the retries are the bill.
Set a re-evaluation date. Models, prices and limits change within months. An evaluation with no expiry date slowly turns into an assumption.
A compact scorecard for one use case might look like this. The values are examples; yours should come from the use case.
- Use case and owner: one sentence describing the job, and one named person who signs the result.
- Weights, fixed before the shortlist: output quality 40%, cost per accepted output 25%, ease of adoption 15%, speed 10%, support and docs 10%.
- Evidence per criterion: quality from blind-rated outputs on 20 fixed inputs, three runs each; cost from the plan you would buy divided by accepted outputs; speed timed on the same inputs.
- Withhold rule: a criterion without evidence is recorded as not assessed, and a tool with a missing criterion gets no total.
- Disqualifiers: data-handling terms your security review rejects, or no way to export your work.
- Re-evaluation date: 90 days out, or sooner on a major model or pricing change.
The discipline is the same one we apply to AI in our own delivery work: a senior engineer writes the specification, a senior engineer reviews every output, and the person who owns the outcome signs it. A tool evaluation deserves the same named owner and the same review.
What to take from this
The supply of AI tools now exceeds anyone’s capacity to try them, so the scarce skill is evaluation. Know which of the five kinds of evidence you are looking at and what it can prove. Before trusting a published rating, check the six things its own pages should tell you. Then run a small, written pilot on your own inputs, because that is the only evidence about your use case. Our methodology page describes how iPlus applies senior review to AI-assisted delivery for clients.
Need help with this?
Explore the iPlus Solution services most relevant to this article.
