On October 23, Ivo and micro1 published a benchmark designed to test how AI systems review contracts.

The leading model scored 58.3% on the benchmark’s weighted rubric. The next two scored 58.1% and 57.1%. Every model tested finished below 60%.

Those numbers do not mean an AI tool gets four out of ten contracts “wrong.” The benchmark was built by companies with a commercial interest in legal AI, and its scoring method is not an independent malpractice study.

It is still useful because it tests something firms can miss during an AI pilot: producing a redline is not the same as exercising legal judgment.

The benchmark tested decisions, not document formatting

ContractBench includes 350 contract-review tasks across 35 scenarios and five agreement types. The tasks cover master services agreements, IP licenses, research partnerships, commercial leases, and marketing agreements.

Some tasks ask the model to review a clean agreement. Others ask it to respond to a counterparty’s markup. The facts, jurisdiction, business priorities, and availability of a company playbook vary.

Seven models ran through the same evaluation process. Grok 4.7 led at 58.3%, followed by Opus 5.5 at 58.1% and Kimi K3 at 57.1%.

The benchmark grades four possible actions for each issue:

Edit the language
Preserve it
Comment for the counterparty
Escalate it for internal approval

That matters because a contract reviewer is not paid to change as much text as possible. Sometimes the correct action is to leave acceptable language alone. Sometimes the issue needs a business decision rather than another redline. A model that edits everything may look busy while creating unnecessary friction.

A polished redline can hide weak judgment

Most legal AI demonstrations show the output. A clause changes, a comment appears, and the document looks ready for an attorney.

The harder questions sit behind the visible edits:

Did the system identify the provisions that create meaningful risk?
Did it spend time on low-value wording while missing a high-priority issue?
Did it follow the firm’s approved fallback position?
Did it recognize when the playbook did not answer the question?
Did it escalate a decision that required client or business approval?
Did it preserve language that was already acceptable?

ContractBench calls these issue prioritization, negotiation restraint, escalation judgment, situational judgment, and standard-position execution.

Those categories are more useful for a law firm than a generic claim that a tool can “review contracts.” They describe the work that should be tested before attorneys rely on the output.

The results need context

Ivo provides AI contract-review software, and micro1 provides expert data and evaluation services. They created the benchmark, selected the tasks, defined the rubrics, and published the results. Three other AI models served as judges for rubric items.

That does not make the benchmark worthless. It means the scores should be treated as one structured vendor study, not a universal ranking of legal ability.

The tasks also focus on five agreement types and a defined review environment. A firm’s employment agreements, loan documents, estate-planning forms, or litigation settlements may produce different results. A model may perform better when a playbook is clear and the contract resembles its testing material. It may perform worse when the facts are incomplete or the firm’s position depends on local practice.

The useful takeaway is not that one model won by two-tenths of a point. It is that the evaluation separated visible editing from the decisions that make the editing useful.

Build a pilot around your firm’s decisions

A firm considering contract AI should not start by uploading a current client agreement and asking whether the output looks impressive.

Create a test set from completed matters or purpose-built examples. Remove client information, preserve the business context needed for review, and include agreements that contain both obvious and subtle issues.

For each test, record what a qualified attorney expects:

Which provisions should change
Which provisions should remain untouched
Which comments should go to the counterparty
Which questions require client or internal approval
Which playbook position applies
Which sources or facts support the decision

Then compare the AI output with that record. Track missed high-priority issues, unnecessary edits, incorrect escalations, and the attorney time required to make the result usable.

This produces evidence that applies to the firm’s work. It is more informative than relying on a vendor’s overall accuracy percentage or a demonstration built around the vendor’s best example.

Version and configuration belong in the test record

Legal AI tools change quickly. A vendor may update the model, prompt, retrieval method, connector, or playbook feature without changing the product name.

Record which version and configuration produced the tested result. Save the playbook used, the source agreement, the expected actions, the output, and the reviewing attorney’s assessment.

If the tool connects to a document-management or practice-management platform, include the access and data flow in the pilot. As discussed in our MyCase and Claude connector review, permission scope and audit history matter alongside output quality.

Repeat a smaller set of benchmark matters after a material product change. The goal is not to freeze the technology. It is to notice when the behavior a firm approved has changed.

Human review needs a defined purpose

“Attorney review required” is too vague to be a control.

The reviewer should know what the AI was asked to do, which playbook governed the task, where the model was expected to escalate, and which parts of the output carry the most risk. The firm should also decide what evidence of that review remains in the matter file.

ABA Formal Opinion 512 ties lawyers’ use of generative AI to existing duties including competence, confidentiality, communication, supervision, candor, and reasonable fees. A benchmark result does not take responsibility for the work away from the lawyer.

A strong tool may save time locating clauses, comparing language, drafting alternatives, and applying a well-defined fallback. It can still make a poor judgment about which issue matters to this client in this transaction.

ContractBench does not settle which legal AI product a firm should buy. It offers a better way to ask the question.

Can the tool edit the document? Can it also preserve acceptable language, follow the firm’s position, recognize uncertainty, and hand the right decision back to a person?

Test those decisions before the polished redline becomes part of the workflow.

If your firm is evaluating legal AI from demonstrations rather than its own matters and playbooks, Artech can help structure a controlled pilot with access, review, and change records.