How to Benchmark AI Patent Draft Quality

Every AI drafting demonstration you sit through will produce a competent-looking specification. That is now the baseline, not the achievement: fluent patent prose is cheap, and it has been cheap for about two years. The difficulty is that a fluent draft and a defensible draft look identical for the first twenty minutes of reading, which is roughly as long as anyone spends in a vendor call. So you need a way to benchmark AI patent draft quality that does not depend on how the output reads. This article sets out one: a small, blind, repeatable test built from measures a vendor cannot rehearse. It is written to be run against any product in the category, including ours. There is a good reason to insist on measuring rather than trusting the marketing. In the first preregistered empirical evaluation of commercial legal AI research tools, researchers at Stanford's RegLab and Institute for Human-Centered AI found that products marketed as hallucination-free were nothing of the kind, with error rates in the range of roughly one query in six to one in three depending on the tool. That study concerned legal research rather than patent drafting, so it does not transfer directly. What transfers is the lesson: the vendors' own claims about reliability were tested and found overstated, and nobody would have known without a structured test.

Why the obvious metrics fail

Three measures will present themselves and all three are misleading. Generation speed. Time to first draft measures the least valuable part of the process. What you care about is time to a draft a signing attorney will accept, and those two numbers are related only loosely. Automated text scores. BLEU, ROUGE and similar measures compare surface text similarity to a reference. A patent claim can be nearly identical to your baseline and still be indefinite, unsupported, or positioned against the wrong art. These tools measure resemblance, not defensibility. "It reads well." This is the dangerous one, because it is the only assessment most evaluations actually make. Fluency is now free. Treating it as evidence of quality is how a bad tool gets bought.

Building the sample set

Use eight to twelve real invention disclosures from your own portfolio. Not the vendor's sample invention: never the vendor's sample invention, which has been drafted against hundreds of times. Span the range you actually file: at least one mechanical, one software or AI, one chemical or life-sciences case if you file them. Include at least two you already know are difficult, either because the art is dense or because the inventor wrote badly. Difficult disclosures are where products separate; clean ones make everything look the same. For each disclosure, retrieve the application your team eventually filed. That is your baseline, and it goes into the review set unlabelled alongside the machine drafts. If your reviewer cannot reliably identify and rank your own attorney's work first, that is information: possibly uncomfortable information, and worth having either way. Blind the review. Strip formatting and headers, assign each draft a code, and have the scoring done by an attorney who did not run the tools and does not know which draft came from where.

Seven measures worth recording

Measure How to record it What it tells you How a demo hides it Instruction count Count every prompt, edit, and accept/reject decision needed from a qualified person to reach a reviewable draft Your real throughput ceiling The vendor drives, so the count is invisible Support-trace time Pick five limitations; time how long to find the disclosure passage supporting each The review burden you are buying Never tested; the demo shows generation, not verification Unsupported limitations Count limitations with no disclosure support, per draft §112 and Article 123(2) exposure at filing Reviewer reads for sense, not for support §112 defects Count antecedent basis errors, indefinite terms, and claim-to-spec vocabulary drift Prosecution cost already baked in A clean sample disclosure produces few Held-back reference Withhold one close reference you already know about; check whether the tool found it and whether the claim set accounts for it Whether the search actually reaches the drafting Search is demonstrated separately from drafting Figure set Count reference numerals needing manual correction; judge whether figures show the distinguishing features Hidden downstream cost Figures shown as thumbnails, never checked Correction time Have the attorney who would sign take one draft to filing-ready, and time it The actual price of the product Nobody corrects anything in a demo The fifth measure is the one worth building the whole exercise around. Take a matter where you know a piece of close art: one your team found late, ideally one that forced an amendment. Do not supply it. Then look at what came back: did the search surface it, and more importantly, does the claim set show any sign of having been positioned against it? Does the specification carry a fallback where that art is dense? A product can have an excellent search and still produce claims drafted as though the search never happened, because in most architectures the search output is a report a human is expected to carry into the drafting by hand. This test detects that gap directly, and it is the single hardest thing for a vendor to rehearse. The last measure is the one that decides purchases and the one almost nobody records. Correction time, in the minutes of the person who would actually sign, is the real price. A product that produces a draft in a quarter of the time and needs two-thirds of it rewritten has cost you money.

What this test is designed to expose, and how Esgenix is built

We publish this framework because it is the test Esgenix was built to pass, and because the measures above are the ones that separate operating models rather than feature lists. Esgenix is an agentic patent workflow platform. An agent here means a piece of software given a defined job, the tools to do it, and a standard to meet: it completes the job and reports back rather than waiting for the next instruction. Several run in sequence on different underlying models: one searches the prior art, one drafts the claims, one writes the description, one produces the figures, one grades the result. That structure maps onto the measures directly. On instruction count. Patent AI is moving from assistance toward workflow execution. An assistance-shaped tool sits beside the attorney while the attorney drafts, proposing text section by section. Esgenix runs the drafting work itself and stops at five gates for the attorney's decision: intake, where you confirm the invention, answer the technical gap flags and set jurisdictions; strategy, where you set claim breadth and choose fallbacks; claims, where you approve or amend the set and nothing proceeds without it; specification and figures, where you review with provenance visible; and sign-off, which is a natural person's signature and cannot be delegated. Where a tool is shaped around assistance, the professional stays present through the drafting itself. Here are five decisions with production in between. Human involvement is not reduced; it moves from paragraph assembly to decisions worth an attorney's time. On support-trace time. Every claim element carries a provenance identifier: a pointer to the passage of the invention disclosure the element came from. Verification is following a pointer rather than reading two documents in parallel. This is the measure where the difference is largest and easiest to time in a room. On the held-back reference. The search runs across 220 million documents from 108 jurisdictions and matches on meaning rather than keywords, so it surfaces the reference that never uses your vocabulary. The part that matters for this test is architectural: the search output feeds the drafting agents in the same session. The claim set is positioned against the closest art from the first draft, and the specification carries fallback positions where the art is dense. On unsupported limitations and §112 defects. An independent model grades the finished draft against the standards a senior patent attorney would apply and returns it to be rewritten when it fails. This is LLM as a judge: a quality gate that catches both the flat, generic texture machine-written prose falls into and the substantive defects underneath it. It runs before your reviewer sees anything. On figures. They are generated with no baseline drawing required. The system reads the specification, decides what the figure set should be, draws it, and reconciles every reference numeral against the text. The common approach in the category is to convert an uploaded sketch, photograph or CAD file, which produces figures illustrating what the inventor drew rather than what the specification claims. Also relevant to a fair test: at intake, gap detection identifies what the disclosure is missing (an unexplained step, an undefined term, a result asserted with no mechanism) and proposes how to close it. When you run difficult disclosures, this is where the difference shows up, because a badly written disclosure produces a badly supported draft in any system that does not ask about it first. Esgenix has completed ISO 27001 Stage 1, covering the platform and the handling of invention and client data: worth confirming before unfiled disclosures enter anyone's system, ours included.

Mistakes that will invalidate your benchmark

Using the vendor's sample invention. Using only clean disclosures. Letting the person who ran the tools score the output. Testing one jurisdiction when you file in three. Scoring the draft without attempting to verify support for a single limitation. And measuring generation time instead of correction time, which is the error that produces the most confident wrong conclusions. One more: do not run the test without your own attorney's filed draft in the blind set. Without it you are ranking products against each other rather than against the standard you actually have to meet.

What no benchmark settles

The attorney remains responsible. Papers filed at the USPTO carry a natural person's signature, and that signature is a personal certification about the contents. Inventorship is a separate question with a firm answer: in Thaler v. Vidal the Federal Circuit held that an inventor must be a natural person, and USPTO guidance requires a significant human contribution to conception. Drafting assistance does not bear on conception. And there is no general duty to disclose that AI assisted with drafting. The duty of disclosure concerns information material to patentability. Any vendor presenting an audit trail as a compliance requirement is describing an obligation that does not exist. Provenance is worth having because it saves review hours, which is a sufficient reason.

A specific next step

Pick eight disclosures from the last eighteen months, including two difficult ones and at least one where a late-found reference forced an amendment. Withhold that reference. Add your own filed drafts to the set unlabelled, and give the whole thing to a colleague who did not run the tools. Then record two numbers per draft: instruction count, and the minutes a signing attorney needs to take it to filing-ready. Those two, together, answer the question. Bring the same eight disclosures to a working session with us and we will run them through the five gates while you count.