1What accuracy means here, and why the metric decides the answer

Three different numbers get called accuracy in patent search, and they answer different questions.

Hit rate asks whether at least one correct answer appeared near the top. It is reported at a cut-off, so a hit rate at the top 5 means the proportion of test cases where a correct document was among the first five results. It is the number closest to the experience of opening a result list and finding something usable immediately.

Recall asks how much of the known-correct set was found at all, within some larger window such as the top 100. It is the number closest to the question a searcher is actually being paid to answer, which is whether anything relevant was left behind.

Precision asks what proportion of returned documents were relevant. It governs how much reading a result list costs, and it is the metric most information retrieval research has historically optimised for.

The mismatch between that last fact and the nature of patent work is not a new observation. In 2010 Magdy and Jones set out the problem in a SIGIR paper, noting that while precision and recall “are often discussed with equal importance, in practice most attention has been given to precision focused metrics,” and that “even for recall-oriented IR tasks of growing importance, such as patent retrieval, these precision based scores remain the primary evaluation measures.” They proposed PRES, a metric that accounts for “recall and the user’s search effort,” and demonstrated it on 48 runs from the CLEF-IP 2009 patent retrieval track.⁷

The practical consequence is simple. A tool can look excellent on a precision-weighted score and still be the wrong instrument for a novelty or freedom-to-operate search, because the document that kills the claim is the one it did not return. When a vendor quotes an accuracy figure, the first question is which of these three it is.

2The answer key: what counts as a correct result

Search reports classify citations by how they bear on patentability, using categories defined in WIPO Standard ST.14. A category X document means “the claimed invention cannot be considered novel or cannot be considered to involve an inventive step when the document is taken alone.” A category Y document means the invention “cannot be considered to involve an inventive step when the document is combined with one or more other such documents, such combination being obvious to a person skilled in the art.” A category A document merely defines the general state of the art and is “not considered to be of particular relevance.”⁶

That gives a graded target rather than a binary one, and the X and Y references are the part worth measuring against. They are the documents that changed the outcome.

PatentBench builds its ground truth this way: X and Y references cited by examiners at different receiving offices are “collected, deduplicated, and normalized by patent family.”¹ The family normalisation matters more than it sounds. The same disclosure is typically published several times across offices, so a system that returns three members of one family has found one piece of art, not three. Counting publications instead of families is one of the easiest ways to make a search score look better than it is.

Two properties make this target useful. It is defined by patent offices rather than by whoever is running the test, and it is independent of the retrieval method, so a keyword search, an embedding model and an agent are all scored against the same thing. Its limitation is that examiner-cited art is what one examiner found in the time available, which makes recall measured against it a floor rather than a ceiling.

3What the published numbers actually say

Novelty search against examiner-cited art

PatentBench, dated July 2026, runs on 340 cross-jurisdiction patent family samples, 68% English and 32% Chinese, with US and Chinese applications at roughly 32% each and EP and WIPO at roughly 18% each, distributed across IPC classifications. Two metrics are reported, and the page glosses both in plain terms. X Hit Rate is the share of test cases in which the agent “successfully identified at least one relevant X document”; X Recall Rate is the share of “all relevant X documents” it retrieved.¹

The page reports that Patsnap’s Novelty Search AI Agent “achieved an 85% X Hit Rate and a 37% X Recall Rate within the top 100 results,” and that “Claude Opus 4.8 ranked second, with a 52.37% X Hit Rate and an 11.68% X Recall Rate.” The remaining systems evaluated are also general-purpose models with web search enabled: Perplexity Pro, ChatGPT 5.4, Gemini 3.1 Pro and DeepSeek 3.2.¹

The system behind those runs is an agent workflow rather than a query box, described on the product page as turning an invention disclosure into ranked prior art and feature-level evidence. Coverage is 200M+ patents across 174 jurisdictions, with SOC 2 and ISO 27001 certification, GDPR and CCPA compliance and no AI training on user data, which is the part that matters when the input is an unfiled invention.³

Read the two numbers together rather than picking the flattering one, because they count different things. A hit rate counts samples: did at least one correct answer come back for this application. A recall rate counts documents: how much of the whole known-correct set came back. A single application often carries several X and Y references, which is why the page can report finding at least one in 85% of cases while retrieving 37% of all of them.¹ The distance between the two numbers is exactly that difference, and it is the honest picture of where retrieval currently sits: dependable at surfacing a relevant document, some way from exhaustively recovering every one.

Design freedom to operate, where the target is an image

A separate benchmark covers design rights, where the query is a product appearance rather than a text disclosure and the ground truth is a confirmed infringing registration. It runs on 261 samples spread roughly evenly across China, the European Union and the United States, covering 26 Locarno primary classifications, split between e-commerce infringement images at 64.8% and invalidation scenarios at 35.2%, with the infringement relationships “rigorously verified by a human annotation team.” The metric is a High-Risk Patent Hit Rate, defined as the “proportion of test samples where the confirmed infringing patent appears in the top K results,” reported within the top 200. The Patsnap Design FTO agent records 77%, against 3.71% for Gemini 3.1 Pro and 0.93% for ChatGPT 5.4. Across offices the page reports performance strongest at the EU office, where the agent reached a 92% hit rate.²

That page also reports a PRES score of 0.7 for what it describes as end-to-end infringement determination quality, evaluating not only whether the infringing registration was found but how highly it was ranked, and it credits the recall-oriented PRES metric Magdy and Jones published in 2010.^{2, 7}

What the academic literature reports

Independent benchmarks give the wider picture, and it is a sobering one. DAPFAM, a 2025 family-level dataset built specifically to test cross-domain retrieval, ran 249 controlled experiments across lexical, dense and rank-fusion configurations over 1,247 query families and 45,336 target families, and found that “OUT-domain performance remains roughly five times lower than IN-domain across all configurations.”¹² PatenTEB, a companion benchmark of 15 patent embedding tasks and 2.06 million examples from the same group, reports its strongest model reaching 0.377 nDCG@100 on DAPFAM.¹³ Those are the numbers for off-the-shelf retrieval, and they are a long way from solved.

4The baseline nobody quotes: how accurate is human search?

Asking whether AI search is accurate enough is only answerable against something. The something is usually assumed to be an exhaustive expert search, and the published evidence does not support that assumption.

No office searches everything. WIPO’s own explanation of why supplementary international search exists is direct: “no single Office is capable of searching the whole of the PCT minimum documentation in its original language,” and a supplementary search “should further reduce the risk of new prior art being cited in the national phase.”⁸ The international system is designed around the expectation that one authoritative search will miss things.

The time available is finite. Frakes and Wasserman, studying USPTO examination, report that “on average, an examiner spends only nineteen hours reviewing an application, including reading the patent application, searching for prior art, comparing the prior art with the patent application, writing a rejection, responding to the patent applicant’s arguments, and often conducting an interview with the applicant’s attorney.” Searching is one line item inside that budget, not the whole of it. Their central finding is that when promotion reduces the time allotted, grant rates rise substantially and the share of citations originating from examiner search falls, with obviousness rejections, the most search-intensive kind, dropping first.⁹

Applicant-supplied art rarely does the work. Cotropia, Lemley and Sampat found that examiners “focus almost exclusively on art they find themselves,” with roughly 87% of references used in rejections coming from examiner searches, only about 2% of applicant-submitted references ever cited in a rejection, and over 90% of the US patents used in rejections emanating from examiner search.¹⁰ Whatever the search process is, it is one person’s search under time pressure.

The offices are automating too. The USPTO has deployed an AI-based Similarity Search inside its internal PE2E examiner system, which uses trained AI models to output a list of documents similar to the application being searched. A memorandum to the Patent Examining Corps dated 24 October 2025 states that “examiners are required to use Similarity Search during examination of a plant or utility application and record the conducted Similarity Search in the application file wrapper.”^{23, 24} The question of whether machine retrieval belongs in prior art search has effectively been answered by the office that sets the standard.

The misses surface later. In USPTO PTAB trials for fiscal year 2025, 22% of challenged claims and 47% of instituted claims were found unpatentable by a preponderance of the evidence.¹¹ Those outcomes rest largely on art that a motivated challenger found and the original search did not.

None of this is an argument that automation is therefore good enough. It is an argument that the comparison should be honest. The useful question is not whether a tool is perfect, but whether it recovers more of the known-correct art per hour than the process it is replacing, and whether a professional can check its work document by document.

5Where AI prior art search still fails

Art from another field. This is the documented hard case and it is also the commercially expensive one, because the reference that invalidates a mechanical claim is often sitting in a different technology area entirely. DAPFAM measured the gap at roughly five times across every configuration tested, and concluded that neither dense retrieval nor lexical search closes it.¹²

Coverage nobody can search around. Machine-readable full text exists for a minority of offices. The EPO’s worldwide bibliographic dataset contains abstracts, citations and families but “no full text or images.”¹⁸ Google Patents indexes “over 120 million patent publications from 100+ patent offices” but lists full text for 22 of them.¹⁹ A search over abstracts and a search over claims are different instruments, and no model compensates for text that was never indexed. The free official collections describe their own boundaries plainly for the same reason: Espacenet publishes more than 150 million documents from more than 100 patent authorities, PATENTSCOPE publishes a per-collection coverage table listing over 128 million documents across its national and regional collections, and USPTO Patent Public Search states that it does not use semantic searching and offers no access to foreign patent databases.^{20, 21, 22}

Finding art is not judging it. A 2025 study built the first dataset for novelty evaluation from real examination cases and found that generative models “make predictions with a reasonable level of accuracy” with explanations good enough to follow the relationship between claim and reference, while classification models “struggle to effectively assess novelty.”¹⁴ Reasonable is not the same as decisive. On a hands-on patent attorney examination, the strongest model scored 0.82 accuracy and 0.81 F1 while smaller open models landed between 0.50 and 0.55, and the study reports that accuracy “never exceeded the average threshold of 0.90 required for professional-level standards.” Its human patent experts, it notes, valued clarity and legal rationale over the raw correctness of the answers.¹⁵

Fabrication when nothing is retrieved. A model that generates a citation instead of returning one from an index will produce something plausible and wrong. A research team profiling legal hallucinations, measuring general-purpose models on verifiable questions about real court cases across more than 800,000 queries, found hallucination rates from 58% to 88% depending on the model.¹⁶ Retrieval augmentation reduces the problem without removing it: the same group found purpose-built commercial legal research tools still “hallucinate between 17% and 33% of the time.”¹⁷ For patent work the practical rule follows directly. A result that does not carry a publication number you can open in an official register is not a result.

6How to measure accuracy on your own art

Every published benchmark is measured on someone else’s sample. Your technology area, your claim language and your jurisdictions are the only test that predicts anything about your work, and you already own the answer key, because your granted families come with search reports. Running the test takes about a day.

1. Assemble 20 to 50 families you have already prosecuted. Prefer cases where the examiner cited X or Y art, and spread them across the technology areas you actually file in rather than the one you know best.
2. Build the answer key before you run anything. Take the X and Y references from every office that examined the family, deduplicate them, and normalise to patent family so that one disclosure counts once.
3. Use the disclosure, not the granted claims. Feed the tool the invention as it looked before drafting. Feeding it the granted claim set leaks the examiner’s hindsight into the query and inflates every score.
4. Record two numbers, not one. Hit rate at the top 5 tells you what the first screen looks like. Recall at the top 100 tells you what a full review recovers. Reporting only the first is how a weak system looks strong.
5. Include cases you expect it to fail. Deliberately load in families whose killer reference came from another technology field. That subset is where the differences between tools are real.
6. Score the analysis separately from the retrieval. Whether the decisive passage was correctly mapped to the claim feature is a different question from whether the document was returned, and a tool can be strong on one and weak on the other.
7. Re-run it when anything changes. A model update, a re-chunk or a coverage change can move results in either direction, and the only way to know is to keep the set and repeat.

If you want this inside your own harness rather than a browser, the Patsnap Open Platform exposes patent research, novelty and freedom-to-operate agents as 32 MCP servers callable from Claude, Cursor or a custom client, which makes a scripted evaluation loop over your own families straightforward to build.^{4, 5}

What no accuracy number settles

• A benchmark is a distribution, not a promise about your case. A hit rate measured across hundreds of samples says nothing about whether the decisive reference for the disclosure in front of you was among them.
• The answer key is itself incomplete. Examiner citations are what one examiner found in the time available, so a score measured against them understates how much art exists and overstates how much has been ruled out.
• Novelty is not patentability. Novelty is assessed one reference at a time. Inventive step is a separate analysis under a named jurisdictional framework, and no retrieval score speaks to it.
• Ask what produced any figure you are quoted. Sample size, the definition of a correct answer, the cut-off and the metric. A percentage without those four is not a measurement.
• The certification stays with the person who signs. The USPTO states that “simply relying on the accuracy of an AI tool is not a reasonable inquiry,” and under 37 CFR 11.18(b) whoever signs certifies that factual contentions have evidentiary support, to the best of their knowledge, information and belief “formed after an inquiry reasonable under the circumstances.”^{25, 26}