Categories
Articles

Patent Searching: (Almost) Everything You Need to Know

Reproduced here are materials presented at the AIPLA CLE webinar Patent Searching: (Almost) Everything You Need to Know on September 14, 2021. The presentation slides, which include links to patent searching resources available online, are available for download below.


Introduction

Rather than a tutorial or “how-to” guide in the mechanics of searching, these program materials address important considerations that inform the planning, execution, and validation of all types of patent searches, including patentability, freedom-to-operate, invalidity, landscape, and due diligence searches. Readers will learn about timing considerations, budgeting and staffing models, export controls, privileges, evidentiary considerations for invalidity/patentability challenges, metrics for evaluating the effectiveness of a search, and more. This overview will be useful for those who perform searches themselves and those who outsource their searches to others.

I. Types of Patent Searches

The purpose of a patent “landscape” search is to ascertain the overall state of patents in a particular technology area.  While “state-of-the-art” is sometimes used synonymously with “landscape” here, a distinction can be drawn between a patent landscape search limited exclusively to patents and published patent applications and a state-of-the-art search that includes patent documents as well as non-patent publications and potentially also commercially-available products.  These searches are frequently used for competitive/business intelligence purposes.  Landscape searches may be limited to particular jurisdictions in some cases.  Reports and documentation for landscape searches are often focused on only high-level bibliographic information and the like and may omit detailed discussion of the content of specific patent claims.  Landscape search reports can be very basic but also can be highly elaborate with infographics and other more elaborate visual elements aimed at an audience of businesspeople—especially higher-level executives—rather than patent attorneys. 

Landscape searches are exploratory searches performed for business-related purposes and they are not legally required. 

A patentability search identifies potential prior art against a particular invention, and is performed in order to assess the likelihood that a patent would be granted on the invention and the potential scope of allowable/patentable subject matter.  They allow more informed judgments as to whether the costs and effort of a patent application will be worthwhile.  Results of a patentability search can also be utilized by the person(s) preparing a patent application on the invention to improve the quality of the application through better awareness of the closest prior art.[1]  The patentability search, as such, is the gathering of relevant prior art for purposes of a patentability analysis, which involves legal judgments beyond mere searching effort. 

The scope of a patentability search may be based on the contents of an invention disclosure document; changes or further developments to the invention that are not contained in such a document—or are simply not explicitly identified in a meaningful way—may not be captured by the search.  Patentability searches should capture disclosures contained anywhere in a given prior art document or product, including inherent disclosures and things visible only in the drawings.  Patentability searches ideally encompass both patent documents and non-patent publications, as well as commercially available products, available anywhere in the world.  However, as a practical matter, patentability searches are usually subject to significant constraints on their scope.  Most patentability searches are budget-constrained, whether in terms of a monetary budget or a time budget (e.g., needed by tomorrow).  Moreover, some patentability searches might be limited to only a search of patent documents, or in a database that covers only certain key jurisdictions or non-patent literature (NPL) publications.

Patentability searches are exploratory searches performed to enable a legal evaluation of patentability as well as to inform business decision-making but they are not legally required by the USPTO before filing a patent application.  Sometimes a patentability search is combined with a freedom-to-operate (FTO) search. 

Due diligence searches are performed in various contexts where it is desired to verify the current ownership/assignment of given patent(s) or application(s) and to identify potential clouds on title, to establish family and other bibliographic data, and to vet corporate mergers & acquisitions (M&A) that involve patent assets and other patent licensing or acquisition opportunities (e.g., to assess scope, validity, valuation).  In the due diligence context, searching may be focused on ascertaining ancillary data for a known or limited universe of patents.  Due diligence searches may shade into other types of searches.  For instance, a freedom-to-operate search may be performed to assess infringement risks associated with a product line or technology acquisition.  Also, sometimes invalidity searches are performed during acquisitions in order to assess the strength of acquisition targets or even to proactively attack the patents to create leverage to drive down acquisition cost.  Another situation where due diligence searches can be utilized is in preparation for enforcement. 

Due diligence searches can be considered exploratory searches.  There is generally no affirmative legal duty to perform a due diligence patent search.[2] 

An FTO search, also called a “clearance” or “right-to-use” search, is used to identify patents and pending patent applications that pose potential infringement risks associated with a particular product or process.[3]  “Contract clearance” searches are a subset of FTO searches for compliance with contract provisions.  The FTO search, as such, is the gathering of relevant patents and applications for purposes of an FTO analysis, which involves legal judgments beyond mere searching effort. 

The scope of an FTO search must generally first be ascertained by performing a “feature identification” or “product decomposition” analysis to identify discrete, potentially patentable features to be searched.  FTO searches focus on the claimed subject matter, and need not address non-patent materials.  Moreover, FTO searches may be limited to only jurisdictions where the product or process will be made, used, sold, or offered for sale—bearing in mind that contributory or induced infringement risks may expand the list of relevant jurisdictions in some situations. 

In order to have a reasonable basis to rule out potential infringement risks an FTO search must be reasonably comprehensive and reliable, meaning that FTO searches should ideally not be subject to arbitrary constraints.  Though in some situations an FTO search on particular feature(s) might be deferred or foregone if there is a reasonable basis to believe those feature(s) will present relatively low risk of infringement, such as where the particular feature of the product or process in question is confidently known to have been in commercial use (identically) for more than twenty years—making any recent patents directed to such a feature likely to be invalid. 

FTO searches are defensive searches performed to enable a legal evaluation of infringement risk.  Sometimes an FTO search is combined with a patentability or due diligence search.  A potential infringer is not subject to an adverse inference for failing to affirmatively conduct an FTO search and/or seek an opinion of counsel.[4]   Though the range of options to avoid liability is much greater if you find a potentially problematic patent before it finds you. 

An invalidity search is used to identify prior art that renders one or more claims of a given patent invalid.  Invalidity searches are often used to develop an invalidity defense to an allegation of infringement and/or to help establish a good faith belief in invalidity to avoid a charge of willful infringement through an advice of counsel defense.[5]  Depending on the context, such as for a challenge at the USPTO, the word “unpatentability” may apply instead of “invalidity”, in a strict sense.  In other countries, terms like “nullity” or “opposition” might be used instead of “invalidity” or “unpatentability”.  An invalidity/unpatentability search can further be used to identify prior art against one or more claims of a pending patent application, for potential use for a third-party pre-grant submission or third-party observations made to the relevant patent office. 

Invalidity searches should seek relevant disclosures contained anywhere in prior art documents or products, including inherent disclosures and things visible only in the drawings.  Invalidity searches ideally encompass both patent documents and non-patent publications, as well as commercially available products, available anywhere in the world.  Though the status of certain activities or documents as “prior art” may vary.  Research into commercially available products may take on the character of an investigation more than an exercise in database searching.  Because invalidity searches may be performed in high-stakes situations, they are often the most involved and comprehensive searches, because the extra effort and expense is justified.  For example, it may sometimes be worthwhile to commission multiple invalidity searches in different languages. 

Invalidity searches are defensive searches performed to enable a legal defense to be developed against an actual or potential allegation of infringement. Although invalidity searches are not legally mandated per se, they are generally necessary in a practical sense in order to develop and assert an invalidity defense or affirmatively challenge validity/patentability.[6] 

II. Timelines

Searches can be informally divided into two general categories: prospective and reactive

Prospective searches include landscape, patentability, and FTO searches.  The prospective category refers to a given search being performed in advance of some key decision point or action, such as before the launch of a product (FTO search), or the preparation and filing of a patent application (patentability).  Prospective searches inform such decisions and actions and, accordingly, should be timed so that completed search results are available to be analyzed prior to any deadlines for a decision or action. 

With respect to FTO searches, in particular, searching may be performed on an ongoing or periodic basis, such as to monitor patenting activity by particular competitor(s) and/or in particular technology area(s).  Additionally, there may be a need to watch the prosecution status of certain pending patent applications, to ascertain the scope of what, if anything, is later allowed or granted.  Some patent database tools include features to help facilitate various ongoing monitoring efforts.  Furthermore, FTO searches may need to be updated at a later point in time.  For instance, changes in product configuration may arise (e.g., manufacturability concerns arise close to a schedule product release that prompt a slightly different product configuration, or a new version of a product is developed years later) and require a new search—or at least a renewed analysis of existing search results.  Additionally, because patent applications are typically published 18 months from filing, though potentially only upon grant, at the time of any given FTO search there may be relevant but unpublished patent applications pending that can only be captured by a subsequent search. 

Invalidity and due diligence searches are reactive, in the sense that they are performed after some concrete or potential infringement risk comes to light or some need for confirmation arises.  There may still be deadlines for filing an answer to a complaint before a district court, filing a patent office challenge (e.g., PGR, IPR, opposition, observation), completing an M&A deal, or the like that constrains aspects of an invalidity search timeline.  One benefit of conducting FTO studies is that early identification of potentially problematic patents can help provide a longer time period for invalidity searching. 

III. Budgeting and Staffing

In the most general sense, patent searches are about solving information-retrieval problems.  While it is always (theoretically) desirable to conduct a “perfect” search, with hypothetical 100% recall and 100% precision, in the real world all searches are subject to constraints and a “perfect” search is not possible.  Those constraints may be in the form of deadlines or budget limits.  For instance, few if any patentability searches have an unlimited budget, and most operate on the premise of making reasonable efforts within some predetermined time and budget limits.  Any given search also usually includes express or implied limits on the databases, jurisdictions, and languages in which a search is conducted. 

The project management discipline has developed something called the “iron triangle”, which can be applied to the searching context.  In its most basic form, the “iron triangle” has three components: cost, speed, and quality.  Project management says you must pick only two of the three on any given project—you cannot have all three, even though those requesting searches often want all three.[7]  But the two factors selected will generally vary depending on the type of search, with patentability searches often sacrificing quality in order to achieve cost and speed objectives and with FTO and invalidity searches accepting slower timelines and/or greater costs in order to maintain high quality, for example.  Also note that if you do not pick two of the three, someone else will pick them for you.

For any given search, the amount of effort required varies depending on the density of patenting or publication in a given technology area.  Though at the beginning of a search that density may be unknown—usually this is only something that can be estimated well if numerous searches have previously been performed on related subject matter. 

graphic showing continuums of patent/publication density and search cost/effort
Continuums of Patent/Publication Density and Searching Cost/Effort

Searches can be performed by dedicated non-attorney searchers.  Though attorneys can perform searches themselves and it may be surprising to some that attorney-conducted searches may be more cost effective in some situations because legal analysis and searching per se can be done concurrently and features of search tools can help aid the legal analysis (e.g., by informing the attorney which particular keyword(s) returned a given reference and highlighting that term in a searchable document).  Prize/award-based “crowdsourcing” is another approach, usually best suited to invalidity searches. 

When retaining a person or firm to conduct a search, another common decision point is choosing between an experienced subject-matter expert (e.g., an in-house staff person) versus a “hired gun” (e.g., a professional searcher with little or no knowledge of the specific subject matter).  Because experience has a cost this is generally a cost vs. quality tradeoff. 

Export controls and similar provisions (e.g., ITAR, sanctions, DoD tranches) may prohibit (or limit) who can perform a given search, which may prohibit outsourcing a search abroad.[8]  For conventional export controls, the Commerce Control Lists (CCLs) establish what is subject to export restrictions.  The “EAR99” CCL designation generally applies to low-tech consumer goods that do not require export license in many situations.  However, “dual-use” export controls apply to things that have both military and non-military applications, and can be esoteric.  The Wassenaar Arrangement Control Lists also provide guidance on what is currently implicated by dual-use export controls.  Reexports and retransfers are also prohibited.[9]  As an example of reexport, consider that a person from a foreign country sends technical information to a searcher in the USA; the searcher may not be able to send (reexport) the same information back to that same person—here you can think of the USA like a “fly trap” for export-controlled information.  Lastly, even if you are operating entirely within the USA, a “deemed export” involves providing controlled information to a non-Green Card foreigner (or non-protected person) located in the USA.  That is, providing information to a foreigner physically located in the USA is “deemed” to be the same as exporting that information to the foreigner’s home country. 

IV. Resources for Conducting Searches

A. Introduction

It is often said that patent searching is more an art than a science.  There is no one-size-fits-all formula for conducting effective and reliable searches.  While searching does not become “easy” simply because you wish it was easy, and there is a learning curve, it is a skill that can be learned.  Fortunately, there are many resources available on the mechanics of how to conduct searches in the patent context.[10] 

There are many patent searching tools available, most of which are computer databases accessible via the Internet, though patent offices and affiliated libraries do offer some additional on-site-only resources for patent searching.[11]  Search engine functionality varies widely, with advanced features like proximity searching, bulk export of selected data, and highlighting of keywords within search results available only with certain tools.  In general, paywalled proprietary databases tend to have bells and whistles that can be helpful and save time.  Though user licenses can be significant and searchers who perform searches only occasionally will likely find yearly license fees cost-prohibitive. 

Whatever particular tool is used it is important to familiarize oneself with the scope of coverage (which jurisdictions are covered? what time periods are full-text searchable?) and the database’s search syntax, which refers to the particular operators used to command the search engine to run queries in particular ways.  While basic Boolean operators like “AND” and “OR” tend to be fairly universal, some tools require those operators to be in all caps.  More advanced operators like proximity commands or wildcard characters can be very different from one tool to the next.  Search syntax matters because a given search string or query inputted verbatim into different database tools may be parsed quite differently. 

B. Principal Database Searching Methods

While it is still possible to search for information by visiting a library and locating relevant physical publications, most patent-related searching today is performed using computer databases, most of which are now accessible via the Internet.  Computer databases useful for patent searches can be queried in a number of different ways that are ultimately a function of both the database’s search engine and metadata structure and content. 

Keyword searching, which is generally synonymous with Boolean searching, involves inputting keywords and Boolean operators (“AND”, “OR”, etc.) into a keyword string or query that the database’s search engine parses against its contents.  Database search engines have vastly different capabilities.  For instance, proximity operators are available with some databases that can be used to greatly increase the precision of keyword-based queries.  Moreover, some databases are only full-text searchable back to a particular historical date.  But be aware that not all databases or search engines are perfect, even the most full-featured tools.  A database populated by text obtained through optical character recognition (OCR) may contain errors that frustrate efforts to use the database’s search engine.  Databases that utilize machine translations of foreign patent documents for search queries may produce unreliable or unexpected machine translations in some instances.  Also, and perhaps less widely known, is that database providers may parse keyword strings in ways that deviate from what a searcher has inputted in given query.  For instance, in order to manage server computer hardware loads, to “enhance” searches, or for another reason, some search engines will include automatic thesaurus and/or pluralization functions or will parse defined phrases with spaces in between characters or as proximity searches (without the user inputting proximity operators).  Such features may increase recall but at the same time may decrease precision.  Conversely, some search engines may limit proximity searches to individual paragraphs, omitting from search results the presents of terms that are as proximate as specified but in two different paragraphs, which may reduce recall. Thesaurus functions can be helpful but cannot be relied upon exclusively.  Not only are the thesaurus lists generally hidden within a “black box” of the database software, but they often are inadequate to capture variations in technical terms used in the patent context.  For instance, an automatic thesaurus is unlikely to recognize that while “shopping cart” is a term used commonly in the USA, in Great Britain and Australia the term “shopping trolley” is used instead, or be able to ascertain whether “backhoe” and “excavator” are synonyms within the context of a given patent search.  Translations can also be a factor here.  For that matter, patentees can act as their own lexicographer.  So, a search for automatic license/number plate recognition (ALPR/ANPR) tools using conventional terminology may not locate a patent that describes such a tool as a “parkulator” (a completely invented word).[12] 

Classification searching involves reviewing patent documents according to subject-matter classifications made by patent offices.  Historically, before the advent of computerized databases, searching hard copies of patent documents by classification (stored in “shoes” at the USPTO) was the only way to search patents.  Today, there are a number of different classification systems currently in use, as well as ones that are no longer in use but can still be utilized in some situations.  The main classification systems currently in use are the International Patent Classification (IPC) system, the Cooperative Patent Classification (CPC) system, the Japanese File Index/Facet (FI) and File Forming Term (F-term) systems, and the Locarno system for design patents (and industrial designs).  The main discontinued classifications that may still have some value are the U.S. Patent Classification (USPC) system and the European Patent Classification (EPC) system, both of which were retired by the end of 2014.  While patent offices have made efforts to re-classify old patent documents under the current classification systems, the nature of differences between the old and new classification systems means that there is at most a statistical correspondence between the two that varies for different subject matter areas. 

In general, patent classification systems establish a taxonomy that includes multiple levels of detail, with highest-level classes and then multiple sub-classes within them (plus main groups/subgroups for the IPC).  Manuals or guides are available explaining these taxonomies, including descriptions of the subject matter that each class/subclass is intended to cover.  Classifications are assigned to individual patent documents by examiners, generally at an initial or pre-examination stage.  Because the process of classification necessarily involves certain judgments, classifications can be prone to human error.  Moreover, even in the best of circumstances, classifications are never perfect.  For instance, at any given point in time classification systems may not recognize certain emerging technology areas in the sense that there may not be specific sub-classes in which to categorize such emerging technologies.  Also, there may be subtle disclosures, such as things shown only in figures and/or left up to implication in the text, that may not be completely captured by a given patent office’s classifications.   And yet, classifications can be a useful tool for patent searching.  It is possible to combine classifications with particular keywords/terms as part of Boolean searches, which can be a powerful strategy to increase the precision of queries.

Forward and backward searching refers to searching references that are cited in the “references cited” section of a given relevant patent (backward searching) or searching for later patents in which the “references cited” section cites back to a given relevant patent (forward searching).  The essence of forward and backward searching is that you begin with one or more patent documents that you have already determined to contain relevant subject matter and then you piggyback on prior searches that others have performed in relation to other patents, leveraging the past efforts of others to compile groupings of technologically-related references.  In other words, forward and backward searching treats citations within and back to a given relevant patent as a kind of quasi-classification system. But forward and backward searching can never be the initial starting point for a search, because you must begin with at least one relevant patent—ideally the most relevant patent(s) you could find through other avenues.  Yet forward and backward searching can sometimes be the most effective method for increasing search recall. The same principles can be applied to citations and bibliographies in non-patent literature.

Other available patent searching tools utilize natural language searching, semantic searching, machine learning, artificial intelligence (AI), or the like.  Typically, such search tools utilize some type of proprietary algorithms to parse and execute search queries.  With natural language, semantic, or AI searching, the general idea is often that a searcher inputs words in a narrative format—unlike formal Boolean search syntax—and the tool applies its algorithms to convert those inputs into one or more queries that are run against the database to produce a result set.  Of note is that the use of proprietary algorithms mean that the search tool runs these searches through what can be called a “black box”—the searcher typically has no idea what happens inside that black box and thus no real sense of what might be omitted or excluded from the result set.  Some evidence from the legal research context suggests that experienced attorneys do not trust “black box” search algorithms.[13]  There are some tools, potentially most applicable to design patent searching (or trade dress searches), in which a searcher uploads an image file and the search tool utilizes image matching software to try to locate documents containing similar images.  As of today, these patent searching tools are still “bleeding edge” technologies and there is generally a lack of independent and peer-reviewed studies to validate the effectiveness of AI (etc.) them.  Vendors frequently make claims about studies they have done themselves, though it is difficult to place any trust in such studies used for marketing purposes when often there is no disclosure of the methodology employed and vendors have strong incentives to succumb to self-interest bias.  Much of the available literature on the subject can be characterized as marketing “advertorials”.  And for that matter, because AI-powered tools almost always rely on proprietary “black box” algorithms there is no way to know how a study involving one black box relates to another black box—or even if a study of a given search tool in the past still applies to that search tool, given that changes to that proprietary tool may occur within the black box without any way for a user or investigator to know.  These concerns echo the reasons experienced attorneys do not trust black box legal research tools or with concerns over law enforcement usage of AI-powered facial-recognition tools or the like[14]—control is being surrendered over aspects of search queries to algorithms premised on undisclosed assumptions that may be important to outcomes.  Perhaps as AI-powered patent searching tools become more sophisticated they will play a larger role.  But as of today, a good rule of thumb is to rely principally on human-guided searches and utilize AI-type search tools only in a secondary sense, such as for initial “quick-start” or “drunk walk” search orientation efforts before running substantive queries, last-step search validation or auditing, or supplemental, almost “hail Mary pass” efforts to perhaps get lucky and find additional art otherwise missed through other avenues. 

C. INID and Kind Codes

“INID” is an acronym for “Internationally agreed Numbers for the Identification of (bibliographic) Data” on patents.[15]  Also referred to informally as “field codes”, these codes embody a system for the international harmonization of bibliographic data (or metadata) appearing on published patent documents, such as for the title (54), assignee name(s) (73), filing date (22), issue/grant date (45), etc. 

INID CodeData Field Content
(10)Publication/Patent Number
(22)Filing Date
(43)Application Publication Date
(45)Grant/Issue Date
(54)Title
(71)Applicant(s)
(73)Assignee(s)
(86)PCT Filing Details
Table of Selected INID “Field” Codes

These INID/field codes can be tremendously useful when fetching individual patents in another language.  For example, either by referring to a table that identifies INID code numbers and the associate data content description, or simply referring to a sample patent in a known language, the location of similar data on a foreign-language patent document can be located.  While machine translations of patent documents are now frequently available, machine translations of very old patent documents are not always available.  And many machine translation tools may be unable to reliably parse PDFs of old patent documents, though it may be possible to perform limited machine translations of just particular text—such as within a given INID field—to ascertain some basic information about a given patent document.  

Comparison of INID “Field” Codes Between Patents in Different Languages
Camera-Based Partial Machine Translation of Selected Patent INID “Field” Contents

“Kind codes” identify type of patent and generally appear as a letter or letter and number (e.g., “A1” or “B2”) immediately after the patent or publication number.[16]  The USPTO only began printing kind codes on U.S. patent documents on January 2, 2001, so older U.S. patents lack a kind code.  Some countries have different types of patents—like utility models—that do not exist in the USA.  And systems for assigning numbers to patents has, historically, not been particularly harmonized internationally. 

Perhaps most confusing to U.S. practitioners is the Japanese patent numbering system, which in the past has assigned the same numbers to different patents differentiated only by the kind code. 

Another potentially confusing practice involves the treatment of old U.S. patents that were issued without any kind code on their face to which the “A” kind code (without a numerical digit) is retroactively applied.  This is potentially confusing because WIPO kind codes “A1”, “A2”, and “A9” are currently used by the USPTO to designate published patent applications.  In other words, the old granted patent “A” designation retroactively applied in searching databases can easily be confused with the new “A1”, “A2”, and “A9” designations that apply only to published applications and never to granted patents. 

V. Metrics for Evaluating Searches

The key metrics for evaluating information retrieval effectiveness in the patent context (or other library science contexts) are “precision” and “recall”.  Recall is the proportion (%) of all relevant documents retrieved in a given search query or overall search; recall is hypothetically expressed as:

Recall = relevant documents actually retrieved divided by universe of all relevant documents

Precision is the proportion (%) of relevant documents versus irrelevant documents retrieved in a given search query or overall search; precision is hypothetically expressed as:

Precision = relevant documents retrieved divided by total relevant & irrelevant documents retreived

Put yet another way, perhaps more informally, “recall” measures how completely relevant materials have been located while “precision” measures how much unwanted or useless material has been captured.  Library science has shown that recall and precision are tradeoffs with any given search query.[17]  That is, increasing recall generally decreases precision and vice-versa

In the patent searching context, 100% recall is always desired but 100% recall is only a theoretical objective.  There is no way to know, definitively, the universe of all relevant materials against which to compare a given query or overall search.  After all, if you knew at the outset what all the relevant materials were, why would you bother conducting a search merely to confirm what you already know?  And yet, if further searching returns additional relevant results than you can retroactively determine that a prior query or set of queries had less than 100% recall.  The rub is that this aspect of ascertaining 100% recall is like the impossibility of “proving a negative”: no matter how much you have searched, you can never know for certain whether or not there is at least one additional relevant reference that you simply have not yet located.  At best, certain protocols can be employed as proxies for 100% recall (e.g., searching X number of databases for Y number of hours, finding no more relevant results after a certain number of queries or period of time, etc.) but such proxies necessarily involve certain assumptions that can always be questioned. 

When is search complete? graphic, with learning curve-like graph

Practically speaking, precision is about efficiency and reliability.  Search results that include irrelevant materials can raise questions about the reliability of the search—did the searcher not understand the specific information being sought?—and can also waste time—both in terms of requiring more efforts by the searcher but also impeding later work by anyone else utilizing imprecise, cluttered search results for a legal analysis or to make business decisions. 

Some patent searching databases assign “relevancy” scores to individual results in a given query.  Such “relevancy” scores are generally determined by proprietary algorithms baked into the database’s software.  Whether the judgments embodied in those algorithms suit a searcher’s purpose is hard to know.  Most vendors do not make public the algorithms or weighting used to assign relevancy scores.  Searchers can anecdotally attest to performing searches where the “silver bullet” prior art was assigned a low relevancy score by the database tool.  Database “relevancy” scores are not meaningful as metrics for assessing search quality.  Though they might still be useful in (theoretically) presenting voluminous query results in an order that is more helpful for review. 

VI. Evidentiary Burdens, Privileges, and Disclosure Duties

A. Evidentiary Issues for NPL References and Public Use/On-Sale Evidence

District court proceedings where invalidity is alleged follow the Federal Rules of Evidence (FRE).[18]  PTAB trials at the USPTO (PGRs, IPRs, Derivations) also generally follow the FRE.[19]  Under the FRE, a document introduced as “prior art” must satisfy rules requiring authentication and prohibiting hearsay (unless an exception applies).[20]  These evidentiary burdens are not a concern for U.S. patents and published patent applications, which as “public records” are self-authenticating (FRE 902) and fall in a hearsay exception (FRE 803(8)) to qualify them as prior art.  In practical terms, authentication/hearsay objections to foreign patents and published patent applications are unlikely—and a party bringing such objections is just going to annoy the judge.  Evidentiary issues surrounding domestic and foreign patents should really only arise in rare if not unheard-of situations where forgeries are being proffered.  But evidentiary issues for NPL references, on-sale activities, and the like cannot be taken for granted.[21]  For instance, the Federal Circuit has held that “corroboration is required of any witness whose testimony alone is asserted to invalidate a patent.”[22] 

However, during ordinary examination and reexamination lower evidentiary standards apply to NPL references and compliance with the FRE (or similar PTAB trial procedures) is unnecessary.[23]

B. Qualifying a Non-Patent Literature (NPL) Reference as “Prior Art”

In order to qualify as “prior art”, an NPL reference must be established as a “printed publication” that was sufficiently generally “publicly accessible” before the critical date—and that must be done in accordance with the hearsay and authentication requirements discussed above for district court litigation and PTAB trials.  “‘[P]ublic accessibility’ has been called the touchstone in determining whether a reference constitutes a ‘printed publication[.]’”[24]  “A reference will be considered publicly accessible if it was disseminated or otherwise made available to the extent that persons interested and ordinarily skilled in the subject matter or art exercising reasonable diligence[] can locate it.”[25]  “Whether a reference is publicly accessible is determined on a case-by-case basis based on the ‘facts and circumstances surrounding the reference’s disclosure to members of the public.’”[26]  Limited distribution, even to those skilled in the art, may not amount to “publication” unless the material is otherwise so situated that “anyone who chooses may avail himself of the information it contains.”[27]  While evidence of “indexing” can establish public accessibility, it is merely one among many ways to establish “printed publication” status.[28]

In the invalidity context, it cannot be assumed that a copyright date, printing date, Internet server upload date, or the like appearing on the face of the NPL reference will necessarily satisfy public availability evidentiary requirement.  Courts and the PTAB have rejected such arguments in numerous cases.[29]  But clear indications on or within an NPL reference evincing publication by an established publisher with traditional hallmarks of publication (ISBN, etc.) might be sufficient.  Courts and the PTAB have occasionally found such evidence sufficient, particularly when dealing with very conventional publications like a (hard copy) book from a well-known publisher or an industry standard published by a well-known standards-setting organization.[30]  Commonly, the publication of NPL materials is substantiated by testimony from a librarian who cataloged or indexed a book or magazine as of a particular date.  In those situations, merely finding a particular disclosure is not enough to qualify the NPL reference as “prior art”—it may also be necessary to locate a witness to corroborate public accessibility. 

Materials from Internet web sites constitute an important category of NPL references where hearsay concerns can be significant.  Once again, the mere appearance of a date on the face of a web page (or in the URL, etc.) will not ordinarily be sufficient to establish the web page as a printed publication that qualifies as “prior art” before a critical date.  It is possible to offer witness testimony to establish web page publication, just as with books, magazines, and the like.  But one unique avenue for satisfying evidentiary burdens to qualify a web page as prior art is to utilize archival records from the Internet Archive’s “Wayback Machine” (archive.org) or its equivalent.[31] 

Other ephemeral or temporarily displayed materials, such as posters, videos, or presentation slides temporarily displayed at trade shows or conferences, raise other unique issues regarding their status as printed publications that qualify as “prior art”.[32]  But they may qualify as prior art under various circumstances.

Also remember that standards of proof differ depending on where a validity or unpatentability challenge is lodged.  A preponderance of the evidence standard is applied at the USPTO whereas a clear and convincing evidence standard is applied in district courts.  These different standards of proof can make a difference in whether or not available evidence establishes something as “prior art”.  Some reported decisions are less than explicit about the standard of proof being applied, which may help make sense of decisions that otherwise might seem inconsistent. 

C. Privileges and Their Limits

The attorney-client privilege and work-product protections may shield certain materials from discovery.  Yet materials generated in connection with searches are not always privileged. 

Attorney-client privilege only applies to communications actually sent, and only when seeking or providing legal advice.[33]  Such communications must involve an attorney or someone acting under the supervision and control of an attorney; and just because a communication involves an attorney does not automatically mean it is privileged.  Privilege law varies across district courts and circuits but some courts have held that the acts of counsel, the general topics of discussion, and the ultimate legal conclusions are not privileged.  Moreover, communications involving in-house counsel are often scrutinized more closely and may be subject to a rebuttable presumption that they contain business rather than legal advice.[34]  In general, anything that would have to be disclosed on a privilege log in litigation is not privileged.[35]  For instance, a privilege log may need to identify patents by number, and documents found during patent infringement/prior art searches or reviewed by an expert witness might need to be identified or disclosed.[36]

Work product protection can preclude discovery of documents and tangible things that are “prepared in anticipation of litigation or for trial” by or for another party or its representative (including the other party’s attorney, consultant, or agent).[37]  Work product protections can apply to the work product of non-attorneys but will not apply where business considerations predominate (regardless of the involvement of any attorney(s)).[38]  Also, work product protection is not absolute, and can be overcome if the materials are otherwise discoverable and the party seeking discovery shows that it has substantial need for the materials to prepare its case and cannot, without undue hardship, obtain their substantial equivalent by other means.[39] 

D. Duty of Disclosure

The duty of disclosure applicable to pending U.S. patent applications may necessitate some additional steps when relevant prior art is located in a search.  “Each individual associated with the filing and prosecution of a patent application has a duty of candor and good faith in dealing with the [USPTO], which includes a duty to disclose  . . . all information known to that individual to be material to patentability . . . .”[40]  Deliberately withholding prior art material to patentability can render any resulting patent unenforceable.[41]  Anyone substantively involved[42] with pending U.S. patent application(s) who receives search results on related subject matter should consider the need to submit search results on an information disclosure statement (IDS). 

VII. Considerations for USPTO Post Grant Proceedings

Strategic and legal considerations regarding the type of post-grant challenge that might be leveled against a given patent will vary depending upon the available prior art.  For that reason, it is usually desirable to locate prior art, or at least most of it, before deciding which type of post-grant proceeding will be sought.  For example, certain types of post-grant proceedings can only be based on patents and printed publications.  Moreover, the thresholds for institution of different types of proceedings vary;[43] though one constant is that all USPTO post-grant proceedings are based on a preponderance of the evidence standard for patentability while district court actions are based on the higher clear and convincing evidence standard for invalidity.  Lastly, IPR institutions are on an all-or-nothing basis.[44] 

PTAB trials involve estoppel that bars any further challenge based on any ground the petitioner “raised or reasonably could have raised” in a prior proceeding.[45]  This amounts to a “use it or lose it” (or “cite it or forgetaboutit”) opportunity to present prior art in a PTAB trial petition for institution, and requires that essentially all prior art searching be completed before the petition is filed—and therefore, practically speaking, before a petition can be prepared by legal counsel.  The PTAB and some district courts have interpreted and applied the “raised or reasonably could have raised” standard by asking whether a skilled searcher conducting a diligent search reasonably could have been expected to discover the prior art reference in question.[46]  The PTAB has applied this standard in a rather strict and formalistic way, showing no sympathy if locating a patent document would have required “brute force” manual review of tens or hundreds of thousands of patents not in any apparently relevant class/subclass.[47]  Courts have also considered whether particular prior art, such as a physical product that is not a “printed publication” but was potentialyl was on sale or in public use, reasonably could have been raised in a PTAB trial.[48] 

On the other hand, estoppel does not arise from ex parte reexamination.  Though collateral estoppel (issue preclusion) from prior litigation can still apply to reexams.[49]  Additionally, it is more difficult to sustain the unchanging clear and convincing evidence burden for subsequent invalidity challenges in district court when the USPTO already considered the same art.[50] 

VIII. Considerations for Design Patents

For design searching, “brute force” manual review of all designs in relevant classifications is recommended.  A major reason is that design patents have so little text that keyword searching is frequently unreliable.  There is so little text that explanatory statements that normally appear in utility patents to provide an enabling disclosure for functional features is absent in design patents focused on ornamentation.  And what little text there is in design patents might use words that are not especially helpful or reliable for purposes of a keyword search.  Though initial keyword searching can still be helpful when seeking to identify relevant Locarno classifications.  Another tip is to be sure to check for alternate embodiments that are not depicted in a representative figure on the front page of the design patent (or shown in representative thumbnail views by search tool databases).  Another point to consider for design patents is that some subject matter areas may have relatively few prior art design patents, even in areas where commercial products have been sold and publicly used for decades.  In such situations it may be necessary to investigate non-patent prior art in the form of evidence of on-sale activities, etc.  Such investigations may call for a substantially different approach and timeline.  Actually, such investigations may require considerable effort to locate and authenticate suitable evidence including witnesses/declarants

Another challenge with design patent searching is that the Federal Circuit looks to the perspective of the ordinary designer (rather than the ordinary observer) for obviousness analyses and applies a two-part Durling test for obviousness.[51] 

Somewhat unique to design patents is the opportunity to present prior art for a three-way comparison between images of the asserted patent, the prior art, and the accused product in “close” cases.[52]  The burden is on the accused infringer to bring forward evidence of the closest prior art for a three-way comparison.[53] 


Photo of Austen Zuege

Austen Zuege is an attorney at law and registered U.S. patent attorney in Minneapolis whose practice encompasses patents, trademarks, copyrights, domain name cybersquatting, IP agreements and licensing, freedom-to-operate studies, client counseling, and IP litigation. If you have patent, trademark, or other IP issues, he can help.


Book available from the ABA. Includes extended treatment of patent searching techniques.


Endnotes:

[1] See Paul C. Haughey, “Patentability Searches – Bust or Save In-house Counsel Budgets?” Lexology (April 9, 2020).

[2] E.g., Digeo, Inc. v. Audible, Inc., 505 F.3d 1362, 1369-70 (Fed. Cir. 2007) (case not exceptional [frivolous] because defendant did not establish plaintiff knew or should have known it lacked legal title due to alleged forgeries); Q-Pharma, Inc. v. Andrew Jergens Co., 360 F.3d 1295, 1303 (Fed. Cir. 2004) (reasonable belief in presumption of validity made suit nonfrivolous under Rule 11). 

[3] See, generally, Patent Freedom to Operate Searches, Opinions, Techniques, and Studies (Austen Zuege, ed., ABA 2017).

[4] 35 U.S.C. § 298; see also Knorr-Bremse Systeme Fuer Nutzfahrzeuge GmbH v. Dana Corp., 383 F.3d 1337, 1345-46 (Fed. Cir. 2004) (en banc) abrogated by In re Seagate Tech., LLC, 497 F.3d 1360, 1371 (Fed. Cir. 2007) (en banc); Seagate, 497 F.3d at 1371 abrogated on other grounds by Halo Elecs., Inc. v. Pulse Elecs., Inc., 979 U.S. ___, 136 S. Ct. 1923 (2016).

[5] See, e.g., The Sedona Conference, “Commentary on Patent Litigation Best Practices: Willful Infringement Chapter” (July 2020 Public Comment Version).

[6] There may be rare situations in which relevant prior art is already known without conducting an invalidity search, though even then a search to locate additional prior art may have value. 

[7] E.g., Mike Morrison, “The Project Management Triangle – Time, Quality, Cost – You Can Have Any Two,” (March 23, 2017).

[8] See 15 C.F.R. Part 730; 22 C.F.R. Part 120; 31 C.F.R. Part 501; Bureau of Industry and Security, “Export Administration Regulations (EAR)”; U.S. Department of State, “Understand the ITAR and Export Controls”; Krauland et al., “State Department Publishes Long-awaited ITAR Rule on Encryption and Other Excluded Activities” Lexology (Dec. 24, 2019); U.S. Dept. of the Treasury, “Sanctions Programs and Country Information”; U.S. Dept. of the Treasury, “Consolidated Sanctions List Data Files”; U.S. Dept. of Defense, “DOD Releases List of Additional Companies, In Accordance with Section 1237 of FY99 NDAA” (Jan. 14, 2021).

[9]  15 C.F.R. § 734.14; 22 C.F.R. §§ 120.19 and 120.51.

[10] E.g., Patent Freedom to Operate Searches, Opinions, Techniques, and Studies, supra; Stephen P. Harter, Online Information Retrieval: Concepts, Principles, and Techniques (1986); David Hunt et al., Patent Searching: Tools & Techniques (2007); WIPO, “Guidelines for Preparing Patent Landscape Reports” (2015).

[11] See WIPO, “INSPIRE” at <https://inspire.wipo.int>.

[12] U.S. Pat. No. 6,243,029 B1.

[13] Brian Sheppard, “Does Machine-Learning-Powered Software Make Good Research Decisions? Lawyers Can’t Know for Sure,” ABA Journal New Normal (Nov. 22, 2016).

[14] See, e.g., Grother et al., National Institute of Standards and Technology (NIST), “Face Recognition Vendor Test (FRVT) – Part 3: Demographic Effects,” NISTIR 8280 (Dec. 2019); “What is Math Washing?,” (last visited Jan. 29, 2021); Lyle Moran, “Pretrial Risk-Assessment Tools Should Only Be Used if They’re Transparent and Unbiased, Warns ABA House” (Feb. 14, 2022).  Note also Jacques Lacan’s concept of “university discourse.”  See, e.g., Jeffrey & Dolar, “The Sting of Knowledge,” The Point Magazine, Issue 16 (April 23, 2018); Bruce Fink, The Lacanian Subject (1995), pp. 132-33.

[15] WIPO, “Standard ST.9: Recommendation Concerning Bibliographic Data on and Relating to Patents and SPCS” (June 2013); WIPO, “Standard ST.80: Recommendation Concerning Bibliographic Data Relating to Industrial Designs” (February 2004); MPEP § 901.05(b)

[16] WIPO, “Standard ST.16: Recommended Standard Code for the Identification of Different Kinds of Patent Documents” (Oct. 2016); MPEP § 901.04(a); USPTO, “‘Kind Codes’ Included on the USPTO Patent Documents” (Sept. 18, 2013).

[17] David M.W. Powers, “Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation,” 2(1) J. Machine Learning Techs. 37, 37-63 (2011).

[18] See also, generally, Evidence in Patent Cases (Dorsney, ed., Bloomberg Law 2018).

[19] 37 CFR § 42.62; see also USPTO, “Hearsay and Authentication” (Dec. 6, 2018).

[20] Article VIII – FRE 801-807: Hearsay; Article IX – FRE 901-903: Authentication; but see, e.g., 37 CFR § 42.61 (admissibility in PTAB trials). 

[21] Cf. Wi-LAN Inc. v. Sharp Elecs. Corp., 992 F.3d 1366 (Fed. Cir. 2021) (proffered evidence of infringement inadmissible at summary judgment for lack of suitable authentication; 3d Circuit law applied).

[22] Texas Digital Sys., Inc. v. Telegenix, Inc., 308 F.3d 1193, 1217 (Fed. Cir. 2002); see also Juicy Whip, Inc. v. Orange Bang, Inc., 292 F.3d 728, 741-42 (Fed. Cir. 2002) (“rule of reason” and factors for assessing sufficiency of corroboration).

[23] E.g., MPEP § 2128 (date on a document can establish its publication for examination purposes unless applicant challenges it); Ex parte Grillo-López, Appeal No. 2018-006082 at *2-3 (PTAB, Jan. 31, 2020) (precedential) (lower examination standards vs. higher IPR standards to establish publication).

[24] In re Hall, 781 F.2d 897, 899 (Fed. Cir. 1986).

[25] Medtronic, Inc. v. Barry, 891 F.3d 1368, 1380 (Fed. Cir. 2018).

[26] In re Lister, 583 F.3d 1307, 1311 (Fed. Cir. 2009) (quoting In re Klopfenstein, 380 F.3d 1345, 1350 (Fed. Cir. 2004)); see also Blue Calypso, LLC v. Groupon, Inc., 815 F.3d 1331, 1348 (Fed. Cir. 2016) (“Whether a reference qualifies as a printed publication is a legal conclusion based on underlying factual determinations.”).

[27] In re Bayer, 568 F.2d 1357, 1360, 1362 (CCPA 1978).

[28] E.g., In re Lister, 583 F.3d at 1312 (“neither cataloging nor indexing is a necessary condition for a reference to be publicly accessible.”).

[29] E.g., Samsung Electronics Co., Ltd. v. InfoBridge Pte. Ltd., IPR2017-00099,-00100, Paper 43 (PTAB, Nov. 13, 2020) (remanded from Samsung Elecs. Co., Ltd. v. InfoBridge Pte. Ltd., 929 F.3d 1363, 1372 (Fed. Cir. 2019)); Acceleration Bay, LLC v. Activision Blizzard Inc., 908 F.3d 765, 773 (Fed. Cir. 2018); Blue Calypso, LLC v. Groupon, Inc., 815 F.3d 1331, 1348-49 (Fed. Cir. 2016); Open Text SA v. Box, Inc., No. 3:13-cv-04910, 2015 WL 4940798 at *7 (N.D. Cal. Aug. 19, 2015); CNET Networks, Inc. v. Etilize, Inc., 584 F. Supp. 2d 1260, 1274 (N.D. Cal. 2008).

[30] E.g., VidStream LLC v. Twitter, Inc., 981 F.3d 1060, 1066-67 (Fed. Cir. 2020); Hulu, LLC v. Sound View Innovations, LLC, Case IPR2018-01039, Paper 29 at *17-21 (PTAB, Dec. 20, 2019) (precedential); Ericsson Inc. v. Intellectual Ventures I LLC, IPR2014-00527, Paper 41 (PTAB, May 18, 2015); Kyocera Wireless Corp. v. Int’l Trade Comm’n, 545 F.3d 1340, 1351 (Fed. Cir. 2008).

[31] Valve Corp. v. Ironburg Inventions Ltd., Nos. 2020-1315, 2020-1316, and 2020-1379, Slip Op. at *16  (Fed. Cir., Aug. 17, 2021); James L. Quarles III and Richard A. Crudo, “[Way]Back to the Future: Using the Wayback Machine in Patent Litigation,” ABA Landslide, Vol. 6, No. 3 (Jan./Feb. 2014); see also Internet Archive, “Frequently Asked Questions”; Archive.Today at <https://archive.is>.

[32] See, e.g., Medtronic, 891 F.3d at 1379-83; Initiative for Medics., Access & Knowledge (I-MAK), Inc. v. Gilead Pharmasset LLC, IPR2018-00123, Paper 7 at *8-11 (PTAB, June 13, 2018); In re Klopfenstein, 380 F.3d at 1347-50; Ecolochem, Inc. v. S. Cal. Edison Co., 227 F.3d 1361, 1369-70 (Fed. Cir. 2000).

[33] See, generally, Jackie Unger, “Maintaining the Privilege: A Refresher on Important Aspects of the Attorney-Client Privilege,” Business Law Today (Oct. 2013).

[34] E.g., Lindley v. Life Inv’rs Ins. Co. of Am., 267 F.R.D. 382, 389 (N.D. Okla. 2010), aff’d in part as modified, No. 08-CV-0379-CVE-PJC, 2010 WL 1741407 (N.D. Okla., Apr. 28, 2010).

[35] See F.R. Civ. P. 26(b)(5); see also, e.g., Travis S. Hunter and Sara M. Metzler, “Is It Privileged? A Young Lawyer’s Guide to Preparing a Privilege Log in Commercial Litigation,” ABA Practice Points (June 29, 2018); Michael Downey and Paige Tungate, “Practical Advice on Privilege Logs,” Law Practice Today (Sept. 14, 2018). 

[36] E.g., Baxter Int’l, Inc. v. Becton, Dickinson and Co., No. 17-C-7576 (N.D. Ill, July 26, 2019); BASF Catalysts LLC v Aristo, Inc., No. 2:07-cv-222, 2009 WL 187808 (N.D. Ind., Jan. 23, 2009); Adobe Inc. v. RAH Color Techs. LLC, Nos. IPR2019-00627, -00628, -00629, -00646, Paper 59 (PTAB, Dec. 12, 2019).

[37] F.R. Civ. P. 26(b)(3)(A)

[38] E.g., Sioux Steel Co. v. Prairie Land Millwright Servs., Inc., No. 1:16-cv-02212 (N.D. Ill., May 19, 2021) (factual information in technical drawings related to attempts to design around patent discoverable, but counsel’s mental impressions and opinions about those efforts and legal advice sought or received not discoverable); In re Google Inc., 462 F. App’x 975, 976-79 (Fed. Cir. 2012) (nonprecedential) (email captioned “Attorney Work Product” expressing need for license was a technical/business investigation not protected work product, nor attorney-client privileged); Takeda Chem. Inds., Ltd. v. Alphapharm Pty., Ltd., No. 04-cv-1966, 2005 WL 1678001 (S.D.N.Y., July 19, 2005) (routine patent searches in ordinary course of business, irrespective of litigation, not protected work product).

[39] F.R. Civ. P. 26(b)(3)(A)

[40] 37 C.F.R. § 1.56; see also MPEP Chapter 2000.

[41] Therasense, Inc. v. Becton, Dickinson & Co., 649 F.3d 1276 (Fed. Cir. 2011 (en banc); see also, e.g., GS Cleantech Corp. v. Adkins Energy LLC, 951 F.3d 1310, 1328-32 (Fed. Cir. 2020).

[42] See Avid Identification Sys. Inc. v. Crystal Import Corp. SA, 603 F.3d 967, 974 (Fed. Cir. 2010).

[43] See USPTO, “Major Differences Between IPR, PGR, and CBM”.

[44] SAS Institute, Inc. v. Iancu, 584 U.S. ___, 138 S.Ct. 1348 (2018).

[45] 35 U.S.C. §§ 315(e) and 325(e); see also Wi-LAN Inc. v. LG Elecs. Inc., 421 F. Supp. 3d 911, 923-24 (S.D. Cal. 2019).

[46] Wasica Finance GmbH v. Schrader Int’l, 432 F. Supp. 3d 448, 453 (D. Del. 2020); IBM Corp. v. Intellectual Ventures II LLC, No. IPR2014-01465, Paper 32 at 5 (PTAB, Nov. 6, 2015).

[47] Valve Corp. v. Ironburg Inventions Ltd., IPR2017-00137, Paper 43 (PTAB, Jan. 25, 2018).

[48] E.g., The California Inst. of Tech. v. Broadcom Ltd., No. 2:16-cv-03714, Slip. Op. at *7-10 (C.D. Cal., Aug. 9, 2019); Wasica, 432 F. Supp. 3d at 452-55; Star Envirotech, Inc. v. Redline Detection, LLC, No. 8:12-cv-01861, 2015 WL 4744394, at *4 (C.D. Cal. Jan. 29, 2015).

[49] In re Freeman, 30 F.3d 1459 (Fed. Cir. 1994).

[50] Am. Hoist & Derrick Co. v. Sowa & Sons, Inc., 725 F.2d 1350, 1360 (Fed. Cir. 1984); accord Microsoft Corp. v. i4i Ltd. P’ship, 564 U.S. 91, 110 (2011); see also Ingersoll-Rand Co. v. Brunner & Lay, Inc., 474 F.2d 491, 496 (5th Cir. 1973).

[51] Campbell Soup Co. v. Gamon Plus, Inc., 939 F.3d 1335, 1340-41 (Fed. Cir. 2019); High Point Design LLC v. Buyers Direct, Inc., 730 F.3d 1301, 1313 (Fed. Cir. 2013); Durling v. Spectrum Furniture Co., 101 F.3d 100, 103 (Fed. Cir. 1996); contrast KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 415-18 (2007).

[52] Egyptian Goddess, Inc. v. Swisa, Inc., 543 F.3d 665, 678 (Fed. Cir. 2008) (en banc); see also, e.g., Wallace v. Ideavillage Prods. Corp., No. 2015-107, 2016 WL 850860 (Fed. Cir., March 3, 2016) (nonprecedential).

[53] Egyptian Goddess, 543 F.3d at 678.