Your TV is watching you – Where does it end?

News reports (extremetech) indicate that Microsoft has filed for a patent whereby

“The users consuming the content on a display device are monitored so that if the number of user-views licensed is exceeded, remedial action may be taken.”

No doubt George Orwell’s Telescreen (from the novel 1984) will be cited as prior art in opposition. The Telescreen allowed the Party to keep its subjects under constant surveillance thereby encouraging a climate of self surveillance. Replace ‘the Party’ with MPAA and you pretty much have it.

HT: Francis K – who brought this story to my attention.

The Imaginary Conflict Between Fair Use and International Copyright Law


Fair use opponents and skeptics often question whether an open standard that relies on judicial application is compatible with international treaty obligations. In my view, outside the European Union, there is no merit in that contention. This post is my first attempt to clarify why there is no conflict between fair use and international copyright law.

Copyright is international

Since 1886, national copyright laws have been regulated by international agreements. In the era of globalization it is not surprising that copyright law is now covered by a multitude of overlapping international, regional and bilateral agreements.

The foundational international agreement concerning copyright law is the Berne Convention for the Protection of Literary and Artistic Works of 1886. The Berne Convention has been revised many times and is now supplemented by a host of international trade agreements including the TRIPs Agreement (Agreement on Trade-Related Aspects of Intellectual Property Rights) (part of the WTO framework), the WIPO Copyright Treaty and various regional (e.g., NAFTA) and bilateral free trade agreements (e.g., the Australia-US FTA). These agreements provide for mutual recognition of copyrights and establish minimum standards for copyright protection.

Questioning Fair Use

Fair use opponents and skeptics often question whether an open standard that relies on judicial application is compatible with the so-called “three step test” for limitations and exceptions contained in Berne and many subsequent agreements.

The Berne convention first came into being in 1886, but prior to 1967 the rights guaranteed by the convention were expressed quite specifically – one article dealt with works “published in the newspapers or periodicals”, another dealt with the “the exclusive right of authorizing the reproduction and public representation of their works by cinematography.” (Article 9 and Article 14). In 1967 the members of the Berne Convention adopted a very broadly expressed reproduction right in Article 9(1) that states simply:

“Authors of literary and artistic works protected by this Convention shall have the exclusive right of authorizing the reproduction of these works, in any manner or form.”

As a counterweight to this omnibus reproduction right, the new Article 9(2) provided:

“It shall be a matter for legislation in the countries of the Union to permit the reproduction of such works in certain special cases, provided that such reproduction does not conflict with a normal exploitation of the work and does not unreasonably prejudice the legitimate interests of the author.”

Those who doubt the legitimacy of fair use under the Berne Convention argue that fair use is too broad and uncertain and is thus not properly confined to “certain special cases” as Article 9(2) requires. In my view, this argument falls somewhere along the continuum between misguided and mischievous. Fair use is fundamentally consistent with the three step test as a matter of international law; moreover, so long as the U.S. retains the fair use doctrine, no other country could seriously be challenged for its adoption of fair use as a matter of international politics. This second proposition is so obvious that I will confine the remainder of my observations to the more interesting legal question.

The three step test is broadly applicable standard, consider …

   1. Drafting History

The drafting history of Berne Article 9(2) reveals that it was not intended as a rigid prohibition on limitations and exceptions to copyright, but rather as an abstract open formula capable of encompassing a wide range of exceptions. (See generally, Martin Senftleben, Copyright, Limitations, and the Three-Step Test: Analysis of the Three-Step Test in International and EC Copyright Law, Kluwer Law International, 2004.) Martin Senftleben has made a detailed study of the history of the three step test, beginning with its adoption in 1967. Senftleben’s research shows that an early draft of 9(2) used the language “in certain particular cases where the reproduction is not contrary to the legitimate interests of the author.” This text was modified at the suggestion of the United Kingdom delegation to the now familiar language — “in certain special cases where the reproduction does not unreasonably prejudice the legitimate interests of the authors.”

Although fair dealing in the U.K. by the late 1960’s was arguably narrower than fair use in the U.S., it was nonetheless an abstract standard (applied to particular circumstances) requiring judicial application and development. It is inconceivable that the U.K. intended to abandon fair dealing when it suggested that limitations and exceptions be limited to “certain special cases”. Read as a single sentence, Article 9(2) is a general statement that does as much to enable limitations and exceptions as it does to confine them. As is so often the case in international agreements, this generality was necessary in order to reconcile the many different types of exceptions various nations had already adopted.

   2. Subsequent International Agreements

Since the Berne Convention, several different versions of the three step test have been incorporated into international agreements such as TRIPs, the WIPO Copyright treaty, NAFTA and U.S. Free Trade Agreements with Australia, Bahrain, Chile, Jordan, Morocco, Singapore, and South Korea.

In some ways this proliferation seems to confuse the question of how the standard should be interpreted, but in other ways it is also clarifying. Admittedly, variations on the exact text of the three step test are confusing. Article 13 of the TRIPs Agreement of 1994 (part of the World Trade Organization) contains a version of the three step test which tacks the language Berne except that rather than “permit[ing] reproduction in certain special cases”, TRIPs instructs member nations to “confine” any limitations and exceptions to “certain special cases”. TRIPs also changes Berne’s reference to “the author” to “the right holder”. These transformations are repeated in Article 10 of the WIPO Copyright Treaty of 1996. TRIPs also contains a slightly different three step test for trademark law and another for patent law but the drafting history offers no explanation as to the motivation or significance of these subtle differences. (See, Annette Kur, Oceans, Islands, and Inland Water- How much Room for Exceptions and Limitations under the Three-step Test?)

If there was ever any doubt as to the flexibility of the three step test framework, that doubt should have been dispelled by Agreed Statement Concerning Article 10 WIPO Copyright Treaty. That statement reads in part:

It is understood that the provisions of Article 10 permit Contracting Parties to carry forward and appropriately extend into the digital environment limitations and exceptions in their national laws which have been considered acceptable under the Berne Convention. Similarly, these provisions should be understood to permit Contracting Parties to devise new exceptions and limitations that are appropriate in the digital network environment.

   3. State practice

The initial three step test was adopted before the U.S. joined the Berne Convention (this only happened in 1989). The U.S. made a number of changes to its law to fit into the Berne framework, principally concerning notice and registration requirements. The fair use doctrine does not appear to have been considered as an obstacle to Berne compliance at the time and it is hard to imagine that the U.S. would have agreed to the convention if it believed that such a central aspect of its copyright law was not Berne compatible. It is even harder to imagine that the U.S. would actively promote the incorporation of a three step test into TRIPs in 1994, the WCT in 1996 and FTAs with Australia, Bahrain, Chile, Jordan, Morocco, Singapore, and South Korea if the fair use doctrine really presented a fundamental conflict.

How should the three step test be applied?

The three-step test is an open-ended norm. There is nothing in the history or text of Berne, TRIPs or the WIPO Copyright Treaty that indicates any intention to abandon common law adjudication for limitations and exceptions. Even the one WTO panel decision that is frequently cited for a restrictive view of the words “certain special cases” also notes: “However, there is no need to identify explicitly each and every possible situation to which the exception could apply, provided that the scope of the exception is known and particularised. This guarantees a sufficient degree of legal certainty.” (Panel Rep. of 15 June 2000, United States-Article 110 (5) of the US Copyright Act, WT/DS160/R. at 6.108)

In a complex world, rules are not the only path to certainty.

In a fair use system, the contours of limitations and exceptions to exclusive rights are developed through adjudication – as the jurisprudence matures, the degree of legal certainty increases. Rule-based limitations and exceptions are quite vulnerable to technological rigidity and their application can hinge on arcane debates over taxonomy – these features can make rules perennially uncertain.

What does it mean to be special?

Like the fair use doctrine itself, the three step test about harm and justification. The three step test allows members to create their own limitations and exceptions to copyright so long as the limitation does not conflict with a normal exploitation of the work and does not unreasonably prejudice the legitimate interests of the author (Berne Article 9(2)). These requirements are preceded in Berne by the words “in certain special cases”, but these words are as much descriptive as they are limiting. Copyright exceptions that do not conflict normal exploitation and do not unreasonably prejudice the legitimate interests of the author are not the norm, they are “special cases.”

Consider the implications of reprinting a novel. Ordinarily copyright law gives the novelist the exclusive right to reproduce her own work and this exclusive right is the means by which profit is obtained – she can sell that right, or sell copies of the work – this potential profit is the incentive that urges her to write the novel in the first place. Usually, a law that allows others to copy the novel without permission directly challenges the novelists control over her work and her ability to monetize her creation. But there are cases where copying should be allowed because it does not conflict with a normal exploitation of the work and does not unreasonably prejudice the legitimate interests of the author – in the classic printing-press paradigm of copyright law these cases are unusual, some would even say “special”. In the printing press paradigm, justified exceptions allowing wholesale reproduction would be rare. In the digital age justified exceptions to the reproduction right are not quite as rare, but they are still “special” in the relevant sense. Many private uses, many uses already where access has already been paid for, de minimus uses, fractional uses for socially beneficial purposes and non-expressive uses should all readily be seen as special.

Recasting 107 in the language of Berne

To understand the consistency of fair use with the three step test, consider how the former doctrine can be reformulated in the language of the latter. There is nothing magical or sacrosanct about the particular language that the U.S. used to try to codify fair use in 1976. A Berne compliant fair use provision could just as well read:

“… the fair use of a copyrighted work in certain special cases, such as news reporting, quotation, commentary, criticism and analysis, scholarly research, is not infringement. In determining whether a use is a fair use a court shall consider: (1) whether the use conflicts with a normal exploitation of the work and thus tends to substitute for the work of the copyright owner; (2) whether the use is likely to prejudice the legitimate copyright interests of the copyright owner to a degree that is unreasonable in light of the purpose and nature of the use; and (3) whether the extent of the use is reasonable in light of the purpose and nature of the use.”


Countries like Australia that are currently considering the adoption of their own fair use doctrine should not be deterred by an imaginary conflict between fair use and international copyright law.

x-post on running data

I posted a few comments on the ownership of running data and some implications for IP and Antitrust law a few days ago on “Reading, wRiting and Running“. The post is repeated below, but to see the comments, go to

I started running in March last year when I borrowed someone else’s ipod nano and went for a jog in Menlo Park, California. I went running because I was just visiting and had no gym work out in. I took the nano to listen to music. But the experience of having the nano tell me every kilometer that passed and give me my run stats at the end got me hooked.

Running with data is way more fun than just running! Using a website like Nikeplus to track your data over time is even more fun.

But my frustration with the nano – it kept ending my runs early when I drenched it with sweat – led to me to want to edit the data. Nike does not let you do this. They don’t even let you merge two runs together. This led me to a very cool site that takes your nikeplus data and gives you some different data visualizations and lets you manually add and edit runs.

Smashrun have worked out how to get your data from the Nike website, but they are still working on Garmin. I now use a Garmin GPS watch to track my runs and calibrate my iPod nano at the end of each run. (I find that the nano is only accurate +/- 5%) I don’t use the Garmin website much because the interface is not great and they don’t analyze the data in ways I find interesting.

Some observations

Nike and Garmin obviously want obviously want to lock their users into their platforms as a means of giving you more reason to buy their gadgets and not to switch every time something better comes along.

Nikeplus is mostly a great website, but Smashrun is better in many ways. To compete with Nike as a data platform Smashrun needs access to the my data. But when I say ‘my data’ is that legally accurate? Do I really own my data to the extent that I have the right to port it somewhere else.

In the course of writing the post I have probably violated the “Platform Use Restrictions” of Nike’s Terms of Service because my “User Generated Content” is inextricably linked with their IP. I am confident that Nike won’t mind, but given that some U.S. prosecutors interoperate the Computer Fraud and Abuse Act as being violated by a TOS violation, maybe I should think twice.

Smashrun is probably not contractually bound by Nike’s TOS (quite a debatable point), but some courts might think that it is. Smashrun might be deemed to be in violation of the Nike TOS which provide that “You agree not to use any data mining, robots, scraping or similar data gathering methods.” (I might have also violated the TOS that reads “Keep your username/password secure and do not allow anyone else to use your username/password to access the Platform.”)

In the wake of the prosecution and subsequent suicide of Aaron Swartz, Representative Zoe Lofgren has drafted a bill that would exclude terms of service violations from the 1984 Computer Fraud and Abuse Act and from the wire fraud statute. This reform is long overdue.

As a matter of consumer protection, if not competition law, data the right of users to have third parties liberate their data aught to be the norm. Then we could have device competition between Nike and Garmin and data platform between Nike, Garmin, Smashrun and host of new entrants.



The Lost Tradition of Fair Use in English and Colonial Copyright Law, Comments on Ariel Katz, Fair Use 2.0

Ariel Katz, Fair Use 2.0: The Rebirth Of Fair Dealing In Canada (Draft, Jan. 24, 2013)

In previous work I have highlighted the English origins of the modern fair use doctrine in abridgement cases from 1710 to 1841 (“The Pre-History of Fair Use” (2011) 76:4 Brooklyn Law Review 1371). The question that I failed to address is, if fair use was part of the English copyright law tradition, why do England and her former colonies now adhere to a much narrower concept of fair dealing? Ariel Katz’s new paper gives us some answers this question.

Conventional wisdom holds that in Commonwealth jurisdictions like England, Australia, New Zealand and Canada fair dealing cannot apply beyond the explicitly enumerated purposes. In the U.S. by contrast, the statutory purposes are just illustrations. Thus we are left with (in Katz’s words) an “omnipresent flexible fair use regime in the United States, and a seemingly rigid and restrictive fair dealing tradition in the Commonwealth countries.”

Katz’s bold claim is that the conventional wisdom is wrong!

“…the history of fair use and fair dealing and shows that … the enactment of the Imperial Copyright Act of 1911 [was] not designed to cause any major alteration in the common law of fair dealing, and the explicit recognition of five enumerated purposes in the (then) newly-enacted fair dealing provision was not intended to limit the principle of fair dealing exclusively to those five purposes.” (page 3)

Katz makes a strong argument that University of London Press, Ltd. v. University Tutorial Press, Ltd. (1916), 2 1916 Ch 601, the first reported case on the newly enacted English “fair dealing” provision of the 1911 Copyright Act may have been misread over the years. But I think that the strongest parts of the paper are his treatment of the legislative history of the 1911 Act and its contemporary reception.

The legislative history of the Imperial Copyright Act of 1911

The 1911 Act provided that: “Any fair dealing with any work for the purposes of private study, research, criticism, review, or newspaper summary” shall not constitute an infringement of copyright. Arguing that there was indication in the legislative history that the 1911 Act was meant to curtail fair use or freeze it in time. Katz has studied the introduction of the bill in Parliament and the House of Lords, he notes (at page 26) that

“If the Bill contemplated major reform with respect to fair dealing, it would have been expected that such change would be mentioned, but it was not. Nor did Viscount Haldane, who introduced the Bill to the Lords, mention any contemplated change with respect to fair dealing.”

Quite the contrary, Viscount Haldane in the House of Lords stated:

“All we propose to do is to declare that for the future the principle of fair dealing which the Courts have established is to be the law of the Code. … The principle of fair dealing is a principle which the Courts have applied with the greatest care. … All that is done here is to make a plain declaration of what the law is and to put all copyright works under the same wording.”

Why codify fair dealing if no change to fair use was intended?

Katz notes (page 25) that

“a simple explanation [for the codification of fair dealing] might be that since the 1911 Act was mainly a project of consolidation of different acts and codification of different common law rules, it seemed prudent not to leave fair use without any statutory basis. … Another explanation … if the Act only recognized the expansion of the copyright but remained silent about limitations to those expanded rights, court might have interpreted that as a signal that Parliament had decided to abolish fair use.”

Katz argues that it is even possible that the fair dealing provisions in the 1911 Act may even have been an attempt to expand fair use.

“… it is possible that the purpose of specifying the five categories was not only to remove any doubts that fair dealing applied to those already recognized in the case law, but also to ensure that it applied to those who lacked solid grounding in the case law. In particular, the addition of ‘newspaper summary’ and ‘private study’, categories that had no direct precedent in the case law, can support this explanation.” (pages 25-26).

Reaction of Treatise Writers

Katz also does a wonderful job of surveying the contemporary reaction of copyright treatise authors to the 1911 Act. He summarizes (at page 30)

“… if by enacting the fair dealing provision Parliament had intended to modify the existing doctrine of fair use by confining it to five enumerated categories exclusively, most of the contemporaneous commentators failed to notice that intention.”

Some examples lifted from Katz’s paper:

J.M. Easton, Copinger on Copyright, 5th edition.

“[a]ny fair dealing, with, any work for the purposes of private study, research, criticism, review, or newspaper summary is also expressly permitted by the Act.”

“fair dealing for other purposes has always been … permitted and, presumably, it was not intended to cut down the rights of fair user previously enjoyed under the old law.”

JB Richardson, The Law of Copyright (London: Jordan & Son, 1913)

“The passing of The Copyright Act, 1911, has completely recast the Law of Copyright, at any rate those parts which depend primarily on Statute Law, such as the term of protection and ownership of copyright. Only those parts of the law which are practically judge-made—such as the questions as to infringement by a new work other than an exact copy—have remained to any great extent unaltered, and even they are not untouched.”

LCF Oldfield, The Law of Copyright (London: Butterworth & Co., 1912)

“[w]hat is fair dealing with a work depends upon the circumstances of each particular case”

How did the restrictive view of the 1911 Act come to dominate?

In terms of copyright law treatises, Katz’s research indicates that “[t]he view that Parliament had intended to restrict fair dealing to the five enumerated purposes began appearing later. … In 1927, the sixth edition of Copinger was published. This edition was no longer authored by Easton, but penned by F. E. Skone James and published by a different publisher.” (page 30)

Katz argues that University of London case of 1916 which is treated as confirming the narrow scope of fair dealing has long been misunderstood. If he is correct, what deserves further exploration is why such a misunderstanding should have taken such firm hold of copyright law in England, Australia, New Zealand … and until recently, Canada.


Embracing the Digital Economy – the 2013 Australian Digital Alliance Copyright Forum

The Australian Digital Alliance will hold its 2013 annual copyright forum, ‘Embracing the Digital Economy: creative copyright for a creative nation’. The forum will considers how Australia’s copyright framework fits in with the ‘digital world’. A timely contribution given that the Australian Law Reform Commission is in the middle of an Inquiry into Australia’s copyright framework, to determine whether existing exceptions are adequate and appropriate in the digital environment.

Australia’s copyright framework has not kept pace with technology or society. The digital age has profoundly changed to the nature of creation and distribution. The divide between ‘producers’ and ‘consumers’ has been blurred in some cases and eliminated all together in others. The production and distribution of creative works is more democratic and more chaotic than ever before.

The ADA forum explores a variety of new technologies, business models and education and cultural services being provided online, and how they fit into our existing copyright framework.

I am honored to be presenting one of the keynote addresses at this important event. The other keynote will be New Zealand internet law expert and District Court Judge David Harvey. I plan to discuss the results of my empirical study of fair use litigation, Predicting Fair Use.

This year’s forum takes place at the National Portrait Gallery in Canberra on Friday 1 March 2013 from 8:30am – 4:30pm, with pre-forum drinks on the evening of Thursday 28 February at the National Library of Australia.

More info is available at

2012 Global Congress on Intellectual Property and the Public Interest #gcongress

The Global Congress on Intellectual Property and the Public Interest is part of an attempt to define a positive agenda for policy reform in IP by building a global network of scholars and advocates.

I attended the Congress as part of the “Global Network on Copyright Limitations and Exceptions” an international group of copyright law experts that is drafting a set of exceptions and limitations to copyright law that (ideally) any country to use to modernize its law, comply with international obligations such as the Berne Convention and TRIPs and take advantage of the under-utilized flexibility those agreements allow. Our objective is

“to promote discussion of employing ‘open-ended’ limitations in national copyright legislation” and to support “the development of binding international agreements providing for mandatory minimum limitations and exceptions.”

Our view is that balancing mechanisms and user rights are an integral part of the copyright policy, not an afterthought.

We were also extremely privileged to have a whole day of private meetings with Marcos Souza, Head of Copyright, Brazil Ministry of Culture. In this meeting members of the L&E Network discussed proposals to amend Brazil’s copyright with Brazilian copyright scholars and officials.

  • Model Flexible Use Clause, Version 4.0 (PDF)
  • Introduction to Text (PDF)
  • Appendix I: Presumptively Lawful Purposes (PDF)
  • Appendix II: Examples of Flexible Limitations and Exceptions from Existing and Proposed Laws (PDF)
  • Appendix III: Responding to Frequently Asked Questions About Flexible Use Provisions (PDF)

Sean Flynn, Michael Carroll, Peter Jaszi and Meredith Jacobs from American University have done a fantastic job coordinating this.

Other L&E Network members include: Ahmed Abdel Latif, Alberto Cerda Silva, Allan Souza, Andrew Rens, Bruno Lewicki, Carlos Affonso, Carolina Botero, Caroline Ncube, Denis Barbosa, Gwen Hinze, Hong Xue, Jennifer Urban, Jonathan Band, Leon Felipe Sanchez Ambia, Matthew Sag, Niva Elkin-Koren, Oliver Metzger, Pedro Mizukami., Pedro Paranagua, and Pranesh Prakash.

These people are all awesome!

The Authors Guild Does Not Speak for Academic Authors

Academic authors are being asked to stand by an watch as the Authors Guild litigates against their wishes and interests, but supposedly on their behalf.

This hubris is not exactly unprecedented. The plaintiffs in Hansberry v. Lee 311 U.S. 32 (1940) sought to enforce a racially restrictive covenant on behalf of a broad class of landowners including African-American’s who would be harmed by enforcement and whites who simply objected. Like the land-owners in Hansberry many academic authors disagree with Authors Guild’s crusade against book digitization. The Supreme Court did not allow the plaintiffs to hijack the class in Hansberry, hopefully the Second Circuit will not allow the Authors Guild to do so in Authors Guild v. Google. 

Pamela Samuelson and David Hansen (both of the University of California, Berkeley – School of Law) have filed a very important amicus brief on behalf of over 150 academic authors* in the Second Circuit Court of Appeals in Authors Guild v. Google. (Available on ssrn)

The brief in support of defendant-appellant Google argues that class certification should have been denied by the District Court because the named plaintiffs don’t represent the interests of academic authors who comprise a large proportion of the class.

The Authors Guild cloaks its lawsuit in the mantel of authorship, yet in reality it represents only a small fraction of the the class it has constructed. Most of the books that Google scanned from major research library collections were written by academics.

The basic problem is that the three individual plaintiffs who claim to be class representatives are not academics and do not share the commitment to broad access to knowledge that predominates among academics.

The plaintiffs’ request for an injunction to stop Google from making the Book Search corpus available would be harmful to academic author interests. The only way for the interests of academic authors to be vindicated in this litigation, given the positions that the plaintiffs have taken thus far, is for Google to prevail on its fair use defense and for the named plaintiffs to lose.

As we explained in the Digital Humanities Amicus Brief in the district court, “[m]ass digitization, like that employed by Google, is a key enabler of socially valuable computational and statistical research (often called “data mining” or “text mining”),”  which allows researchers to discover and use the non-copyrightable facts and ideas that are contained within the collection of copyrighted works themselves.

The Authors Guild are bad representatives of the interests of academic authors because

  1. Academic authors would generally prefer their books be findable using Google Book Search.
  2. If the Authors Guild wins, academic authors will be deprived of a valuable resource, in the form of the Google Book Search Engine and the HathiTrust Digital Library.
  3. If the Authors Guild wins, text mining — the most basic tool of the Digital Humanities — will have been declared to be prima facie illegal.
* I was one of the signatories.



My CELS Comments on “Standards of Proof in Civil Litigation: An Experiment from Patent Law”

Comments on

Standards of Proof in Civil Litigation: An Experiment from Patent Law” by David L. Schwartz (Chicago-Kent College of Law) and Christopher B. Seaman (Washington and Lee University School of Law)

I commented on this excellent paper at the 7th Annual Conference on Empirical Legal Studies Paper at Stanford yesterday (November 9, 2012). The authors ran an online experiment to try to parse out the effect of a recent U.S. Supreme Court case, Microsoft Corp. v. i4i Limited Partnership.

In i4i the Court held that a patent’s presumption of validity can only be overcome by clear and convincing evidence, but, the Court also explained that the jury should be instructed that it may be easier to satisfy this standard when the party challenging the patent’s validity offered evidence that was not previously been considered by the U.S. Patent & Trademark Office.

The experiment tested the same fact pattern on 500 respondents, assigning them into three groups. Each group was asked to apply a different standard of proof. The clear and convincing evidence standard resulted in far fewer findings of obviousness – no surprise there. But the authors main finding of interest was that a mock jury instruction directed by the i4i decision was not statistically different from the (lower) preponderance of the evidence standard explicitly rejected by the Court.

To try to understand this puzzling result, I took their data and produced some graphs to show the extent of disorder in the three experiment conditions. In the figures below, each bar is a single response. The more red on the image, the more respondents thought the patent in question was invalid due to being obvious in light of the prior art. If subjects were being consistent, all the red bars should be on the right and all black bars on the left.

[Updated and revised November 10, 2012]