EFTA01113316.pdf

Ambiguity, the Certainty Illusion, and Gigerenzer's Natural Frequency Approach to Reasoning with Inverse Probabilities John Fountains and Philip Gtmbyl 2 February 2010 Version: 06 April 2010 Abstract People have difficulty reasoning with information to do with uncertain situations, including when making economic decisions. This is especially true when decisions require the calculation of conditional probabilities. Putting the data in terms of natural frequencies as promoted by Gigerenzer (2002) makes it easier for people to reason in situations of uncertainty. Unfortunately, it invokes the normally false assumption that the frequency information is precise. The use of simple graphical techniques can help to resolve this problem, providing a tool that can be used for making decisions in uncertain situations when the probabilistic information is imprecise and thus ambiguity exists. Keywords: Ambiguity; certainty illusion; inverse probability; natural frequencies; uncertainty. JEL Classification: A200, D100, D800. Draft. The opinions and conclusions expressed are solely those of the authors. All errors are our own. Department of Economics, University of Canterbury. r Department of Economics Universit of Canterbu . "Corresponding Author: ph: fax: EFTA01113316  Ambiguity, the Certainty Illusion, and Gigerenzer's Natural Frequency Approach to Reasoning with Inverse Probabilities Abstract People have difficulty reasoning with information to do with uncertain situations, including when making economic decisions. This is especially true when decisions require the calculation of conditional probabilities. Putting the data in terms of natural frequencies as promoted by Gigerenzer (2002) makes it easier for people to reason in situations of uncertainty. Unfortunately, it invokes the normally false assumption that the frequency information is precise. The use of simple graphical techniques can help to resolve this problem, providing a tool that can be used for making decisions in uncertain situations when the probabilistic information is imprecise and thus ambiguity exists. Keywords: Ambiguity; certainty illusion; inverse probability; natural frequencies; uncertainty. JEL Classification: A200, DI00, D800. EFTA01113317 1. Introduction The world is an uncertain place and many decisions people make depend on accurate information about the risks they face as well as clear thinking with the information available to them. If a medical treatment has the potential to improve a person's life but also potential side-effects which could worsen their life, should they go ahead with it? Medical tests providing information used in making this decision are prone to both false negatives and false positives, how much faith should a person place in a test result? When facing a choice about what risky financial assets to put their savings into, where the returns are uncertain and the capital investment is at risk, which assets should people choose? As with medical decisions, informative but imperfect signals are available, such as the credit ratings for the financial assets or the organisations issuing them. Other equally important decisions under uncertainty routinely occur such as when and how much to gamble, which party to vote for in an election, which consumption items to purchase, whether someone accused of a crime is guilty or innocent, and basing decisions on weather forecasts. These decisions all depend on clear thinking about probabilities. Mistakes made in many of these situations can prove very costly. The Sally Clark case in the United Kingdom in which she was accused of killing her two children and was subsequently convicted, partly on the basis of incorrect calculation of probabilities by an expert witness, is an obvious example. Whether or not a woman should undergo a mastectomy if a mammography is positive, or a man should have his prostrate removed if his PSA test is positive are not as clear cut as it first seems. Both procedures involve risks of serious side-effects of physical discomfit, psychological stress, a perceived loss of femininity See The Guardian, (2007), 8 November, p. II. The title of the article was "Sally Clark, Mother Wrongly Convicted of Killing Her Sons, Found Dead at Home." and her inability to recover from the accusations and initial conviction shows the potential cost of getting a probability calculation wrong. 2 EFTA01113318 for a mastectomy, and incontinence and impotence for removal of a prostate. Mistakes relating to how risky financial assets are can also prove very costly as seen with the recent financial crisis. For example, the California Public Employees Retirement System is reportedly suing three credit rating agencies for "...hundreds of millions in losses..." it made over what it saw as inaccurate credit ratings of financial assets. Irrespective of whether or not these assessments were accurate, they clearly affected beliefs about the riskiness of the assets. If people incorrectly adjust their beliefs because of poor statistical thinking, then serious consequences can result as is evident. Thinking about how people make decisions in the face of risk is also very important to economic thought. The subjective expected utility framework is predicated on people calculating probabilities correctly when faced with decisions where they face uncertain situations. If people do not correctly reason about uncertainty then the validity of the current mainstream description of people's decision making under uncertainty is called into question. That possible problems may exist with the subjective expected utility model in economics as a descriptive summary of individuals' decisions under uncertainty has been around since the Allais and Ellsberg paradoxes were discovered. Both results showed that the observed choices of people conflict with the predictions of expected utility theory, although for different reasons. This resulted in generalisations of expected utility theory rather than a throwing out the general approach, as neither paradox was caused by incorrect statistical reasoning. The possibility that people might have trouble reasoning about risk was however raised in work by ICahneman et al (1982) who questioned the ability of people to make accurate inferences from statistical information. This finding was observed in other pieces of 3 EFTA01113319 research in psychology with the added observation that people had difficulties in calculating probabilities from statistical information, particularly conditional probabilities. These results do have the potential to challenge the subjective expected utility framework since they challenge the notion that people's thinking about uncertainty obey the laws of probability. Good summaries of this topic are Gilboa (2009), Machina (1987, 2005), Starmer (2000), and Wakker (2004). Making matters even more complicated is the possibility of ambiguity about the statistical information. Imprecise knowledge of probabilities is not by itself fatal to expected utility as shown by Klibanoff et al (2005). But ambiguity combined with violations of the laws of probability surely is. Thankfully, all is not lost. What has be found to be important for how well people calculate and reason statistically is how the statistical information is communicated to them. Simply put, people find it easier to make better inferences if information is communicated to them in certain types of forms rather than others. Framing matters as all students of behaviourial economics know. In this article we present a tool that makes it easier (and thus in some sense cheaper) for people to think accurately about situations of uncertainty even in the presence of ambiguity about the information. The approach has broader applications and would be useful in the classroom as a tool to help students understand and think about probabilities, and more generally in private and public organisations by those making decisions or communicating risk information when facing uncertainty about the resulting outcomes. 4 EFTA01113320 2. Can People Reason Accurately in Uncertain Situations? Given the importance of understanding how people think about risk, much research has been undertaken about how people reason statistically, particularly whether or not their reasoning is accurate, and the conditions under which their accuracy and statistical reasoning processes might be improved. might be expected, the key issues have no simple answers, although much has been learnt along the way in trying to find the answers. 2.1 Problems People Have in Reasoning About Uncertainty Even though correct statistical reasoning is important for people's welfare, the evidence is that people have trouble reasoning with information about risks. Research in cognitive psychology and other areas have found strong evidence of biases in statistical reasoning by people.. The human brain has evolved in a way that can give the illusion of certainty where none exists, making people prone to preferring certainty over uncertainty. Single event probabilities are prone to misinterpretation since reference classes are typically unstated, or even worse, the event is unique in which case any probability given is likely to be a guess (and not necessarily an "educated" one). People are typically confused by what conditional probabilities mean, finding it difficult to calculate and interpret them. Information given as relative risks is open to misinterpretation since it does not indicate if the numbers involved are meaningful. The British Medical Journal and the Financial Times even included columns on how poorly people seem to think about uncertain situations, particularly when having to calculate conditional probabilities.t A sample of this research is Bimbaum et al (1990), Chen and Crask (1998), Dougherty and Sprenger (2006), Lewis and Keren (1999), and Reyna and Brainerd (2008) . Watkins (2000) and the Financial Times, (2003), 19 June, p.21. Difficulties people have in estimating conditional probabilities has been known for over four decades , see Bauer (1972). 5 EFTA01113321  These findings are more than just academic curiosities. The possibility of drawing incorrect inferences from data in courtrooms, one known as the prosecutor's fallacy and the other the defendant's fallacy, was first highlighted by William Thompson and Edward Schumann in their classic 1987 article: In the Sally Clark case an expert witness made a mistake in calculating a joint probability which was later pointed out by the Royal Statistical Society in a press release "...expressing its concern at the misuse of statistics in the courts."t The expert calculated a joint probability on the basis that the events involved were independent whereas the evidence was overwhelmingly against this assumption. The presence of systematic and predictable difficulties in reasoning with statistical information found by researchers, such as the illusion of certainty or misinterpretation of relative risks, creates strategic incentives to exploit it. For example, pharmaceutical companies have an incentive to report the use of relative risk information as it will be more likely to convince civil servants, doctors, and potential consumers that their drugs successfully treat medical conditions, increasing the demand for them and consequently their prices. This shortcoming in people's thinking also creates doubts about the subjective expected utility model to characterise people's decision making under uncertainty (see the references in Section 1). If people's understanding and calculations of probabilities violate the laws of probability then how exactly do you model their choices under uncertainty? 2.2 Natural Frequencies and the Importance of How Statistical Information is Presented and Communicated Though the evidence strongly suggests that humans are poor at statistical reasoning all is not lost, and thankfully for economists this includes the subjective expected utility framework. Some researchers, such as Gigerenzer (2002), claim that the primary cause of the Thompson and Schumann (1987). t Online. Available: www.rss.org.uk/docsfRoyal%20Statistical%20Society.doc. 4 February, 2010. 6 EFTA01113322 miscalculation of probabilities or the misunderstanding of statistical information is because it is presented in ways that do not suit the evolutionary structure of the human brain.. In other words, as we know from behavioural economics, the framing of the information matters. Gigerenzer argues that presenting statistical information as frequencies (this is explained in Section 3.2) is far more natural for humans given their evolutionary past. Frequency based information naturally specifies a reference class and also clearly highlights that in most situations belief in certainty is an illusion. Statistical information in the form of natural frequencies suits our evolutionary past where risk information was not in normalised forms such a probabilities and percentages. Summary count or frequency based information reduces the number of mental calculations involved in working out probabilities as well as making it easier to calculate conditional probabilities. It also means people are more likely to understand what they mean in their interpersonal communication about uncertainties. Other advantages of information in a frequency form include it making clear how meaningful changes are, because the information is in absolute risk changes, and how much evidence underlies the information at hand, because it shows how many observations have occurred. Like most things, statistical information presented as natural frequencies is not a universal panacea. For example, Chapman and Liu (2009) find evidence that a minimal level of numeracy is needed before the beneficial effects of presenting probabilistic information as frequencies occur. Barbey and Sloman (2007) argue that natural frequencies do help people to think about risk, but that how it does so is more complicated than is put forward by Gigerenzer and that it works better in some situations than others. Overall, it does seem that people are perfectly capable of statistical reasoning but the accuracy of their results is strongly dependent on how the information is presented. This feature of people's thinking is • For other examples supporting natural frequencies as an effective way of communicating statistical information see also Brase (2008), Gigerenzer and Hoffrage (1998), Kurzenhauser and Hertwig (2006), and Sedlmeier (2002) . 7 EFTA01113323 best captured in notion of bounded rationality in the sense of Simon (1957). The presentation of information as frequencies is relatively cheap in terms of emotional and cognitive effort for people to process, and so they calculate probabilities accurately. The presentation of information in other forms is relatively expensive to process in terms of the same effort, so calculations of probabilities are based on heuristics and other devices and as a consequence are less accurate. 2.3 Ambiguity and Imprecise Probabilities Complicating the use of natural frequencies and other forms of communication of statistical information to improve people's thinking about uncertain situations is the presence of ambiguity. The effect of ambiguity on people's behaviour is well known in economics from the Ellsberg paradox which showed that people prefer known risks over unknown risks. This has led to a literature in economics which includes not only preferences about risk but also preferences about ambiguity. Examples of this approach are Klibanoff et al. (2005) and Baillon et al. (2010). Mukerji (2000) and especially Nau (2007) present good summaries of recent work trying to incorporate ambiguity in economics. Klibanoff et al (p. 1849) define ambiguity in way that allows them to model indifference curves as smooth and their approach is shown in this quote from their paper: "One advantage of this model is that the well- developed machinery for dealing with risk attitudes can be applied as well to ambiguity attitudes." Ambiguity in this approach essentially means people have subjective beliefs about the values of the probabilities, in effect a two-stage approach to uncertainty. There is uncertainty about states that can occur, with probabilities of these occurring, and there is uncertainty about the values of these probabilities. One result being that just as there can be risk aversion in situations of uncertainty, there can be ambiguity aversion when people are not sure about the probabilities of the risks they face. 8 EFTA01113324  This of course pre-supposes that what is meant by ambiguity is a lack of knowledge about the possible outcomes that can occur. But when psychologists think of ambiguity, the meaning can be wider, with people not only lacking knowledge of the values of probabilities, but also lacking knowledge about the set of possible states or not being able to calculate probabilities correctly. An example of the former are novel situations, such as new technologies. The de Havilland Comet jet crashes that occurred in 1954 is a well known case of this type.. An obvious example of the latter is the Sally Clark case. Interestingly, Mosleh and Bier (1996) show that ambiguity arising from the lack of knowledge of probabilities is consistent with the subjective theory of probability, but that ambiguity stemming from "cognitive imprecision" is not. This suggests the smooth approach to ambiguity is not necessarily a panacea in capturing ambiguity in decision making, at least not in all situations. 3. Graphical Techniques to the Rescue What we are considering is people making decisions in uncertain situations when there exists ambiguity from cognitive imprecision plus uncertainty about the accuracy of the information being presented. We know that the framing of probabilistic information matters in how people think about it. We also know that people may have doubts about how precise the information they are getting actually is (see Fairman (2006) for examples). The evidence suggests that Gigerenzer's frequency based presentation of data substantially reduces problems with cognitive imprecision. But the same cognitive imprecision based ambiguity After two years of safe operation a Comet crashed after take-off in Rome. Thirty-five people died. Flights were briefly suspended, but a second plane crashed shortly after they resumed. An intensive investigation eventually came to the conclusion that the fault lay with metal fatigue from the high speeds and high altitudes, conditions which had previously been unknown to aeronautical engineers, and which they had thus simply not considered. Stanley (1986, p. 54) commenting on this disaster reports that "At the end of the war de Havilland had been sufficiently courageous to venture into the unknown and design and build the world's first jet aircraft. The two accidents had been the result of factors beyond the limit of contemporary knowledge..." 9 EFTA01113325 can make it difficult for people to cope with imprecise frequency information. Thankfully, tools do exist to help people calculate probabilities and to draw inferences from evidence correctly, that is to lower the costs of analysing information and thus increase the amount and accuracy of the analysis of it. We know from cognitive psychology that graphical data can be superior in terms of communicating information and decision making accuracy than other forms of data presentation. For instance, Speier (2006) finds that graphic representation of data results in more accurate decisions for tasks about comparisons, trends, and such like, than tabular data (which is superior for precise numerical tasks). Graphic data in most cases resulted in faster decisions times. This finding is mirrored in Coll et al (1994). Burkell (2004) also presents evidence that graphical data (pictorial in this study) is easier to understand than numerical data. Regarding probability calculations, Cole and Davidson (1989) find that graphic representation of probabilistic information can substantially improve the time it takes to form conditional probabilities and their accuracy, more so than tabular depiction of the information. Overall, the available evidence suggests that tools that present information graphically, particularly when the decisions to be made involve deep understanding and comparative assessments and not merely exact numerical calculations, can help people make more accurate and faster decisions. Teachers of economics are well aware of the advantages of graphical over tabular methods in the presentation of the most basic Marshallian analysis of demand and supply. This advantage of graphical methods over tabular methods holds for making decisions in uncertain situations. Finally, Natter and Berry (2005) present evidence that people who actively process information are significantly more accurate in their frequency and probability estimates. This means that a graphical tool which allows people to manipulate elements of it to process and show probabilistic information to make decisions are 10 EFTA01113326 even more likely to make accurate decisions than if they receive the information passively. What we do in the rest of this paper is demonstrate a graphical tool that takes advantage of these findings in cognitive psychology and elsewhere and which can be used by people to think about situations of uncertainty where ambiguity about is present. Furthermore, the tool is simple to use. Apart from making it easier for people to make better decisions, the tool would also be useful in the classroom to teach students about probabilistic information. 3.1 Statistical Background Basics The method we use to represent Gigerenzer style natural frequency methods graphically is based on Lad's (1996) geometrical exposition of de Finetti's Fundamental Theorem of Prevision. This little known but extremely powerful theorem in statistics facilitates the identification of coherent beliefs over a "larger" finite state space that are consistent with a "smaller" number of expectations and probability assessments about operational quantities of interest related to this state space. While a general formulation of de Finetti's theorem requires knowledge of linear algebra and convexity, when the quantities are two binary valued variables, simple 2-dimensional graphical techniques suffice. Table 1 sets out an example of the basic information we will be working with. For ease of interpretation we use a health example, but the method is applicable for any two binary variables. That is, for all situations which have an unknown state where an imperfect diagnostic test exists which gives information about the value of the state. Let S, the binary variable in the first row of the table, be the logical truth value (1 or 0) of a proposition about the health state "person A has a disease X". Let D, the binary variable in the second row, be 11 EFTA01113327 the logical truth value of a proposition about a diagnostic health test such as "person A has a positive diagnostic test for the disease X". We assume that truth or falsity of each proposition can be confirmed by an operational measurement, but that knowledge of the outcomes of these measurement procedures might be uncertain, in whole or in part. Table I: Truth Table and Natural Frequencies False True True False Negativ Negativ Positive Positive e e S I 1 D I 0 1 0 Frequency 16 4 24 56 The top two rows in the table set out the four logically possible combinations of the truth values for a pair of propositions (S,D) in the familiar, logical truth table format used in deductive logic (see Suppes (1957, p.1 I)). Each column of possible (S,D) values is labelled with their conventional epidemiological name: (S,D)=(1,1) identifies the situation that person A has the disease and the diagnostic signal for the disease is positive (a "true positive"); (S,D)=(1,0) identifies the situation where person A has the disease but the diagnostic signal is negative for that disease, a "false negative" (in this binary context "not positive" for the disease means "negative" for the disease); (S,D)=(0,I) indicates that person A doesn't have the disease but the diagnostic signal for the disease is positive (a `false positive"); (S,D)=(0,0) indicates that the person doesn't have the disease and the diagnostic signal for the disease is negative (a "true negative .). The final row of the table is an illustrative set of (precise) frequency numbers of the 12 EFTA01113328 kind used to express uncertainties in the context of an inference task. The individual column entries for row 3 of the table are called counts or cases. An aggregate count across all logical possibilities (columns) is also pre-specified, in this case 100, but is in general a variable component of the way information about the state and the diagnostic is represented. The actual frequency numbers in Table 1 are chosen to make the initial construction of the graph representing them easy computationally. Once the method of converting from table to graph is grasped, the frequency numbers can be easily varied to suit the inference task at hand. Note that while there are four conceptually distinct, non-negative case counts, one for each column, there are really only three logically independent counts, since they must sum to a pre-specified total count. Note also that these same frequency numbers can, after simple scaling by the total count, be interpreted as probability assessments for a joint probability mass function, P(S,D), on the discrete space for the two random variables, (S,D). 3.2 Frequency Versus Probability Representations of Probabilistic Information The truth table just presented is simply a vehicle for representing relevant precise numerical information and it is itself neutral between frequency or probability formats. It is thus unable to help answer the question of whether ordinary intelligent people make better inferences using a frequency format or a probability format; "format" here meaning "representation". But before proceeding any further it makes sense to first explain the difference between a frequency format and a probability format. Gigerenzer and Hoffrage (1995) explain the difference between a standard probabilityformat and standardfrequency format, in the context of an inference task specified in English textual terms as:. The sentences are quotes from Table 1 of Gigerenzer and Hoffrage (1995) with the italicized words and frequencies indicating changes from the original in order to conform to the health example and frequency numbers in our example. For example, our "disease X" replaces their "breast cancer", and so on. 13 EFTA01113329  Standard probabilityformat The probability of disease X is 20 percent for women at age forty who participate in routine screening. If a woman has disease X, the probability is 80 percent that she will get a positive diagnostic test. If a woman does not have disease X, the probability is 30 percent that she will also get a positive diagnostic test. A woman in this age group had a positive diagnostic test in a routine screening. What is the probability that she actually has disease X? % Standardfrequencyformat 20 out of every 100 women at age forty who participate in routine screening have disease X. 16 of the 20 women with disease X will get a positive diagnostic test. 24 out of the 80 women without disease X will also get a positive diagnostic test. Here is a new representative sample of women at age forty who got a positive diagnostic test in routine screening. How many of these women do you expect to actually have disease X? Posing an inference problem in text form is not the only way quantitative information in frequency or probability formats can be, or is, presented to subjects in experimental research. Figure 1 below, based on Figure 4-2 in Gigerenzer (2002, p.45), but adapted to our tabular frequencies, illustrates the difference between the two formats in a tree diagram superimposed on tabular information (the bold emphasis, serving to draw attention to particular numbers, is in the original). The upper panels in Figure 1 for each format are different, albeit logically equivalent, ways of representing information relevant to the inference task at hand: what are the chances of having a disease given a positive test result, 14 EFTA01113330 on the basis of the precise information provided? The boxes in the lower panels show the kinds of calculations that need to be made to arrive at a correct inference (40 percent is the posterior probability of disease given a positive diagnostic test result). Once someone learns to "read" the table and the superimposed tree in the frequency format in the left-hand panel, so that the 16+24=40 cases or counts of positive diagnostics can be readily identified and classified into those diseased or not, then the task of working out what fraction, proportion, or number of these 40 actually having the disease is greatly simplified. Certainly in a comparative sense this computational task is much easier than the inverse probability calculations required in the right hand probability format panel. As Gigerenzer (2002, p.45) puts it: "The representation does part of the computation." The dynamic graphical method for representing uncertainties we present later on in the paper will take this insight farther where the representation itself does the whole computation. 3.3 Graphically Representing Natural Frequencies As noted previously, once the total count is specified there are really only three logically independent entries in the column entries of Table 1 available for representing uncertainty about (S,D). But three distinct cell counts aren't the only, nor the most useful, way of identifying the three relevant bits of information. For example the text formats in Section 3.2 specify three other numbers as a way of communicating information relative to reference: a sensitivity number, a specificity number, and a base rate number. These three numbers mix and match uncertainties both about the state variable S and about the diagnostic variable D viewed conditional on information about the state variable S. The sensitivity number expresses uncertainty about whether the diagnostic test will be positive, or L; 1, assuming that S=1 is true. In our example this is the 80 percent probability asserted in the sentence "If a woman has disease X, the probability is 80 percent that she will get a positive diagnostic 15 EFTA01113331  Figure 1: Frequency and Probability Representation Formats Natural Frequencies Probabilities 100 people 20 80 P(disease) = 0.2 disease no disease P(positiveldisease) = 0.8 P(positivelno disease) = 0.3 /\ 16 /\ 4 24 56 positive negative positive negative 16 0.8x 0.2 P(disease I positive) P(disease I positive) 16+24 (0.8x 0.2) + (0.3 x 0.8) test" The specificity number expresses an uncertainty about whether the diagnostic test will be negative, or D=0, assuming that S=0 is true. In our example this is a 70 percent probability, or 1 minus the 30 percent probability asserted in the sentence: "If a woman does not have disease X, the probability is 30 percent that she will get a positive diagnostic test." Probabilistically sophisticated people understand these numbers as conditional probabilities, P(D=IIS=I) for sensitivity and P(D=OIS=0) for specificity. The base rate number for the state variable characterizes uncertainty about the binary state variable S in the absence of, or prior to, learning any diagnostic information D. In our example this is the 20 percent probability asserted in the sentence "The probability of disease X is 20 percent for women at age forty who participate in routine screening." Probabilistically sophisticated people understand the 16 EFTA01113332 base rate number for the state variable as a marginal or unconditional probability, P(S). Probabilistically sophisticated people also understand that specifying a full joint probability distribution, P(S,D), as in the cells of Table 1, or specifying the sensitivity number P(D=115=1), the specificity number P(DOIS=0), and the base rate number P(S=1), are logically equivalent ways of saying the same thing. But, what about probabilistically unsophisticated people? Or, probabilistically sophisticated people in situations where there is ambiguity about the source of the probabilistic information? What we do next is present a graphical method to make it easier for such people who find themselves in these situations to perform statistical calculations, particular statistical calculations. In essence the method lowers the costs of performing the underlying statistical calculations, meaning in situations of bounded rationality there is less reliance on previously less costly but less accurate heuristics, with commensurate increases in the speed and accuracy of the calculations. Figure 2 below is the first step of a three stage process in how to represent a specific table of natural frequencies graphically. Conditional or marginal probabilities (or counts) for the presence of the disease, S=1, are plotted in the x-axis direction and conditional or marginal probabilities (or counts) for the diagnostic D being positive, are plotted in the y-axis direction. At first we will view the graphical representation of the frequency table simply as a collection of three dots, but a second more insightful approach is to connect the dots and view the frequency table as the intersection of several linear coherency constraints on beliefs. The many nuances to the concept of "probabilistic sophistication" are well discussed in Gilboa (2009). In our paper we use the term simply to describe an ability to quantify uncertainties and to reason consistently within the boundaries of the laws ofprobability. 17 EFTA01113333  Figure 2: Graphical Representation of Sensitivity, Specificity, and Base Rate Derived from a Natural Frequency Table natural Frequencies True False False True Positive Negative Positive Negative S D Frequency 16 4 24 56 100 Probabilities of 0=1 • l u 7 a N ci ii ›.• II 0.8 I I I t is in Co -- ..- ''' — i5 i 0.6 ..- ...- O a c— -- ..- v) e... I 0.4 aj .,,,- -- y= 0.8 x + 0.3 (1-x) ---- e cs V t• o 0.2 I •O 14 i Base rate for S • o P(5=1)=0.2 0 V x 0 0.2 0.4 0.6 0.8 I Probabilities of 5=1 The first two columns of the table in Figure 2, where Srl, show that 16+4=20 out of 100 cases are associated with the proposition being true that person A has the disease. This, gives a base rate, P(S=1), equal to 20 out of 100 or 20 percent, shown as a triangle on the x- axis along a vertical line (dash-dot) at the point (0.2,0) on the horizontal axes in Figure 2. Still focusing on the first two columns and the 20 cases where the disease is present, the sensitivity of the diagnostic test, P(D=1IS=1), is 16 out of 20 or 80 percent, shown as a circle 18 EFTA01113334 on the right hand margin of the graph with coordinates (1,0.8). The diagnostic test is good, albeit imperfect, at detecting the presence of the disease state. If the disease is there with 100 percent surety (P(S=1)=1 along the right hand margin), the diagnostic picks this up with a high (80 percent) chance in this circumstance. But the diagnostic test does get it wrong with a 20 percent chance (a false negative type of wrong). This false negative rate, P(D=OIS=1)=0.2, is also indicated by the circle on the right hand margin of the graph, reading down from the top right corner. That one dot indicates two interesting uncertainties about the diagnostic D, under the assumption that the disease is there with 100 percent surety (P(S=l)=1 along the right hand margin). Using the last two columns of the table where the disease is absent, or S=0, 24 cases are false positives and 56 are true negatives. These two numbers determine the false positive rate and the specificity of the test. The specificity of the test, P(EOIS=0), equal to 56 out of 80, or 70 percent, is shown as a circle on the left hand margin of the graph at a height of 30 percent, the false positive rate of the test. The diagnostic test is good, albeit imperfect, at detecting an absence of the disease state. If the disease is absent with 100 percent surety (P(S=1)=0 along the left hand margin), it picks this up with a high (70 percent) chance. But the diagnostic test does get it wrong in these circumstances with a 30 percent chance (a false positive type of wrong). The false positive rate, P(D=IIS4)=0.3, is indicated by the circle on the left hand margin of the graph, reading up from the origin. The specificity of the diagnostic test is indicated by the same circle on the left hand margin of the graph, reading down from the top left corner. That one dot indicates two interesting uncertainties about the diagnostic D, under the assumption that the disease is absent with 100 percent surety (P(S=1)=0 along the left hand margin). 19 EFTA01113335  The dashed line joining the two circular dots on the left and right hand margins, with equation y = 0.8x + 0.3(1-4, is a linear coherency constraint on overall probability assessments." It is based on the mathematical fact that the base rate for the diagnostic test, the unconditional probability of having a positive diagnostic, P(E1), should be a weighted average of the chances of having a positive diagnostic when disease is present, P(D=IIS=1), and the chances of having a positive diagnostic when the disease is absent, P( IIS=0), with weights being the chances of presence, P(S), or absence, 1-P(S), of the disease state. In algebraic terms: P(D= 1) = P =1 IS = = + P(D I IS O)(1 - P(5=1)). Letting P(D=l) be the y-axis variable and P(S=l) be the x-axis variable, and using P(D=1IS=1)=0.8 and P(D=l IS=0)=0.3 from the table, the line y = 0.8x + 0.3(1-x) traces out all combinations of (x,y). In this context it shows all combinations of (P(S=1) and P(D=1)) consistent with the given sensitivity and specificity numbers of 0.8 and 0.7. For example when the base rate for the state variable is P(S=1)=0.2, the base rate for the diagnostic will be 0.8*0.2 + 0.3(1-0.2) = 0.4, the intersection of the dashed line and the vertical dot-dashed line through the base rate triangle on the graph. The important point to realize is that once the two conditional probabilities, the sensitivity and the specificity, are fixed, the permissible combinations of P(S) and P(D) must lie along that dashed line. Exactly the same logic can be applied to posterior inferences about the state variable • Coherency here can be interpreted simply in the formalist sense of consistency with the laws of probability or in the operational subjective sense of being unwilling to assert probabilities that would make you a sure loser. See Ladd (1996) for more about this concept. A short webcast explainaing how to download and use the dynamic interface can be found here: httplluctv.canterbury.ac.nz/post/4/1049. As explained in the webcast and associated documentation, the Mathematica Player used to run the interactive demonstration is freely downloadable and no prior experience with Mathematica is required. 20 EFTA01113336  Figure 3: Graphical Representation of Posterior Inferences Based on the Natural Frequency Table Table and Natural Frequencies True False False True Positive Negative Positive Negative S 1 1 0 0 D 1 0 1 0 Frequency 16 4 24 56 100 Posterior for disease given positive diagnostic Probabilities of D=1 P(S=11D=1)=0.4 y 0.8 0.6 Base rate for D P(C1=-1)=0.4 0.4 1 f // I I 1 0.2 ' _I —1 1 I Posterior for disease / given negative diagnostic 0 i P(S=11D=0)=-4/60 V 0 0.2 0.4 0.6 0.8 Probabilities of S=1 being true, S=1, given diagnostic test information, either positive, L; 1, or negative, ,C0.* The squared box in Figure 3 along the upper boundary x-axis at 0.4 is the posterior or inverse inference about the chances of the disease state being present when a positive diagnostic is observed, P(S=IID=1):1 It is derived from columns 1 and 3 in the table in Figure 3, where D=1, with 16 out of the total 16+24=40 cases of a positive diagnostic being associated with By "posterior" we mean after learning the outcome of the diagnostic test but while still being uncertain about the state variable S. P(S=IID=1) is known as an inverse conditional probability since the role of S and D and "inverted" compared to the sensitivity assessment P(D=11S=1). 21 EFTA01113337 the disease being present and the other 24 cases associated with the disease being absent in these circumstances. The smaller squared box in Figure 3 along the lower boundary x-axis at 4/6020.067 is the posterior or inverse inference about the chances of the disease state being present when a negative diagnostic is observed, that is P(S=1ID=0). It derives from columns 2 and 4 in the table, where D=0, with four out of the total 4+56=60 cases of a negative diagnostic being associated with the disease being present and the other 56 cases associated with the disease being absent in these circumstances. The base rate for the diagnostic test being positive, P(E1), is plotted as a triangle and horizontal dash-dotted line at height 0.4. The line x = 0.4y + 4/60(1-y) between the two conditional posterior inferences (the dot- dashed line between the squared boxes) is another linear