Takes you to the main page for this section

Main Journals Page | Ends and Means Home Page

New Technologies And The Ethics Of Extreme Risks

Martin Peterson

1 Ethics and Decision Theory

In this paper I intend to discuss social decision-making involving extreme risks. By an extreme risk, I mean a potential outcome of an act for which the probability is low, but whose negative value is high. Extreme risks are often discussed when new technologies are introduced into society. Nuclear power and genetic engineering are two well-known examples.

The major paradigm in normative decision theory, social decision-making included, is the expected utility rule. According to this rule, society ought to choose an act for which the probability-weighed sum of utilities for all outcomes is at least as large as that of every other alternative. Hence, if the expected utility of accepting a new technology associated with an extreme risk is larger than that of not accepting it, society ought to accept the technology in question.

Before I render the specific normative question to be discussed in this paper more precise, I would like to make a few remarks about the relation between ethics on the one hand, and decision theory on the other hand. From an ethical point of view it is far from obvious why decision theoretic principles, such as e.g. the expected utility rule, are at all relevant in social decision-making about extreme risks. Why not instead apply traditional act utilitarianism, or a rights-based moral theory?

The reason why I think (act) utilitarianism fails as a normative theory for extreme risks, is its well-known lack of action guidance: An ethical theory should advice its subscribers how to behave, but when extreme risks are involved we do not know (by definition) the actual consequences of e.g. introducing a new technology; hence, no practical recommendation is generated by the act utilitarian doctrine. To defend utilitarianism by making use of the old distinction between criteria of rightness and decision-making procedures is pointless. Because a proposed decision-making procedure cannot be evaluated relative to a given criterion of rightness unless we already know what act(s) this criterion of rightness would in fact prescribe in the real-world cases we are interested in - and (again by definition) we do not know what act(s) would be prescribed by act utilitarianism in a situation involving extreme risks. Hence, no decision-making procedure can be justified on act utilitarian grounds.

As an illustration of this argument, suppose that society has to make a decision about nuclear power. In order to determine whether a given decision-making procedure X is better than another Y from a utilitarian point of view, we have to compare the prescriptions yielded by X and Y to the prescriptions yielded by the act utilitarian criterion of rightness. But since we cannot know (neither in advance, nor "afterwards") whether the total consequences of using nuclear power are better than those of the alternatives, this comparison cannot be made even in principle.

The reason why rights-based ethical theories are of little relevance for social decisions involving extreme risks is that such risks tend to affect all members of society, and are usually accepted through democratic elections. Hence, it is seldom the case that some members of society impose unwanted extreme risks on others. Nuclear power is, again, a well-known example. We are all exposed to approximately the same extreme risks from nuclear power, and we have all agreed in democratic elections to allow this technology. Therefore, no relevant rights seem to be violated if society democratically agrees to continue accepting this extreme risk.

For the two reasons outlined above, I suggest that the relevant "sort" of normative principles to be used as a point of departure for an ethical analysis of extreme risks, are the kind of principles discussed by decision theorists and risk analysists. Such principles certainly provide action-guidance. When decision theoretic principles, such as e.g. the expected utility rule, the maximin rule and the de minimis principle, are discussed in relation to social decisions involving extreme risks, they should be interpreted as normative criteria of rightness. No additional decision method is needed.

Let me now return to the specific normative problem that is in focus of the present paper. Remember that well-known arguments for the expected utility rule have been developed by e.g. von Neumann & Morgenstern, and Savage. However, despite the formal elegance of those arguments, the application of the expected utility rule to extreme risks is nevertheless problematical. It can, more precisely, be challenged from at least two perspectives. The adherents of the precautionary principle claim that the mere fact that e.g. the introduction of a new technology might have catastrophic consequences is enough reason for prohibiting the technology in question. And the adherents of the de minimis principle say that sufficiently unlikely scenarios should be disregarded, even if they may involve catastrophic outcomes. Consequently, the normative implications of the de minimis principle are often inconsistent with those of the precautionary principle, while they both differ from the recommendations yielded by the expected utility rule.

A significant normative problem that deserves to be addressed is thus: Who is right when it comes to social decisions involving extreme risks; is it the advocates of the expected utility rule, or the advocates of the precautionary principle, or the advocates of the de minimis principle, or someone else?

2 Expected Utility

As indicated above; expected utility rule recommends decision-makers to choose an act for which is maximal, where is the probability of state and the utility of outcome . When applied to social decision making, the decision maker is to be thought of as society as a whole.

An obvious problem with applying the expected utility rule to social decisions is how to construct the appropriate utility function: this seems to presuppose interpersonal comparisons of utility. In contemporary cost-benefit analysis (CBA), in which the expected utility rule is routinely used for comparing risky benefits to costs, this problem is often "solved" by using monetary values instead of utilities, or by counting the total number of human lives saved by various alternatives. In the present paper I shall make no attempt to attack the problem with interpersonal comparisons of utility.

Arguments in favour of the expected utility rule can be divided into two groups. First, arguments based on the Law of Large Numbers remind us that in case we face a large number of similar decisions, we will probably be better off in the long run if we apply the expected utility rule than if we apply any other decision rule. However, this argument seems to be of little relevance in many authentic decision situations, especially when extreme risks are involved. Consider, for instance, the case with nuclear power. From a statistical perspective there are not that many reactors in the world, and all reactors are not similar in all relevant aspects. Furthermore, since the probability of a meltdown is very small, but the consequences enormous, the Law of Large Numbers seems to provide no support at all for the use of the expected utility rule in this case.

The second group of arguments for the expected utility rule does not rely on the Law of Large Numbers. Instead, the basic idea is to show that the expected utility rule can be derived from a number of normatively reasonable postulates. For instance, Savage (1954, 1972) famously formulated a set of postulates from which he derived a representation theorem, roughly saying that there is a utility function u and a probability function p such that if and only if > . Savage's argument applies both to individual decision-making under uncertainty and to decision making under risk, since the probability function p is derived "within" the formalization by using subjective probability assignments. A few years earlier von Neumann & Morgenstern (1947) proved a similar representation theorem based on objective probabilities.

Unfortunately, the value of Savage's and von Neumann & Morgenstern's arguments, and other arguments developed from their work, is commonly regarded as dubious. First, the normative relevance of the representation theorems is unclear. Second, it is hard to acquire intuitive support for some of the postulates used for deriving the representation theorems, namely Savage's so called sure-thing principle, and von Neumann & Morgenstern's independence postulate. However, given that sufficiently plausible and relevant postulates could be formulated, there seems to be nothing wrong with their general approach: if the expected utility rule can be deduced out of a set of true postulates, then it will certainly be true too. Of course, the postulates do not have to be some kind of fundamental, or logical truths; what matters is just that they are true.

3 The Precautionary Principle

Despite the mathematical elegance of the formal arguments for the expected utility rule, most people seem to reject its recommendations when it comes to decisions involving a possibility for large losses. For instance, John Maynard Keynes famously argued that the expected utility rule should be abandoned "because it ... assumes that an even chance of heaven or hell is precisely as much to be desired as the certain attainment of a state of mediocrity." Arguably, the precautionary principle can be seen as a response to the demand of a more risk averse decision rule. It was famously adopted by the Rio Conference in 1992, but its origins can be traced back at least to German environmental legislation from the 1970's. As has been pointed out by Sandin and others there appears to be little consensus about the exact meaning of the precautionary principle, even though a common feature of all interpretations seems to be that acts involving too severe risks (in terms of negative values) ought to be avoided, even in case there is not full scientific evidence that the risk in question might actually realize. The following formulations of the precautionary principle were adopted by the Rio conference, respectively proposed by Fleming (1996):

Where there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent environmental degradation. (UNCED 1993, Principle 15)

The costs of avoiding unacceptable environmental consequences must be borne, even though they may turn out in the event to have been greater than necessary. (Fleming 1996, p 147)

In decision theoretic explorations of the precautionary principle, aiming at bringing more precision into the debate, it is often agreed that the precautionary principle should be interpreted either as maximin or as minimax regret. The maximin rule recommends the decision-maker to choose an alternative for which the worst possible outcome is at least as high as that of every alternative act, and the minimax regret rule recommends the decision maker to choose an act for which the maximum regret value is as low as possible. However, as far as I can understand, both these decision theoretic interpretations are incorrect. The reason is that the precautionary principle does not - unlike the maximin and minimax regret rules - recommend any particular act or set of acts. Rather, the precautionary principle excludes certain acts, namely those acts that are "too dangerous". More precisely, I think the precautionary principle is best interpreted as a rule that is used in the framing of a decision problem, e.g. by transforming an initial representation of a decision problem into another, in which the set of "too dangerous" alternatives has been deleted. Thereby, the decision-maker is prevented from choosing any of the "too dangerous" acts.

But how do we determine if an act is "too dangerous" then? Are all acts involving extreme risks too dangerous, or just some? (Which?) I think it might be wise to not pay too much attention to this problem here. Let us just note that empirical studies show that people are generally more concerned about technological risks than natural ones. For instance, many people seem to be more afraid for electromagnetic radiation from mobile phones than natural radon. Yet, this standpoint appears to be hard to defend from a normative perspective. Because what ultimately matters is, arguably, our well-being - and there are many reasons to believe that natural radon affects our well-being more than mobile phones. Despite this, it can perhaps be argued that our level of knowledge about new technologies is usually not as high as that about (old) natural risks, and consequently, that we therefore ought to take more precaution when facing new technologies. (For a decision theoretic discussion of this topic, see Gärdenfors & Sahlin 1982.)

4 De Minimis

The normative intuitions underlying the de minimis principle are very different from those underlying the precautionary principle. Somewhat roughly put, the general idea is that some extreme risks are so unlikely to materialize that they ought to be disregarded. A trivial issue to which the de minimis principle can be applied is, for instance, the very unlikely event in which a space shuttle crashes into my bedroom tomorrow night: Since the probability of this event is so small one should, given that the de minimis principle is accepted, neglect it and consider it as being beyond concern.

The phrase 'de minimis' originates from the Latin sentence 'de minimis non curat lex', which is a well-established principle in American common law meaning that courts should not concern themselves with trifles. An unpaid debt of five cents could be an example of a legal matter in which an American court would choose to apply the de minimis clause, because of the smallness of the debt. The concept of 'de minimis risk' was derived from this legal principle in the early eighties. It was originally used by the U.S. Food and Drug Administration (FDA) as a motive for not obeying the so called Delaney Amendment, which prohibits the use of all cancerogenic food additives, including such substances that only increase the cancer risk by a very tiny fraction: If a risk is considered de minimis, FDA claimed, we are justified not to take any notice of it at all. Later on, the de minimis principle has been applied to other health and environmental decisions as well, for example to regulatory decisions concerning small doses of radiation and pesticides.

In 1984 Derek Parfit presented an argument against the de minimis principle in his influential book Reasons and Persons. In this argument, which he claims to be the third in a set of five mistakes in "moral mathematics", he asks us to assume that nuclear engineers actually did ignore all probabilities at or below e.g. one-in-a-million. Then:

It might be the case that, for each of the many components in a nuclear reactor, there is a one-in-a-million chance that, in any day, this component would fail in a way that would cause a catastrophe..... If there are many reactors, each with many such components, it would not take many days before the one-in-a-million risk had been run a million times. There would fairly soon be a catastrophe. (Parfit 1984, pp 74-75.)

The main point in this argument seems to be that a large number of very small risks together can constitute a large risk, which is definitely not de minimis. But does this show that there is something wrong with the de minimis principle, as Parfit believes? No, I don't think so. In my view, Parfit's example only shows that the de minimis principle is sensitive to the individuation of risks: Since the outcome of a failure in a reactor is roughly the same no matter which (critical) component breaks down, it seems reasonable to say that the nuclear technology is associated with one rather large risk for catastrophe, not thousands of very small risks. Hence, the de minimis principle should not be applied in the example described by Parfit.

However, this does not exclude that there really are situations in which it is reasonably to apply the de minimis principle. For instance, in the 1980's FDA decided that the 1-in-19-billion risk of getting cancer from the food additive R&D Orange no 17 should be considered de minimis. Since there is just a limited amount people eating food contaminated with this additive, and there is a limited number of other additives in the food, the total risk for the consumers is not very large.

5 Ecumenism About Extreme Risks

In the previous sections I have not tried to entirely refute any of the three principles discussed so far. The reason is that I believe they all contain some elements of truth. More precisely, I think that these three principles, viz. the expected utility rule, the precautionary principle and the de minimis principle, should be combined into a single composite decision principle, which differs in substantial ways from traditional decision principles such as e.g. the maximin rule and the minimax regret rule. The composite principle that I am proposing is constructed as follows. First, the decision-maker applies the de minimis principle to his (or her) decision problem and thereby remove sufficiently improbable scenarios; then the precautionary principle is applied for removing too dangerous acts; and finally the expected utility rule is applied for choosing among the remaining alternatives. The reason why the de minimis principle is applied before the precautionary principle is that it counterbalances the effects of the latter, thereby making its implications less extreme. This motivation for applying the de minimis principle is different from the traditional motivation, that it decreases the decision-maker's decision costs.

In order to see how my composite principle works in practice, consider a government agency that has to decide among three alternative programs for vermin control. Program 1 and 2 involve the use of one of the two pesticides A and B respectively, whereas Program 3 involves the introduction of genetically modified mayflies. Program 2 (i.e. the use of pesticide B) is associated with a very small non-zero probability for toxic bioaccumulation in parts of the human population leading to death, but is economically more favourable than program 1 (i.e. pesticide A). The most economically favourable alternative is the introduction of genetically modified mayflies, i.e. program 3, but this alternative is unfortunately associated with a non-negligible risk for a new severe disease to spread uncontrolled, having a great negative value. Now, the composite principle advocated here first discharges the de minimis risk associated with alternative 2; then the precautionary principle eliminates alternative 3, i.e. the genetically modified mayflies, since this option is (here defined to be) too dangerous; finally, the decision among pesticides A and B (alternatives 1 and 2) is based on expected utility, favouring, we assume, pesticide B.

I wish to emphasize that I have not offered any direct argument for my composite principle. However, since each of the three principles discussed above appears to be attractive taken alone, we have an indirect argument for the composite principle, namely that it makes most sense of our intuitions about social decisions involving extreme risks. In case we opt for some other decision rule, we have to give up at least one of the involved intuitions. The indirect argument for my composite rule is thus an appeal to conservatism: do not give up any normative intuitions unless you are absolutely forced to do so!

Of course, this composite rule has to be worked out more in detail before it can be used by authentic decision-makers. For instance, more has to be said about the cut-off levels for negligible probabilities and disastrous consequences. (Can we use real numbers for describing those cut-off levels, or are they perhaps instances of vagueness?) However, even at this preliminary stage I think my composite rule is sufficiently precise for helping us focus on the right kind of issues: It is not the actual consequences of extreme risks that matters from a normative point of view, but the ones that are judged by the decision-maker as possible.

Martin Peterson
Royal Institute of Technology,
Stockholm, Sweden

References

Bales, R E 1971: "Act-Utilitarianism: Account of Right-Making Characteristics or Decision-Making Procedure?", American Philosophical Quarterly 8, pp 257-65.

Blumenthal, D 1990: "Red No. 3 and Other Colorful Controversies", FDA Consumer.

Broome, J 1999: Ethics out of Economics, Cambridge University Press.

Fleming, D 1996: "The Economics of Taking Care." In The Precautionary Principle and International Law: The Challenge of Implementation, Eds. David Freestone & Ellen Hey. International Environmental Law and Policy Series, vol. 31. The Hague: Kluwer Law International, 147-167.

Gärdenfors, P & Sahlin. N-E 1982: "Unreliable Probabilities, Risk-Taking, and Decision Making", in Syntese 53, pp 361-386.

Hansson, S O 2001: "The Causal Dilution Problem", pp 131-139 in Eric Carlson and Rysiek Sliwinski (eds.) Omnium-gatherum. Philosophical Essays Dedicated to Jan Österberg on the Occasion of his Sixtieth Birthday, Uppsala Philosophical Studies vol 50.

Hansson, S O 1999: "The Moral Significance of Indetectable Effects", Risk 10, pp 101-108.

Hansson, S O 1997: "The Limits of Precaution", Foundations of Science 2, pp 293-306.

Keynes, J M 1921: A Treatise on Probability, MacMillan & Co.

Parfit, D 1984: Reasons and Persons, Oxford UP.

Peterson, M 2001: "Formal Argunments for the Principle of Maximizing Expected Utility: Two Approaches", manuscript.

Sandin P 1999: "Dimensions of the Precautionary Principle", Human and Ecological Risk Assessment 5, 889-907.

Savage, L J 1954: The Foundations of Statistics, Wiley and Sons. 2^nd edition 1972, Dover.

Sjöberg, L: 1999 "Perceived Risk and Tampering with Nature", Journal of Risk Research 3, pp 353-367.

Stern, Alan H. 1996: "Derivation of a Target Concentration of Pb in Soil Based on deviation of Adult Blood Pressure", Risk Analysis 16, pp 201-10.

Suter II et al 1995: "An Approach for Balancing Health and Ecological Risks at Hazardous Waste Sites", Risk Analysis 15, pp 221-231.

UNCED 1993: The Earth Summit: The United Nations Conference on Environment and Development (1992: Rio De Janeiro). Introduction and commentary by Stanley P. Johnson. London: Graham & Trotman.

von Neumann, J & Morgenstern, O 1947, Theory of Games and Economic Behavior, 2^ndedition, Princeton UP.

Weinberg, A M. 1985: "Science and its limits: The Regulator's Dilemma",Issues in Science and Technology 2, pp 59-72.

Whipple, C (ed.) 1987: De Minimis Risk, Plenum Press.

View this page as text only

University Home · Prospective students · Prospectuses · A to Z Index · Search
Email & Telephone Directories · Contacts/Help · Maps · Privacy Policy & Disclaimer · Accessibility Policy