What prompted me to create this page was the observation that there are gross misconceptions out there as to what constitutes "Responsible" decision-making under conditions of severe uncertainty.
The specific trigger was this beauty:
|
Engineering Design and Strategic Planning Under Severe Uncertainty What happens when the uncertainties facing a decision maker are so severe that the assumptions in conventional methods based on probabilistic decision analysis are untenable? Jim Hall and Yakov Ben-Haim describe how the challenges of really severe uncertainties in domains as diverse as climate change, protection against terrorism and financial markets are stimulating the development of quantified theories of robust decision making. |
This is the opening paragraph of a paper posted on the web site FloodRiskNet in the UK since November 2007. I submitted my critical comments on this paper twice, but never got any response other than an "out of office" AutoReply (Feb 8, 2008).
So, no big deal.
I'll comment on this paper -- and related issues -- here. This is, and will continue to be, "work in progress". For the time being all I do is post here the comments I sent Jim Hall in Feb 2008.
The point of Hall and Ben-Haim's paper is to advocate the use of Info-Gap decision theory for responsible decision-making under severe uncertainty.
It is important therefore to point out especially, for the benefit of readers who are not familiar with Info-Gap Decision Theory that this theory turns a blind eye to the universal
|
|
As a consequence it does not follow the well known maxim:
|
|
Hence, the info-Gap rhetoric would have us believe that, (apparently) by some miracle, an analysis conducted in the immediate neighborhood of a wild guess can generate results that are .... meaningfull/worthwhile/useful etc.!!!
But the most astounding thing of all is that, in their paper, Hall and Ben-Haim (2007) do not provide even a single reference to the very relevant and thriving field of Robust Optimization. How about this!!!
See my Voodoo Decision-Making Campaign for details. I also discuss this issue in my contribution to
| WIKIPEDIA article on Info-Gap Decision Theory |
and in
| The Mighty Maximin! |
where I show, among other things, that Info-Gap robustness model is in fact a ... simple Maximin model (circa 1945) in disguise.
In short, not only do Hall and Ben-Haim (2007) unwittingly re-invent a very famous 1945 wheel, they do not use it properly.
In any case, here is the comment I sent Jim in February 2008.
|
The title of the article suggests that the methodology that it promotes, namely Info-Gap, is almost too good to be true. After all, we are led to believe that there is a new methodology out there that is capable of generating robust decisions in situations where "... the uncertainties facing a decision maker are so severe that the assumptions in conventional methods based on probabilistic decision analysis are untenable ..." With this as the appetizer, what should we expect for the main course? Would it be a description of a breakthrough in decision-making under severe uncertainty or ... a recipe for voodoo decision making? Let's see. Classical decision theory offers the Maximin paradigm as a "natural" framework for dealing with robust decision-making under severe uncertainty. And as we know only too well, the price tag for the ultimate robustness provided by this paradigm is significant: the worst case philosophy underpinning Maximin can result in extremely conservative decisions. This is the familiar consequence of over-protection. So the question arises: how is it that Maximin, the stalwart of robust decision-making, is not mentioned, let alone discussed, in the article? Indeed, how is it that this paradigm is not mentioned in the two editions of the Info-Gap book? Readers interested in this aspect of Info-Gap are welcome to visit my website to read more about this intriguing phenomenon. Here it suffices to point out that Info-Gap's generic model is -- surprise surprise -- a simple Maximin model (The formal proof is 2 lines long). This is the good news. The bad news is that Info-Gap's generic model conducts its analysis in the immediate neighborhood of the nominal value of the parameter of interest, hence the worst-case analysis a la Maximin conducted by Info-Gap is local in nature. In other words, Info-Gap deploys a definition of robustness that does not attempt to explore the entire region of uncertainty. It focusses on the nominal value of the parameter of interests and its immediate neighborhood. Since under severe uncertainty the nominal value is a poor indication of the true value of the parameter of interest and is likely to be substantially wrong, it follows that actually robustness a la Info-Gap does not deal with severe uncertainty: it simply ignores it. This fundamental flaw can be vividly illustrated by inspection: the results generated by the generic Info-Gap model are invariant with an increase in the actual size of the region of uncertainty. Thus, if you increase the size of the actual region of uncertainty, say ten-fold, the analysis and the results are not sensitive to this change. And if you discover that actually the region of uncertainty should be increased 100-fold, or 2034578-fold, still this has no impact whatsoever on the analysis and the results generated by the generic Info-Gap model. In short, the "breakthrough" reported on in the article is not a breakthrough at all. Rather, the article promotes a methodology that is based on a simple Maximin model whose effective region of uncertainty is concentrated around a nominal value of the parameter of interest. Since under severe uncertainty this nominal value is a "wild guess", we have no choice but assume that the results generated by this methodology are also wild guesses. Now back to the title of the article. Given that Info-Gap's generic model is a simple Maximin model and given the local nature of the uncertainty analysis, it would be more appropriate to change "Responsible" to "Irresponsible". A full paper inspired by this article will be completed soon. |
The Black Swan
|
|
It is too early to evaluate the long term impact, if any, of Nassim Taleb's recent popular and controversial book The Black Swan: The Impact of the Highly Improbable on the field of decision-making under severe uncertainty.
However, I do hope that the issues raised in this book and its predecessor, Fooled by Randomness: The Hidden Role of Chance in the Markets and in Life, will be instrumental in helping decision-makers identify voodoo decision theories that promise robust decisions under severe uncertainty.
On the other hand, based on my experience over the past 40 years, I believe that the danger is that the Black Swan will inspire a new wave of voodoo decision theories, purportedly capable of ... "domesticating" black swans and preempting the discovery of ... purple swans!
We shall have to wait and see.
This topic is related to some of my other campaigns, namely the Worst-Case Analysis / Maximin Campaign, Robust Decision-Making Campaign and the Info-Gap Campaign.
There are also references to this issue in the articles/presentations listed below below:
Articles, Working Papers, Notes
- Sniedovich, M. (2008) FAQS about Info-Gap Decision Theory, Working Paper No. MS-12-08, Department of Mathematics and Statistics, The University of Melbourne, (PDF File)
- Sniedovich, M. (2008) A Call for the Reassessment of the Use and Promotion of Info-Gap Decision Theory in Australia (PDF File)
- Sniedovich, M. (2008) Info-Gap decision theory and the small applied world of environmental decision-making, Working Paper No. MS-11-08
This is a response to comments made by Mark Burgman on my criticism of Info-Gap (PDF file) - Sniedovich, M. (2008) Wald's Maximin model: a treasure in disguise!, Journal of Risk Finance, 9(3), 287-291.
- Sniedovich, M. (2008) Anatomy of a Misguided Maximin formulation of Info-Gap's Robustness Model (PDF File)
In this paper I explain, again, the misconceptions that Info-Gap proponents seem to have regarding the relationship between Info-Gap's robustness model and Wald's Maximin model. - Sniedovich. M. (2008) The Mighty Maximin! (PDF File)
This paper is dedicated to the modeling aspects of Maximin and robust optimization. - Sniedovich, M. (2007) The art and science of modeling decision-making under sever uncertainty, Decision Making in Manufacturing and Services, 1-2, 111-136. (PDF File) .
- Sniedovich, M. (2007) Crystal-Clear Answers to Two FAQs about Info-Gap (PDF File)
In this paper I examine the two fundamental flaws in Info-Gap decision theory, and the flawed attempts to shrug off my criticism of Info-Gap decision theory. - My reply (PDF File) to Ben-Haim's response to one of my papers. (April 22, 2007)
This is an exciting development!
- Ben-Haim's response confirms my assessment of Info-Gap. It is clear that Info-Gap is fundamentally flawed and therefore unsuitable for decision-making under severe uncertainty.
- Ben-Haim is not familiar with the fundamental concept point estimate. He does not realize that a function can be a point estimate of another function.
So when you read my papers make sure that you do not misinterpret the notion point estimate. The phrase "A is a point estimate of B" simply means that A is an element of the same topological space that B belongs to. Thus, if B is say a probability density function and A is a point estimate of B, then A is a probability density function belonging to the same (assumed) set (family) of probability density functions.
Ben-Haim mistakenly assumes that a point estimate is a point in a Euclidean space and therefore a point estimate cannot be say a function. This is incredible!
- A formal proof that Info-Gap is Wald's Maximin Principle in disguise. (December 31, 2006)
This is a very short article entitled Eureka! Info-Gap is Worst Case (maximin) in Disguise! (PDF File)
It shows that Info-Gap is not a new theory but rather a simple instance of the famous Wald's Maximin Principle dating back to 1945, which in turn goes back to von Neumann's work on Maximin problems in the context of Game Theory (1928). - A proof that Info-Gap's uncertainty model is fundamentally flawed. (December 31, 2006)
This is a very short article entitled The Fundamental Flaw in Info-Gap's Uncertainty Model (PDF File).
It shows that because Info-Gap deploys a single point estimate under severe uncertainty, there is no reason to believe that the solutions it generates are likely to be robust. - A math-free explanation of the flaw in Info-Gap. ( December 31, 2006)
This is a very short article entitled The GAP in Info-Gap (PDF File).
It is a math-free version of the paper above. Read it if you are allergic to math. - A long essay entitled What's Wrong with Info-Gap? An Operations Research Perspective (PDF File) (December 31, 2006).
This is a paper that I presented at the ASOR Recent Advances in Operations Research (PDF File) mini-conference (December 1, 2006, Melbourne, Australia).
Lectures, Seminars
If your organization is promoting Info-Gap, I suggest that you invite me for a seminar in your place. I promise to deliver a lively, informative, entertaining and convincing presentation explaining why it is not a good idea to use -- let alone promote -- Info-Gap as a decision-making tool.
Here is a list of relevant lectures/seminars on this topic that I gave in the last two years.
- The Rise and Rise of Voodoo Decision Theory.
ASOR Recent Advances, Deakin University, November 26, 2008. This presentation was based on the pages on my website (voodoo.moshe-online.com). - Responsible Decision-Making in the face of Severe Uncertainty (PDF File) .
(Singapore Management University, Singapore, September 29, 2008) - A Critique of Info-Gap's Robustness Model (PDF File) .
(ESREL/SRA 2008 Conference, Valencia, Spain, September 22-25, 2008) - Robust Decision-Making in the Face of Severe Uncertainty (PDF File) .
(Technion, Haifa, Israel, September 15, 2008) - The Art and Science of Robust Decision-Making (PDF File) .
(AIRO 2008 Conference, Ischia, Italy, September 8-11, 2008 ) - The Fundamental Flaws in Info-Gap Decision Theory (PDF File) .
(CSIRO, Canberra, July 9, 2008 ) - Responsible Decision-Making in the Face of Severe Uncertainty (PDF File) .
(OR Conference, ADFA, Canberra, July 7-8, 2008 ) - Responsible Decision-Making in the Face of Severe Uncertainty (PDF File) .
(University of Sydney Seminar, May 16, 2008 ) - Decision-Making Under Severe Uncertainty: An Australian, Operational Research Perspective (PDF File) .
(ASOR National Conference, Melbourne, December 3-5, 2007 ) - A Critique of Info-Gap (PDF File) .
(SRA 2007 Conference, Hobart, August 20, 2007) - What exactly is wrong with Info-Gap?
A Decision Theoretic Perspective (PDF File) .
(MS Colloquium, University of Melbourne, August 1, 2007) - A Formal Look at Info-Gap Theory (PDF File) .
(ORSUM Seminar , University of Melbourne, May 21, 2007) - The Art and Science of Decision-Making Under Severe Uncertainty (PDF File) .
(ACERA seminar, University of Melbourne, May 4, 2007) - What exactly is Info-Gap? An OR perspective. (PDF File)
ASOR Recent Advances in Operations Research mini-conference (December 1, 2006, Melbourne, Australia).
Disclaimer: This page, its contents and style, are the responsibility of the author (Moshe Sniedovich) and do not represent the views, policies or opinions of The University of Melbourne.
