On Heuristics: Trial and Error in Data Science

"We know the laws of trial and error, of large numbers and probabilities. We know that these laws are part of the mathematical and mechanical fabric of the universe, and that they are also at play in biological processes. But, in the name of the experimental method and out of our poor knowledge, are we really entitled to claim that everything happens by chance, to the exclusion of all other possibilities?" (Albert Claude, [Nobel Prize Lecture], 1974)

"Heuristics are rules of thumb that help constrain the problem in certain ways (in other words they help you to avoid falling back on blind trial and error), but they don't guarantee that you will find a solution. Heuristics are often contrasted with algorithms that will guarantee that you find a solution - it may take forever, but if the problem is algorithmic you will get there. However, heuristics are also algorithms." (S Ian Robertson, "Problem Solving", 2001)

"We can simplify the relationships between fragility, errors, and antifragility as follows. When you are fragile, you depend on things following the exact planned course, with as little deviation as possible - for deviations are more harmful than helpful. This is why the fragile needs to be very predictive in its approach, and, conversely, predictive systems cause fragility. When you want deviations, and you don’t care about the possible dispersion of outcomes that the future can bring, since most will be helpful, you are antifragile. Further, the random element in trial and error is not quite random, if it is carried out rationally, using error as a source of information. If every trial provides you with information about what does not work, you start zooming in on a solution - so every attempt becomes more valuable, more like an expense than an error. And of course you make discoveries along the way." (Nassim N Taleb, "Antifragile: Things that gain from disorder", 2012)

"Another crowning achievement of deep learning is its extension to the domain of reinforcement learning. In the context of reinforcement learning, an autonomous agent must learn to perform a task by trial and error, without any guidance from the human operator." (Ian Goodfellow et al, "Deep Learning", 2015)

"Bayesian networks provide a more flexible representation for encoding the conditional independence assumptions between the features in a domain. Ideally, the topology of a network should reflect the causal relationships between the entities in a domain. Properly constructed Bayesian networks are relatively powerful models that can capture the interactions between descriptive features in determining a prediction." (John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, worked examples, and case studies", 2015) 

"A learning algorithm for a robot or a software agent to take actions in an environment so as to maximize the sum of rewards through trial and error." (Tomohiro Yamaguchi et al, "Analyzing the Goal-Finding Process of Human Learning With the Reflection Subtask", 2018)

"Reinforcement learning is also a subset of AI algorithms which creates independent, self-learning systems through trial and error. Any positive action is assigned a reward and any negative action would result in a punishment. Reinforcement learning can be used in training autonomous vehicles where the goal would be obtaining the maximum rewards." (Vijayaraghavan Varadharajan & Akanksha Rajendra Singh, "Building Intelligent Cities: Concepts, Principles, and Technologies", 2021)

"Methodologically, much of modern machine learning practice rests on a variant of trial and error, which we call the train-test paradigm. Practitioners repeatedly build models using any number of heuristics and test their performance to see what works. Anything goes as far as training is concerned, subject only to computational constraints, so long as the performance looks good in testing. Trial and error is sound so long as the testing protocol is robust enough to absorb the pressure placed on it." (Moritz Hardt & Benjamin Recht, "Patterns, Predictions, and Actions: Foundations of Machine Learning", 2022)

On Heuristics: Trial and Error in Mathematics

"It is common sense to take a method and try it. If it fails, admit it frankly and try another. But above all, try something." (Franklin D Roosevelt, Address at Oglethorpe University, 1932)

"The more difficult and novel the problem, the greater is likely to be the amount of trial and error required to find a solution. At the same time, the trial and error is not completely random or blind; it is, in fact, rather highly selective. The new expressions that are obtained by transforming given ones are examined to see whether they represent progress toward the goal. Indications of progress spur further search in the same direction; lack of progress signals the abandonment of a line of search. Problem solving requires selective trial and error." (Herbert A Simon, "The Sciences of the Artificial". 1969)

"We know the laws of trial and error, of large numbers and probabilities. We know that these laws are part of the mathematical and mechanical fabric of the universe, and that they are also at play in biological processes. But, in the name of the experimental method and out of our poor knowledge, are we really entitled to claim that everything happens by chance, to the exclusion of all other possibilities?" (Albert Claude, [Nobel Prize Lecture], 1974)

"Mathematics is not a deductive science - that’s a cliché. When you try to prove a theorem, you don’t just list the hypotheses, and then start to reason. What you do is trial and error, experimentation, guesswork." (Paul R Halmos, "I Want to Be A Mathematician", 1985)

"Heuristics are rules of thumb that help constrain the problem in certain ways (in other words they help you to avoid falling back on blind trial and error), but they don't guarantee that you will find a solution. Heuristics are often contrasted with algorithms that will guarantee that you find a solution - it may take forever, but if the problem is algorithmic you will get there. However, heuristics are also algorithms." (S Ian Robertson, "Problem Solving", 2001)

“Another feature of Bourbaki is that it rejects intuition of any kind. Bourbaki books tend not to contain explanations, examples, or heuristics. One of the main messages of the present book is that we record mathematics for posterity in a strictly rigorous, axiomatic fashion. This is the mathematician’s version of the reproducible experiment with control used by physicists and biologists and chemists. But we learn mathematics, we discover mathematics, we create mathematics using intuition and trial and error. We draw pictures. Certainly, we try things and twist things around and bend things to try to make them work. Unfortunately, Bourbaki does not teach any part of this latter process.” (Steven G Krantz, “The Proof is in the Pudding: The Changing Nature of Mathematical Proof”, 2010)

On Heuristics: Trial and Error in Biology

"Higher organisms are able to learn through trial and error how a certain problem should be solved. We may say that they too make testing movements - mental testings - and that to learn is essentially to tryout one testing movement after another until one is found that solves the problem. We might compare the animal's successful solution to an expectation and hence to a hypothesis or a theory. For the animal's behaviour shows us that it expects (perhaps unconsciously or dispositionally) that in a similar case the same testing movements will again solve the problem in question." (Karl R Popper, "The Logic and Evolution of Scientific Theory", [in "All Life is Problem Solving", 1999] 1972)

"The natural as well as the social sciences always start from problems, from the fact that something inspires amazement in us, as the Greek philosophers used to say. To solve these problems, the sciences use fundamentally the same method that common sense employs, the method of trial and error. To be more precise, it is the method of trying out solutions to our problem and then discarding the false ones as erroneous. This method assumes that we work with a large number of experimental solutions. One solution after another is put to the test and eliminated." (Karl R Popper, "The Logic and Evolution of Scientific Theory", [in "All Life is Problem Solving", 1999] 1972)

"Whatever the system, adaptive change depends upon feedback loops, be it those provided by natural selection or those of individual reinforcement. In all cases, then, there must be a process of trial and error and a mechanism of comparison. […] By superposing and interconnecting many feedback loops, we (and all other biological systems) not only solve particular problems but also form habits which we apply to the solution of classes of problems." (Gregory Bateson, "Steps to an Ecology of Mind", 1972)

"When we examine this suggestion, we see that it is no more than a formal acknowledgement of a problem, the problem of how (by what institutional arrangement, by what organization of affairs) the equilibrium prices are to be discovered. Repeated trial and error, while the market stands in suspense awaiting the outcome, is not a practical resort. The number of distinct trials, even if confined to discrete steps of price and quantity, would be so immense that the necessary 'market day' would extend beyond human life-times [... The] theoretical ideal applies to mutually isolated days or moments, each to be treated as perfectly self-contained and looking to no yesterday and no tomorrow. But the real market is dealing with goods inherited from yesterday, and in means of production whose products will not be ready till tomorrow. Meanwhile the non-economic circumstances are changing and rendering each successive equilibrium obsolete." (George L S Shackle, "Epistemics and Economics", 1972)

"We know the laws of trial and error, of large numbers and probabilities. We know that these laws are part of the mathematical and mechanical fabric of the universe, and that they are also at play in biological processes. But, in the name of the experimental method and out of our poor knowledge, are we really entitled to claim that everything happens by chance, to the exclusion of all other possibilities?" (Albert Claude, "The Coming Age of the Cell", Nobel Prize Lecture] 1974)

"Survival machines that can simulate the future are one jump ahead of survival machines that who can only learn of the basis of trial and error. The trouble with overt trial is that it takes time and energy. The trouble with overt error is that it is often fatal. [...] The evolution of the capacity to simulate seems to have culminated in subjective consciousness. Why this should have happened is, to me, the most profound mystery facing modern biology." (Richard Dawkins, "The Selfish Gene Source: The Selfish Gene", 1976)

"In the process of the evolution of life, as far as we know, the total mass of living matter has always been and is now increasing and growing more complex in its organization. To increase the complexity of the organization of biological forms, nature operates by trial and error. Existing forms are reproduced in many copies, but these are not identical to the original. Instead they differ from it by the presence of small random variations."  (Valentin F Turchin, "The Phenomenon of Science: A cybernetic approach to human evolution", 1977)

"I believe people can solve complex problems eventually. By repeated trial and error they will get there; but they need a long time. At this point I agree with Herbert Simon. People do not learn immediately, as those rational expectations models seem to imply. I don't believe that. The statement that assumptions do not matter is nonsense. It is funny. Yes, I assume people are consistent in their behavior. I assume that not because I believe everybody actually is, but because I believe, on the average, you do not get too far from it." (Franco Modigliani, "Conversations with Economists", 1983)

"Heuristics are rules of thumb that help constrain the problem in certain ways (in other words they help you to avoid falling back on blind trial and error), but they don't guarantee that you will find a solution. Heuristics are often contrasted with algorithms that will guarantee that you find a solution - it may take forever, but if the problem is algorithmic you will get there. However, heuristics are also algorithms." (S Ian Robertson, "Problem Solving", 2001)

On Heuristics: Trial and Error in Science

"The one lesson that comes out of all our theorizing and experimenting is that there is only one really scientific progressive method; and that is the method of trial and error." (George B Shaw, "The Doctor's Dilemma: Preface on Doctors", 1909)

"There are many men now living who were in the habit of using the age-old expression: 'It is as impossible as flying.' The discoveries in physical science, the triumphs in invention, attest the value of the process of trial and error. In large measure, these advances have been due to experimentation." (Louis Brandeis, "Dissent, New State Ice Co. v. Liebmann, 285 U.S. 262", 1932)

"The discoveries in physical science, the triumphs in invention, attest the value of the process of trial and error. In large measure, these advances have been due to experimentation." (Louis Brandeis, "Judicial opinions", 1932)

"But I believe that there is no philosophical high-road in science, with epistemological signposts. No, we are in a jungle and find our way by trial and error, building our road behind us as we proceed. We do not find signposts at cross-roads, but our own scouts erect them, to help the rest." (Max Born, "Experiment and Theory in Physics", 1943)

"The method of learning by trial and error - of learning from our mistakes - seems to be fundamentally the same whether it is practised by lower or by higher animals, by chimpanzees or by men of science." (Karl Popper, "Conjectures and Refutations: The Growth of Scientific Knowledge", 1963)

"The difference between the amoeba and Einstein is that, although both make use of the method of trial and error elimination, the amoeba dislikes erring while Einstein is intrigued by it [...]" (Karl R Popper, "Objective Knowledge: An Evolutionary Approach", 1972) 

"Science progresses by trial and error, by conjectures and refutations. Only the fittest theories survive." (Alan Chalmers, "What Is This Thing Called Science?", 1976)

"The great revolutions in science are almost always the result of unexpected intuitive leaps. After all, what is science if not the posing of difficult puzzles by the universe? Mother Nature does something interesting, and challenges the scientist to figure out how she does it. In many cases the solution is not found by exhaustive trial and error […] or even by a deduction based on the relevant knowledge." (Martin Gardner, "Aha! Insight", 1978)

"I believe people can solve complex problems eventually. By repeated trial and error they will get there; but they need a long time. At this point I agree with Herbert Simon. People do not learn immediately, as those rational expectations models seem to imply. I don't believe that. The statement that assumptions do not matter is nonsense. It is funny. Yes, I assume people are consistent in their behavior. I assume that not because I believe everybody actually is, but because I believe, on the average, you do not get too far from it." (Franco Modigliani, "Conversations with Economists", 1983)

"Science usually amounts to a lot more than blind trial and error. Good statistics consists of much more than just significance tests; there are more sophisticated tools available for the analysis of results, such as confidence statements, multiple comparisons, and Bayesian analysis, to drop a few names. However, not all scientists are good statisticians, or want to be, and not all people who are called scientists by the media deserve to be so described." (Robert Hooke, "How to Tell the Liars from the Statisticians", 1983)

"Whatever humans have learned had to be learned as a consequence only of trial and error experience. Humans have learned only through mistakes." (R Buckminster Fuller, "Intuition", 1983)

"Growth is a process of experimentation, a series of trials, errors, and occasional victories. The failed experiments are as much as part of the process as the experiments that work." (Chérie Carter-Scott, "If Life Is a Game, These Are the Rules", 1998)

"The natural as well as the social sciences always start from problems, from the fact that something inspires amazement in us, as the Greek philosophers used to say. To solve these problems, the sciences use fundamentally the same method that common sense employs, the method of trial and error. To be more precise, it is the method of trying out solutions to our problem and then discarding the false ones as erroneous. This method assumes that we work with a large number of experimental solutions. One solution after another is put to the test and eliminated." (Karl R Popper, "All Life is Problem Solving", 1999)

"We can simplify the relationships between fragility, errors, and antifragility as follows. When you are fragile, you depend on things following the exact planned course, with as little deviation as possible - for deviations are more harmful than helpful. This is why the fragile needs to be very predictive in its approach, and, conversely, predictive systems cause fragility. When you want deviations, and you don’t care about the possible dispersion of outcomes that the future can bring, since most will be helpful, you are antifragile. Further, the random element in trial and error is not quite random, if it is carried out rationally, using error as a source of information. If every trial provides you with information about what does not work, you start zooming in on a solution - so every attempt becomes more valuable, more like an expense than an error. And of course you make discoveries along the way." (Nassim N Taleb, "Antifragile: Things that gain from disorder", 2012)

On Problem Solving XVII: Heuristics

"Devising the plan of the solution, we should not be too afraid of merely plausible, heuristic reasoning. Anything is right that leads to the right idea. But we have to change this standpoint when we start carrying out the plan and then we should accept only conclusive, strict arguments." (George Pólya, "How to solve it", 1945)

"Heuristic reasoning is reasoning not regarded as final and strict but as provisional and plausible only, whose purpose is to discover the solution of the present problem. We are often obliged to use heuristic reasoning. We shall attain complete certainty when we shall have obtained the complete solution, but before obtaining certainty we must often be satisfied with a more or less plausible guess. We may need the provisional before we attain the final. We need heuristic reasoning when we construct a strict proof as we need scaffolding when we erect a building." (George Pólya, "How to solve it", 1945)

"Fast and frugal heuristics employ a minimum of time, knowledge, and computation to make adaptive choices in real environments. They can be used to solve problems of sequential search through objects or options, as in satisficing. They can also be used to make choices between simultaneously available objects, where the search for information (in the form of cues, features, consequences, etc.) about the possible options must be limited, rather than the search for the options themselves. Fast and frugal heuristics limit their search of objects or information using easily computable stopping rules, and they make their choices with easily computable decision rules." (Gerd Gigerenzer & Peter M Todd, "Fast and Frugal Heuristics: The Adaptive Toolbox" [in "Simple Heuristics That Make Us Smart"], 1999)

"Heuristics are rules of thumb that help constrain the problem in certain ways (in other words they help you to avoid falling back on blind trial and error), but they don't guarantee that you will find a solution. Heuristics are often contrasted with algorithms that will guarantee that you find a solution - it may take forever, but if the problem is algorithmic you will get there. However, heuristics are also algorithms." (S Ian Robertson, "Problem Solving", 2001)

"Heuristics are needed in situations where the world does not permit optimization. For many real-world problems (as opposed to optimization-tuned textbook problems), optimal solutions are unknown because the problems are computationally intractable or poorly defined." (Christoph Engel & Gerd Gigerenzer, "Law and Heuristics: An interdisciplinary venture" [in "Heuristics and the Law", 2006)

"Less apparent is that qualitatively different problem-solving techniques are required at high levels of complexity than at low ones. Purely analytical techniques, powerful for the lower levels, can be overwhelmed at the higher ones. At higher levels, architecting methods, experience-based heuristics, abstraction, and integrated modeling must be called into play."  (Mark W Maier, "The Art Systems of Architecting" 3rd Ed., 2009)

Previous <<||>> Next

On Heuristics IV

"Design problems - generating or discovering alternatives - are complex largely because they involve two spaces, an action space and a state space, that generally have completely different structures. To find a design requires mapping the former of these on the latter. For many, if not most, design problems in the real world systematic algorithms are not known that guarantee solutions with reasonable amounts of computing effort. Design uses a wide range of heuristic devices - like means-end analysis, satisficing, and the other procedures that have been outlined - that have been found by experience to enhance the efficiency of search. Much remains to be learned about the nature and effectiveness of these devices." (Herbert A Simon, "The Logic of Heuristic Decision Making", [in "The Logic of Decision and Action"], 1966)

"Heuristics are rules of thumb that help constrain the problem in certain ways (in other words they help you to avoid falling back on blind trial and error), but they don't guarantee that you will find a solution. Heuristics are often contrasted with algorithms that will guarantee that you find a solution - it may take forever, but if the problem is algorithmic you will get there. However, heuristics are also algorithms." (S Ian Robertson, "Problem Solving", 2001)

"A heuristic is defined as a simple rule that exploits both evolved abilities to act fast and struc- tures of the environment to act accurately and frugally. The complexity and uncertainty of an environment cannot be determined independently of the actor. What matters is the degree of complexity and uncertainty encountered by the decision maker." (Christoph Engel & Gerd Gigerenzer, "Law and Heuristics: An interdisciplinary venture" [in "Heuristics and the Law", 2006)

"Heuristics are needed in situations where the world does not permit optimization. For many real-world problems (as opposed to optimization-tuned textbook problems), optimal solutions are unknown because the problems are computationally intractable or poorly defined." (Christoph Engel & Gerd Gigerenzer, "Law and Heuristics: An interdisciplinary venture" [in "Heuristics and the Law", 2006)

"Perception and memory are imprecise filters of information, and the way in which information is presented, that is, the frame, influences how it is received. Because too much information is difficult to deal with, people have developed shortcuts or heuristics in order to come up with reasonable decisions. Unfortunately, sometimes these heuristics lead to bias, especially when used outside their natural domains." (Lucy F Ackert & Richard Deaves, "Behavioral Finance: Psychology, Decision-Making, and Markets", 2010)

"A rule of thumb, or heuristic, enables us to make a decision fast, without much searching for information, but nevertheless with high accuracy. [...] A heuristic can be safer and more accurate than a calculation, and the same heuristic can underlie both conscious and unconscious decisions." (Gerd Gigerenzer, "Risk Savvy: How to make good decisions", 2014)

"A heuristic is a strategy that ignores part of the information, with the goal of making decisions more quickly, frugally, and/or accurately than more complex methods." (Gerd Gigerenzer et al, "Simply Rational: Decision Making in the Real World", 2015)

"Probability theory is not the only tool for rationality. In situations of uncertainty, as opposed to risk, simple heuristics can lead to more accurate judgments, in addition to being faster and more frugal. Under uncertainty, optimal solutions do not exist (except in hindsight) and, by definition, cannot be calculated. Thus, it is illusory to model the mind as a general optimizer, Bayesian or otherwise. Rather, the goal is to achieve satisficing solutions, such as meeting an aspiration level or coming out ahead of a competitor."  (Gerd Gigerenzer et al, "Simply Rational: Decision Making in the Real World", 2015)

"Judgments made in difficult circumstances can be based on a limited number of simple, rapidly-arrived-at rules ('heuristics'), rather than formal, extensive algorithmic calculus and programs. Often, even complex problems can be solved quickly and accurately using such 'quick and dirty' heuristics. However, equally often, such heuristics can be beset by systematic errors or biases." (Jérôme Boutang & Michel De Lara, "The Biased Mind", 2016)

On Fuzzy Logic IV

"Fuzzy systems are excellent tools for representing heuristic, commonsense rules. Fuzzy inference methods apply these rules to data and infer a solution. Neural networks are very efficient at learning heuristics from data. They are 'good problem solvers' when past data are available. Both fuzzy systems and neural networks are universal approximators in a sense, that is, for a given continuous objective function there will be a fuzzy system and a neural network which approximate it to any degree of accuracy." (Nikola K Kasabov, "Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering", 1996)

"Fuzzy set theory [...] is primarily concerned with quantifying and reasoning using natural language in which words can have ambiguous meanings. It is widely used in a variety of fields because of its simplicity and similarity to human reasoning." (Tzung-Pei Hong et al, "Genetic-Fuzzy Data Mining Techniques", 2009)

"Fuzzy systems are rule-based expert systems based on fuzzy rules and fuzzy inference. Fuzzy rules represent in a straightforward way 'commonsense' knowledge and skills, or knowledge that is subjective, ambiguous, vague, or contradictory." (Nikola K Kasabov, "Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering", 1996)

"A concept which has a position of centrality in fuzzy logic is that of a fuzzy set. Informally, a fuzzy set is a class with a fuzzy boundary, implying a gradual transition from membership to nonmembership. A fuzzy set is precisiated through graduation, that is, through association with a scale of grades of membership. Thus, membership in a fuzzy set is a matter of degree. Importantly, in fuzzy logic everything is or is allowed to be graduated, that is, be a matter of degree. Furthermore, in fuzzy logic everything is or is allowed to be granulated, with a granule being a clump of attribute-values drawn together by indistinguishability, equivalence, similarity, proximity or functionality. Graduation and granulation form the core of fuzzy logic. Graduated granulation is the basis for the concept of a linguistic variable – a variable whose values are words rather than numbers. The concept of a linguistic variable is employed in almost all applications of fuzzy logic." (Lofti A Zadeh, "Fuzzy Logic", 2009)

"[Fuzzy logic p]rovides formal means for the representation of, and inference based on imprecisely specified premises and rules of inference; can be understood in different ways, basically as fuzzy logic in a narrow sense, being some type of multivalued logic, and fuzzy logic in a broad sense, being a way to formalize inference based on imprecisely specified premises and rules of inference. (Janusz Kacprzyk, "Foundations of Fuzzy Sets Theory", 2009)

"Granular computing is a general computation theory for using granules such as subsets, classes, objects, clusters, and elements of a universe to build an efficient computational model for complex applications with huge amounts of data, information, and knowledge. Granulation of an object a leads to a collection of granules, with a granule being a clump of points (objects) drawn together by indiscernibility, similarity, proximity, or functionality. In human reasoning and concept formulation, the granules and the values of their attributes are fuzzy rather than crisp. In this perspective, fuzzy information granulation may be viewed as a mode of generalization, which can be applied to any concept, method, or theory." (Salvatore Greco et al, "Granular Computing and Data Mining for Ordered Data: The Dominance-Based Rough Set Approach", 2009)

"In essence, logic is concerned with formalization of reasoning. Correspondently, fuzzy logic is concerned with formalization of fuzzy reasoning, with the understanding that precise reasoning is a special case of fuzzy reasoning." (Lofti A Zadeh, "Fuzzy Logic", 2009)

"Science deals not with reality but with models of reality. In large measure, scientific progress is driven by a quest for better models of reality. In the real world, imprecision, uncertainty and complexity have a pervasive presence. In this setting, construction of better models of reality requires a better understanding of how to deal effectively with imprecision, uncertainty and complexity. To a significant degree, development of fuzzy logic has been, and continues to be, motivated by this need." (Lofti A Zadeh, "Fuzzy Logic", 2009)

"Unlike a conventional set, in a fuzzy set, a fuzzy membership function is used to define the degree of an element belonging to the set. Fuzzy logic is a superset of conventional (Boolean) logic that has been extended to handle the concept of partial truth as defined by membership functions. Fuzzy logic contributes to the machinery of granular computing." (Zhengxin Chen, "Philosophical Foundation for Granular Computing", 2009)

"Unlike the classic set theory where a set is represented as an indicator function to specify if an object belongs or not to it, a fuzzy set is an extension of a classic set where a subset is represented as a membership function to characterize the degree that an object belongs to it. The indicator function of a classic set takes value of 1 or 0, whereas the membership function of a fuzzy set takes value between 1 and 0." (Jianchao Han & Nick Cerone, Principles and Perspectives of Granular Computing, 2009) 

On Heuristics III

"Intelligence has two parts, which we shall call the epistemological and the heuristic. The epistemological part is the representation of the world in such a form that the solution of problems follows from the facts expressed in the representation. The heuristic part is the mechanism that on the basis of the information solves the problem and decides what to do." (John McCarthy & Patrick J Hayes, "Some Philosophical Problems from the Standpoint of Artificial Intelligence", Machine Intelligence 4, 1969)

"Heuristic (it is of Greek origin) means discovery. Heuristic methods are based on experience, rational ideas, and rules of thumb. Heuristics are based more on common sense than on mathematics. Heuristics are useful, for example, when the optimal solution needs an exhaustive search that is not realistic in terms of time. In principle, a heuristic does not guarantee the best solution, but a heuristic solution can provide a tremendous shortcut in cost and time." (Nikola K Kasabov, "Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering", 1996)

"Heuristic methods may aim at local optimization rather than at global optimization, that is, the algorithm optimizes the solution stepwise, finding the best solution at each small step of the solution process and 'hoping' that the global solution, which comprises the local ones, would be satisfactory." (Nikola K Kasabov, "Foundations of Neural Networks, Fuzzy Systems, and Knowledge Engineering", 1996)

"A proof of a mathematical theorem is a sequence of steps which leads to the desired conclusion. The rules to be followed [...] were made explicit when logic was formalized early in the this century [...] These rules can be used to disprove a putative proof by spotting logical errors; they cannot, however, be used to find the missing proof of a [...] conjecture. [...] Heuristic arguments are a common occurrence in the practice of mathematics. However... The role of heuristic arguments has not been acknowledged in the philosophy of mathematics despite the crucial role they play in mathematical discovery. [...] Our purpose is to bring out some of the features of mathematical thinking which are concealed beneath the apparent mechanics of proof." (Gian-Carlo Rota, "Indiscrete Thoughts", 1997)

"A heuristic is ecologically rational to the degree that it is adapted to the structure of an environment. Thus, simple heuristics and environmental structure can both work hand in hand to provide a realistic alternative to the ideal of optimization, whether unbounded or constrained." (Gerd Gigerenzer & Peter M Todd, "Fast and Frugal Heuristics: The Adaptive Toolbox" [in "Simple Heuristics That Make Us Smart"], 1999)

"Fast and frugal heuristics employ a minimum of time, knowledge, and computation to make adaptive choices in real environments. They can be used to solve problems of sequential search through objects or options, as in satisficing. They can also be used to make choices between simultaneously available objects, where the search for information (in the form of cues, features, consequences, etc.) about the possible options must be limited, rather than the search for the options themselves. Fast and frugal heuristics limit their search of objects or information using easily computable stopping rules, and they make their choices with easily computable decision rules." (Gerd Gigerenzer & Peter M Todd, "Fast and Frugal Heuristics: The Adaptive Toolbox" [in "Simple Heuristics That Make Us Smart"], 1999)

"Theories of choice are at best approximate and incomplete. One reason for this pessimistic assessment is that choice is a constructive and contingent process. When faced with a complex problem, people employ a variety of heuristic procedures in order to simplify the representation and the evaluation of prospects. These procedures include computational shortcuts and editing operations, such as eliminating common components and discarding nonessential differences. The heuristics of choice do not readily lend themselves to formal analysis because their application depends on the formulation of the problem, the method of elicitation, and the context of choice." (Amos Tversky & Daniel Kahneman, "Advances in Prospect Theory: Cumulative Representation of Uncertainty" [in "Choices, Values, and Frames"], 2000)

"A heuristic is a rule applied to an existing solution represented in a perspective that generates a new (and hopefully better) solution or a new set of possible solutions." (Scott E Page, "The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools and Societies", 2008)

"A heuristic is a decision rule that utilizes a subset of the information set. Since in virtually all cases people must economize and cannot analyze all contingencies, we use heuristics without even realizing it." (Lucy F Ackert & Richard Deaves, "Behavioral Finance: Psychology, Decision-Making, and Markets", 2010)

On Heuristics II

"Models of bounded rationality describe how a judgement or decision is reached (that is, the heuristic processes or proximal mechanisms) rather than merely the outcome of the decision, and they describe the class of environments in which these heuristics will succeed or fail." (Gerd Gigerenzer & Reinhard Selten [Eds., "Bounded Rationality: The Adaptive Toolbox", 2001)

"A second class of metaphors - mathematical algorithms, heuristics, and models - brings us closer to the world of computer science programs, simulations, and approximations of the brain and its cognitive processes." (Diego Rasskin-Gutman, "Chess Metaphors: Artificial Intelligence and the Human Mind", 2009)

"In particular, the accurate intuitions of experts are better explained by the effects of prolonged practice than by heuristics. We can now draw a richer and more balanced picture, in which skill and heuristics are alternative sources of intuitive judgments and choices." (Daniel Kahneman, "Thinking, Fast and Slow", 2011)

"This is the essence of intuitive heuristics: when faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution." (Daniel Kahneman, "Thinking, Fast and Slow", 2011)

"Heuristics are an evolutionary solution to an ongoing problem: we have limited mental resources. As such, they have a very long and thoroughly time-tested history of helping us - on average - make better decisions." (Peter H Diamandis, "Abundance: The Future is Better Than You Think", 2012)

"Heuristics are simplified rules of thumb that make things simple and easy to implement. But their main advantage is that the user knows that they are not perfect, just expedient, and is therefore less fooled by their powers. They become dangerous when we forget that." (Nassim N Taleb, "Antifragile: Things that gain from disorder", 2012)

"The art of reasoned persuasion is an iterative, recursive heuristic, meaning that we must go back and forth between the facts and the rules until we have a good fit. We cannot see the facts properly until we know what framework to place them into, and we cannot determine what framework to place them into until we see the basic contours of the facts." (Joel P Trachtman, "The Tools of Argument", 2013)

"Heuristic decision making is fast and frugal and is often based on the evaluation of one or two salient bits of information." (Amitav Chakravarti, "Why People (Don’t) Buy: The Go and Stop Signals", 2015)

"A heuristic is a strategy we derive from previous experience with a similar problem." (Darius Foroux, "Think Straight", 2017)

"The social world that humans have made for themselves is so complex that the mind simplifies the world by using heuristics, customs, and habits, and by making models or assumptions about how things generally work (the ‘causal structure of the world’). And because people rely upon (and are invested in) these mental models, they usually prefer that they remain uncontested." (Dr James Brennan, "Psychological  Adjustment to Illness and Injury", West of England Medical Journal Vol. 117 (2), 2018)

On Heuristics I

"The materialistic point of view in psychology can claim, at best, only the value of an heuristic hypothesis." (Wilhelm Wundt, "Principles of Physiological Psychology", 1874)

"Heuristic reasoning is good in itself. What is bad is to mix up heuristic reasoning with rigorous proof. What is worse is to sell heuristic reasoning for rigorous proof." (George Pólya, "How to Solve It", 1945)

"Heuristic, or heuretic, or 'ars inveniendi' was the name of a certain branch of study, not very clearly circumscribed, belonging to logic, or to philosophy, or to psychology, often outlined, seldom presented in detail, and as good as forgotten today. The aim of heuristic is to study the methods and rules of discovery and invention. [...] Heuristic, as an adjective, means 'serving to discover'." (George Pólya, "How to Solve It", 1945)

"Heuristic reasoning is reasoning not regarded as final and strict but as provisional and plausible only, whose purpose is to discover the solution of the present problem. We are often obliged to use heuristic reasoning. We shall attain complete certainty when we shall have obtained the complete solution, but before obtaining certainty we must often be satisfied with a more or less plausible guess. We may need the provisional before we attain the final. We need heuristic reasoning when we construct a strict proof as we need scaffolding when we erect a building." (George Pólya, "How to Solve It", 1945)

"The attempt to characterize exactly models of an empirical theory almost inevitably yields a more precise and clearer understanding of the exact character of a theory. The emptiness and shallowness of many classical theories in the social sciences is well brought out by the attempt to formulate in any exact fashion what constitutes a model of the theory. The kind of theory which mainly consists of insightful remarks and heuristic slogans will not be amenable to this treatment. The effort to make it exact will at the same time reveal the weakness of the theory." (Patrick Suppes," A Comparison of the Meaning and Uses of Models in Mathematics and the Empirical Sciences", Synthese  Vol. 12 (2/3), 1960)

"Factoring big numbers is a strange kind of mathematics that closely resembles the experimental sciences, where nature has the last and definitive word. […] as with the experimental sciences, both rigorous and heuristic analyses can be valuable in understanding the subject and moving it forward. And, as with the experimental sciences, there is sometimes a tension between pure and applied practitioners." (Carl B Pomerance, "A Tale of Two Sieves", The Notices of the American Mathematical Society 43, 1996)

"[…] mathematics does not come to us written indelibly on Nature’s Tablets, but rather is the product of a controlled search governed by metaphorical considerations, the premier instance being the heuristics of the conservation principles." (Philip Mirowski, "More Heat than Light: Economics as Social Physics: Physics as Nature’s Economics", 1989)

"Mathematicians, like the rest of us, cherish clever ideas; in particular they delight in an ingenious picture. But this appreciation does not overwhelm a prevailing skepticism. After all, a diagram is - at best - just a special case and so can't establish a general theorem. Even worse, it can be downright misleading. Though not universal, the prevailing attitude is that pictures are really no more than heuristic devices; they are psychologically suggestive and pedagogically important - but they prove nothing. I want to oppose this view and to make a case for pictures having a legitimate role to play as evidence and justification - a role well beyond the heuristic.  In short, pictures can prove theorems." (James R Brown, "Philosophy of Mathematics: An Introduction to the World of Proofs and Pictures", 1999)

"In the language of mental models, such past experience provided the default assumptions necessary to fill the gaps in the emerging and necessarily incomplete framework of a relativistic theory of gravitation. It was precisely the nature of these default assumptions that allowed them to be discarded again in the light of novel information - provided, for instance, by the further elaboration of the mathematical formalism - without, however, having to abandon the underlying mental models which could thus continue to function as heuristic orientations." (Jürgen Renn, "Before the Riemann Tensor: The Emergence of Einstein’s Double Strategy", [in "The Universe of General Relativity"] 2000)

"You can often hear from non-mathematicians, especially from philosophers, that mathematics consists exclusively in drawing conclusions from clearly stated premises; and that in this process, it makes no difference what these premises signify, whether they are true or fa1se, provided only that they do not contradict one another. But a per. son who has done productive mathematical work will talk quite differently. In fact these people [the non-mathematicians] are thinking only of the crystallized form into which finished mathematica1 theories are finally cast. However, the investigator himself, in mathematics as in every other science, does not work in this rigorous deductive fashion. On the contrary, he makes essential use of his imagination and proceeds inductively aided by heuristic expedients. One can give numerous examples of mathematicians who have discovered theorems of the greatest importance which they were unable to prove. Should one then refuse to recognize this as a great accomplishment and in deference to the above definition insist that this is not mathematics? After all it is an arbitrary thing how the word is to be used, but no judgment of value can deny that the inductive work of the person who first announces the theorem is at least as valuable as the deductive work. of the one who proves it. For both are equally necessary and the discovery is the presupposition of the later conclusion." (Felix Klein)

Mental Models LVIII

"We form in the imagination some sort of diagrammatic, that is, iconic, representation of the facts, as skeletonized as possible. The impression of the present writer is that with ordinary persons this is always a visual image, or mixed visual and muscular; but this is an opinion not founded on any systematic examination." (Charles S Peirce, "Notes on Ampliative Reasoning", 1901)

"A mental model is conceived here as a knowledge structure possessing slots that can be filled not only with empirically gained information but also with 'default assumptions' resulting from prior experience. These default assumptions can be substituted by updated information so that inferences based on the model can be corrected without abandoning the model as a whole. Information is assimilated to the slots of a mental model in the form of 'frames' which are understood here as 'chunks' of knowledge with a well-defined meaning anchored in a given body of shared knowledge." (Jürgen Renn, "Before the Riemann Tensor: The Emergence of Einstein’s Double Strategy", [in "The Universe of General Relativity"] 2000)

"In the language of mental models, such past experience provided the default assumptions necessary to fill the gaps in the emerging and necessarily incomplete framework of a relativistic theory of gravitation. It was precisely the nature of these default assumptions that allowed them to be discarded again in the light of novel information - provided, for instance, by the further elaboration of the mathematical formalism - without, however, having to abandon the underlying mental models which could thus continue to function as heuristic orientations." (Jürgen Renn, "Before the Riemann Tensor: The Emergence of Einstein’s Double Strategy", [in "The Universe of General Relativity"] 2000)

"A mental model represents a possibility, or, to be precise, the structure and content of the model capture what is common to the different ways in which the possibility could occur [...]" (Philip N Johnson-Laird, Mental Models, Sentential Reasoning, and Illusory Inferences, [in "Mental Models and the Mind"], 2006)

"According to mental model theory, human reasoning relies on the construction of integrated mental representations of the information that is given in the reasoning problem's premises. These integrated representations are the mental models. A mental model is a mental representation that captures what is common to all the different ways in which the premises can be interpreted. It represents in "small scale" how "reality" could be— according to what is stated in the premises of a reasoning problem. Mental models, though, must not be confused with images. A mental model often forms the basis of one or more visual images, but some of them represent situations that cannot be visualized. Instead, mental models are often likened to diagrams since, as with diagrams, their structure is analogous to the structure of the states of affairs they represent." (Carsten Held et al, "Mental Models and the Mind", 2006)

"Mental models are mental representations of a certain type. The main problem in the philosophy of mental representation is to characterize the relation between a mental representation and the represented object. Naively speaking, a mental representation is an entity that 'stands for' another—the represented object - , but here 'stands for' is just a metaphoric place-holder for 'represents', thus requires further explanation." (Carsten Held et al, "Mental Models and the Mind", 2006)

"Prom the processing view, the model theory distinguishes between three different operations. In the construction phase, reasoners construct the mental model that reflects the information from the premises. In the inspection phase, this model is inspected to find new information that is not explicitly given in the premises. In most variants of the model theory, the inspection process is conceptualized as a spatial focus that scans the model to find new information not given in the premises.. In the variation phase, reasoners try to construct alternative models from the premises that refute the putative conclusion. If no such model is found, the putative conclusion is considered true." (Carsten Held et al, "Mental Models and the Mind", 2006)

"The model theory postulates that mental models are parsimonious. They represent what is possible, but not what is impossible, according to assertions. This principle of parsimony minimizes the load on working memory, and so it applies unless something exceptional occurs to overrule it." (Philip N Johnson-Laird, Mental Models, Sentential Reasoning, and Illusory Inferences, [in "Mental Models and the Mind"], 2006)

"Just as physicists have created models of the atom based on observed data and intuitive synthesis of the patterns in their data, so must designers create models of users based on observed behaviors and intuitive synthesis of the patterns in the data. Only after we formalize such patterns can we hope to systematically construct patterns of interaction that smoothly match the behavior patterns, mental models, and goals of users. Personas provide this formalization." (Alan Cooper et al, "About Face 3: The Essentials of Interaction Design", 2007)

"We tend to form mental models that are simpler than reality; so if we create represented models that are simpler than the actual implementation model, we help the user achieve a better understanding. […] Understanding how software actually works always helps someone to use it, but this understanding usually comes at a significant cost. One of the most significant ways in which computers can assist human beings is by putting a simple face on complex processes and situations. As a result, user interfaces that are consistent with users’ mental models are vastly superior to those that are merely reflections of the implementation model." (Alan Cooper et al,  "About Face 3: The Essentials of Interaction Design", 2007)

On Diagrams (1975-1999)

"Pencil and paper for construction of distributions, scatter diagrams, and run-charts to compare small groups and to detect trends are more efficient methods of estimation than statistical inference that depends on variances and standard errors, as the simple techniques preserve the information in the original data." (W Edwards Deming, "On Probability as Basis for Action", American Statistician, Volume 29, Number 4, November 1975)

"The formalist makes a distinction between geometry as a deductive structure and geometry as a descriptive science. Only the first is regarded as mathematical. The use of pictures or diagrams, or even mental imagery, all are non- mathematical. In principle, they should be unnecessary. Consequently. he regards them as inappropriate in a mathematics text, perhaps even in a mathematics class." (Philip J Davis & Reuben Hersh, "The Mathematical Experience", 1981)

"The thinking person goes over the same ground many times. He looks at it from varying points of view - his own, his arch-enemy’s, others’. He diagrams it, verbalizes it, formulates equations, constructs visual images of the whole problem, or of troublesome parts, or of what is clearly known. But he does not keep a detailed record of all this mental work, indeed could not. […] Deep understanding of a domain of knowledge requires knowing it in various ways. This multiplicity of perspectives grows slowly through hard work and sets the state for the re-cognition we experience as a new insight." (Howard E Gruber, "Darwin on Man", 1981)

"[The diagram] is only an heuristic to prompt certain trains of inference; [...] it is dispensable as a proof-theoretic device; indeed, [...] it has no proper place in the proof as such. For the proof is a syntactic object consisting only of sentences arranged in a finite and inspectable array." (Neil Tennant, "The withering away of formal semantics", Mind and Language Vol. 1 (4), 1986)

"We distinguish diagrammatic from sentential paper-and-pencil representations of information by developing alternative models of information-processing systems that are informationally equivalent and that can be characterized as sentential or diagrammatic. Sentential representations are sequential, like the propositions in a text. Diagrammatic representations are indexed by location in a plane. Diagrammatic representations also typically display information that is only implicit in sentential representations and that therefore has to be computed, sometimes at great cost, to make it explicit for use. We then contrast the computational efficiency of these representations for solving several. illustrative problems in mathematics and physics." (Herbert A Simon, "Why a diagram is (sometimes) worth ten thousand words", 1987) 

"People who have a casual interest in mathematics may get the idea that a topologist is a mathematical playboy who spends his time making Möbius bands and other diverting topological models. If they were to open any recent textbook in topology, they would be surprised. They would find page after page of symbols, seldom relieved by a picture or diagram." (Martin Gardner, "Hexaflexagons and Other Mathematical Diversions", 1988)

"The value of diagram techniques even at this rudimentary level should be clear by now: it is easier to visualize where simplifications may be found in a complicated network by searching for a reducible linkage than by examining a complicated algebraic expression."(Geoffrey E Stedman, "Diagram Techniques in Group Theory", 1990)

"Diagrams are physical situations. They must be, since we can see them. As such, they obey their own set of constraints. […] By choosing a representational scheme appropriately, so that the constraints on the diagrams have a good match with the constraints on the described situation, the diagram can generate a lot of information that the user never need infer. Rather, the user can simply read off facts from the diagram as needed." (Jon Barwise & John Etchemendy, "Visual information and valid reasoning", [in "Visualization in Teaching and Learning Mathematics"], 1991)

"It has been said that the art of geometry is to reason well from false diagrams." (Jean Dieudonné, "Mathematics - The Music of Reason", 1992)

"A mental model is not normally based on formal definitions but rather on concrete properties that have been drawn from life experience. Mental models are typically analogs, and they comprise specific contents, but this does not necessarily restrict their power to deal with abstract concepts, as we will see. The important thing about mental models, especially in the context of mathematics, is the relations they represent. We will use diagrams to depict mental models for a variety of concepts, and it is important to keep in mind that any diagram, or even a non-diagrammatic representation that represents the same essential relations would be equally effective." (Lyn D English & Graeme S Halford," "Mathematics Education: Models and Processes", 1995) 

"Schematic diagrams are more abstract than pictorial drawings, showing symbolic elements and their interconnection to make clear the configuration and/or operation of a system." (Ernest O Doebelin, "Engineering experimentation: planning, execution, reporting", 1995)

"Given particular tasks of reasoning, different types of diagrams show different degrees of suitedness. For example, Euler diagrams are superior in handling certain problems concerning inclusion and membership among classes and individuals, but they cannot be generally applied to such problems without special provisos. Diagrams make many proofs in geometry shorter and more intuitive, while they take certain precautions of the reasoner's to be used validly. […] Mathematicians experience that coming up with the 'right' sorts of diagrams is more than half-way to the solution of most complicated problems." (Atsushi Shimojima, "Operational Constraints in Diagrammatic Reasoning" , [in "Logical Reasoning with Diagrams"], 1996)

"Making a good choice of representational conventions is always important in solving a problem, but especially true of charts. This sensitivity of type of chart to the particularities of the task at hand makes a very general logic of charts useless." (Jon Barwise & Eric Hammer, "Diagrams and the Concept of Logical System", [in "Logical Reasoning with Diagrams"], 1996)

"Mathematicians, like the rest of us, cherish clever ideas; in particular they delight in an ingenious picture. But this appreciation does not overwhelm a prevailing skepticism. After all, a diagram is - at best - just a special case and so can't establish a general theorem. Even worse, it can be downright misleading. Though not universal, the prevailing attitude is that pictures are really no more than heuristic devices; they are psychologically suggestive and pedagogically important - but they prove nothing. I want to oppose this view and to make a case for pictures having a legitimate role to play as evidence and justification - a role well beyond the heuristic.  In short, pictures can prove theorems." (James R Brown, "Philosophy of Mathematics: An Introduction to the World of Proofs and Pictures", 1999)

On Metaphors VI

"Progress in truth - truth of science and truth of religion - is mainly a progress in the framing of concepts, in discarding artificial abstractions or partial metaphors, and in evolving notions which strike more deeply into the root of reality." (Alfred N Whitehead, "Religion in the Making", 1926)

"Science advances metaphorically. It does not proceed in an orthogenic fashion moving inexorably forward in a straight line. It does not radiate along many branches like a growing tree. It moves from one view to another by a large leap, preceded by a radical shift in the scientist’s mode of thought." (François Jacob, "The Logic of Life: A History of Heredity", 1970)

"Clever metaphors die hard. Their tenacity of life approaches that of the hardiest micro-organisms. Living relics litter our language, their raisons d’etre forever past, ignored if not forgotten, and their present fascination seldom impaired by the confusions they may create." (James R Moore, "The Post-Darwinian Controversies", 1979)

"Metaphor, the life of language, can be the death of meaning. It should be used in moderation, like vodka. Writers drunk on metaphor can forget they are conveying information and ideas." (Robert Fulford, 1996)

"So much of science consists of things we can never see: light ‘waves’ and charged ‘particles’; magnetic ‘fields’ and gravitational ‘forces’; quantum ‘jumps’ and electron ‘orbits’. In fact, none of these phenomena is literally what we say it is. Light waves do not undulate through empty space in the same way that water waves ripple over a still pond; a field is only a mathematical description of the strength and direction of a force; an atom does not literally jump from one quantum state to another, and electrons do not really travel around the atomic nucleus in orbits. The words we use are merely metaphors." (K C Cole, "On Imagining the Unseeable", Discover Magazine, 1982)

"Metaphor and simile are the characteristic tropes of scientific thought, not formal validity of argument." (Rom Harré, "Varieties of Realism", 1986)

"[…] mathematics does not come to us written indelibly on Nature’s Tablets, but rather is the product of a controlled search governed by metaphorical considerations, the premier instance being the heuristics of the conservation principles." (Philip Mirowski, "More Heat than Light: Economics as Social Physics: Physics as Nature’s Economics", 1989)

"Today’s quarks and leptons can be viewed as metaphors of the underlying reality of nature, though metaphors that are objectively and rationally defied and are components of theories that have great predictive power. And that’s the difference between the metaphors of science and those of myth: scientific metaphors work." (Victor J Stenger, "Physics and Psychics: The Search for a World Beyond the Senses", 1990)

"[…] we see from these formulations how difficult it is when we try to push new ideas into an old system of concepts belonging to an earlier philosophy, or, to use an old metaphor, when we attempt to put new wine into old bottles. Such attempts are always distressing, for they mislead us into continually occupying with the inevitable cracks in the old bottles, instead of rejoicing over the new wine." (Werner K Heisenberg)

Mental Models X: (Limitations III)

“The moment a person forms a theory, his imagination sees, in every object, only the traits which favor that theory.” (Thomas Jefferson, [letter to Charles Thompson] 1787)

"Man is always prey to his truths. Once he has admitted them, he cannot free himself from them." (Albert Camus, "The Myth of Sisyphus and Other Essays", 1942)

"Although we can never devise a pictorial representation which shall be both true to nature and intelligible to our minds, we may still be able to make partial aspects of the truth comprehensible through pictorial representations or parables. As the whole truth does not admit of intelligible representation, every such pictorial representation or parable must fail somewhere. The physicist of the last generation was continually making pictorial representations and parables, and also making the mistake of treating the half-truths of pictorial representations and parables as literal truths.” (James H Jeans, “Physics and Philosophy” 3rd Ed., 1943)

"Once we give serious consideration to the hypothesis of the unconscious, it follows that our view of the world can be but a provisional one; for if we effect so radical an alteration in the subject of perception and cognition as this dual focus implies, the result must be a world view very different from any known before." (Carl Gustav Jung, "The Structure And Dynamics Of The Psyche", 1960)

"Ideas that require people to reorganize their picture of the world provoke hostility." (James Gleick, "Chaos: Making a New Science" , 1987)

“Metaphysics in philosophy is, of course, supposed to characterize what is real - literally real. The irony is that such a conception of the real depends upon unconscious metaphors.” (George Lakoff,  “Philosophy in the Flesh: The Embodied Mind and its Challenge to Western Thought”, 1999)

“Your mental models shape the way you see the world. They help you to quickly make sense of the noises that filter in from outside, but they can also limit your ability to see the true picture. [...] We eventually lose all awareness that these ‘models’ are in fact internal illusions. We accept them as external reality and act on them as if they were. If they are good models, in most circumstances they more than adequately permit the mind to handle external reality. But here a danger creeps in. When the world changes in important ways, we can find ourselves with a model that is completely irrelevant to the current situation. We find ourselves wearing our street clothes when we are thrown off the deck of a ship. What we need at that point is a wet suit and lifejacket.” (Colin Cook & Yoram R Wind, “The Power of Impossible Thinking: Transform the Business of Your Life and the Life of Your Business”, 2006)

“Our inner working models, therefore, function as interpretation schemes, on the basis of which we organize our experiences. But, such schemes also distort reality in the direction of our pattern of expectations. In short: such working models organize and screen our experiences. This means that such an inner working model organizes and colours our perception of things in such a way that it can be extremely stimulating but can also sometimes slow us down considerably.” (M H M de Wolf, “Freud and Mahler”, 2007)

“Thinking in models has enormous advantages for us as a species, in representing the unknowable world in a form in which we can locate ourselves and with which we can engage. But it also has disadvantages for us whether as natural scientists, psychoanalytic theorists, practising analysts, or simply as individuals. We can become in Wittgenstein’s phrase the fly in the ‘fly bottle’ of our own model, with its own language from which philosophy might have a part to play in rescuing us.”  (Ronald Britton, “Between Mind and Brain: Models of the mind and models in the mind”, 2015)

On Theorems (1990-1999)

"[…] mystery is an inescapable ingredient of mathematics. Mathematics is full of unanswered questions, which far outnumber known theorems and results. It’s the nature of mathematics to pose more problems than it can solve. Indeed, mathematics itself may be built on small islands of truth comprising the pieces of mathematics that can be validated by relatively short proofs. All else is speculation." (Ivars Peterson, "Islands of Truth", 1990)

"A distinctive feature of mathematics, that feature in virtue of which it stands as a paradigmatically rational discipline, is that assertions are not accepted without proof. […] By proof is meant a deductively valid, rationally compelling argument which shows why this must be so, given what it is to be a triangle. But arguments always have premises so that if there are to be any proofs there must also be starting points, premises which are agreed to be necessarily true, self-evident, neither capable of, nor standing in need of, further justification. The conception of mathematics as a discipline in which proofs are required must therefore also be a conception of a discipline in which a systematic and hierarchical order is imposed on its various branches. Some propositions appear as first principles, accepted without proof, and others are ordered on the basis of how directly they can be proved from these first principle. Basic theorems, once proved, are then used to prove further results, and so on. Thus there is a sense in which, so long as mathematicians demand and provide proofs, they must necessarily organize their discipline along lines approximating to the pattern to be found in Euclid's Elements." (Mary Tiles, "Mathematics and the Image of Reason", 1991)

"[...] there is no criterion for appreciation which does not vary from one epoch to another and from one mathematician to another. [...] These divergences in taste recall the quarrels aroused by works of art, and it is a fact that mathematicians often discuss among themselves whether a theorem is more or less 'beautiful'. This never fails to surprise practitioners of other sciences: for them the sole criterion is the 'truth' of a theory or formula." (Jean Dieudonné, "Mathematics – The Music of Reason", 1992)

"An intuitive proof allows you to understand why the theorem must be true; the logic merely provides firm grounds to show that it is true." (Ian Stewart, "Concepts of Modern Mathematics",  1995)

"Mathematics is about theorems: how to find them; how to prove them; how to generalize them; how to use them; how to understand them. […] But great theorems do not stand in isolation; they lead to great theories. […] And great theories in mathematics are like great poems, great paintings, or great literature: it takes time for them to mature and be recognized as being 'great'." (John L Casti, "Five Golden Rules", 1995)

"To be an engineer, and build a marvelous machine, and to see the beauty of its operation is as valid an experience of beauty as a mathematician's absorption in a wondrous theorem. One is not ‘more’ beautiful than the other. To see a space shuttle standing on the launch pad, the vented gases escaping, and witness the thunderous blast-off as it climbs heavenward on a pillar of flame - this is beauty. Yet it is a prime example of applied mathematics.” (Calvin C Clawson, “Mathematical Mysteries”, 1996)

"The lack of beauty in a piece of mathematics is of frequent occurrence, and it is a strong motivation for further mathematical research. Lack of beauty is associated with lack of definitiveness. A beautiful proof is more often than not the definitive proof (though a definitive proof need not be beautiful); a beautiful theorem is not likely to be improved upon or generalized." (Gian-Carlo Rota, "The phenomenology of mathematical proof", Synthese, 111(2), 1997)

"The most common instance of beauty in mathematics is a brilliant step in an otherwise undistinguished proof. […] A beautiful theorem may not be blessed with an equally beautiful proof; beautiful theorems with ugly proofs frequently occur. When a beautiful theorem is missing a beautiful proof, attempts are made by mathematicians to provide new proofs that will match the beauty of the theorem, with varying success. It is, however, impossible to find beautiful proofs of theorems that are not beautiful." (Gian-Carlo Rota, "The Phenomenology of Mathematical Beauty", 1997)

"Mathematical truth is found to exceed the proving of theorems and to elude total capture in the confining meshes of any logical net." (John Polkinghorne, "Belief in God in an Age of Science", 1998)

"A mathematician experiments, amasses information, makes a conjecture, finds out that it does not work, gets confused and then tries to recover. A good mathematician eventually does so – and proves a theorem." (Steven Krantz, "Conformal Mappings", American Scientist, Sept.–Oct. 1999)

"Let us regard a proof of an assertion as a purely mechanical procedure using precise rules of inference starting with a few unassailable axioms. This means that an algorithm can be devised for testing the validity of an alleged proof simply by checking the successive steps of the argument; the rules of inference constitute an algorithm for generating all the statements that can be deduced in a finite number of steps from the axioms." (Edward Beltrami, "What is Random?: Chaos and Order in Mathematics and Life", 1999)

"Mathematicians, like the rest of us, cherish clever ideas; in particular they delight in an ingenious picture. But this appreciation does not overwhelm a prevailing skepticism. After all, a diagram is - at best - just a special case and so can't establish a general theorem. Even worse, it can be downright misleading. Though not universal, the prevailing attitude is that pictures are really no more than heuristic devices; they are psychologically suggestive and pedagogically important - but they prove nothing. I want to oppose this view and to make a case for pictures having a legitimate role to play as evidence and justification - a role well beyond the heuristic.  In short, pictures can prove theorems." (James R Brown, "Philosophy of Mathematics: An Introduction to the World of Proofs and Pictures, 1999)

See also:
Theorems I, II, III, IV, V, VI, VII, VIII, X
Proofs I, II, III, IV, V,. VI, VII, VIII, IX