On Heuristics: Trial and Error in Data Science

"We know the laws of trial and error, of large numbers and probabilities. We know that these laws are part of the mathematical and mechanical fabric of the universe, and that they are also at play in biological processes. But, in the name of the experimental method and out of our poor knowledge, are we really entitled to claim that everything happens by chance, to the exclusion of all other possibilities?" (Albert Claude, [Nobel Prize Lecture], 1974)

"Heuristics are rules of thumb that help constrain the problem in certain ways (in other words they help you to avoid falling back on blind trial and error), but they don't guarantee that you will find a solution. Heuristics are often contrasted with algorithms that will guarantee that you find a solution - it may take forever, but if the problem is algorithmic you will get there. However, heuristics are also algorithms." (S Ian Robertson, "Problem Solving", 2001)

"We can simplify the relationships between fragility, errors, and antifragility as follows. When you are fragile, you depend on things following the exact planned course, with as little deviation as possible - for deviations are more harmful than helpful. This is why the fragile needs to be very predictive in its approach, and, conversely, predictive systems cause fragility. When you want deviations, and you don’t care about the possible dispersion of outcomes that the future can bring, since most will be helpful, you are antifragile. Further, the random element in trial and error is not quite random, if it is carried out rationally, using error as a source of information. If every trial provides you with information about what does not work, you start zooming in on a solution - so every attempt becomes more valuable, more like an expense than an error. And of course you make discoveries along the way." (Nassim N Taleb, "Antifragile: Things that gain from disorder", 2012)

"Another crowning achievement of deep learning is its extension to the domain of reinforcement learning. In the context of reinforcement learning, an autonomous agent must learn to perform a task by trial and error, without any guidance from the human operator." (Ian Goodfellow et al, "Deep Learning", 2015)

"Bayesian networks provide a more flexible representation for encoding the conditional independence assumptions between the features in a domain. Ideally, the topology of a network should reflect the causal relationships between the entities in a domain. Properly constructed Bayesian networks are relatively powerful models that can capture the interactions between descriptive features in determining a prediction." (John D Kelleher et al, "Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, worked examples, and case studies", 2015) 

"A learning algorithm for a robot or a software agent to take actions in an environment so as to maximize the sum of rewards through trial and error." (Tomohiro Yamaguchi et al, "Analyzing the Goal-Finding Process of Human Learning With the Reflection Subtask", 2018)

"Reinforcement learning is also a subset of AI algorithms which creates independent, self-learning systems through trial and error. Any positive action is assigned a reward and any negative action would result in a punishment. Reinforcement learning can be used in training autonomous vehicles where the goal would be obtaining the maximum rewards." (Vijayaraghavan Varadharajan & Akanksha Rajendra Singh, "Building Intelligent Cities: Concepts, Principles, and Technologies", 2021)

"Methodologically, much of modern machine learning practice rests on a variant of trial and error, which we call the train-test paradigm. Practitioners repeatedly build models using any number of heuristics and test their performance to see what works. Anything goes as far as training is concerned, subject only to computational constraints, so long as the performance looks good in testing. Trial and error is sound so long as the testing protocol is robust enough to absorb the pressure placed on it." (Moritz Hardt & Benjamin Recht, "Patterns, Predictions, and Actions: Foundations of Machine Learning", 2022)

On Heuristics: Trial and Error in Technology

"Every detection of what is false directs us towards what is true: every trial exhausts some tempting form of error. Not only so; but scarcely any attempt is entirely a failure; scarcely any theory, the result of steady thought, is altogether false; no tempting form of error is without some latent charm derived from truth." (William Whewell, "Lectures on the History of Moral Philosophy in England", 1852)

"It is common sense to take a method and try it. If it fails, admit it frankly and try another. But above all, try something." (Franklin D Roosevelt, Address at Oglethorpe University, 1932)

"[...] human problem solving, from the most blundering to the most insightful, involves nothing more than varying mixtures of trial and error and selectivity."  (Herbert A Simon, "The Sciences of the Artificial". 1969)

"Whatever humans have learned had to be learned as a consequence only of trial and error experience. Humans have learned only through mistakes." (R Buckminster Fuller, "Intuition", 1983)

"The idea that no one really knew how to run a government led to the idea that we should arrange a system by which new ideas could be developed, tried out, and tossed out if necessary, with more new ideas brought in - a trial and error system." (Richard P Feynman, "What Do You Care What Other People Think". Book by Richard P. Feynman, 1988)

"Reinventing the wheel and getting it wrong is more valuable than nailing it first time. There are lessons learned from trial and error that have an emotional component to them that reading a technical book alone just cannot deliver!" (Jason P Sage, [in Kevlin Henney’s "97 Things Every Programmer Should Know", 2010])

"Technology is the result of antifragility, exploited by risk-takers in the form of tinkering and trial and error, with nerd-driven design confined to the backstage." (Nassim N Taleb, "Antifragile: Things that gain from disorder", 2012)

"Another crowning achievement of deep learning is its extension to the domain of reinforcement learning. In the context of reinforcement learning, an autonomous agent must learn to perform a task by trial and error, without any guidance from the human operator." (Ian Goodfellow et al, "Deep Learning", 2015)

On Heuristics: Trial and Error in Biology

"Higher organisms are able to learn through trial and error how a certain problem should be solved. We may say that they too make testing movements - mental testings - and that to learn is essentially to tryout one testing movement after another until one is found that solves the problem. We might compare the animal's successful solution to an expectation and hence to a hypothesis or a theory. For the animal's behaviour shows us that it expects (perhaps unconsciously or dispositionally) that in a similar case the same testing movements will again solve the problem in question." (Karl R Popper, "The Logic and Evolution of Scientific Theory", [in "All Life is Problem Solving", 1999] 1972)

"The natural as well as the social sciences always start from problems, from the fact that something inspires amazement in us, as the Greek philosophers used to say. To solve these problems, the sciences use fundamentally the same method that common sense employs, the method of trial and error. To be more precise, it is the method of trying out solutions to our problem and then discarding the false ones as erroneous. This method assumes that we work with a large number of experimental solutions. One solution after another is put to the test and eliminated." (Karl R Popper, "The Logic and Evolution of Scientific Theory", [in "All Life is Problem Solving", 1999] 1972)

"Whatever the system, adaptive change depends upon feedback loops, be it those provided by natural selection or those of individual reinforcement. In all cases, then, there must be a process of trial and error and a mechanism of comparison. […] By superposing and interconnecting many feedback loops, we (and all other biological systems) not only solve particular problems but also form habits which we apply to the solution of classes of problems." (Gregory Bateson, "Steps to an Ecology of Mind", 1972)

"When we examine this suggestion, we see that it is no more than a formal acknowledgement of a problem, the problem of how (by what institutional arrangement, by what organization of affairs) the equilibrium prices are to be discovered. Repeated trial and error, while the market stands in suspense awaiting the outcome, is not a practical resort. The number of distinct trials, even if confined to discrete steps of price and quantity, would be so immense that the necessary 'market day' would extend beyond human life-times [... The] theoretical ideal applies to mutually isolated days or moments, each to be treated as perfectly self-contained and looking to no yesterday and no tomorrow. But the real market is dealing with goods inherited from yesterday, and in means of production whose products will not be ready till tomorrow. Meanwhile the non-economic circumstances are changing and rendering each successive equilibrium obsolete." (George L S Shackle, "Epistemics and Economics", 1972)

"We know the laws of trial and error, of large numbers and probabilities. We know that these laws are part of the mathematical and mechanical fabric of the universe, and that they are also at play in biological processes. But, in the name of the experimental method and out of our poor knowledge, are we really entitled to claim that everything happens by chance, to the exclusion of all other possibilities?" (Albert Claude, "The Coming Age of the Cell", Nobel Prize Lecture] 1974)

"Survival machines that can simulate the future are one jump ahead of survival machines that who can only learn of the basis of trial and error. The trouble with overt trial is that it takes time and energy. The trouble with overt error is that it is often fatal. [...] The evolution of the capacity to simulate seems to have culminated in subjective consciousness. Why this should have happened is, to me, the most profound mystery facing modern biology." (Richard Dawkins, "The Selfish Gene Source: The Selfish Gene", 1976)

"In the process of the evolution of life, as far as we know, the total mass of living matter has always been and is now increasing and growing more complex in its organization. To increase the complexity of the organization of biological forms, nature operates by trial and error. Existing forms are reproduced in many copies, but these are not identical to the original. Instead they differ from it by the presence of small random variations."  (Valentin F Turchin, "The Phenomenon of Science: A cybernetic approach to human evolution", 1977)

"I believe people can solve complex problems eventually. By repeated trial and error they will get there; but they need a long time. At this point I agree with Herbert Simon. People do not learn immediately, as those rational expectations models seem to imply. I don't believe that. The statement that assumptions do not matter is nonsense. It is funny. Yes, I assume people are consistent in their behavior. I assume that not because I believe everybody actually is, but because I believe, on the average, you do not get too far from it." (Franco Modigliani, "Conversations with Economists", 1983)

"Heuristics are rules of thumb that help constrain the problem in certain ways (in other words they help you to avoid falling back on blind trial and error), but they don't guarantee that you will find a solution. Heuristics are often contrasted with algorithms that will guarantee that you find a solution - it may take forever, but if the problem is algorithmic you will get there. However, heuristics are also algorithms." (S Ian Robertson, "Problem Solving", 2001)

On Heuristics: Trial and Error in Science

"The one lesson that comes out of all our theorizing and experimenting is that there is only one really scientific progressive method; and that is the method of trial and error." (George B Shaw, "The Doctor's Dilemma: Preface on Doctors", 1909)

"There are many men now living who were in the habit of using the age-old expression: 'It is as impossible as flying.' The discoveries in physical science, the triumphs in invention, attest the value of the process of trial and error. In large measure, these advances have been due to experimentation." (Louis Brandeis, "Dissent, New State Ice Co. v. Liebmann, 285 U.S. 262", 1932)

"The discoveries in physical science, the triumphs in invention, attest the value of the process of trial and error. In large measure, these advances have been due to experimentation." (Louis Brandeis, "Judicial opinions", 1932)

"But I believe that there is no philosophical high-road in science, with epistemological signposts. No, we are in a jungle and find our way by trial and error, building our road behind us as we proceed. We do not find signposts at cross-roads, but our own scouts erect them, to help the rest." (Max Born, "Experiment and Theory in Physics", 1943)

"The method of learning by trial and error - of learning from our mistakes - seems to be fundamentally the same whether it is practised by lower or by higher animals, by chimpanzees or by men of science." (Karl Popper, "Conjectures and Refutations: The Growth of Scientific Knowledge", 1963)

"The difference between the amoeba and Einstein is that, although both make use of the method of trial and error elimination, the amoeba dislikes erring while Einstein is intrigued by it [...]" (Karl R Popper, "Objective Knowledge: An Evolutionary Approach", 1972) 

"Science progresses by trial and error, by conjectures and refutations. Only the fittest theories survive." (Alan Chalmers, "What Is This Thing Called Science?", 1976)

"The great revolutions in science are almost always the result of unexpected intuitive leaps. After all, what is science if not the posing of difficult puzzles by the universe? Mother Nature does something interesting, and challenges the scientist to figure out how she does it. In many cases the solution is not found by exhaustive trial and error […] or even by a deduction based on the relevant knowledge." (Martin Gardner, "Aha! Insight", 1978)

"I believe people can solve complex problems eventually. By repeated trial and error they will get there; but they need a long time. At this point I agree with Herbert Simon. People do not learn immediately, as those rational expectations models seem to imply. I don't believe that. The statement that assumptions do not matter is nonsense. It is funny. Yes, I assume people are consistent in their behavior. I assume that not because I believe everybody actually is, but because I believe, on the average, you do not get too far from it." (Franco Modigliani, "Conversations with Economists", 1983)

"Science usually amounts to a lot more than blind trial and error. Good statistics consists of much more than just significance tests; there are more sophisticated tools available for the analysis of results, such as confidence statements, multiple comparisons, and Bayesian analysis, to drop a few names. However, not all scientists are good statisticians, or want to be, and not all people who are called scientists by the media deserve to be so described." (Robert Hooke, "How to Tell the Liars from the Statisticians", 1983)

"Whatever humans have learned had to be learned as a consequence only of trial and error experience. Humans have learned only through mistakes." (R Buckminster Fuller, "Intuition", 1983)

"Growth is a process of experimentation, a series of trials, errors, and occasional victories. The failed experiments are as much as part of the process as the experiments that work." (Chérie Carter-Scott, "If Life Is a Game, These Are the Rules", 1998)

"The natural as well as the social sciences always start from problems, from the fact that something inspires amazement in us, as the Greek philosophers used to say. To solve these problems, the sciences use fundamentally the same method that common sense employs, the method of trial and error. To be more precise, it is the method of trying out solutions to our problem and then discarding the false ones as erroneous. This method assumes that we work with a large number of experimental solutions. One solution after another is put to the test and eliminated." (Karl R Popper, "All Life is Problem Solving", 1999)

"We can simplify the relationships between fragility, errors, and antifragility as follows. When you are fragile, you depend on things following the exact planned course, with as little deviation as possible - for deviations are more harmful than helpful. This is why the fragile needs to be very predictive in its approach, and, conversely, predictive systems cause fragility. When you want deviations, and you don’t care about the possible dispersion of outcomes that the future can bring, since most will be helpful, you are antifragile. Further, the random element in trial and error is not quite random, if it is carried out rationally, using error as a source of information. If every trial provides you with information about what does not work, you start zooming in on a solution - so every attempt becomes more valuable, more like an expense than an error. And of course you make discoveries along the way." (Nassim N Taleb, "Antifragile: Things that gain from disorder", 2012)

On Numbers: Large Numbers I

"The calculation of probabilities is of the utmost value, […] but in statistical inquiries there is need not so much of mathematical subtlety as of a precise statement of all the circumstances. The possible contingencies are too numerous to be covered by a finite number of experiments, and exact calculation is, therefore, out of the question. Although nature has her habits, due to the recurrence of causes, they are general, not invariable. Yet empirical calculation, although it is inexact, may be adequate in affairs of practice." (Gottfried W Leibniz [letter to Bernoulli], 1703)

"Further, it cannot escape anyone that for judging in this way about any event at all, it is not enough to use one or two trials, but rather a great number of trials is required. And sometimes the stupidest man - by some instinct of nature per se and by no previous instruction (this is truly amazing) - knows for sure that the more observations of this sort that are taken, the less the danger will be of straying from the mark." (Jacob Bernoulli, "The Art of Conjecturing", 1713)

"If thus all events through all eternity could be repeated, by which we would go from probability to certainty, one would find that everything in the world happens from definite causes and according to definite rules, and that we would be forced to assume amongst the most apparently fortuitous things a certain necessity, or, so to say, FATE." (Jacob Bernoulli, "The Art of Conjecturing", 1713)

"And thus in all cases it will be found, that although Chance produces Irregularities, still the odds will be infinitely great that in the process of time, those Irregularities will bear no proportion to the recurrency of that Order which naturally results from ORIGINAL DESIGN." (Abraham de Moivre, "The Doctrine of Chances", 1718)

"Things of all kinds are subject to a universal law which may be called the law of large numbers. It consists in the fact that, if one observes very considerable numbers of events of the same nature, dependent on constant causes and causes which vary irregularly, sometimes in one direction, sometimes in the other, it is to say without their variation being progressive in any definite direction, one shall find, between these numbers, relations which are almost constant." (Siméon-Denis Poisson, "Poisson’s Law of Large Numbers", 1837)

"Huge numbers are commonplace in our culture, but oddly enough the larger the number the less meaningful it seems to be." (Albert Sukoff, "Lotsa Hamburgers", Saturday Review of the Society, 1973)

"We know the laws of trial and error, of large numbers and probabilities. We know that these laws are part of the mathematical and mechanical fabric of the universe, and that they are also at play in biological processes. But, in the name of the experimental method and out of our poor knowledge, are we really entitled to claim that everything happens by chance, to the exclusion of all other possibilities?" (Albert Claude, [Nobel Prize Lecture], 1974)

"The logarithm is an extremely powerful and useful tool for graphical data presentation. One reason is that logarithms turn ratios into differences, and for many sets of data, it is natural to think in terms of ratios. […] Another reason for the power of logarithms is resolution. Data that are amounts or counts are often very skewed to the right; on graphs of such data, there are a few large values that take up most of the scale and the majority of the points are squashed into a small region of the scale with no resolution." (William S. Cleveland, "Graphical Methods for Data Presentation: Full Scale Breaks, Dot Charts, and Multibased Logging", The American Statistician Vol. 38 (4) 1984)

"The trouble with integers is that we have examined only the small ones. Maybe all the exciting stuff happens at really big numbers, ones we can’t get our hand on or even begin to think about in any very definite way. So maybe all the action is really inaccessible and we’re just fiddling around. Our brains have evolved to get us out of the rain, find where the berries are, and keep us from getting killed. Our brains did not evolve to help us grasp really large numbers or to look at things in a hundred thousand dimensions." (Paul Hauffman, "The Man Who Loves Only Numbers", The Atlantic Magazine, Vol 260, No 5, 1987)

"The law of truly large numbers states: With a large enough sample, any outrageous thing is likely to happen." (Frederick Mosteller, "Methods for Studying Coincidences Journal of the American Statistical Association, Volume 84, 1989)

Previous Post <<||>> Next Post